Configuring Dual-Homed FEX modules
Lately at work I've been testing dual-homed FEX connectivity in an Active/Standby configuration on the Nexus 9300 platform. Dual-Homed FEX are only supported on specific Nexus switch models and NX-OS versions etc and depending on the Nexus platform in various configuration types. There's a Cisco article on which topologies are supported on which platforms that can be found here. For this particular scenario, the Dual Homed Host (Active/Standby) and Active-Active FEX (VPC) design is used. When I was trying to deploy this I found very little information about how to get this working so decided to write this article. Below is the network topology that is used for this article. As you can see I have 2 Nexus 9300 switches in a vPC pair and have connected two FEX modules, 101 and 122, to each of the vPC pairs. This design allows you to take out/reboot a vPC pair of switches without impacting host connectivity to the FEX modules. In this design however, not that the ESX host can only have active/passive connectivity to the FEX modules unless you are using LAG on the host side and even then there are a number of caveats around exactly what is supported.
When configuring a Dual-Homed FEX, while it's not a requirement, it is highly recommended to setup Cisco's Config Sync. Config Sync allows you to automatically configure various settings on vPC pairs from a single switch. Personally I like the idea but have found it a bit temperamental and finicky to get to work. Config Sync requires you to configure and enable Cisco Fabric Services (VFS). CFS can be configured to operate over multicast, fibre channel, IPv4 or IPv6 and the various elements of the configuration that you can configure using config sync can will utilise different methods. You can check this in the output of the show cfs application command. Physical-eth refers to the use of CFS over Ethernet (CFSoE) to synchronise actions and data and can be used for features like STP and IGMP. Physical-fc-ip refers to CFS over Fibre Channel however if the fibre channel method fails, it will fall back to Ethernet. Physical-ip will distribute application data across the IP network and is typically used when FC is not available.
WRMEMSWA# sh cfs application
----------------------------------------------
Application Enabled Scope
----------------------------------------------
arp Yes Physical-eth
mld Yes Physical-eth
ntp No Physical-fc-ip
stp Yes Physical-eth
vpc Yes Physical-eth
igmp Yes Physical-eth
l2fm Yes Physical-eth
role No Physical-fc-ip
icmpv6 Yes Physical-eth
radius No Physical-fc-ip
tacacs No Physical-fc-ip
syslogd No Physical-fc-ip
callhome No Physical-fc-ip
vlan_mgr Yes Physical-eth
session-mgr Yes Physical-ip
eth_port_chann Yes Physical-eth
Total number of entries = 16
Enabling CFS and Config Sync
In the above scenario, I will be configuring CFS over IPv4 and configuring config-sync to utilise the management interfaces of the Nexus switches to sync the application data. The first thing to do is to configure cfs to distribute over IPv4. To do that, you will need to fist disable CFS distribute, then configure the multicast address (Also not required as you can use the pre-defined Multicast address however I decided to so that I could configure a different multicast address for each vPC pair of switches). This must be configured on both switches.
WRMEMSWA# conf t
WRMEMSWA(config)# no cfs ipv4 distribute
WRMEMSWA(config)# cfs ipv4 mcast-address 239.255.1.102
Distribution over this IP type will be affected.
Change multicast address for CFS-IP ?
Are you sure? (y/n) [n] y
WRMEMSWA(config)# cfs ipv4 distribute
WRMEMSWB# conf t
WRMEMSWB(config)# no cfs ipv4 distribute
WRMEMSWB(config)# cfs ipv4 mcast-address 239.255.1.102
Distribution over this IP type will be affected.
Change multicast address for CFS-IP ?
Are you sure? (y/n) [n] y
WRMEMSWB(config)# cfs ipv4 distributeYou can verify that CFS over IPv4 is enabled and that the multicast address has changed using the sh cfs status command.
WRMEMSWA# sh cfs status
Distribution : Enabled
Distribution over IP : Enabled - mode IPv4
IPv4 multicast address : 239.255.1.102
IPv6 multicast address : ff15::efff:4653
Distribution over Ethernet : EnabledOnce CFS has been configured and enabled, configure the Config Sync switch profile on both switches. This switch-profile name can be anything you want however it needs to be exactly the same on both switches.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWB# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWB(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1With the switch-profile configured, configure the management IP's of the vPC pair as the sync-peer destination
WRMEMSWA(config-sync-sp)# sync-peer destination 10.1.1.10
WRMEMSWA(config-sync-sp)# end
WRMEMSWB(config-sync-sp)# sync-peer destination 10.1.1.11
WRMEMSWB(config-sync-sp)# endWith the sync-peer configured, you can exit out of config sync mode and verify the status of the CFS and config sync configuration using the commands sh cfs peer and verify the switch-profile connectivity using the sh switch-profile status command.
WRMEMSWA# sh cfs peer
Physical Fabric
-------------------------------------------------------------------------
Switch WWN IP Address
-------------------------------------------------------------------------
20:00:a4:10:c4:18:a9:97 10.1.1.10 [Local]
WRMEMSWA
20:00:c4:7e:e0:86:b2:6f 10.1.1.11
Total number of entries = 2
WRMEMSWA# sh switch-profile status
switch-profile : WRMEMSWA-SWB
----------------------------------------------------------
Start-time: 82691 usecs after Tue May 20 12:49:47 2025
End-time: 891085 usecs after Tue May 20 12:50:00 2025
Profile-Revision: 3
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Sync Success
Local information:
----------------
Status: Commit Success
Error(s):
Peer information:
----------------
IP-address: 10.1.1.11
Sync-status: In sync
Status: Commit Success
Error(s):
WRMEMSWB# sh cfs peer
Physical Fabric
-------------------------------------------------------------------------
Switch WWN IP Address
-------------------------------------------------------------------------
20:00:c4:7e:e0:86:b2:6f 10.1.1.11 [Local]
WRMEMSWB
20:00:a4:10:c4:18:a9:97 10.1.1.10
Total number of entries = 2
WRMEMSWB# sh switch-profile status
switch-profile : WRMEMSWA-SWB
----------------------------------------------------------
Start-time: 84808 usecs after Tue May 20 12:49:47 2025
End-time: 992844 usecs after Tue May 20 12:50:00 2025
Profile-Revision: 3
Session-type: Commit
Session-subtype: -
Peer-triggered: Yes
Profile-status: Sync Success
Local information:
----------------
Status: Commit Success
Error(s):
Peer information:
----------------
IP-address: 10.1.1.10
Sync-status: In sync
Status: Commit Success
Error(s):I would like to mention at this point that config-sync utilises multicast to communicate with it's peers and I was originally experiencing peer status unreachable errors and eventually found that it was due to a multicast issue even though IGMP snooping was enabled on the TOR switches, I didn't have a PIM RP point which from what I was able to determin would have resolved my issue without having to connect the management interfaces to the same TOR Switch. I didn't bother configuring a PIM RP as this was in a prod network that I couldn't make those kinds of changes on but you could have just configured ip pim sparse-dense-mode on your switch and not have to worry about this.
Configure the FEX
With CFS and Config Sync configured, you can start configuring the FEX. If you haven't already, you can enable the fex feature set on the Nexus platform. Once that's done, configure the FEX module on the switches. In this scenario it's FEX 101 and 122.
WRMEMSWA# conf t
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config)# fex 101
WRMEMSWA(config-fex)#description FEX101
WRMEMSWA(config)# fex 122
WRMEMSWA(config-fex)#description FEX122
WRMEMSWB# conf t
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWB(config)# fex 101
WRMEMSWB(config-fex)#description FEX101
WRMEMSWB(config)# fex 122
WRMEMSWB(config-fex)#description FEX122Once the FEX number is configured, configure the port-channels that will be used for the FEX modules. For this, I'll be using Po101 and Po122 for the FEX modules. To do this, you first need to enter into config sync mode, and then into the switch-profile. Once in the profile, configure the port channel interface then verify and commit the changes. Note that these changes only need to be done on one switch as these will be synced between the vPC pairs. Once the commit is completed the status should say Commit Successful. If the commit fails, you will need to review the commands and the reason for the failure. I'll provide a small overview of this later.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int po101
WRMEMSWA(config-sync-sp-if)# int po122
WRMEMSWA(config-sync-sp-if)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulWith the Port-Channel interfaces configured, next configure the physical interfaces as part of the port channels using the command channel-group XXX force where XXX refers to the port-channel number. Once again, this should be done from config-sync mode and verified then commited.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int eth1/90-91
WRMEMSWA(config-sync-sp-if-range)# channel-group 101 force
WRMEMSWA(config-sync-sp)# int eth1/92-93
WRMEMSWA(config-sync-sp-if-range)# channel-group 122 force
WRMEMSWA(config-sync-sp-if-range)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulWith the physical ports configured, configure the port-channel mode as fex-fabric switchport mode. As usual, verify and commit the changes. Note that I found that if I tried to enter these commands all at once and commit I received a commit error. I found that I had to configure small parts of the port configuration and commit them in order to successfully complete the process.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int po101
WRMEMSWA(config-sync-sp-if)# switchport mode fex-fabric
WRMEMSWA(config-sync-sp-if)# int po122
WRMEMSWA(config-sync-sp-if)# switchport mode fex-fabric
WRMEMSWA(config-sync-sp-if-range)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulNext configure the rest of the FEX port-channel settings. Note that the order these commands are configured matters and if it's incorrect will fail to commit.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int po101
WRMEMSWA(config-sync-sp-if)# switchport mode fex-fabric
WRMEMSWA(config-sync-sp-if)# fex associate 101
WRMEMSWA(config-sync-sp-if)# vpc 101
WRMEMSWA(config-sync-sp-if)# int po122
WRMEMSWA(config-sync-sp-if)# switchport mode fex-fabric
WRMEMSWA(config-sync-sp-if)# fex associate 122
WRMEMSWA(config-sync-sp-if)# vpc 122
WRMEMSWA(config-sync-sp-if-range)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulWith the FEX port-channel vPC and associate commands completed, you can now configure the rest of the port-channel settings such as the desction and mtu. Once again, I found that If I tried to configure too many commands at once and commit the configuration, it would fail and I then had to configure a single line at a time.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int po101
WRMEMSWA(config-sync-sp-if)# descr FEX 101
WRMEMSWA(config-sync-sp-if)# int po122
WRMEMSWA(config-sync-sp-if)# desc FEX 122
WRMEMSWA(config-sync-sp-if)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit Successful
WRMEMSWA(config-sync-sp)# int po101
WRMEMSWA(config-sync-sp-if)# mtu 9216
WRMEMSWA(config-sync-sp-if)# int po122
WRMEMSWA(config-sync-sp-if)# mtu 9216
WRMEMSWA(config-sync-sp-if)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulWith all of the physical port and port-channel configuration completed, you can now enable the physical interfaces to the FEX modules.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int eth1/90-91
WRMEMSWA(config-sync-sp-if-range)# no shut
WRMEMSWA(config-sync-sp-if-range)# int eth1/92-93
WRMEMSWA(config-sync-sp-if-range)# no shut
WRMEMSWA(config-sync-sp-if)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulWith the ports enabled, you can verify that the FEX is online using the sh fex command.
WRMEMSWA# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------------------
122 FEX0122 Image Download N2K-C2232TM-E-10GE SSI18496BBL
--- -------- Connected N2K-C2232TM-E-10GE SSI1841394G
WRMEMSWB# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------------------
101 FEX0101 Image Download N2K-C2232TM-E-10GE SSI1841394G
--- -------- Connected N2K-C2232TM-E-10GE SSI18496BBLOnce the FEX modules have finally downloaded the software image and booted, you should see both FEX modules available on both switches.
WRMEMSWA# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------------------
101 FEX0101 Online N2K-C2232TM-E-10GE SSI1841394G
122 FEX0122 Online N2K-C2232TM-E-10GE SSI18496BBL
WRMEMSWB# sh fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------------------
101 FEX0101 Online N2K-C2232TM-E-10GE SSI1841394G
122 FEX0122 Online N2K-C2232TM-E-10GE SSI18496BBLThat's it, you now have Dual-Homed FEX modules. The next thing to do is to start configuring the port-profiles if you use them and configuring the host ports. With Config Sync enabled, you will need to configure this only one the primary switch.
Configuring Port-Profiles and Interfaces
Configuring interfaces and port-profiles etc is the same process as the previous steps to configure port-channels and physical interfaces for the FEX modules. Start by entering config sync mode and then enter into the switch-profile on your primary switch.
WRMEMSWA# config sync
Enter configuration commands, one per line. End with CNTL/Z.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1Next configure the required port profile settings and apply them to a physical interface.
WRMEMSWA(config-sync-sp)# port-profile type ethernet SERVER-TRUNK
WRMEMSWA(config-sync-port-prof)# switchport
WRMEMSWA(config-sync-port-prof)# switchport mode trunk
WRMEMSWA(config-sync-port-prof)# switchport trunk native vlan 999
WRMEMSWA(config-sync-port-prof)# switchport trunk allowed vlan 15-16,20,25
WRMEMSWA(config-sync-port-prof)# spanning-tree port type edge trunk
WRMEMSWA(config-sync-port-prof)# no shutdown
WRMEMSWA(config-sync-port-prof)# description Server Trunk
WRMEMSWA(config-sync-port-prof)# state enabled
WRMEMSWA(config-sync-port-prof)# int eth101/1/10
WRMEMSWA(config-sync-sp-if)# inherit port-profile SERVER-TRUNK
WRMEMSWA(config-sync-sp-if)# description PROD SERVER 1
WRMEMSWA(config-sync-sp-if)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Commit SuccessfulThat's all there is to it. This will automatically configure the ports on both switches. Also note that if you save the configuration, it only needs to be done on the primary switch as this will automatically sync to the second vPC pair.
Commit Failures
As I mentioned earlier I've found config sync to be a bit finicky and temperamental to get to work and commit changes properly. If you receive a commit failed error like below there are a few things to check.
WRMEMSWA(config-sync)# switch-profile WRMEMSWA-SWB
Resyncing db before starting Switch-profile.Re-synchronization of switch-profile db takes a few minutes...
Re-synchronize switch-profile db completed successfully.
Switch-Profile started, Profile ID is 1
WRMEMSWA(config-sync-sp)# int po101
WRMEMSWA(config-sync-sp-if)# descr FEX 101
WRMEMSWA(config-sync-sp-if)# storm broad lev 10
WRMEMSWA(config-sync-sp-if)# storm multica lev 10
WRMEMSWA(config-sync-sp-if)# int po122
WRMEMSWA(config-sync-sp-if)# desc FEX 122
WRMEMSWA(config-sync-sp-if)# storm broad lev 10
WRMEMSWA(config-sync-sp-if)# storm multica lev 10
WRMEMSWA(config-sync-sp-if)# verify
Verification Successful
WRMEMSWA(config-sync-sp)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
Failed to Commit: Commit Failed
You can check which commands in particular the error was on by using the command show switch-profile status.
WRMEMSWA(config-sync-sp)# sh switch-profile status
switch-profile : WRMEMSWA-SWB
----------------------------------------------------------
Start-time: 81485 usecs after Tue May 20 17:44:23 2025
End-time: 24231 usecs after Tue May 20 17:44:33 2025
Profile-Revision: 8
Session-type: Commit
Session-subtype: -
Peer-triggered: No
Profile-status: Commit Failed
Local information:
----------------
Status: Commit Failure
Error(s):
Failed command(s):
config terminal
` interface port-channel101`
` interface port-channel101`
` description FEX 101
` interface port-channel101`
Peer information:
----------------
IP-address: 10.1.1.11
Sync-status: In sync
Status: Commit Failure
Error(s):
Failed command(s):
config terminal
` interface port-channel101`
` description FEX 101
` interface port-channel101`
` mtu 9216`Once you have checked where the error occurred, verify the commands that you entered are correct. If you want to show a list of all commands that you entered, you can view the commit buffer using the command show switch-profile buffer.
WRMEMSWA(config-sync-sp)# show switch-profile buffer
switch-profile : WRMEMSWA-SWB
----------------------------------------------------------
Seq-no Command
----------------------------------------------------------
1 interface port-channel101
1.1 description FEX 101
1.3 storm-control broadcast level 10.00
1.4 storm-control multicast level 10.00
2 interface port-channel122
2.1 description FEX 122
2.3 storm-control broadcast level 10.00
2.4 storm-control multicast level 10.00
If the commands look correct, the next thing to do would be to make sure those commands can be applied to the interfaces. To do that, exit out of config sync mode and enter into conf t mode and verify that the commands are correct. In the example above for instance, the FEX port-channels in this version of code, no longer support the storm-control commands so I had to remove them from the commit. You can remove all of the buffered configuration or specify to remove individual lines from the commit buffer by using the command buffer-delete.
WRMEMSWA(config-sync-sp)# buffer-delete ?
<1-9999> Range(whole-number) of command id(s) to be deleted from switch-profile buffer
WORD Exact command id (x.x.x format) to be deleted from switch-profile buffer (Max Size 64)
all Delete all buffered commands
WRMEMSWA(config-sync-sp)# buffer-delete 1.4
WRMEMSWA(config-sync-sp)# buffer-delete 1.3
WRMEMSWA(config-sync-sp)# buffer-delete 2.4
WRMEMSWA(config-sync-sp)# buffer-delete 2.3
WRMEMSWA(config-sync-sp)# show switch-profile buffer
switch-profile : WRMEMSWA-SWB
----------------------------------------------------------
Seq-no Command
----------------------------------------------------------
1 interface port-channel101
1.1 description FEX 101
2 interface port-channel122
2.1 description FEX 122Once you've removed the erroneous commands, you can then verify and commit the changes again. Another thing I found was that if in some instances, if I configured too many options at once, or configured specific items out of order, the commit failed. I got around this by applying fewer lines at once and in some situations a single line at a time.
Thanks for checking out my blog. If you've noticed anything missing or have any issues setting this up, please leave a comment and let me know.
Add new comment