Cisco Link State Tracking

We have just deployed a new rack in our primary data centre running a cluster of ten VMware servers. Each server connects to a pair of Cisco 4948E TOR switches with 4 x 1Gb from each server to each of the 4948 switches. The 4948 switches connect to pair of 6509E switches. The uplinks from the TOR switch to the core switches are currently 4 x 1Gb links but can be updated to 10Gb links in the future if required.

We initially planned to cross connected the switches to both core switches to provide redundancy in the event that a core switch should failure. Unfortunately as our 6509 switches do not run VSS this would have meant losing half the bandwidth due to spanning tree blocking the redundant uplinks. We therefore decided to connect each 4948 switch to only one of the core switches:

The problem with this design is that should a 6509 switch fail the VMware servers would continue to send frames to the now orphaned 4948. What we needed was a way to let the VMware servers know about the lack of upstream connectivity.

One way to do this is to enable the beaconing feature on the VMware servers. Essentially each server will regularly send beacon-probe broadcasts out of each interface in the NIC team for every VLAN. If the beacon-probes fail to arrive back at the server the interface would be considered unavailable and a failover would take place to another NIC in the team.

When Cisco switches are used, a better alternative is to use link state tracking.

Simply put, link state tracking monitors the interface(s) to the upstream switch (in our case the 6509) and if all the links become inactive, all the downstream interfaces (in our case the links to the VMware servers) would be shut (with error-disabled). As soon as an uplink interface becomes available the downstream interfaces would be re-activated.

There are three steps to configuring link state tracking:

• Globally enable link state tracking
• Define the upstream interfaces
• Define the downstream interfaces

The first step is to globally enable link state tracking for a particular group number. The number of groups supported on a particular switch is model dependant. In this example, group 1 is used.

HQ4948A(config)#link state track 1

The second step is to define the upstream interfaces. In our case the upstream interface is a single port  channel.

HQ4948A(config)#int po11

HQ4948A(config-if)#link state group 1 upstream

The final step is to define the downstream interfaces.

HQ4948A(config)#int range g1/1 – 40

HQ4948A(config-if-range)#link state group 1 downstream

If the servers are already in production, be sure to configure the upstream interfaces before the downstream interfaces to avoid an outage.

You can check the status using the ‘show link state’ command

HQ4948A#show link state group 1 detail
Link State Group: 1 Status: Enabled, 
UpUpstream Interfaces : Po11(Up) 
Downstream Interfaces: Gi1/1(Up) Gi1/2(Up) Gi1/3(Up) Gi1/4(Up) 
                       Gi1/5(Up) Gi1/6(Up) Gi1/7(Up) Gi1/8(Up) 
                       Gi1/9(Up) Gi1/10(Up) Gi1/11(Up) Gi1/12(Up) 
                       Gi1/13(Up) Gi1/14(Up) Gi1/15(Up) Gi1/16(Up)
                       Gi1/17(Up) Gi1/18(Up) Gi1/19(Up) Gi1/20(Up)
                       Gi1/21(Up) Gi1/22(Up) Gi1/23(Up) Gi1/24(Up)
                       Gi1/25(Up) Gi1/26(Up) Gi1/27(Up) Gi1/28(Up)
                       Gi1/29(Up) Gi1/30(Up) Gi1/31(Up) Gi1/32(Up)
                       Gi1/33(Up) Gi1/34(Up) Gi1/35(Up) Gi1/36(Up)
                       Gi1/37(Up) Gi1/38(Up) Gi1/39(Up) Gi1/40(Up)

(Up):Interface up (Dwn):Interface Down (Dis):Interface disabled

We did have one issue with the configuration on our switches. When entering the link state command for the port channel interface the tab keys did not complete the current keyword and typing a ? produced an error message:

HQ4948A(config-if)#link state ?
% Ambiguous command:  "link state "

However after entering the command in full, the command was accepted and a test confirmed that the link state tracking was working as expected.