TCP State bypass on a Cisco ASA

We have a few remote sites that directly connect to the primary data centre with fibre optic cables. A backup connection is provided using MPLS.

StateLess

If the fibre optic cable is cut (thankfully a not too common event) traffic is rerouted through the managed MPLS network and enters the data centre through an ASA firewall. Because the ASA has no information in its state table about in-flight connections, the TCP packets are dropped, temporarily disrupting the operation of the remote site.

For this scenario and for others where asymmetric traffic flow can occur it is possible to configure the ASA to bypass TCP state checking. Note that the traffic still has to be allowed by access-lists.

Step 1 – Identify the traffic

The first step is to identify the traffic for which you want to bypass TCP state inspection. This is done by creating an access list. In this example traffic flowing between subnets 10.1.0.0/16 and 10.2.0.0/16 is identified:

access-list StatelessTraffic remark Outbound Traffic
access-list StatelessTraffic extended permit tcp 10.1.0.0 255.255.0.0 10.12.0.0 255.255.0.0 log disable
access-list StatelessTraffic remark Inbound Traffic
access-list StatelessTraffic extended permit tcp 10.2.0.0 255.255.0.0 10.1.0.0 255.255.0.0 log disable

Step 2 – Create a class map

A class map is created to match traffic using the access-list created in step 1

class-map tcp_bypass
  match access-list StatelessTraffic

Step 3 – Create the policy maps

A policy map defines the actions to be taken for traffic matching specified class maps. The policy map is applied to one or more interfaces or can be applied as a global policy. In this example two policy maps are used, one for the outgoing traffic (entering the Inside interface) and one for the incoming traffic (entering the MPLS interface):

policy-map Inside-policy
  class tcp_bypass 
    set connection timeout idle 0:15:00  
    set connection advanced-options tcp-state-bypass

policy-map MPLS-policy
  class tcp_bypass 
    set connection timeout idle 0:15:00  
    set connection advanced-options tcp-state-bypass

The two policies apply two settings for matching traffic. The first set statement modifies the idle timeout from the default (usually one hour) to 15 minutes (see below). The second set statement requests that state inspection of matching TCP traffic be bypassed.

Step 4 – Apply the policy maps to the interfaces

The last step is to apply the policy maps to the appropriate interfaces, in this case Inside and MPLS:

service-policy Inside-policy interface Inside
service-policy MPLS-policy interface MPLS

Idle Timeout Interval

The idle timeout interval is used to remove entries from the ASA state table for any connections which have been idle for a specified time, usually one hour. Normally TCP entries will be deleted when the ASA sees that the connection has been terminated (FIN or RST). However when TCP state bypass is enabled, this check is not done and all TCP sessions will remain in the state table until the idle timeout is reached. Reducing the idle timeout to 15 minutes will delete closed sessions earlier and reduce memory requirements.

Limitations

There are a number of limitations introduced when TCP bypass is enabled on an ASA. You should refer to the latest Cisco documentation to understand how they may impact your security before using this feature. In our case we only enable TCP bypass for traffic in  fairly rare and short lived cases and accept the limitations to provide a seamless failover to our remote users.

Cisco Link State Tracking

We have just deployed a new rack in our primary data centre running a cluster of ten VMware servers. Each server connects to a pair of Cisco 4948E TOR switches with 4 x 1Gb from each server to each of the 4948 switches. The 4948 switches connect to pair of 6509E switches. The uplinks from the TOR switch to the core switches are currently 4 x 1Gb links but can be updated to 10Gb links in the future if required.

We initially planned to cross connected the switches to both core switches to provide redundancy in the event that a core switch should failure. Unfortunately as our 6509 switches do not run VSS this would have meant losing half the bandwidth due to spanning tree blocking the redundant uplinks. We therefore decided to connect each 4948 switch to only one of the core switches:

LinkTrack

The problem with this design is that should a 6509 switch fail the VMware servers would continue to send frames to the now orphaned 4948. What we needed was a way to let the VMware servers know about the lack of upstream connectivity.

One way to do this is to enable the beaconing feature on the VMware servers. Essentially each server will regularly send beacon-probe broadcasts out of each interface in the NIC team for every VLAN. If the beacon-probes fail to arrive back at the server the interface would be considered unavailable and a failover would take place to another NIC in the team.

When Cisco switches are used, a better alternative is to use link state tracking.

Simply put, link state tracking monitors the interface(s) to the upstream switch (in our case the 6509) and if all the links become inactive, all the downstream interfaces (in our case the links to the VMware servers) would be shut (with error-disabled). As soon as an uplink interface becomes available the downstream interfaces would be re-activated.

There are three steps to configuring link state tracking:

  • Globally enable link state tracking
  • Define the upstream interfaces
  • Define the downstream interfaces

The first step is to globally enable link state tracking for a particular group number. The number of groups supported on a particular switch is model dependant. In this example, group 1 is used.

HQ4948A(config)#link state track 1

The second step is to define the upstream interfaces. In our case the upstream interface is a single port  channel.

HQ4948A(config)#int po11

HQ4948A(config-if)#link state group 1 upstream

The final step is to define the downstream interfaces.

HQ4948A(config)#int range g1/1 – 40

HQ4948A(config-if-range)#link state group 1 downstream

If the servers are already in production, be sure to configure the upstream interfaces before the downstream interfaces to avoid an outage.

You can check the status using the ‘show link state’ command

HQ4948A#show link state group 1 detail
Link State Group: 1 Status: Enabled, 
UpUpstream Interfaces : Po11(Up) 
Downstream Interfaces: Gi1/1(Up) Gi1/2(Up) Gi1/3(Up) Gi1/4(Up) 
                       Gi1/5(Up) Gi1/6(Up) Gi1/7(Up) Gi1/8(Up) 
                       Gi1/9(Up) Gi1/10(Up) Gi1/11(Up) Gi1/12(Up) 
                       Gi1/13(Up) Gi1/14(Up) Gi1/15(Up) Gi1/16(Up)
                       Gi1/17(Up) Gi1/18(Up) Gi1/19(Up) Gi1/20(Up)
                       Gi1/21(Up) Gi1/22(Up) Gi1/23(Up) Gi1/24(Up)
                       Gi1/25(Up) Gi1/26(Up) Gi1/27(Up) Gi1/28(Up)
                       Gi1/29(Up) Gi1/30(Up) Gi1/31(Up) Gi1/32(Up)
                       Gi1/33(Up) Gi1/34(Up) Gi1/35(Up) Gi1/36(Up)
                       Gi1/37(Up) Gi1/38(Up) Gi1/39(Up) Gi1/40(Up)

(Up):Interface up (Dwn):Interface Down (Dis):Interface disabled

We did have one issue with the configuration on our switches. When entering the link state command for the port channel interface the tab keys did not complete the current keyword and typing a ? produced an error message:

HQ4948A(config-if)#link state ?
% Ambiguous command:  "link state "

However after entering the command in full, the command was accepted and a test confirmed that the link state tracking was working as expected.