ASA – Nigel’s Networking Nuggets

A Cisco ASA is a router, except when its not.

We recently had a planned total shutdown of our primary data centre in order to make emergency power repairs. After the restart, everything was working except for an unexpected message in our log:

Jan 19 00:09:49 EdgeFirewall %ASA-4-106023: Deny udp src Outside:172.31.0.6/123 dst Inside:10.1.2.3/123 by access-group "Outside_access_in"
Jan 19 00:09:53 EdgeFirewall %ASA-4-106023: Deny udp src Outside:172.31.0.6/123 dst Inside:10.1.2.3/123 by access-group "Outside_access_in"

The log entries were repeated around every minute or so. The ASA in question has three interfaces, Inside, Outside and DMZ. The log entry indicates that a UDP packet entering the Outside interface (172.31.0.6) attempting to access an NTP server through the Inside interface was being blocked by the inbound access list (Outside_access_in). The problem is that 172.31.0.6 is the IP address of switch in the DMZ.

As we have a well locked down edge router, I knew that we would not allow RFC1918 addresses through to our firewall, so this was not a spoofed address. My first thought was that the route to the NTP server had been lost, causing the NTP packets to be sent out to the Internet (where they would be sent straight back by the edge router). A display of the routing table showed the route was present:

EdgeFirewall# show route
  Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
         D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default, U - per-user static route, o - ODR
         P - periodic downloaded static route

  Gateway of last resort is InternetRouter to network 0.0.0.0

  C 172.31.0.0 255.255.255.0 is directly connected, DMZ
    10.0.0.0 255.0.0.0 [1/0] via 10.1.31.1, Inside
    0.0.0.0 0.0.0.0 [1/0] via InternetRouter, Outside

I then did a packet trace and confirmed that the packets were indeed entering the DMZ interface, being sent out the Outside interface to be routed straight back by the edge router. So why was the routing table being ignored. Because, the ASA is not a router.

A display of the connection entry confirmed the problem:

  EdgeFirewall# show conn address 172.31.0.6
    214 in use, 1566 most used
    UDP DMZ 172.31.0.6:123 Outside 10.1.2.3:123 idle 0:00:05, bytes 1762, flags UIO

The connection entry clearly shows that the egress interface is Outside, not Inside as would be expected. Clearing the connection entry solved the problem and allowed the DMZ switch to get the correct time.

So what caused this? This is my theory:

After the planned shutdown, the power was restored
The ASA, DMZ switch and edge router restarted.
The Inside switch was still powering up (it is a modular switch and takes a while)
The DMZ switch started sending NTP packets every 64 seconds to the NTP server
The ASA created a connection entry and as the Inside interface was not yet up, the default route was used to send the packet out the Outside interface
The 6500 switch became available and the ASA Inside interface came up
The route to 10.0.0.0/8 was added to the routing table
Because the NTP packets were sent every 64 seconds and the defualt UDP timeout is 2 minutes, the connection entry never timed out and packets continued to be sent out the Outside interface

ASA 9.0 AnyConnect License Change

I have been looking at the new features in ASA 9.0. Two in particular that caught my eye were:

Allow up to eight ASA’s to be clustered
VPN and dynamic routing are now supported per context.

It is to be hoped that clustering, initially only available on the high-end boxes, becomes available on the smaller boxes sooner rather than later.

In the Q&A document I noticed the following:

Q. Is next generation encryption available on all ASA platforms?

A. No. Next Generation Encryption is fully supported on the ASA 5585-X, 5500-X Series, and 5580, as well as on the Catalyst 6500 Series ASA Services Module. It can only be partially supported on the ASA 5505, 5510, 5520, 5540, and 5550 due to hardware limitations. AnyConnect 3.1 or greater and an AnyConnect Premium License are also required to use next generation encryption for remote access connections.

If I read this correctly, AnyConnect with the previous generation encryption only needs an Essentials License. AnyConnect with the next generation encryption needs the much more expensive Premium License’.

The word ‘Sneaky’ springs to mind.

Annoying Cisco ASA/ASDM Bugs–Part 2

You might get an argument from Cisco that this one is a bug, but it sure is annoying:

On an ASA you can apply firewall rules in two places (in ASA 8.3 and above). Rules can be applied on an interface or globally. When applied on an interface they can operate on traffic entering the interface (incoming rules) or on traffic egressing the interface (outgoing rules).

When using ASDM to create a new rule, you can right-click on a rule and select Insert After to add a new rule after the existing right-clicked rule. When you do that you are presented with a dialog box allowing you enter the required details:

The dialog box presented allows the common details to be entered with an option to expand the dialog to include further options if needed. Selecting More Option reveals the following additional fields:

The second additional option allows the traffic direction to be set. In this example, the direction is set to In – that is the rule is an Incoming rule.

Unfortunately, the Traffic Direction is set to ‘In’ even if you are inserting a rule after an outgoing rule. If you don’t notice this, and with the unexpanded dialog box why should you, the rule is added in the wrong place and will not work as expected.

Annoying Cisco ASA/ASDM Bugs–Part 1

We have a number of Cisco ASA devices and on one high availability pair we run OSPF. We do this to allow us to exchange routes with our MPLS provider. When we first set this up we made use of ASDM to display the external routes in the OSPF database – that is type 5 LSAs. This was useful to verify that the expected routes were being redistributed to us from the providers BGP.

Unfortunately the display only listed a few LSAs with the majority missing. After some time chasing our tails to try to resolve the problem, we used the CLI to display the OSPF database and were surprised, but happy, to see all the external routes we had expected.

After some investigating we discovered that the list displayed by ASDM was being truncated at the first LSA with an address that matched an ASA object name. We reported this problem to Cisco and they accepted it as a bug (CSCtq48552 if you are interested).

Well, over a year later it still has not been fixed. I guess the moral from the story is: if in doubt use the CLI and having a bug accepted by Cisco does not mean it will ever get fixed.

TCP State bypass on a Cisco ASA

We have a few remote sites that directly connect to the primary data centre with fibre optic cables. A backup connection is provided using MPLS.

If the fibre optic cable is cut (thankfully a not too common event) traffic is rerouted through the managed MPLS network and enters the data centre through an ASA firewall. Because the ASA has no information in its state table about in-flight connections, the TCP packets are dropped, temporarily disrupting the operation of the remote site.

For this scenario and for others where asymmetric traffic flow can occur it is possible to configure the ASA to bypass TCP state checking. Note that the traffic still has to be allowed by access-lists.

Step 1 – Identify the traffic

The first step is to identify the traffic for which you want to bypass TCP state inspection. This is done by creating an access list. In this example traffic flowing between subnets 10.1.0.0/16 and 10.2.0.0/16 is identified:

access-list StatelessTraffic remark Outbound Traffic
access-list StatelessTraffic extended permit tcp 10.1.0.0 255.255.0.0 10.12.0.0 255.255.0.0 log disable
access-list StatelessTraffic remark Inbound Traffic
access-list StatelessTraffic extended permit tcp 10.2.0.0 255.255.0.0 10.1.0.0 255.255.0.0 log disable

Step 2 – Create a class map

A class map is created to match traffic using the access-list created in step 1

class-map tcp_bypass
  match access-list StatelessTraffic

Step 3 – Create the policy maps

A policy map defines the actions to be taken for traffic matching specified class maps. The policy map is applied to one or more interfaces or can be applied as a global policy. In this example two policy maps are used, one for the outgoing traffic (entering the Inside interface) and one for the incoming traffic (entering the MPLS interface):

policy-map Inside-policy
  class tcp_bypass 
    set connection timeout idle 0:15:00  
    set connection advanced-options tcp-state-bypass

policy-map MPLS-policy
  class tcp_bypass 
    set connection timeout idle 0:15:00  
    set connection advanced-options tcp-state-bypass

The two policies apply two settings for matching traffic. The first set statement modifies the idle timeout from the default (usually one hour) to 15 minutes (see below). The second set statement requests that state inspection of matching TCP traffic be bypassed.

Step 4 – Apply the policy maps to the interfaces

The last step is to apply the policy maps to the appropriate interfaces, in this case Inside and MPLS:

service-policy Inside-policy interface Inside
service-policy MPLS-policy interface MPLS

Idle Timeout Interval

The idle timeout interval is used to remove entries from the ASA state table for any connections which have been idle for a specified time, usually one hour. Normally TCP entries will be deleted when the ASA sees that the connection has been terminated (FIN or RST). However when TCP state bypass is enabled, this check is not done and all TCP sessions will remain in the state table until the idle timeout is reached. Reducing the idle timeout to 15 minutes will delete closed sessions earlier and reduce memory requirements.

Limitations

There are a number of limitations introduced when TCP bypass is enabled on an ASA. You should refer to the latest Cisco documentation to understand how they may impact your security before using this feature. In our case we only enable TCP bypass for traffic in fairly rare and short lived cases and accept the limitations to provide a seamless failover to our remote users.