We recently had a planned total shutdown of our primary data centre in order to make emergency power repairs. After the restart, everything was working except for an unexpected message in our log:
Jan 19 00:09:49 EdgeFirewall %ASA-4-106023: Deny udp src Outside:172.31.0.6/123 dst Inside:10.1.2.3/123 by access-group "Outside_access_in"
Jan 19 00:09:53 EdgeFirewall %ASA-4-106023: Deny udp src Outside:172.31.0.6/123 dst Inside:10.1.2.3/123 by access-group "Outside_access_in"
The log entries were repeated around every minute or so. The ASA in question has three interfaces, Inside, Outside and DMZ. The log entry indicates that a UDP packet entering the Outside interface (172.31.0.6) attempting to access an NTP server through the Inside interface was being blocked by the inbound access list (Outside_access_in). The problem is that 172.31.0.6 is the IP address of switch in the DMZ.
As we have a well locked down edge router, I knew that we would not allow RFC1918 addresses through to our firewall, so this was not a spoofed address. My first thought was that the route to the NTP server had been lost, causing the NTP packets to be sent out to the Internet (where they would be sent straight back by the edge router). A display of the routing table showed the route was present:
EdgeFirewall# show route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is InternetRouter to network 0.0.0.0
C 172.31.0.0 255.255.255.0 is directly connected, DMZ
10.0.0.0 255.0.0.0 [1/0] via 10.1.31.1, Inside
0.0.0.0 0.0.0.0 [1/0] via InternetRouter, Outside
I then did a packet trace and confirmed that the packets were indeed entering the DMZ interface, being sent out the Outside interface to be routed straight back by the edge router. So why was the routing table being ignored. Because, the ASA is not a router.
A display of the connection entry confirmed the problem:
EdgeFirewall# show conn address 172.31.0.6
214 in use, 1566 most used
UDP DMZ 172.31.0.6:123 Outside 10.1.2.3:123 idle 0:00:05, bytes 1762, flags UIO
The connection entry clearly shows that the egress interface is Outside, not Inside as would be expected. Clearing the connection entry solved the problem and allowed the DMZ switch to get the correct time.
So what caused this? This is my theory:
- After the planned shutdown, the power was restored
- The ASA, DMZ switch and edge router restarted.
- The Inside switch was still powering up (it is a modular switch and takes a while)
- The DMZ switch started sending NTP packets every 64 seconds to the NTP server
- The ASA created a connection entry and as the Inside interface was not yet up, the default route was used to send the packet out the Outside interface
- The 6500 switch became available and the ASA Inside interface came up
- The route to 10.0.0.0/8 was added to the routing table
- Because the NTP packets were sent every 64 seconds and the defualt UDP timeout is 2 minutes, the connection entry never timed out and packets continued to be sent out the Outside interface
