Dice Update

After updating our 6500 switches with SUP2T supervisors and with some Nexus switches on the way, I decided to play around with ERSPAN. Being able to capture traffic from a remote data centre directly to my workstation is very useful so I took the opportunity to update Dice to correctly format ERSPAN packets.


When the ERSPAN capture is sent directly to the host running the packet capture software, the captured data is encapsulated within GRE and an ERSPAN header as above. I thought it would be useful to be able to extract the original frames from the capture so added this option as well. When you save a capture you can now request that ERSPAN captures are extracted. For example, the frame above becomes:


Dice updated to version 3.0.22

I have just updated my free Windows packet capture and analysis tool to version 3.0.22. The updates include:

  • Support for the openflow protocol
  • Updates for the protocols 802.11,DNS and LISP
  • Minor other changes and bug fixes

The support for the openflow protocol is somewhat incomplete and only supports version 1.0 due to the lack of suitable captures to test with. I hope to improve the support in future updates.

Why is the network so slow?

If you are a network engineer you are probably used to being blamed for every problem under the sun. If not blamed, your network is sure to be top of the list of suspects. Well, in this case, it was the network, or at least the ISP.

Like most techies I can access work from home using a remote access VPN. This allows me to check problems without having to go to the office. In my case this would not be too much of a problem as I only live half a mile from the office, but when I get a 2am call, I would much rather just fire up the VPN than drive into work.

Well, my VPN has got slow, real slow. My Internet access is delivered using WiMax and this does get slow occasionally, requiring a restart of the router to fix the problem. In this case that did not help. So my next step was to do a simple trace route:

tracert 212.x.x.x
Tracing route to 212.x.x.x
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms
  2   152 ms    84 ms    73 ms
  3    74 ms    78 ms    79 ms
  4    53 ms    79 ms    78 ms
  5    88 ms    78 ms    79 ms
  6    59 ms    79 ms    82 ms
  7   295 ms   209 ms   224 ms
  8   231 ms   223 ms   198 ms  ae-3-4.bar2.Marseille1.Level3.net []
  9   218 ms   228 ms   229 ms  ae-15-15.ebr1.Frankfurt1.Level3.net []
 10   219 ms   228 ms   228 ms  ae-84-84.csw3.Frankfurt1.Level3.net []
 11   237 ms   228 ms   228 ms  ae-83-83.ebr3.Frankfurt1.Level3.net []
 12   224 ms   228 ms   228 ms  ae-47-47.ebr1.Dusseldorf1.Level3.net []
 13   219 ms   229 ms   229 ms  ae-23-23.ebr2.Dusseldorf1.Level3.net []
 14   219 ms   228 ms   229 ms  ae-46-46.ebr1.Amsterdam1.Level3.net []
 15   218 ms   229 ms   228 ms  ae-57-112.csw1.Amsterdam1.Level3.net []
 16   218 ms   228 ms   238 ms  ae-1-51.edge4.Amsterdam1.Level3.net []
 17   227 ms   229 ms   228 ms  OMAN-TELECO.edge4.Amsterdam1.Level3.net []
 18   373 ms   388 ms   442 ms
 19   453 ms   398 ms   393 ms
 20   379 ms   400 ms   396 ms  xxxxxx.omantel.net.om [212.x.x.x.x]
 21   402 ms   387 ms   404 ms  212.x.x.x

And there you have it. To get from my apartment to the office half a mile away, I go from Oman to France on to Germany into Holland and back to Oman. Unfortunately this means that half the in-country users of our web services take the same tour of Europe.

A big thanks to my ISP. Their motto, ironically, is Get Closer.


A Cisco ASA is a router, except when its not.

We recently had a planned total shutdown of our primary data centre in order to make emergency power repairs. After the restart, everything was working except for an unexpected message in our log:

Jan 19 00:09:49 EdgeFirewall %ASA-4-106023: Deny udp src Outside: dst Inside: by access-group "Outside_access_in"
Jan 19 00:09:53 EdgeFirewall %ASA-4-106023: Deny udp src Outside: dst Inside: by access-group "Outside_access_in"

The log entries were repeated around every minute or so. The ASA in question has three interfaces, Inside, Outside and DMZ. The log entry indicates that a UDP packet entering the Outside interface ( attempting to access an NTP server through the Inside interface was being blocked by the inbound access list (Outside_access_in). The problem is that is the IP address of  switch in the DMZ.

As we have a well locked down edge router, I knew that we would not allow RFC1918 addresses through to our firewall, so this was not a spoofed address. My first thought was that the route to the NTP server had been lost, causing the NTP packets to be sent out to the Internet (where they would be sent straight back by the edge router). A display of the routing table showed the route was present:

EdgeFirewall# show route
  Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
         D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
        N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
        E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
         i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
         * - candidate default, U - per-user static route, o - ODR
         P - periodic downloaded static route

  Gateway of last resort is InternetRouter to network

  C is directly connected, DMZ [1/0] via, Inside [1/0] via InternetRouter, Outside

I then did a packet trace and confirmed that the packets were indeed entering the DMZ interface, being sent out the Outside interface to be routed straight back by the edge router. So why was the routing table being ignored. Because, the ASA is not a router.

A display of the connection entry confirmed the problem:

  EdgeFirewall# show conn address
    214 in use, 1566 most used
    UDP DMZ Outside idle 0:00:05, bytes 1762, flags UIO

The connection entry clearly shows that the egress interface is Outside, not Inside as would be expected. Clearing the connection entry solved the problem and allowed the DMZ switch to get the correct time.

So what caused this? This is my theory:

  • After the planned shutdown, the power was restored
  • The ASA, DMZ switch and edge router restarted.
  • The Inside switch was still powering up (it is a modular switch and takes a while)
  • The DMZ switch started sending NTP packets every 64 seconds to the NTP server
  • The ASA created a connection entry and as the Inside interface was not yet up, the default route was used to send the packet out the Outside interface
  • The 6500 switch became available and the ASA Inside interface came up
  • The route to was added to the routing table
  • Because the NTP packets were sent every 64 seconds and the defualt UDP timeout is 2 minutes, the connection entry never timed out and packets continued to be sent out the Outside interface

CBT Nuggets GNS3 Training Series

I have used CBT Nuggets training videos in the past to help pass a number of Cisco exams and also just to learn stuff. Their Cisco training by Jeremy Cioara are especially good – who thought training videos could be fun? Unfortunately I have been put off recently by their licensing changes and rather high prices.

The price of their new GNS3 training series is not a problem though. At $5 for one month streaming access to a 10 hour, 22 video series, it is an absolute bargain. The course covers pretty much everything anyone starting out with GNS3 would need and even experienced users of GNS3 will find it useful. The course is mainly focused on running GNS3 under Windows but also provides videos focused on Linux and MAC OSX. Of particular interest to me was a video on emulating the ASA firewall. The series is presented by Keith Barker who, apart from the singing, does a really excellent job.

If you are thinking of using GNS3, this course will save you hours of time and frustration. Highly recommended.

CBT Nuggets web site is www.cbtnuggets.com