ARGGHH ARRRRRRRP!!!

I run into this issue several times a year with a certain local DSL/Fiber service provider and ASA firewalls. Sometimes it consistently occurs after an outage, others it is seemingly random. This is the relevant debug ARP input (modified for confidentiality):

?arp-req: generating request for 16.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending
?arp-req: generating request for 16.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending
?arp-req: generating request for 216.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending

What makes this a REALLY bad thing is that 16.197.160.254 is the ASA’s default gateway. No internet access for you.

Unfortunately, I have little (no) visibility into the provider settings, but I can hazard a guess that there is some sort of spoofing protection in place. Sometimes a reboot of the ISP modem will fix the problem, but often times we have to call the ISP and, while trying to explain ARP to tier 1 support should NOT be too difficult, it is typically an exercise in frustration. Not surprising, often we are asked to plug a laptop directly into the modem with the ASA’s static IP programmed on its NIC, which works, and causes the ISP to cheer “NOT OUR PROBLEM!” Of course, if I plug a laptop directly into the Provider1 interface on the ASA, it gets an ARP reply right away and communication to the “gateway” also works just fine. Ultimately, I have not been able to find an easy, repeatable fix from the side I have control over (the ASA), but sometimes the ISP clears an ARP table to solve the problem.

Something similar occurs from time to time with this ISP with NAT/PAT IP addresses that are NOT the ASA’s interface address. In this case, we can PCAP traffic leaving the ASA with the appropriate IP and MAC towards the ISP, but the ISP will never forward the return traffic to the ASA – the gateway doesn’t create an ARP entry for that IP.

In this case, an easy fix is to temporarily change the ASA interface IP to match the NAT IP. This causes the ASA to generate a gratuitous ARP and suddenly the return traffic gets delivered.

This is certainly different from the first case, where no amount of restarting or GARP seems to convince the ISP gateway to reply to the ASA ARP request for the gateway’s MAC, but I suspect the cause may be related.

I’m currently waiting for the ISP to determine if an engineer who actually knows how to log into the gateway router and look at an ARP table does indeed exist, or whether I’m more likely to watch a unicorn run a red light on my commute home. In the  meantime, if anyone can explain what this ISP is doing to cause this behaviour, I’d love to hear it.

 

CWNE #190

cwne-brennan-martinI woke to some exciting news last week. An email from CWNP informing me that my CWNE application had been approved and that I am now CWNE #190!

This is a huge honor, and I am humbled to be part of the CWNE community. While I like to think that I have picked up some skills over the past decade, I owe a great deal of thanks to people whom I have met over the past year  who have mentored and encouraged me to engage in the Wi-Fi community.

Thanks to Keith ParsonsBlake KroneDavid WestcottPeter MackenzieMartin Ericson for sharing your insights and invaluable knowledge!

Onwards…

 

Cisco Live! recap – Monday

I had a session booked at 8am. Breakfast started at 7 and with a 30 minute walk to the conference centre, I needed to get up at 6. So much for getting some rest while my kids weren’t waking me up!

BRKEWN-2000 “Design and Deployment of Wireless LANs for real time applications” with Jerome Henry! 

This is where I would finally meet the illustrious Rowell Dionicio of Packet 6 and the Clear to Send podcast; and my Canadian brother from another country, Meru Mitch! (that’s right, I know THE Meru Mitch!). Check out Mitch’s recap of CLUS 2016 as well.

Here are a couple of things that stuck with me from Jerome‘s session:

  • People use only one real time application at a time. You have a FaceTime chat, then hang up and go onto something else. It’s pretty impractical to FaceTime and have a VoIP call at the same time. It’s safe to assume that one real time app will be the only bandwidth need while in use.
  • How to do the math. Some good examples of how to actually calculate the bandwidth need in a cell.
  • What is my client device’s max EIRP. Some more great examples. This info isn’t often published by the manufacturers. An iPhone 6s for example, has a worst max EIRP in the 5 GHz band of 10.3 dBm.
  • Here’s a big one: 802.11 devices decide which transmission rate to use based on the signal strength of transmissions received from the AP

    • Think about this one. This goes towards the concept of matching transmit power.

These four points are covered in the first 30 pages out of 191; so I highly recommend you grab the PDF and have a read through. Lots of excellent concepts explained by Jerome.
IMG_6448Next was off to the opening keynote with the honorary hosers, Hub Holster Robb and Meru Mitch. Lasers, dancers, and Your Time is Now – the theme for CLUS.

28,000 attendees, and don’t forget about Cisco DNA!

IMG_6449

 

With a little time after the keynote before the next session, we couldn’t resist heading to the World of Solutions (WoS).

We caught up with Matt, and I had a specific stop in mind.  We quickly located the giant sign in the distance pointing the direction to the Cisco certification lounge, and made our way over. I admit I couldn’t resist leaving my compadres at the general entrance when I saw the dedicated CCIE entrance! Being my first CLUS, I couldn’t resist trying out the perks.

Note for next year: the CCIE entrance is like the side door. We ended up at the exact same place, ha! Oh well, I got my fancy CCIE ribbon for my badge, but my fingers were crossed for something else… YES. They were in the back of the lounge!

I got my first CLUS tramp stamp. Now I was really on my way to being a CLUS veteran! Thanks to Robb,  Mitch and Steve for putting up with me and generally encouraging the nerdery. Moving on.

We picked up our CLUS 2016 T-shirts and got our picture taken with Elvis:

IMG_6457
@Robb_404, Elvis, @mattouellette, yours truly, and #MeruMitch

We hung around WoS a little longer and started accumulating various vendor swag. Little did I know just how much swag we would end up with. Soon it was off to another session.

BRKRST-2515 “QoS Design and Deployment for Wireless LANs” with Robert Barton

Having recently completed the CWNP CWAP course, much of this was very familiar to me so thankfully it wasn’t hard to follow. QoS is always a complex topic.

Here are a couple of things that stuck with me from Robert’s session:

  • Do not use the Wired QoS Protocol field. This caught my attention because it was a departure from Cisco’s earlier best practices which I was familiar with. Admittedly, it had been a few years since I had really reviewed this topic (see my post When a 6 means 5 – Cisco WLAN QoS), so this stood out for me.
  • A lot of work on QoS has been done in WLC code recently. Time for me to start reading.
  • Introducing Fastlane. The fruits of the Apple/Cisco partnership are ripening. Big benefits to roaming and QoS for Apple devices, often with no admin intervention or config required.
  • EasyQoS. This should be a BIG deal. See @Cantechit’s post here for an excellent overview.

This session was a real brain session, so I enjoyed the change of pace in the next session:

BRKEWN-2019 – “7 Ways to fail as a Wireless Expert” with Steven Heinsius (contender for best Twitter profile pic, btw).

This was a fun session, and there were many great takeaways. Fun = engaging, and it did a great job of reinforcing some concepts we all know well, in a way that might help us explain it to our clients and users.

IMG_6460
You really can’t argue with that.
  • CnHuXXHUsAAaAFP.jpg-largeChannels 1,6 and 11 folks.
  • Get moving to 5 GHz.
  • Don’t use max power.
  • Cisco says use RRM. But Cisco also tells us to tune RRM. Max power should be limited to 17 dBm and min to 5 dBm.
  • Everyone loves Bad-Fi.com.
  • Go take a look at Wigle.netYou can learn a lot from this site. No more TKIP/WPA1 or WEP. Even WPA2-Passphrase is not appropriate for the enterprise.
  • No, 802.11ac is not going to give you better that 1 Gbps throughput. See Andrew von Nagy’s webinar here.
  • Look for the “Best Practices” slides in Steven’s Presentation. Good rules of thumb to keep handy.

This was a great way to end day #1. Next was back to WoS for more swag! There were so many vendors to see, and even better, the food and beer was everywhere we looked. We paused at a table at one point and were joined by a very friendly and unassuming woman looking for a spot to put down her plate.  At luck would have it, we met Cisco’s social media manager Laura Babbili!
IMG_6468

How cool is that. Laura is part of the top brass of Cisco’s social media team, including @CiscoLive. We picked her brain about the inner workings of the Cisco Live twitter team, but I’m sure it’s a complete coincidence that we had several twitter pictures on the big screen at the closing keynote…

 

On our way out of WoS Robb, Meru Mitch, Steve, and I took our commemorative photo with the Cisco Live! sign:

IMG_6470.JPG

But the day doesn’t end there! (See why I am breaking these posts up by day??)

Robb and I met back up with Rowell at a lounge in the Cosmo hotel, where I had the pleasure of meeting Ryan Adzima and Richard Macintosh, two Wi-Fi wranglers who are also well known in the community. I admit I was pumped when Richard saw my twitter handle and pointed out that he had read some of my posts. It was cool to know that, even if I am embarrassing myself online, I am getting a little bit of attention in the community I am hoping to be a part of.

I’m aiming to recap the rest of the week soon! Stay tuned.

my NETSCOUT Aircheck G2 calls me slow :(

As we’re all getting used to our nifty new gadget, we’re bound to scratch our heads over a few things. My honorary Canadian friend MeruMitch and I were noting a particular output from the G2 which seemed odd.

I’m connected to my AP at a data rate of 867Mbps:

wifisignal data rate
Wifi Signal is the shizzle

And a quick Wifi Perf says I can push 115 Mbps of actual throughput (wireless to wireless over one AP):

FullSizeRender

But the G2 is adamant that my connection rate is 24 Mbps:

Screenshot0003
Um… Wha?

Well let’s think about this for a minute…

The G2 shows a “PHY Data Rate” of 780 Mbps connecting itself to that AP:

Screenshot0000.png

And if we check the G2 manual:


Ok, so that makes sense. This is what we’re expecting to see, and if you watch this field, it fluctuates a bit in the the 780-867 Mbps range.

So what then is the “Connection Rate” which is stuck at 24Mbps? Again from the user guide:

Interesting! Now I bet some of you are thinking that 24Mbps must be a basic rate in this WLAN; and you would be correct!


The key here is that the G2 is looking at the data rate of individual frames, no averages, and showing us only the last frame’s rate.

So let’s grab a basic pcap and take a look. Forgive me, this pcap is of my iPhone 6S+ rather than the MBP that I started this post with, but the relevant bits are the same. The iPhone was associated to an AP with a maximum data rate of 300 Mbps, and the G2 stayed fixed on the connection rate of 24 Mbps.

This pcap is of my iPhone doing a speed test, and I’ve filtered it to only show frames from the client, based on that definition from the G2 guide above.

 

Most of the data, by Bytes, is in data packets:

TopProtobyBytes

There are a total of 22,508 Bytes sent by my iPhone, and 8,510 Bytes are data packets using the 300 Mbps data rate (38% of the captured Bytes, matching the pic above). 33.5% of the Bytes are sent at 24Mbps… surely the G2 should have, even briefly, shown something at 300Mbps?

Packet Bytes

But the G2 isn’t looking at Bytes… it’s looking frame by frame! Look at the data rates of each frame:

packets

Better yet, let’s change the previous output to see the numbers for each frame:

Packet Numbers

Ok well my iPhone sent 87 frames at 300 Mbps, and 300 at 24Mbps, clearly the majority of frames. This is a good explanation why the G2 appears to say that the connection rate is always 24Mbps.

It goes to show how important it is to know how to interpret what our tools are telling us. Note that only data frames in my capture were sent at the 300Mbps data rate. The control frames and some of the data frames are all sent at 12/24Mbps – our basic rates. Which is a nice plug for CWAP!

Now I’m not really sure how useful the “connection rate” field really is in its current form. Showing the last frame rate seems less useful than showing the “PHY data rate” that the G2 uses when it connects to an AP itself.

I’d love to hear what anyone else thinks of the “connection rate” field…

Cisco Live! recap – Sunday

Wow – what a trip. My feet have finally stopped smoking. Note to self: next time, stay at the Mandalay bay next to the convention centre, you’ll still get plenty of exercise.

My memory of the finer details are fading fast, so here’s my shotgun recap for Sunday:

Sunday: I met an old buddy, Steve in Calgary, and we carried onto Vegas. We got checked into the hotel, and headed to the convention centre to pick up our badges and CLUS backpacks. Immediately, many of us who had “met” on twitter some time back were coordinating to meet up in person, like an awkward online dating experience, but with fewer expectations 😉

The convention centre is huge. and 3 floors. Steve and I were headed down the escalators from the 3rd floor when lo’ and behold… a wild Rob Boardman appeared riding upwards. A ‘shoot from the hip’ kind of guy, when we realized what was happening, I could see the gears turning. salmon-run-600x401He evaluated his options, turned around on the upwards traveling escalator, and began heading downwards to rendezvous with us. I felt like I was watching the #VegasSalmonRun watching Rob weave his way against the natural flow (even though there was no chance of any spawning). We noted that, for the remainder of the week, there were signs posted at the escalators to remind everyone to pay attention to the direction of travel. Rob made an immediate impression on CLUS.

Robb is the brains behind Hub Holster. If you do Wi-Fi site surveys, check out the site. More cool products are in development too – keep an eye on this sockeye.

Stoked to bask in the warm glow that is Rob’s in-person presence, we carried on our twitter escapade to combine the gravity of our own stars with the rest of the twitter galaxy.  Next thing I know, Steve, Robb and I ended up at “the Hub” of CLUS where we had the pleasure of meeting the Matts – Haedo and Ouellette (NOT a voice guy FYI) – two “CCIE pending”s – follow them to join the struggle. Ask them about the RSv5 written exam.

Hungry, we began wandering into the Mandalay bay looking for sustenance, catching up along the way with Rob #2, whom I refer to as superman. Superman is a longtime mentor of mine, and the nickname is not (just) because he’s a hero. Superman is no here.

Lunch at Citizen’s. One of the last moments of calm for the whole week, and a good start to new friendships.

Rob (superman), Steve and I would later meet up with another old friend, Schwaby for the Canadian world domination conference dinner at gfx-facebookTom Colicchio’s CraftSteak. What an awesome way to burn a hole in your pocket. Amazing steak, amazing food. The four of us were once upon a time all doing time at the same company. Together, we’re kind of like the power rangers, a bunch of Cisco engineers covering RS, Wireless, Collab, Security, and DC.

Sunday was a great day. I learned quickly that the people and new connections will without a doubt end up being the best thing about CLUS.

More recaps and new people to come.

My First Design Part 3 – ADDENDUM

I received some feedback on this post that I’d like to share. The idea behind this series of posts was to share my admittedly novice design process so that others could learn with me; and so, rather than edit the original post to fix errors, I’d like to call them out so others can see what I’m learning.

Here’s a quick recap of the coverage requirements set for this project:

CV-Reqs

Note that the channel overlap is set to a max of two APs at -86 dBm before the requirement is flagged as not being met. We’ll look at this later.

My signal strength heatmap settings were called into question:

SigStrLeg

Specifically, the fact that the grey edge ends at -73 dBm. What I had done was used the signal strength visualization to represent two metrics:

  1. Coverage I want – where the requirement is met, or anything above -67 dBm
  2. Coverage I don’t care about – where the requirement isn’t met, or anything below -67 dBm

Then I arbitrarily set -73 dBm as a threshold to indicate where the requirement wasn’t quite met, but wasn’t terrible overall. This was a poor choice on my part!

I had forgotten about something useful that I had learned in the ECSE course. The signal strength visualization can show us three useful metrics:

    1. Coverage I want – where the requirement is met, or anything above -67 dBm
    2. Coverage I DON’T WANT – where the requirement isn’t met, but the AP is still causing CCI on that channel, or -67 dBm to -86 dBm (-86 dBm to match the channel overlap requirement)
    3. Coverage I don’t care about – where the requirement isn’t met, and the AP signal should be low enough to not cause noticeable CCI; anything below -86 dBm

There’s an important distinction there!

Now this doesn’t change my design much. Let’s take a look with the heatmap coloring rules set to the appropriate “want/don’t want/don’t care” levels. Here again is the signal strength for the main floor showing the where the requirement is measured against the two strongest APs:

SS2-CCI86

and here is what it looked like before:

Main Floor - Signal Strength (2 Strongest APs)

Minor changes showing where CCI flows into the areas I had stopped caring about before hand. For reference, here is the signal heat map for the strongest one AP, with the “don’t want” levels set to go to -86 dBm:

SS1-CCI86.PNG

The real benefit of this though comes when we look at one AP at a time. Here I’m only looking at the coverage for a single AP. The green areas again indicate where our requirement is met, but look at all of that grey! These are areas where this AP is still heard above -86 dBm, meaning we can’t reuse that channel without causing CCI!

SS-OneAPCCI.PNG

Ok, so has there been any effect on my CCI metrics? Let’s take a look. In this heatmap, the “don’t want” signal strength level has been left at -67 to -86 dBm, but we’ve moved to the channel overlap visualization:

Main Floor - Channel Overlap

Looks the same! Phew! Let’s see what it’s telling us:

We’re still using the -86 dBm threshold as the point where we stop counting overlap as CCI, so here we have two APs on the 40 MHz channel covering channels 108 and 112, both heard above -86 dBm (-63 and -74 in this case).

Hopefully this helps to clear up some of the last post! Let me know if anyone has more questions or comments!