Here comes Mobility Field Day 2! 

I’m currently sitting in the Saskatoon airport waiting to board my first flight on the way to Mobility Field Day 2 in San Jose, and I am really excited to be a delegate for the first time!

Side note: as I prepare to spend two days seeing what the best in the mobility industry are up to, I’m disappointed to be sitting at YXE airport using my hotspot because the Wi-Fi is currently completely inoperable. It had been working very well on recent trips. Oh well.

First of all, I’d like to send a little shout out to the Tech Field Day crew – they have been awesome in helping us get ready to travel to San Jose and participate in this fantastic event. They’ve clearly done this before and know all of the steps to make sure everything goes as smoothly as possible. Thank you TFD team!

Tune into MFD2 on Tuesday, July 25 here. Dont forget to follow the hashtag #MFD2 on Twitter during the event to catch our up to date thoughts, and mention me (@CdnBeacon) or any of the other delegates if you have any questions you’d like us to ask!

Here’s what I’m looking forward to:

Mist Systems

I’ve had the chance to play with the Mist APs a little bit recently, and I’ve been impressed. It was very easy to get an SSID spun up, and what was really neat was how easy it was to setup their virtual BLE (Bluetooth low-energy) beacons. It took me longer to find the floor plans for my house (and I already knew where they were) than it did to set up the beacons and use their app to watch myself wander around the house. I wasn’t terribly familiar with Mist’s solutions before this, so I’m really looking forward to going in depth. Keep your ears open for things like “blue-dot experience” and lots about client experience analytics.

Mojo Networks

Mojo has gone through some reinventions over the past years. I was more familiar with them as Airtight, but they have a new focus now. I saw them at WLPC and their mojopackets online PCAP analyzer it pretty neat. I’m very curious to hear about their plans for Wi-Fi on open hardware. We’ve heard that idea before, let’s see if someone can make it a feasible reality for the enterprise, and in my case, industrial use cases as well.

Cape Networks

I was fortunate to come home from WLPC with my own Cape sensor, and I’ve had it running in my office ever since. Cape has a great UI, making it really easy to see the important metrics – how is the Wi-Fi working? I think it’s narrow-minded though to think of the Cape sensor as a Wi-Fi sensor only, because it’s really good at letting me know when the network isn’t the problem. Cape has some great potential and now that I’ve spent some time with it, I’m ready with some ideas for tweaks and improvements – bring your notepad, Drew. 

Nyansa

Nyansa has been doing some really interesting work in gathering huge amounts of Wi-Fi client behaviour data, and turning it into informative insights. This is another one that I am looking forward to learning about a bit more in depth. With so much client variation in our environments, I really love the idea of being able to identify, for example, that any iPhone 6 running iOS 9 has a bad habit of sticking to an AP well past when it should have roamed. 

Netscout

One of my favorite tools is my Netscout Aircheck G2. There is a lot of talk in the Wi-Fi industry these days about how difficult it is to get relevant data about Wi-Fi and client performance from the wireless clients themselves. My G2 is great for giving me data from a client perspective, and sometimes there is no substitute for a handheld tool. I’m still a network engineer as much as a mobility engineer, and I admit I’m trying to find a good reason to buy a LinkRunner AT. Netscout’s tools are great for giving me the answers I need quickly. 

On-premise IT roundtable

Rumor has it there might be a recording of Gestalt IT’s On-premise podcast. I’ve been enjoying these episodes since they started as a “just the right length” dive into various issues and technologies for our industry. Keep an eye out for an episode from MFD2 on iTunes, or the website here.

See you in San Jose!

ESS 9 is here: Demo with Jussi!

We had an excellent opportunity last week at Cisco Live! to visit with the team from Ekahau and hear about several things that they’re working on, including some significant improvements to ESS in V9 (which has now been released – if you have an active service contract, go download and install it now!).

The team from Ekahau has a reputation for really being a part of  the Wi-Fi engineer community, and really listening to what the community is asking for, and ESS 9 is a prime example. Ekahau has spent over a year working closely with Wi-Fi experts to add density and airtime utilization calculations that are grounded in solid RF theory.

Check it out below, and for more info, see the Ekahau blog here.

Changing Wireshark Link-layer Header settings on Mac OS

This is one of those quick posts aiming to save me and (maybe you) some time the next time I forget this.

On my Mac, I use Wireshark primarily to capture Wi-Fi traffic, in monitor mode. I want to see the Radiotap and 802.11 headers. Usually I leave Wireshark set this way.

On occasion, I actually use Wireshark to inspect higher level traffic – I want to see the IP addresses and TCP/UDP ports etc. I might be troubleshooting an issue and am using my Mac as the client trying to recreate the issue – so I don’t need monitor mode for that. Simple enough – turn it off in the interface settings (Find this button on the Main toolbar Wi-Fi. en0 Wireshark, Today at 9.20.38 PM  to access the menu, then scroll to the right to find the Monitor mode drop down and make sure your Wi-Fi interface has this disabled):

Wireshark · Capture Interfaces Wireshark, Today at 9.30.03 PM

Then just set the Link-layer header back to Ethernet, just like your other interfaces:

Wireshark · Capture Interfaces Wireshark, Today at 9.30.57 PM

Except “Ethernet” isn’t an option. I could’ve sworn that’s what it is set to by default after install…

I can’t believe this still trips me up every few months. I spent half an hour the other day scratching my head, when the trick is simply to restart Wireshark. Close it entirely, reopen it and voila:

Wireshark · Capture Interfaces Wireshark, Today at 9.34.30 PM

Ethernet is back! Also, the 802.11 options have disappeared because we’re no longer in monitor mode. Now I can see Ethernet, IP, and TCP/UDP headers again:

Wi-Fi. en0 Wireshark, Today at 9.35.58 PM

In comparison to capturing 802.11 frames in monitor mode:

Wi-Fi. en0 Wireshark, Today at 9.38.08 PM

I keep forgetting the need to restart Wireshark for the Link-layer options to change #facepalm.

Note: you also need to restart Wireshark after enabling monitor mode before the 802.11 options will show up in the Link-layer header drop down option.

Or maybe it’s just me. I’m confident that I’ll still forget all about this post next time I try to show a University Computer Engineering class how many packets it takes to load the Facebook home page.

It was 781 (including DNS lookups and a couple of retransmitted frames), in case you’re wondering…

PING! Not always what you think! – Meraki Wireless troubleshooting

I’m quite fond of the Meraki dashboard. I’ve seen firsthand how it can enable lean and low-skilled IT departments to manage more of their own networks themselves. The dashboard GUI makes it easy to find status and troubleshoot at a basic level, but it’s still important to actually understand what is going on under the hood.

Here’s an example. If you’ve ever seen the Meraki dashboard, you’ve probably seen the Ping tool on every client status page. Here, a Meraki AP is successfully Ping-ing my MacBook Pro:

pign-success-invalid-ip

Pretty straightforward. Ping the client, client responds, client is online and working, right?

If you have a Meraki Security Appliance, you may stumble across this little note on the Addressing & VLANs page:

ping-is-arp

Wait… Ping is based on ARP? What happened to ICMP?

We may have jumped to conclusions here. As it turns out, Meraki is not using ICMP like most of us would assume. Here’s an example of a few PCAP frames of that same Meraki AP ARP-Ping-ing my MacBook Pro:

directed-arp

Notice this is a directed-ARP; the Meraki AP (MAC 13:da:90) is sending an ARP request to the MBP (MAC 91:75:d8) rather than sending a broadcast. That is, the Meraki AP already knows the MBPs MAC address. But the ARP response tells the Meraki AP that the MBP is alive, and online – just like an ICMP Ping.

This brings about an interesting question. We network engineers often use Ping as a way to confirm that the network is working. A successful Ping means that routing, IP addressing and the physical path are all functioning correctly at layer 3. But if we’re doing a Ping at layer 2 with ARP – would we be wrongly assuming all is well when we get a response, just like with ICMP?

There is definitely some potential to make incorrect assumptions here. In fact, even though that screenshot above of the Meraki AP Ping-ing my MBP has a loss of 0%, at the time, my MBP had an incorrect IP address and was not Ping-able by other devices in the network (well, via ICMP at least). Here’s the PCAP’d ARP frames from the same time as that Ping output:

directed-arp-wrong-ip

Almost identical, except that both the ARP request and response are from 10.11.3.1, when the subnet is actually 10.11.30.0/24. However, the client is still responding, albeit at layer 2, and that’s good enough for the Meraki AP.

Now, I do think this is one of those things where the vendor has made an odd decision to label this as a Ping without being clear about what is actually being done, but it is after all our responsibility as the network engineer to know what we’re looking at. There are similar examples where traceroute can use UDP or ICMP, depending on the OS, and now you know that sometimes Ping is ARP instead of ICMP.

Here’s the relevant documentation:

Meraki Ping Tool

ARGGHH ARRRRRRRP!!!

I run into this issue several times a year with a certain local DSL/Fiber service provider and ASA firewalls. Sometimes it consistently occurs after an outage, others it is seemingly random. This is the relevant debug ARP input (modified for confidentiality):

?arp-req: generating request for 16.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending
?arp-req: generating request for 16.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending
?arp-req: generating request for 216.197.160.254 at interface Provider1
arp-req: request for 16.197.160.254 still pending

What makes this a REALLY bad thing is that 16.197.160.254 is the ASA’s default gateway. No internet access for you.

Unfortunately, I have little (no) visibility into the provider settings, but I can hazard a guess that there is some sort of spoofing protection in place. Sometimes a reboot of the ISP modem will fix the problem, but often times we have to call the ISP and, while trying to explain ARP to tier 1 support should NOT be too difficult, it is typically an exercise in frustration. Not surprising, often we are asked to plug a laptop directly into the modem with the ASA’s static IP programmed on its NIC, which works, and causes the ISP to cheer “NOT OUR PROBLEM!” Of course, if I plug a laptop directly into the Provider1 interface on the ASA, it gets an ARP reply right away and communication to the “gateway” also works just fine. Ultimately, I have not been able to find an easy, repeatable fix from the side I have control over (the ASA), but sometimes the ISP clears an ARP table to solve the problem.

Something similar occurs from time to time with this ISP with NAT/PAT IP addresses that are NOT the ASA’s interface address. In this case, we can PCAP traffic leaving the ASA with the appropriate IP and MAC towards the ISP, but the ISP will never forward the return traffic to the ASA – the gateway doesn’t create an ARP entry for that IP.

In this case, an easy fix is to temporarily change the ASA interface IP to match the NAT IP. This causes the ASA to generate a gratuitous ARP and suddenly the return traffic gets delivered.

This is certainly different from the first case, where no amount of restarting or GARP seems to convince the ISP gateway to reply to the ASA ARP request for the gateway’s MAC, but I suspect the cause may be related.

I’m currently waiting for the ISP to determine if an engineer who actually knows how to log into the gateway router and look at an ARP table does indeed exist, or whether I’m more likely to watch a unicorn run a red light on my commute home. In the  meantime, if anyone can explain what this ISP is doing to cause this behaviour, I’d love to hear it.

 

Wireless Professionals Compensation Survey

Today, Wednesday, November 30th there will be a one-day independent survey to gather information on the current state of compensation in the Wireless LAN community. We respectfully request your participation in this 90-second survey. The current results will be available to all survey takers as they complete the survey for instant feedback. Later, the complete results will freely reported back the entire community.
Thank you for your support and participation!
http://surveymonkey.com/r/wlccs2016 – Wireless LAN Community Compensation Benchmark