Hours saved by the AirCheck G2

I really enjoyed visiting Netscout HQ during Mobility Field Day 2. They just might have the coolest social media manager in the tech industry (Hi Kendall!!!) and I was excited to meet some of the people on their wireless team behind my favorite tool, the AirCheck G2.  Chris Hinz pointed out a couple of things I didn’t already know about my Aircheck G2. Check out Chris’s presentations at the Tech Field Day page: Netscout at MFD2

In a similar vein, I thought I’d share my two favorite G2 features. I just spent an entire month on a industrial mine site physically recabling fiber optics and replacing switches.

To get an idea of the environment I’m talking about, take a look at these pics:

This is nothing linear about this facility, and I got turned around every day. Thankfully, my G2 has saved me from  hours of frustration on more than one occasion.

I’m almost ashamed to say that I probably use the G2 to trace wires more than I use it for troubleshooting wireless. Looking back at the pics above, you might imagine how difficult it was to follow cables visually. Without a tool like the G2, it would have been next to impossible to determine what was on the other end of a cable. The Aircheck’s Ethernet test has the ability to read CDP/LLDP and give us the name of the switch and the specific port number we’re connected to.

Screenshot0010

Not to mention confirming that PoE is available, that we’re connected to a Gigabit port, and that DHCP is working. We can see that the G2 has even reached out to Google to confirm that there is a working internet connection, and then uploaded all of the test results to my Link-Live dashboard (a free service).

Screenshot0011

Besides tracing wired endpoints while I was replacing switches, I was also challenged with some actual wireless reconfiguration. I needed to reboot several wireless access points, but few of the switches in this facility were PoE enabled, and all of the (old 1242s) APs were powered by AC adapters. No easy power cycles from the switch port for me. Now I have the opposite problem from above – I know where the switch is, but there’s no way I can trace cables visually to the AP. AirCheck G2 to the rescue again:

Screenshot0012

The AP locator gives me an easy “hot and cold” way to find an AP by signal strength. I could do this with a laptop and Inssider or Wi-Fi Explorer Pro, but in this industrial facility, safety concerns make this an impractical approach (and safety rules actually prohibit having both hands full). I need to be able to have two free hands, not to mention that I am wearing gloves, along with a hard hat, safety glasses, and sometimes earplugs.

The G2 simplifies the whole process, giving me a visual indicator as well as an audible beep like a metal detector. I haven’t tried it, but the spec sheet says you can even plug in a USB headset instead of listening to the beep from the unit, which, looking back, would have been useful for working with ear protection. I can also slide the unit into a pocket in order to have both hands free for going up and down stairs and it’s small enough to operate with one hand once I’ve chosen the AP to track down. These are all big improvements over a laptop when I’m not working in a carpeted office environment.

These APs are also al in NEMA enclosures in a facility that has NEMA enclosures everywhere; and most aren’t protecting APs. Suffice to say, it was much easier to locate the APs I needed with the G2. I am even questioning my earlier decision to not purchase the directional antenna for the G2 (which Chris discusses in the TFD presentation linked above). I will gladly fork out the cash for that tool if I ever have to spend a month doing this kind of work again.

Netscout has done a great job with the AirCheck G2. Best of all, the G2 is still a new tool – so expect plenty of innovations and improvements to the G2 feature set to come out of Netscout soon. I can smell good things cooking in the Netscout kitchen…

Wearing the Cape

Looking at Cape Networks after their presentation at Mobility Field Day 2.

There’s a trend in the networking industry of turning client devices into trustworthy monitoring and troubleshooting devices. In the past it was common for users to call the network team to say “the Wi-Fi isn’t working” and the network team would promptly log into the controller/dashboard/APs or other network gear and say “it looks good from this end.” Especially with Wi-Fi, it can be very difficult to understand the client perspective when you’re looking at the network – it’s like the other side of the fence.

Cape Networks is doing a solid job of putting the network engineer on the client side of the fence. Check out the #MFD2 presentation here:

Cape Networks Presents at Mobility Field Day 2

Cape gives us a client device whose entire purpose is to report standard, trustworthy, and consistent performance metrics to the network engineer, in their own language. It’s an interesting idea if you think about it – they’re making a user talk like a network engineer… usually the network engineers are tasked with putting themselves in the users’ shoes.

No more “It was WAY faster yesterday!”

How do I even measure that…?

No more “The Wi-Fi is down everywhere!”

Does that really mean Wi-Fi, or did DNS crash again?

How about a user that can tell you that how much download throughput they’ve had from YouTube for every ten minutes over the last 24 hours, and even as far back as the last 30 days?

Cape - Youtube perf

How about a user that can give you bar graphs tying RSSI to Channel Utilization, Retry Rate, and BSSID, again for every ten minutes in the last 24 hours to 30 days?

Cape - Wifi stats

Oh and this user has PCAPs every time they have a “bad” experience, triggered by crossing a performance threshold.

Cape - PCAPs

But wait… there’s more! Your superhero user has iPerf stats too.

Cape - iperf setup

Icing on the cake: when this user can’t connect to Wi-Fi, the tests still run, and you are still seeing the results in your monitoring dashboard because the user is sending you the stats via their cellular hotspot. There’s also a battery – so the sensor can still send you a notification via cellular when it loses power. In fact, my office sensor has been the first way I find out someone has blown the 2nd floor breaker every time without fail. Damn 100+ year-old buildings and interns with their space heaters. But I digress…

I have junior engineers that can do this for me. But it’s not productive to have them sitting in a remote office running these tests all the time. Plus my boss would make me feed them and give them bathroom breaks. My Cape sensors dutifully sit on the wall, PoE powered, reporting stats 24 hours a day, 7 days a week, 365 days a year, without a dime in overtime pay.

If you’re like me, you’re also digging this dashboard UI. Cape has some killer UI designers. The whole interface is easy to navigate and configure. Having an intuitive and simple UI is an advantage that is hard to overstate – if the functionality is there, an easy UI makes the Cape sensor the Most Valuable Employee you’ve got – but you never have to promote them and search for months for a replacement that will never be as effective and still a pleasure to deal with.

There is some room for improvement. Email notifications get annoying and are often just noise, but this is an issue with any NMS too. I’ve found myself ignoring most notifications, because I’ve learned that those I do check into are often back to normal by the time I log into the dashboard, usually just a couple of minutes. Tuning alert thresholds can help with this. Better yet, I’ve suggested to Cape that something like a mobile app that can give me the green light/red light behaviour without emailing me to death would be a welcome addition – and I’ve been told this is being explored.

Cape - thresholds

Which is another strength – like all of the good cloud based services these days, Cape is pushing out new firmware and features on a regular basis. Both iPerf testing and PCAPs are features that have appeared since I first got a sensor to test in February, and the Cape team is hard at work making improvements all the time.

I’ve had Cape in the office and lab since then, and it definitely beats every infrastructure based NMS I’ve worked with. Keep an eye on these guys and reach out to their team if you’d like to try them out. They’ve been very great to deal with as a company.

In the interest of full disclosure, I won a Cape sensor with free service as a door prize at the WLPC conference in February, and was also provided a sensor as a #MFD2 delegate. I have not otherwise been paid to review their product but have chosen to do so because I have been impressed with them as a solution and company.

 

 

Here comes Mobility Field Day 2! 

I’m currently sitting in the Saskatoon airport waiting to board my first flight on the way to Mobility Field Day 2 in San Jose, and I am really excited to be a delegate for the first time!

Side note: as I prepare to spend two days seeing what the best in the mobility industry are up to, I’m disappointed to be sitting at YXE airport using my hotspot because the Wi-Fi is currently completely inoperable. It had been working very well on recent trips. Oh well.

First of all, I’d like to send a little shout out to the Tech Field Day crew – they have been awesome in helping us get ready to travel to San Jose and participate in this fantastic event. They’ve clearly done this before and know all of the steps to make sure everything goes as smoothly as possible. Thank you TFD team!

Tune into MFD2 on Tuesday, July 25 here. Dont forget to follow the hashtag #MFD2 on Twitter during the event to catch our up to date thoughts, and mention me (@CdnBeacon) or any of the other delegates if you have any questions you’d like us to ask!

Here’s what I’m looking forward to:

Mist Systems

I’ve had the chance to play with the Mist APs a little bit recently, and I’ve been impressed. It was very easy to get an SSID spun up, and what was really neat was how easy it was to setup their virtual BLE (Bluetooth low-energy) beacons. It took me longer to find the floor plans for my house (and I already knew where they were) than it did to set up the beacons and use their app to watch myself wander around the house. I wasn’t terribly familiar with Mist’s solutions before this, so I’m really looking forward to going in depth. Keep your ears open for things like “blue-dot experience” and lots about client experience analytics.

Mojo Networks

Mojo has gone through some reinventions over the past years. I was more familiar with them as Airtight, but they have a new focus now. I saw them at WLPC and their mojopackets online PCAP analyzer it pretty neat. I’m very curious to hear about their plans for Wi-Fi on open hardware. We’ve heard that idea before, let’s see if someone can make it a feasible reality for the enterprise, and in my case, industrial use cases as well.

Cape Networks

I was fortunate to come home from WLPC with my own Cape sensor, and I’ve had it running in my office ever since. Cape has a great UI, making it really easy to see the important metrics – how is the Wi-Fi working? I think it’s narrow-minded though to think of the Cape sensor as a Wi-Fi sensor only, because it’s really good at letting me know when the network isn’t the problem. Cape has some great potential and now that I’ve spent some time with it, I’m ready with some ideas for tweaks and improvements – bring your notepad, Drew. 

Nyansa

Nyansa has been doing some really interesting work in gathering huge amounts of Wi-Fi client behaviour data, and turning it into informative insights. This is another one that I am looking forward to learning about a bit more in depth. With so much client variation in our environments, I really love the idea of being able to identify, for example, that any iPhone 6 running iOS 9 has a bad habit of sticking to an AP well past when it should have roamed. 

Netscout

One of my favorite tools is my Netscout Aircheck G2. There is a lot of talk in the Wi-Fi industry these days about how difficult it is to get relevant data about Wi-Fi and client performance from the wireless clients themselves. My G2 is great for giving me data from a client perspective, and sometimes there is no substitute for a handheld tool. I’m still a network engineer as much as a mobility engineer, and I admit I’m trying to find a good reason to buy a LinkRunner AT. Netscout’s tools are great for giving me the answers I need quickly. 

On-premise IT roundtable

Rumor has it there might be a recording of Gestalt IT’s On-premise podcast. I’ve been enjoying these episodes since they started as a “just the right length” dive into various issues and technologies for our industry. Keep an eye out for an episode from MFD2 on iTunes, or the website here.

See you in San Jose!

ESS 9 is here: Demo with Jussi!

We had an excellent opportunity last week at Cisco Live! to visit with the team from Ekahau and hear about several things that they’re working on, including some significant improvements to ESS in V9 (which has now been released – if you have an active service contract, go download and install it now!).

The team from Ekahau has a reputation for really being a part of  the Wi-Fi engineer community, and really listening to what the community is asking for, and ESS 9 is a prime example. Ekahau has spent over a year working closely with Wi-Fi experts to add density and airtime utilization calculations that are grounded in solid RF theory.

Check it out below, and for more info, see the Ekahau blog here.

Changing Wireshark Link-layer Header settings on Mac OS

This is one of those quick posts aiming to save me and (maybe you) some time the next time I forget this.

On my Mac, I use Wireshark primarily to capture Wi-Fi traffic, in monitor mode. I want to see the Radiotap and 802.11 headers. Usually I leave Wireshark set this way.

On occasion, I actually use Wireshark to inspect higher level traffic – I want to see the IP addresses and TCP/UDP ports etc. I might be troubleshooting an issue and am using my Mac as the client trying to recreate the issue – so I don’t need monitor mode for that. Simple enough – turn it off in the interface settings (Find this button on the Main toolbar Wi-Fi. en0 Wireshark, Today at 9.20.38 PM  to access the menu, then scroll to the right to find the Monitor mode drop down and make sure your Wi-Fi interface has this disabled):

Wireshark · Capture Interfaces Wireshark, Today at 9.30.03 PM

Then just set the Link-layer header back to Ethernet, just like your other interfaces:

Wireshark · Capture Interfaces Wireshark, Today at 9.30.57 PM

Except “Ethernet” isn’t an option. I could’ve sworn that’s what it is set to by default after install…

I can’t believe this still trips me up every few months. I spent half an hour the other day scratching my head, when the trick is simply to restart Wireshark. Close it entirely, reopen it and voila:

Wireshark · Capture Interfaces Wireshark, Today at 9.34.30 PM

Ethernet is back! Also, the 802.11 options have disappeared because we’re no longer in monitor mode. Now I can see Ethernet, IP, and TCP/UDP headers again:

Wi-Fi. en0 Wireshark, Today at 9.35.58 PM

In comparison to capturing 802.11 frames in monitor mode:

Wi-Fi. en0 Wireshark, Today at 9.38.08 PM

I keep forgetting the need to restart Wireshark for the Link-layer options to change #facepalm.

Note: you also need to restart Wireshark after enabling monitor mode before the 802.11 options will show up in the Link-layer header drop down option.

Or maybe it’s just me. I’m confident that I’ll still forget all about this post next time I try to show a University Computer Engineering class how many packets it takes to load the Facebook home page.

It was 781 (including DNS lookups and a couple of retransmitted frames), in case you’re wondering…

PING! Not always what you think! – Meraki Wireless troubleshooting

I’m quite fond of the Meraki dashboard. I’ve seen firsthand how it can enable lean and low-skilled IT departments to manage more of their own networks themselves. The dashboard GUI makes it easy to find status and troubleshoot at a basic level, but it’s still important to actually understand what is going on under the hood.

Here’s an example. If you’ve ever seen the Meraki dashboard, you’ve probably seen the Ping tool on every client status page. Here, a Meraki AP is successfully Ping-ing my MacBook Pro:

pign-success-invalid-ip

Pretty straightforward. Ping the client, client responds, client is online and working, right?

If you have a Meraki Security Appliance, you may stumble across this little note on the Addressing & VLANs page:

ping-is-arp

Wait… Ping is based on ARP? What happened to ICMP?

We may have jumped to conclusions here. As it turns out, Meraki is not using ICMP like most of us would assume. Here’s an example of a few PCAP frames of that same Meraki AP ARP-Ping-ing my MacBook Pro:

directed-arp

Notice this is a directed-ARP; the Meraki AP (MAC 13:da:90) is sending an ARP request to the MBP (MAC 91:75:d8) rather than sending a broadcast. That is, the Meraki AP already knows the MBPs MAC address. But the ARP response tells the Meraki AP that the MBP is alive, and online – just like an ICMP Ping.

This brings about an interesting question. We network engineers often use Ping as a way to confirm that the network is working. A successful Ping means that routing, IP addressing and the physical path are all functioning correctly at layer 3. But if we’re doing a Ping at layer 2 with ARP – would we be wrongly assuming all is well when we get a response, just like with ICMP?

There is definitely some potential to make incorrect assumptions here. In fact, even though that screenshot above of the Meraki AP Ping-ing my MBP has a loss of 0%, at the time, my MBP had an incorrect IP address and was not Ping-able by other devices in the network (well, via ICMP at least). Here’s the PCAP’d ARP frames from the same time as that Ping output:

directed-arp-wrong-ip

Almost identical, except that both the ARP request and response are from 10.11.3.1, when the subnet is actually 10.11.30.0/24. However, the client is still responding, albeit at layer 2, and that’s good enough for the Meraki AP.

Now, I do think this is one of those things where the vendor has made an odd decision to label this as a Ping without being clear about what is actually being done, but it is after all our responsibility as the network engineer to know what we’re looking at. There are similar examples where traceroute can use UDP or ICMP, depending on the OS, and now you know that sometimes Ping is ARP instead of ICMP.

Here’s the relevant documentation:

Meraki Ping Tool