Hours saved by the AirCheck G2

I really enjoyed visiting Netscout HQ during Mobility Field Day 2. They just might have the coolest social media manager in the tech industry (Hi Kendall!!!) and I was excited to meet some of the people on their wireless team behind my favorite tool, the AirCheck G2.  Chris Hinz pointed out a couple of things I didn’t already know about my Aircheck G2. Check out Chris’s presentations at the Tech Field Day page: Netscout at MFD2

In a similar vein, I thought I’d share my two favorite G2 features. I just spent an entire month on a industrial mine site physically recabling fiber optics and replacing switches.

To get an idea of the environment I’m talking about, take a look at these pics:

This is nothing linear about this facility, and I got turned around every day. Thankfully, my G2 has saved me from  hours of frustration on more than one occasion.

I’m almost ashamed to say that I probably use the G2 to trace wires more than I use it for troubleshooting wireless. Looking back at the pics above, you might imagine how difficult it was to follow cables visually. Without a tool like the G2, it would have been next to impossible to determine what was on the other end of a cable. The Aircheck’s Ethernet test has the ability to read CDP/LLDP and give us the name of the switch and the specific port number we’re connected to.

Screenshot0010

Not to mention confirming that PoE is available, that we’re connected to a Gigabit port, and that DHCP is working. We can see that the G2 has even reached out to Google to confirm that there is a working internet connection, and then uploaded all of the test results to my Link-Live dashboard (a free service).

Screenshot0011

Besides tracing wired endpoints while I was replacing switches, I was also challenged with some actual wireless reconfiguration. I needed to reboot several wireless access points, but few of the switches in this facility were PoE enabled, and all of the (old 1242s) APs were powered by AC adapters. No easy power cycles from the switch port for me. Now I have the opposite problem from above – I know where the switch is, but there’s no way I can trace cables visually to the AP. AirCheck G2 to the rescue again:

Screenshot0012

The AP locator gives me an easy “hot and cold” way to find an AP by signal strength. I could do this with a laptop and Inssider or Wi-Fi Explorer Pro, but in this industrial facility, safety concerns make this an impractical approach (and safety rules actually prohibit having both hands full). I need to be able to have two free hands, not to mention that I am wearing gloves, along with a hard hat, safety glasses, and sometimes earplugs.

The G2 simplifies the whole process, giving me a visual indicator as well as an audible beep like a metal detector. I haven’t tried it, but the spec sheet says you can even plug in a USB headset instead of listening to the beep from the unit, which, looking back, would have been useful for working with ear protection. I can also slide the unit into a pocket in order to have both hands free for going up and down stairs and it’s small enough to operate with one hand once I’ve chosen the AP to track down. These are all big improvements over a laptop when I’m not working in a carpeted office environment.

These APs are also al in NEMA enclosures in a facility that has NEMA enclosures everywhere; and most aren’t protecting APs. Suffice to say, it was much easier to locate the APs I needed with the G2. I am even questioning my earlier decision to not purchase the directional antenna for the G2 (which Chris discusses in the TFD presentation linked above). I will gladly fork out the cash for that tool if I ever have to spend a month doing this kind of work again.

Netscout has done a great job with the AirCheck G2. Best of all, the G2 is still a new tool – so expect plenty of innovations and improvements to the G2 feature set to come out of Netscout soon. I can smell good things cooking in the Netscout kitchen…

Mining Borer Wireless Connectivity

See this?

When I started working in mining, all of the mines I worked with were using Cisco APs mounted on the borer to facilitate a network connection to the less mobile equipment. The root of the link is usually the end of an extensible conveyor belt, and the PLCs controlling the belt communicate with the PLCs on the borer. There are things like safety interlocks which shut down the belt if the communication is interrupted. The extensible belt root AP is also uplinked to the rest of the mine network.

The Cisco bridges worked reasonably well at reasonable distances. On the bright side, EIRP was not a concern since we were far enough underground to prevent any sort of radiation into the open atmosphere, so very high gain antennas were standard operating procedure. I tried to find a pic, but just think parabolic dish.
But the borer moves away from the extensible as it does its work. We’re talking thousands of feet, and the cuts that the borer makes don’t stay level over the entire cut – so LoS is not maintained. On one hand, hard rock does an alright job bouncing signal along. On the other, throughput and data rate still faded quickly.

Can’t pull fiber and power in behind a borer – it will need to come back out the way it went in, and moving heavy machinery makes that idea asking for trouble. So ….

Yes, that IS two 1242 APs in a NEMA enclosure with back to back patch antennas, forming a repeater; AND I’m happy to see you!

It worked. No channel reuse (sorry Keith) because it was more important for unskilled miners to be able to move and install them with zero config. The entire set of root, repeaters and end bridge are hard coded to the same frequency.

Oh and if you could see to the ground in that picture, you’d find a car battery and some alligator clips. Fancy.

And yes that’s repeaters plural. The borer just got that far away. Sometimes it goes around a corner and then you need a repeater too.

So it’s not surprising to know that throughput was… not great. When VoIP was first brought out to the borers, we had a problem with the PLC interlock between the borer and the extensible belt dropping anytime someone picked up the phone. Neither the PLC interlock nor the VoIP call required much bandwidth, but there was enough latency or loss to starve the interlock, and VoIP tends to get prioritized by default. We had to do a little QoS reservation for the PLCs to keep them running at the expense of the calls.

So a year or two ago the instrumentation guys on site started playing with these:

This is a Ubiquiti PowerBridge M3. 3.5GHz licensed frequency? No problem – we’re thousands of meters underground! That’s a 20dBi integrated panel antenna, and a 2 SS radio.They use a root AP with a 26dBi antenna, and 12dBi repeaters, if necessary (sorry, I neglected to record the antenna specs on the Cisco gear when we installed them years back).

The results were rather stunning. At distances without repeaters, they can maintain near 100Mbps. With repeaters, they stay stable at distances of six to eight thousand feet without LoS due to elevation change. In the particular set I checked recently, the bridge was operating at the 32.5Mbps data rate (not throughput) over a 5Mhz wide channel, with 25dBm output power on the borer AP (100Mbps data rate was at 20MHz channel widths). The root has a 26dBi antenna, and the repeaters 12dBi. I didn’t have a chance to grab the models and power settings on those though.

Just for fun, here’s an idea of the perspective of the borer AP:

IMG_4655

So these are significant (though unquantified) improvements from the 1242 and 1310 Cisco bridges. Had I thought to keep specs from then, perhaps I would have been able to provide more than just anecdotal commentary. But there are a few points that make it clear we’re not making an apples to apples comparison:

  • 3Ghz is not Wi-Fi At a rough level, it also goes farther with less loss.
  • 5Mhz wide channels are also not Wi-Fi. While there are some benefits to narrower channels being a little more robust against interference, interference generally isn’t a concern in this environment
  • Ubiquiti isn’t exactly known for playing by the rules in general. There are some proprietary enhancements at play – the data sheet for the M3 lists it as a MIMO TDMA protocol device

I’m no Ubiquiti fan. I know their gear is a lower quality hardware, and that they’ve caused some trouble in the unlicensed space. But that’s not the point.

This is a unique environment. Disposable hardware is favoured – service contracts are less useful here, so spares are kept on the shelf for quick deployment, which means low cost is a bonus. Performance is priority, and we’re not constrained by the rules of the surface world.

The engineers and technicians on site like a lot of things about the Ubiquiti bridges, including the simple (yet effective) GUI, and the physical signal strength indicators on the bridge, which simplify aiming.

In the end, I guess it comes back to designing for requirements. Each solution may shine or fail horribly in different scenarios.

I’m not the most experienced Wi-Fi engineer, but I’m learning, and I found this an interesting educational experience. I am curious to hear what the more experienced member of the Wi-Fi community think about Ubiquiti and their place in these less traditional deployments.

 

Building networks where the sun doesn’t shine…

 

When I started building communications networks 11 years ago, I expected to be sitting at a desk. Or maybe in a server closet. If I was really lucky, I figured I’d be in a data centre, or hanging APs in a medical clinic – and I’ve done all of those! BUT I did NOT expect to be underground, attached via console cable to an AP, mounted onto one of these:

IMG_4634
Badass drilling tank

That, is a borer, or miner, or boring machine (not “boring” like “yawn”). It bores into rock, underground, thousands of meters below the surface of the earth. And, yes, it was moving, albeit slowly, whilst I was consoled to that AP, trying to figure out why it couldn’t connect to the root AP back towards the rest of the mine.

My home province of Saskatchewan produces approximately 50% of the world supply of Potash. Potash is a big ingredient in commercial fertilizers – potash helps grow strong food.

This means that potash mining is a big deal here; and since most of it comes from here, it also means that, outside of western Canada, many people have never heard of potash (like perogies and bunny-hugs).

IMG_4635
Raw potash ore underfoot

Over the past 8 years, I have been incredibly fortunate to spend a lot of time building communications networks for underground mining operations. I have steel-toed boots, a hard hat with my name on it, several pairs of safety glasses, and as you can see in the title image, the latest in arc-flash fashion.

This is certainly a unique environment – in addition to the underground mine, the sites include large mill facilities which process the raw ore, train yards where processed potash is loaded into rail cars for shipping, and more typical administrative office buildings.

IMG_4609
At first glance, you might not notice that this shop is underground

I got involved in mining early into what we now call “the internet of things”. Back then, engineers and instrumentation technologists had been trying to adapt to running their controls (PLCs, HMIs, MPUs – valves, belts, motors) over ethernet. These controls had typically all used proprietary protocols and connections, but the vendors had started to see the benefits to using a standardized technology. When I came along, single VLANs stretched the entire length of the mine underground (100s of kilometres); control systems  didn’t quite obey the TCP/IP rules we all know and love; broadcasting constantly like that guy in your office who always talks so loud on the phone that no one else can even read the words on their screens, and these guys were starting to question the impact of running these controls on the same gear as the internet.

Long story short – things are better now! But there are still lots of unique challenges. Mining is a big mix of several networking disciplines:

  • Routing – No more geographically huge VLANs! Also, VRFs to separate control and traditional traffic on the same physical equipment. Best of all – we actually use dynamic routing now (HALLELUJAH).
  • Switching – often regular Catalyst along with industrial switches. The industrial switches in particular come in form factors more suited to installation in moving equipment and power sleds. Where cabling can be pulled, there are increasing numbers of access ports everywhere – including riding on that borer.
IMG_4639
IE3000 switch in a cabinet mounted on the borer, along with some PLCs
  • Security – firewalls are big these days as we have begun to heavily protect those controls. Safety is always #1 and controls handle motors, valves, and high voltage electricity.
  • Wireless – Controls, VoIP, and mobile vehicles all operating where physical cabling doesn’t yet exist or can’t be pulled necessitate a lot of Wi-fi.
IMG_4608
Copper power and fiberoptic communication cabling wherever it can be
  • Data Centre – often not huge DCs, but virtualized servers and networking have been a huge help in maintaining safety and adaptability.

This is certainly the most interesting work I get to do – so I thought I would share some pictures and thoughts. I’ll end this one here, and I’ll write about more specific work in subsequent posts.