Amplex Network Log

Okay, so Microsoft has amazed me in light of their usual track record with software patches, and created a patch that doesn’t appear to blow anything up. Granted, all I use my Windows XP image for is the running of the Copilot remote assistance software, for those especially difficult support sessions. But, I can say that so far, Windows XP Service Pack 3 has broken nothing that I am aware of.

In fact, without reading the documentation for the patch, I would have almost no idea what this patch did accomplish aside from taking an inextricable amount of time to install. That being said, there is one cool feature that I am very happy to see. Windows Vista, being the giant ugly, resource hogging, unusable piece of bloat ware that it is, does have a really cool network level feature called internet black hole detection. Basically, internet black holes are created when less skilled network admins filter way too much of the useful signaling traffic the internet depends on to function properly. This is often done with the careful deployment of many, badly configured firewalls. Don’t get me started on firewalls, they accomplish about one tenth of the things people think they do, but I am getting sidetracked here. The end result of a network that has created a black hole in itself is that a very useful signaling system called Path Maximum Transmission Unit Discovery, or PMTUD is broken. The fallout from breaking this obscure, but tremendously important protocol is that many websites on said black holed network will be randomly inaccessible to many of us end users, who would like to visit those websites. Like I said, Windows Vista has a black hole detection and correction service written into the network level. With the installation of Service Pack 3, Windows XP gains this little feature.

In my opinion, this feature alone makes Service Pack 3 a good choice for immediate installation by most people. PMTUD issues are very frustrating for us, as they are the result of absolutely nothing wrong with our service, but we are usually charged with finding a workaround for them anyways. It is pretty hard for us to fix something that is broken on someone else’s network. But, the addition of this patch into a very mainstream operating system such as Windows XP, means that a general purpose workaround now exists that is readily available by default. The better solution would be to educate other networks about how black holes get created, and why they are so very, very bad. But, I suppose that will never happen. The next best thing is to work around them.

I can honestly say, I am so behind on caring about Microsoft Products, that I had no idea SP3 was even in the works, but it was, and it is here! Windows XP Service Pack 3 has been released as of today. As with all software patches, especially those labeled Microsoft *anything*, I am extremely wary of actually installing this puppy. Oh, I am sure it fixes many, many gloriously awful issues, and of course, causes 3 times as many new ones. I just can’t get excited about trading the bugs I know, for a whole host of new ones. But, I do have an ace in my sleeve today. I don’t run Windows XP as my primary workstation. I keep a copy of it running on a virtualization system called VMWare. VMware allows me to run a virtual computer, on my computer. So, I can have a copy of Windows XP, a copy of Windows Vista, and I can play with the latest Linux distributions, all without gumming up my workstation. It has lots of cool features too, like the ability to back itself up to a known safe state, so I feel pretty confident that installing this service pack won’t annoy me for very long. But I am rambling, I should I stop that.

Like I said, I am wary of any major Microsoft patches, and I strongly suggest you be wary as well. Better that I install SP3 on my virtual computer, and it blow up, rather than you install it on your PC, and it blow up. My virtual computer is very easy to fix, and I wouldn’t care very much if it wasn’t. I will post my impressions of the patch in a few days, after I have had several moments to see what breaks, what is cool, and of course, what makes me shake my head in utter disbelief.

In order to better serve you, the customers, and to keep our own sanity, we are starting to use a trouble ticketing system. This is helpful because, as we have grown, we have discovered we cannot keep all of this stuff in a pile of notes on our desk, nor does it all fit in our brains all that well. So, we have centralized all the case notes in a very easily accessed system. The hope is, we will forget less, and when we have a caller with an ongoing issue that may span several days, one of us does not have to play the, “I wonder what the last tech already tried” game, we can just look at the case notes! I know, nothing terribly exciting, but it is important, and we hope it makes our support service faster and more effective.

Our Oregon Road tower has been driving us nuts for months. Odd things happen every time even a small weather event happens - UPS’s shut off for no apparent reason, power supplies blow out, switch ports dying, etc. I know… all of this sounds like either a power or grounding problem. But where and why?

The tower itself is a water storage tank. Steel. Full of water. Connected to miles of underground pipe. You can’t build anything with a better ground if you wanted to.

Power to the site is standard utility power - nothing special but the other tenants on the tower and the owner don’t seem to have any power problems. Why should we?

The weirdness started last summer (2007) when we installed a UPS (backup power) and a network switch at the tower. When a small storm would come by the UPS would shut down. Run over to the tower and everything looked normal but the UPS would be off. Push the power button and it comes right back on. After the 2nd time this happened we ordered a new UPS and installed it.

Due to some interference issues we replaced the backhaul radios feeding the tower. About 2 weeks later a small snowstorm came through and the site shut down again. When I get there the power supplies for the radios are dead and one of the switch ports is dead. At this point I’m wondering if there is something wrong with the new backhaul radios so we switch back to the old ones and test the new ones at the office - they work fine. Now what the heck does all this mean?

Fast forward to Friday night 3/21 when yet another snow storm comes through. Yep - Oregon Road is dead again. Run over there and I can’t believe what I’m seeing. Sparks. Really big sparks. Sparks coming from the network cables jumping to the UPS. Jumping to me when I get too close. Unplug the cables from the power injectors and sparks jump from the cable to the nearest ground or just crack between pins on the cables. Ack! Where the heck is all this electricity coming from? What’s different about this tower?

Time to backup a little bit. Our access points are a Motorola Canopy radio modified by Last Mile Gear (http://www.lastmilegear.com) with a better antenna and a stronger case called a Cyclone. The older Cyclones had a metal mounting bracket that was a pain to install but was all metal and grounded the radio case to the tower. The bracket design was improved about 2 years ago to one that is much easier to install but does not ground the radio case or the antenna.

We use shielded cable to connect the AP’s to the equipment at the base of the tower. We intentionally do not connect the shield at the top of the tower and ground the shield at the bottom. This design creates a Faraday cage inside the shield and prevents electrical noise from entering (or leaving) the cable. The shield is not intended to be a ground for the equipment. If you connect both ends of a shield you no longer have a shield but a conductor.

So what was creating all of the sparks? The only logical explanation is that the dry snow blowing past the antenna and case generates a static charge on the case. As the case is not grounded the charge builds up until it can jump across either the insulated mounting bracket or onto the shield of the Ethernet cable. The shield being closer that was were the charge was going, traveling down the cable, and discharging into the Ethernet switch at the base of the tower. Can blowing air generate much static? Well - I know it occurs on airplanes when flying in precipitation. Helicopters generate very high static charges on the airframe when flying in snow or dust. Apparently it also happens on isolated Access Points as well.

So why are the Cyclone Access Points (AP) not grounded? The idea was that lightning normally discharges from the ground to a cloud (despite how things look). A lightning bolt starts with both a charged path coming from the ground up toward a cloud that meets the bolt coming down from the cloud. The theory is that if you ground the AP you increase the risk that the leader will start at the AP resulting in a greater chance of damage. All good in theory - except that allowing the AP to accumulate static charges due to wind and snow is causing more damage than lightning itself.

We replace the AP with a new one and grounded it very well to the tower. At the base we installed good surge suppressors on both lines coming from the AP and BH with good grounding at the base. We replaced the switch with a new one since the old one was damaged.

I really hope this is the end of this mess. Now to go address the issue at all of the other towers…..

Updated 3/24/08

I had sent Brian Magnuson from Last Mile Gear an email regarding what we had found and asking about grounding the case given the static issues. Brian was kind enough to call me back this morning and we had a long discussion regarding grounding practices for the radios. Brian indicated he had not seen too many other cases where the Cyclone appeared to be collecting a large static charge. He did suggest grounding the AP using the shield in the STP CAT5 cable rather than leaving it open at one end (the common practice for shields). Brian also suggested putting Motorola 600SS’s (surge protectors) at both the top and bottom of the tower (unless you are using a CTM in which case put the 600SS’s only at the top).

I went back to the tower this afternoon and added the surge suppressors at the top of the tower. I had installed a ground wire to the case of the Cyclone on Friday night and decided to leave that in place as I did not have the proper shielded RJ45 connectors in order to ground the Cyclone using the drain wire. I may modify this the next time I need to climb the tower.

As far as the ground lug on the Cyclone case - Brian indicated that he didn’t think they were doing that yet but were going to be doing so on the units with timing onboard.   The reason for adding the ground was that the timing onboard units have surge suppression built into the case and therefore need a ground reference.   Not providing a ground to the case on the existing units was an intentional decision to try to avoid lightning issues.  In our situation where we are seeing strong static buildup it may be necessary to ground the case.

My latest project around here, in between all the other urgent projects, has been to deploy a new version of our webmail service. Frequent users will be relieved to know that we are not switching platforms, our webmail is still based on the horde system, so the usual features everyone enjoys will be present, but now, there will be some new ones.

The biggest difference between the old software, and this one, is that the old software is just for email. The new version is what is commonly called groupware. it has a lot of the features of MS Exchange or IBM Lotus Notes; the already known address book, but also notes, calendars, appointments, and the ability to share any of these with other users. Their is also a new spam reporting feature, which will tie in with an adaptive filter on the mail servers, and hopefully make our spam tagging much more accurate.

Of course, this is a work in progress, in between all the other more urgent issues I tend to deal with regularly, and as such, is guaranteed to be broken in more ways than one. Just a few:

• Report as spam does nothing at this time, pretty sure you’ll get a bounce notice if you try to use it.
• Address Books have not been migrated from the old database to the new one, I’m kind of dreading this step. I am also regularly blowing away the address book data in the new database, as I test, and retest, and retest, while I try to build a migration script for existing data. So, expect your address book to disappear from the new software quite regularly until I determine I am “done” with this part. No, this won’t hurt your current address book, once I have a migration solution ready, I will mirror the old data into the new software. *UPDATE* The solution for this issue was far easier than I could have dreamed, currently address books from 2008-03-21 are mirrored from the old webmail, and I will re-mirror them at least one more time.
• Filter Rules have all the same issues I just said about address books, hence, I am also fearing migrating existing filters. The process is about the same, so when I finish one, the other should follow close behind.
• Mail Folders are stored in a slightly different place in the new software, which causes it to break all the sent-items folder settings for just about everybody. I’m trying to shoehorn that back into the old spot, so this issue will just go away.
• You might notice that the main page is a customizable portal, with many potential add-ons, one of which is a program called ‘fortune.’ Fortune is a fun program that basically generates odd little quips not unlike a fortune cookie. Some of them are quite hilarious. There is an option to display potentially offensive fortunes, as well as the tamer variety. Please don’t add the offensive ones to your portal if you are offended by just about anything, most of them make a wardrobe malfunction look minor.
• No, I don’t know why there is an ‘@’ symbol next to the user name box on the main login screen. I really want it to go away, but I can’t figure out where in the code it is created. It doesn’t hurt anything, so this is the last issue I will probably take care of, unless I find it while looking for something else.
• And I guarantee, there is much, much more.

So, that being said, if you want to tinker around, with all those issues tucked away in the front of your mind, feel free. The new webmail site is located at, new-webmail.amplex.net. The one thing that shouldn’t be effected is your actual stored email. Letters are stored on the mail servers, the webmail software resides on the web server, which simply accesses your mailboxes. It is the one thing that just works, everything else….. not so much.

It appears from the signal strength graphs that the Curtice tower 900Mhz south facing sector antenna has water in the RF Connector. During the one above freezing day we had this week (Thursday) the signal strength for all the customers off that sector dropped about 12dB when the moisture in the connector turned from ice to water. The problem ‘corrected’ itself around 2am when the moisture refroze. This RF loss that comes and goes with temperature is a pretty good signal that there is water in places it shouldn’t be. Clearing the water may take 1/2 hour as I will need to heat the connectors to drive the moisture out but it needs to be done carefully so as to not damage the connector in the process. My plan is to do this Monday afternoon/early evening. With the other 2 sectors in place at Curtice this should only affect a few customers.