So what can we do to decrease the likelihood of another outage such as that on Friday and minimize the impact on any outage that does happen?
Long term, we know the answer. We need to increase our network density to the point where most of our members have at least two independent paths back to (ideally) different local head ends, (ideally) served by different upstream fiber providers. That’s the only really robust solution I’m aware of and that is our plan for the long term.
Today our pilot network has just one upstream vendor, Cogent, and just one head end, at the John Hancock Tower. In March, we expect to add a second head end and then a third head end by the summer. We’re also investigating a second upstream fiber provider for at least one of the next two local head ends.
Meanwhile, there are a few items I can address:
1. When there is a problem, call Cogent Support if there is any chance their network monitoring tools could help shed light on what’s happening. I wasted over an hour because I thought I’d made a mistake and therefore I shouldn’t bug Cogent. Bad idea, especially as the problem turned out to be in Cogent’s fiber. Even if it had been my mistake, Cogent Support could have helped establish what was going on, so I should have called them immediately.
2. Arrange a better procedure for after hours access at the John Hancock Tower. This is a minor issue, but things could have worked more smoothly.
3. Get proper lighting in and around our rack, and also a chair. This was the first time I’d had to work on our gear after dark and it quickly became apparent that the lighting in the 61st floor mechanical room doesn’t reach to the far corner where our rack is located. During the day, we have natural lighting (and a spectacular view). At night the view is still spectacular, but our rack is in the dark.
4. The moment we get a second local head end on line we need to establish a wireless path between the two so we can provide backup bandwidth in the event of future fiber problems.
PS: Here are daytime views from our rack looking west and northwest.
