Hello? Anyone There? We Heard There Was A Riot!

How to deal with the unexpected.

We get a trouble ticket stating that a remote site in a third world country had no network connectivity. This was reported to us from an automated system rather than a human.

So we start the usual investigations - ping and trace routes.

Always timed out with the last hop being a local ISP. Very odd.

We tried to get in touch with relevant persons on site and at the ISP. Absolutely no joy. What could we do but wait for someone to arrive on site and contact us (we left numerous messages and emails).

During our lunch break one of the lads working with me had a news aggregator and the city where our site was located appeared. Large scale riots reported.

Madness, Mayhem & Anarchy apparantly.

So we knew that it was going to be a very long day waiting for someone to arrive on site. Eventually (I think possibly the following morning) we get a message from the site manager.

"Unable to log on to network, office broken into and damaged"

We then activate our local external engineer who arrives on site and calls us back.

Engineer "I found the problem, the network is gone"

Me "Ok...just reboot the switches and servers and lets see where we are"

Engineer "That could be a problem"

Me "?"

Engineer "Well...unless we can get data across two tins cans a long length of string I am going to need alot of CAT5"

Me "?"

Engineer "Yes...the robbers took all the copper cabling from the building"

Me "!"

Sometimes....things happen you have no control over.