Well guess what. Apparently you need power to run servers, wireless radios and pretty much everything. Not once, but twice now at our main server room and radio tower we had none.
That didn't go so well for us and all our customers. We were just as unhappy as our customers were.
So basically the generator on site that is used to restore power should there be an outage simply didn't work. Nope. Nada. Uh,ah. Really annoying.
The UPS we had on site ran down quickly which took offline all servers, routers, and radio links. We raced to site and restored power as quickly as we could with another temporary generator and power leads. However the outage did affect our servers and bringing them online - so we had to fix those issues which also led to some delays.
Needless to say we got it going. And we learnt alot more about our failure points in our network. And we are fixing them.
We have scheduled the generator to be fixed and re-tested (regularily) with the company that owns and runs the generator. We have also invested in a larger more robust UPS with alarming. We are also re-configuring our systems to allow for faster recoveries and less likely interruptions from power outages.
So - yeh - not great - but what doesn't break you makes you stronger - and this outage will make Dreamtilt stronger as we fix these issues. We have to. Our customers deserve it.