Todays Blip in Service

openmind · #1 (**permalink**) 02-05-2008, 10:54

Sorry about today's minor outage, there was a major issue in London which knocked out the data centre we are in along with about a billion* other sites. More info here:
www.openmindstatus.com

* OK a billion is a lie but it would have been lots

RayB · #2 (**permalink**) 02-05-2008, 10:56

And I resisted the urge to ring you in a state of panic and withdrawal

openmind · #3 (**permalink**) 02-05-2008, 11:03

You could have done along with the other 30 other odd calls I got in less than 30 minutes

Indizine · #4 (**permalink**) 02-05-2008, 13:47

does tis happen often phil?

openmind · #5 (**permalink**) 02-05-2008, 14:13

Thankfully not. The cause of this was one of the main peering points for the UK falling over. Our servers stayed up and running as did the data centre, the problem was the transit provider fell over.

Thankfully this is pretty rare and the DC have safeguards in place which kicked as they should to use alternative transit.

At the end of the day there was nothing I nor the data centre could have done to foresee or prevent this happening.

boxby · #6 (**permalink**) 02-05-2008, 15:57

Quote:

Originally Posted by openmind

At the end of the day there was nothing I nor the data centre could have done to foresee or prevent this happening.

Are you absoluetly sure?

openmind · #7 (**permalink**) 02-05-2008, 16:00

Quote:

Originally Posted by boxby

Are you absoluetly sure?

Absolutely

You know I operate an honest and no bull approach to our services, I could have fudged my way round this today but that's not how we work.

Indizine · #8 (**permalink**) 02-05-2008, 16:01

when you sell fell over? what does that mean exactly? surely they do a risk assdessment which picks up potential issues like that? (ps I know it's not your fault as youre as much the end used as your cusotmer in this respect).

openmind · #9 (**permalink**) 02-05-2008, 16:06

It simply means for reasons that are yet to be cascaded down to me, Linx simply suffered an outage. This then snowballed into other UK ISPs and our data centre.

Luckily our DC do not rely on just one upstream provider, they have several so they were able to switch to an alternative fairly quickly. Other hosts won't have been so lucky.

Sometimes things just snap without warning. Like a lightbulb blowing, it takes time to find and replace it with a new one. Problems like this are so rare though simply because upstream providers as big as Linx are able to detect upcoming problems and act pro-actively. Very, very occasionally though things can just die without warning.

It's the redundancy setup you have, and we do, that dictates how long it takes to recover.

RayB · #10 (**permalink**) 02-05-2008, 16:07

It was only down for a few minutes though Phil?