Facebook outage - lessons for all businesses.

On Monday 4th October Facebook suffered an outage which made all its platforms – Facebook, WhatsApp and Instagram – unavailable to its 3.5bn users for almost 6 hours.

During this time rumours were rife that Facebook had been hacked, but the real reason, it appears, is much more mundane (though no less worrying). And it’s an issue that could be laying dormant in your business without you realising.

You can read in detail all the problems that came together to cause the outage, with a “faulty configuration change” capturing the headlines, but that configuration change was only the beginning.

That’s because that “faulty configuration change” also prevented their engineers making the simple changes that would have fixed the issue.

The config change effectively caused Facebook’s servers to stop telling the internet where it’s data centres were, so user devices couldn’t connect. It’s a simple fix but, crucially, it appears that Facebook’s own engineers needed to use the same network to log in and apply the fix, which meant that when they tried to connect to so, they couldn’t.

Effectively Facebook had created the very things all network designers seek to avoid – a single point of failure.

Clearly, it’s easy to sit here and criticise a huge behemoth of an organisation like Facebook after the fact. So rather than sticking the boot in, what can we learn from this and apply to our own organisations?

Get someone else to audit your network

Without their servers, Facebook doesn’t have a business. Imagine how much money they spend getting the best people available to ensure they are always available. And yet….

Could it be that they couldn’t see the wood from the trees? It’s a common problem in all areas of business, and in life in general, that when you work closely on something day after day, it can be hard to see the structural issues that might be obvious to someone else

So even if you have a crack IT team running your network, it might be useful to get a professional third party to examine your network design from time to time, just in case.

At One2Call we frequently undertake network reviews for customers, and even at their most high level, we always uncover areas for improvement. And quite often there are small and relatively inexpensive changes that can be made to make your network much more resilient.

Commit to regular audits.

Business systems rarely stand still. It’s natural for them to change with the business in order to facilitate growth. But every change, no matter how simple, has the potential to create problems that lay dormant, waiting to strike in the most horrible of ways (just like Facebook’s).

Most IT and Business Managers know that it’s best practice to regularly review and audit business systems and processes. But in times like this, when there are so many stresses and strains on every business, it can be easy to let it slide.

The best way to stop that from happening is to take it external. Sign a contract with external network consultants to undertake regular network reviews. There’s nothing like having committed money to a project to motivate managers to make sure it happens and succeeds!

From full-time network monitoring to annual network reviews, at One2Call we can tailor a service to fit your timescales and budget.

Stress test it – properly.

Full system testing is often something that everyone nods their heads about and agrees how important it is but can be much less enthusiastic about actually doing! And we get it. It’s scary. It’s a step into the unknown. It can be disruptive and involve bringing team in outside work hours.

But how can you know how resilient your business systems are without trying to break them! And fixing a problem you’ve found in controlled circumstances while the business is still merrily working away is much preferable to trying to do it with angry customers and Directors breathing down your neck – just ask Facebook!

From managed backups to failover configuration to a full Disaster Recovery planning, our experienced consultants can help set you off on the right track.

Conclusion.

What outages like this, and many of the headline-grabbing cyber security breaches, show us smaller businesses (well, smaller than Facebook anyway) is that big doesn’t necessarily mean better. It’s within the reach of all of us to run our crucial business systems just as well, if not better than some of the world’s biggest companies!

Schedule a Consulatation

Why not book time with our consultants now, for free?

Book your 15 or 30 minute session directly into their diaries now.