BT Broadband outage blamed on power failure [updated]
BT Broadband has suffered a major outage this morning and it's pointing the finger at a power-outage in one of its central London service providers.
Who pulled the plug on BT?
Upon further investigation, it appears that this mornings BT outage is down to a power failure at Telecity Harbour Exchange, where BT as well as various other ISPs join the LINX peer exchange. The power outage at Telecity lasted for about an hour, from just before 8am until 9:15am, when full connectivity was restored.
Equinix, owner of Telecity said: "This morning between 07:55 BS and 08:17 BST, one of the datacentres that houses equipment for The London Internet Exchange (LINX) experienced a partial power outage. This affected only one of a number of Internet peering nodes that LINX operates at the facility, and service was fully restored on the LINX network at 09:15 BST."
A spokesman for LINX told the BBC's technology correspondent Rory Cellan-Jones, "We take any outage very seriously. We will be having very serious conversations with Telecity about how this happened."
BT Broadband has suffered a major outage this morning, leaving many of its customers without internet access.
As we were set to publish this article, a BT spokesperson told SCMagazineUK.com that the problem had been resolved and full service would be restored soon.
The broadband provider the UK's largest broadband outfit and has put the outage down to a power failure on a site in London. It is unconfirmed where the outage occurred but speculation is, given that it appears to be a DNS outage, that it was a problem with the London Internet Exchange (LINX).
On BT's Service Status updates page, the company said, “A small number of our customers in the areas shown below, may experience a loss of telephone and/or broadband services.”
However many dialing codes were affected, as demonstrated by this map:
I'd say BT broadband is in a spot of bother right now pic.twitter.com/h0g817SVB8— Barry Collins (@bazzacollins) July 20, 2016
The company's customer support Twitter account said it was attempting to restore service as quickly as possible:
Some BT Broadband customers have reported problems accessing websites this morning. Sorry, we're aware of a problem & working to fix asap— BTCare (@BTCare) July 20, 2016
The vulnerabilities of the DNS server system are well known within the IT industry, as Prof Alan Woodward of the department of computer science at the University of Surrey told SC.
“We've known for a long time that this architecture needed to be more robust because it does make it vulnerable to attack,” he said.
In a blog post in 2013, Woodward wrote about the risks inherent in the border gateway protocol (BGP) which sets the rules for how internet peers share routing information.
In particular, he flagged up the vulnerability of the global network of internet exchanges which help facilitate this information sharing.
The London Internet Exchange (LINX) is one of these points of vulnerability, and if it turns out that it was a failure there that caused this morning's outage, then according to Woodward, “it's really not acceptable for such a vital national asset”.
“Where are the backup generators, the graceful degradation of service? And why did it take so long to resolve?” he said. “BT are supposed to be providing a 21st Century network but the fact that they then peer that network via LINX and simply throw up their hands saying ‘problem with our partner' does not fill one with confidence. If what BT say is true [about the cause of the failure] then we don't need attackers: we just need the fuse to blow.”
The outage comes a day after MPs said that BT's broadband infrastructure division, Openreach, should "put its house in order" or face being split from the company itself. The allegations come in relation to BT's quality of service, where it would show favourable treatment to BT services.
In the past half an hour SCMagazineUK.com spoke with BT, and a spokesperson for the company said, “customers were being told that the service was slowly recovering” and advised that "it will take time to reset profiles and for speed/performance to get back to normal."
Paul Heywood, managing director EMEA at Dyn told SC, “Outages such as the one experienced by BT's broadband service this morning need to eliminated in today's digital world. Consumers' expectations have never been higher than they are now and consequently, outages and downtime detract from brand value, customer loyalty and the likelihood that consumers will come back for repeat business. Looking to a secondary DNS provider, for example, will ensure the digital experience you provide for customers and partners will always remain consistent, regardless of time or circumstance.”