Over the weekend of July 19-20, both the BBC website and iPlayer VOD website were hit by various, unexplained ‘technical issues' which prevented users from visiting certain sections on the bbc.co.uk website as well as watching their favourite shows on iPlayer. Even now – almost a week later – some users are still reporting issues with iPlayer Radio.
At the time of the initial incident, a pop-up message on the website said that “technical problems” were the reason for displaying a simplified version of the BBC Homepage adding further that it was working to restore normal service.
A statement issued shortly afterwards to SCMagazineUK.com continued with this same theme: "We're aware of an issue which means some people can't access certain parts of BBC Online. We're working hard to fix this as soon as possible."
A former member of the iPlayer team said that 24 hours was a ‘hugely long time' for the service to be unavailable.
The downtime led some – especially in light of BBC's relative lack of communication - to suggest that the corporation had been affected by a DDoS attack, quite possibly by pro-Palestine hackers unhappy with the news website's Gaza coverage.
Corero, a US-based DDoS and network defence company, put out a statement warning on the dangers of DDoS shortly after the incident and the firm's CTO, Dave Larson, told SCMagazineUK.com that a DDoS attack of some kind was the most likely explanation.
“Their statements around observing spikes in bandwidth would almost certainly indicate a form of DDoS as a component of an attack,” said Larson, who said that volumetric DDoS attacks are evolving in scale and bandwidth. He added that DDoS attacks are often used for a ‘brute force' takedown of a firewalls, IPS and router, or occasionally just for misdirection.
“It looks to me like a hybrid attack – what surprises me is that it took them 48 hours to restore the service in full.”
“My perspective is that this kind of prolonged outage seems to indicate more than DDoS. It leaves us all in the dark.”
Two sources with close connections to the news corporation told SC that the outage was not a DDoS attack but was most likely a software/infrastructure issue that arose during maintenance. Another said that a DDoS attack would have been appeared on the LINX/LONAP/AMSIX peering sites controlled by some ISPs to get online content up as fast as possible.
Despite declining to respond to our emails on the cause of the matter, BBC's controller of digital distribution Richard Cooper put out a detailed blog post on Wednesday on how the situation unravelled.
“As many of you will have noticed we suffered a serious incident over the weekend which impacted BBC iPlayer, BBC iPlayer Radio, and audio and video playback on other parts of bbc.co.uk. We also had to use our emergency homepage for prolonged periods of time.”
He went onto say that BBC's web systems comprises 58 application servers across 10 data servers to provide programme and clip metadata. The system cuts across two data centres in a “hot-hot” fashion (both are running at the same time).
“At 09.30 on Saturday morning (19th July 2014), the load on the database went through the roof, meaning that many requests for metadata to the application servers started to fail.”
But the blog post failed to appease many readers, who criticised BBC for not commenting on the cause of the attack.
“What was the reason for the "through the roof" request spike?” said one commentator. “Interesting, I'd love to know where the load came from, was it a DDoS or a malfunctioning consumer device?” asked another.
There was also some criticism for the BBC in the way it carries out digital forensics on the network, especially in light of its most recent report which admitted (p85) that the corporation needs to develop “effective monitoring and forensics capabilities” for the IT systems to reduce the risks to content delivery.
“Why - in 2014 - does the BBC not already have effective network monitoring? Is the lack of forensic capabilities not going to hamper your investigations, as you repeatedly mention that forensic investigation is essential to finding the root cause?” said the user ‘Kelly Paterson', quoted on Cooper's blog post.