Swedish telecoms company Ericsson has confirmed that it is working to fix a problem that has caused a complete failure of the O2 data network in the UK and the SoftBank data network in Japan.
- UPDATE: Expired certificate identified as cause of O2 failure (see below for statement from Ericsson)
The failure of data networks on this scale calls into question their reliability for business critical data services.
It has also had severe repercussions for personal safety, revealing a worrying dependency on a service that is subject to single point of failure.
A statement on the O2 service status page read: "We’re aware that our customers are unable to use data this morning. One of our third party suppliers has identified a global software issue in their system which has impacted us. We believe other mobile operators around the world are also affected. Our technical teams are working with their teams to ensure this is fixed as quickly as possible. We’d encourage our customers to use Wifi wherever they can and we apologise for the inconvenience caused."
O2 provides phone and data services to 32 million customers, 25 million directly and seven million through third-party providers Sky, Tesco, Giffgaff and Lycamobile.
According to a report in the Financial Times, the outage has been blamed on problems with the data infrastructure provided by Swedish telecoms company Ericsson. Other affected mobile operators include the Japanese mobile network Softbank, also a customer of Ericsson.
An Ericsson spokesperson confirmed to SC Media UK that it is the provider. "We are aware of the issue and are working together with our customers to solve it as soon as possible," the company said.
Expired certificate identified as cause of software failure that took down O2 and other networks
Ericsson has issued the following statement:
Following network disturbances in a number of Ericsson’s customer networks, Ericsson has taken immediate action to minimize impact and support the restoration of services.
During December 6, 2018, Ericsson has identified an issue in certain nodes in the core network resulting in network disturbances for a limited number of customers in multiple countries using two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity).
Börje Ekholm, President and CEO, Ericsson, says: "The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers. We work hard to ensure that our customers can limit the impact and restore their services as soon as possible."
An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers. A complete and comprehensive root cause analysis is still in progress. Our focus is now on solving the immediate issues.
During the course of December 6, most of the affected customers’ network services have been successfully restored. We are working closely with the remaining customers that are still experiencing issues.
Downdetector.uk reported that the outage began at 5.37am.
The SoftBank outage in Japan began at 1.39pm local time.
Services which rely on the O2 data network are being affected. Early reports include the failure of Transport for London’s electronic timetable service at bus stops. Access to electronic maps is also affected, impacting those trying to find their way to unfamiliar places.
Couriers for Yodel and other delivery services which rely on O2 have also been affected. One Twitter user reported that he was unable to download his electronic train ticket for Virgin Trains.
Whilst you're posting jokes about #o2down be aware that not everyone has WiFi at home. A lot of disabled people rely on 4G to get from A to B. A lot of people also rely on being able to text elderly& vulnerable realtives who don't use the internet. It's not all selfies and memes.— Challis Zillwood (@challiszillwood) December 6, 2018
Taxi drivers are reporting that they are unable to connect and it’s costing them hundreds of pounds in lost fares.
@O2 is such a let down. As a taxi driver, I'm constantly using internet. But now message from O2 is I've no internet connection. Which makes me loose £200 worth of work.— Phionah (@phionah_hart) December 6, 2018
Machine to machine communication which relies on 4G networks, such as some alarm systems, will also be affected unless the system employs dual path signalling.
In light of today’s failure, businesses may reconsider plans for moving to new 5G mobile data services which are due to be launched in early 2019.
Around 48 percent of businesses in a recent survey covering North America and Europe reported they had experienced service outages due to internet failures. Some 95 percent of businesses report that they have a disaster recovery plan but a quarter of these (23 percent) have never tested it.
According to a 2013 press release from Ericsson, it provides the LTE-compatible core network for O2 and has deployed multi-standard base stations which support 2G/3G/4G across 50 percent of O2’s mobile phone network.
Ericsson became the sole provider of O2’s evolved packet core (EPC) in the UK. EPC is the mobile core network defined by 3GPP connecting LTE devices to the 4G radio network. The EPC is a simplified software-only, IP-based mobile core network with a flat architecture and support for packet switching only.