Ericsson software issue causes O2 data network outage

News by Tom Reeve

Ericsson has confirmed that it is working on a fix for a network problem that has caused a blackout in the O2 data network in the UK.

Ericsson mobile mast

Swedish telecoms company Ericsson has confirmed that it is working to fix a problem that has caused a complete failure of the O2 data network in the UK and the SoftBank data network in Japan.

  • UPDATE: Expired certificate identified as cause of O2 failure (see below for statement from Ericsson)

The failure of data networks on this scale calls into question their reliability for business critical data services.

It has also had severe repercussions for personal safety, revealing a worrying dependency on a service that is subject to single point of failure.

A statement on the O2 service status page read: "We’re aware that our customers are unable to use data this morning. One of our third party suppliers has identified a global software issue in their system which has impacted us. We believe other mobile operators around the world are also affected. Our technical teams are working with their teams to ensure this is fixed as quickly as possible. We’d encourage our customers to use Wifi wherever they can and we apologise for the inconvenience caused."

O2 provides phone and data services to 32 million customers, 25 million directly and seven million through third-party providers Sky, Tesco, Giffgaff and Lycamobile.

According to a report in the Financial Times, the outage has been blamed on problems with the data infrastructure provided by Swedish telecoms company Ericsson. Other affected mobile operators include the Japanese mobile network Softbank, also a customer of Ericsson.

An Ericsson spokesperson confirmed to SC Media UK that it is the provider. "We are aware of the issue and are working together with our customers to solve it as soon as possible," the company said.

Expired certificate identified as cause of software failure that took down O2 and other networks

Ericsson has issued the following statement:

Following network disturbances in a number of Ericsson’s customer networks, Ericsson has taken immediate action to minimize impact and support the restoration of services.

During December 6, 2018, Ericsson has identified an issue in certain nodes in the core network resulting in network disturbances for a limited number of customers in multiple countries using two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity).

Börje Ekholm, President and CEO, Ericsson, says: "The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers. We work hard to ensure that our customers can limit the impact and restore their services as soon as possible."

An initial root cause analysis indicates that the main issue was an expired certificate in the software versions installed with these customers. A complete and comprehensive root cause analysis is still in progress. Our focus is now on solving the immediate issues.

During the course of December 6, most of the affected customers’ network services have been successfully restored. We are working closely with the remaining customers that are still experiencing issues.

Downdetector.uk reported that the outage began at 5.37am.

The SoftBank outage in Japan began at 1.39pm local time. 

Services which rely on the O2 data network are being affected. Early reports include the failure of Transport for London’s electronic timetable service at bus stops. Access to electronic maps is also affected, impacting those trying to find their way to unfamiliar places.

Couriers for Yodel and other delivery services which rely on O2 have also been affected. One Twitter user reported that he was unable to download his electronic train ticket for Virgin Trains.

People are also reporting problems caring for disabled and elderly people who rely on 4G networks to contact carers and relatives.

Taxi drivers are reporting that they are unable to connect and it’s costing them hundreds of pounds in lost fares.

Machine to machine communication which relies on 4G networks, such as some alarm systems, will also be affected unless the system employs dual path signalling. 

In light of today’s failure, businesses may reconsider plans for moving to new 5G mobile data services which are due to be launched in early 2019.

Around 48 percent of businesses in a recent survey covering North America and Europe reported they had experienced service outages due to internet failures. Some 95 percent of businesses report that they have a disaster recovery plan but a quarter of these (23 percent) have never tested it. 

According to a 2013 press release from Ericsson, it provides the LTE-compatible core network for O2 and has deployed multi-standard base stations which support 2G/3G/4G across 50 percent of O2’s mobile phone network.

Ericsson became the sole provider of O2’s evolved packet core (EPC) in the UK. EPC is the mobile core network defined by 3GPP connecting LTE devices to the 4G radio network. The EPC is a simplified software-only, IP-based mobile core network with a flat architecture and support for packet switching only. 

Find this article useful?

Get more great articles like this in your inbox every lunchtime

Upcoming Events