Ericsson certificate expiry downs mobile data networks

Ericsson has apologised for a software glitch which took down data services for mobile networks across the UK and Japan for most of yesterday, blaming the issue - embarrassingly enough - on an expired security certificate.

Problems with connectivity on the O2 network in the UK, all mobile virtual network operators (MVNOs) which depend on the O2 network, and selected mobile networks in Japan were first reported early yesterday morning with symptoms including a complete loss of all data connectivity and, occasionally, the ability to make calls. Engineers scrambled to fix the problem, but the issue took most of the day to track down - and turned out to fall solidly at Ericsson's door.

According to the company's statement on the outage, the complete loss of data connectivity was caused by a boneheaded mistake: the expiration and failure to renew of a key certificate included in its networking software. Worse still, reports from engineers working on the problem have indicated that the expired certificate prevents remote administration - meaning that each affected system needs to be upgraded to the latest release in-person.

'The faulty software that has caused these issues is being decommissioned and we apologise not only to our customers but also to their customers,' says Ericsson chief executive and president Börje Ekholm of the flaw. 'We work hard to ensure that our customers can limit the impact and restore their services as soon as possible.'

A failure to renew security certificates - a process which can be entirely automated, leaving nothing to chance - is a surprisingly common cause of outages, though rarely to the scale experienced by Ericsson customers yesterday: Back in 2009 Gears of War became briefly unplayable due to a failure to renew the associated Games for Windows Live security certificate, while back in March this year another expired security certificate booted players off their Oculus Rift hardware. These issues, along with a general lack of adoption of TLS encryption for non-payment-processing services, were key in the Electronic Frontier Foundation's decision to launch the automated Let's Encrypt certificate authority service which allows users to request a security certificate and apply it to their servers in under a minute.