Owners of Belkin routers woke up yesterday to discover their modems were unable to access the Internet, as the vendor seems to have pushed out an update to older units, apparently whether or not the user had requested automated updates.
The scale of the problem remains unknown, but reports from the US suggest that many users of Belkin routers supplied by Comcast and Time Warner were affected, whilst Belkin itself posted a series of help files about the issue on its main Web site.
The Ars Technica newswire suggests that ISPs - "inundated with support calls by unhappy users, are directing complaints to Belkin's support line, which appears to have gone into meltdown in response."
"The reason for the massive outages is currently unknown. Initial speculation was that Belkin pushed a buggy firmware update overnight, but on a Reddit thread about the problem, even users who claim to have disabled automatic updates have found their Internet connectivity disrupted," says the newswire.
Some users are reporting that, since the problem appears to stem from malformed/corrupt DNS lookup tables, if they elect to use so-called public DNS services from the likes of Google, the issue can be sidestepped.
Ars Technica quotes Belkin as saying that, beginning around midnight on October 7, it began experiencing an issue with a service configured in certain Belkin router models that causes a failure when it checks for general network connectivity by pinging a site hosted by Belkin.
"We know this issue has affected select older Wireless-N Belkin router models including F9K1102, F9K1105, F9K1113 and F9K1116. We are continuing to investigate other possible routers that may have been affected. We are working to resolve this issue as quickly as possible. Until there is a fix in place, we have identified a workaround for customers that are using the F9K1102 and F9K1105 routers to regain Internet connectivity. That work-around as well as ongoing updates can be found at our outage status page for more information: www.belkin.com/outage," said the firm in its prepared statement.
Belkin later said that the problem had been fixed, asking users to power cycle their routers and wait five minutes for an update.
Nigel Stanley, practice director for cyber security at OpenSky UK, said that, whilst problematic modem updates were nothing new. What he was concerned about was the apparent fact that users were having their routers updated - even where they had explicitly stated they did not want updates to be automatically applied.
This, he said, can cause problems with the way third-party software operates, as the convention within many corporates is to thoroughly test an update in a controlled environment, before it is rolled out to users.
"That's my main concern. Whilst it's good to hear that Belkin moved swiftly to remediate its update, I think the methodology behind the update process needs further investigation," he said.
A bad thingOver at Lancope, Tim Keanini, the security vendor's CTO, said that automatic updates are usually a good thing - that is until they are a bad thing.
"This is a peek into the future of how things will be with more and more devices getting connected to the Internet from the home. The home user will not have the time or ability to be updating 50 different devices from 10 different manufacturers, so automated updates will be a necessity. Yet when things go wrong like this, everything is broken until the updater can provide a fix," he said.
Keanini went on to say that, as more of the workforce starts to work from home, and as more of the home starts to get connected with the Internet of Things, home versions of business continuity plans and disaster recovery routines will need to be practiced.
"This should be as common as your fire drill you do with your family on a regular basis," he explained.
Dwayne Melancon, CTO with Tripwire, meanwhile, said that, if this situation is due to a problem with an automated update then you can probably expect to see more of this in the future from other companies.
"As our reliance increases on the `Internet of Things; devices, automated updates to mass-market devices will become commonplace. The challenge is that these loosely managed devices can sometimes be unpredictable during updates. The big challenge is that when one of the updates fails, the device itself may not be in a condition to accept a corrective update," he said.
"In those cases, we rely on users to take care of the problem and in a lot of cases, the users are not experienced enough to actually complete the task. It is incumbent on device designers to ensure that the design of their device accommodates not only remote updates, but provides mechanisms to recover in the event of an ungraceful fail," he added.