The agency's so-called 'Hacienda' programme, revealed by German publication Heise, started in 2009 when GCHQ decided to apply the ‘standard tool' of port scanning against entire nations.
Documents published by Heise show GCHQ fully trawls 27 countries – meaning it “randomly scans every IP identified for that country” – and partially scans five other nations. The 32 country names are blanked out in the report.
Heise explains that port scanning has been used by attackers since the early days of TCP, exploiting a flaw in the handshake between TCP clients and servers which means the server leaks information without checking the client's authorisation.
Heise warns: “It is not the technology used that is shocking, but rather the gargantuan scale and pervasiveness of the operation. The massive use of this technology can make any server anywhere, large or small, a target for criminal state computer saboteurs.”
The documents show GCHQ uses Hacienda for “vulnerability assessment, network analysis and target discovery”, and to detect ORBs (operational relay boxes) – servers it can use as stepping stones to hide the source of its attacks.
Referring to previously revealed GCHQ hacks, Heise says: “As shown with the penetration of Belgacom and Stellar, when an employee's computer system or network credentials may be useful, those systems and people are targeted and attacked.”
The agency gathers data on any weak server including the host name, banners (which provide system and application information), application names, port status, directory listing and content of the main page. It also profiles the machines concerned, including browser, operating system, PSP and patch history.
The data is then shared with the UK's partners in the Five Eyes group of countries - the US, Canada, Australia and New Zealand. This is done through the cheekily tongue in cheek named “MAILORDER” programme.
Heise says the process of scanning entire countries for vulnerable network infrastructure is consistent with GCHQ's £1 billion programme of "Mastering the Internet", whereby the agency tries to attack every possible system it can.
It concludes: “System and network administrators now face the threat of industrial espionage, sabotage and human rights violations created by nation-state adversaries indiscriminately attacking network infrastructure and breaking into services.
“Every system or network administrator needs to worry about protecting their system against this unprecedented threat level. In particular, citizens of countries outside of the Five Eyes have, as a result of these programmes, greatly reduced security, privacy, integrity and resilience capabilities.
“System administrators need to improve their defensive posture and, in particular, reduce the visibility of non- public services.”
Despite these warnings, UK-based cyber industry experts are less concerned at the revelations.
Professor Mike Jackson of Birmingham City University told SC via email: “For years hackers have used port scanning techniques to look for open ports. Now we hear that GCHQ is also using these techniques to compile lists of vulnerable machines. In many ways it is comforting to know that they are being proactive in this way. Since countless hackers will have already garnered this information, it is only to be expected that our security services should also gather it. “
Information security researcher and author David Lacey was equally sanguine, telling SCMagazineUK.com: “Governments have been gathering intelligence for hundreds of years. You expect intelligence services to gather intelligence, it's their job.”
He added: “In the past it didn't have as much oversight as it does today. Of course it's more powerful than it was before. But it's very hard to find examples of misuse. There's no evidence that there's a big problem from this.”
But as far as security professionals are concerned, Lacey said: “It's another reminder for CISOs in companies operating in international markets that they need to be aware that just about every country gathers intelligence for economic or commercial purposes as well as national security, and you need to understand the risks and the consequences of this.”
Jackson had similar advice: “Any open port means that there is a potential for vulnerability. Whether it is the government or random individuals involved, system administrators should, as a matter of course, check that they know who in their organisation can attach to ports and what software is showing its public face to the internet.”
The Heise article was written by a team of investigative journalists, together with academics at Germany's Technische Universität München (TUM). Alongside the article, TUM has produced a fix for any system admins worried they are being trawled by GCHQ or other attackers - the TCP Stealth port knocking solution. This software can be downloaded at https://gnunet.org/knock.