The Apache Software Foundation has confirmed that it was hit by a direct, targeted attack, specifically the server hosting its issue-tracking software.
The Apache infrastructure team confirmed that it uses a donated instance of Atlassian JIRA as an issue tracker for its projects, which it uses to track issues and requests. In a report, it said that on 5th April, the attackers opened a new issue, INFRA-2591, via a compromised Slicehost server. This issue contained a message claiming ‘I've got this error while browsing some projects in JIRA' with a weblink included via a URL-shortening service.
It claimed that the specific URL redirected back to the Apache instance of JIRA, at a URL containing a cross-site scripting (XSS) attack. The attack was crafted to steal the session cookie from the user logged-in to JIRA and when this issue was opened against the infrastructure team, several of its administrators clicked on the link. This compromised their sessions, including their JIRA administrator rights.
At the same time, the attackers started a brute force attack against the JIRA login.jsp, attempting hundreds of thousands of password combinations, and on 6th April one of these was successful.
The report claimed: “Having gained administrator privileges on a JIRA account, the attackers used this account to disable notifications for a project, and to change the path used to upload attachments.
“The path they chose was configured to run JSP files, and was writable by the JIRA user. They then created several new issues and uploaded attachments to them. One of these attachments was a JSP file that was used to browse and copy the file system. The attackers used this access to create copies of many users' home directories and various files. They also uploaded other JSP files that gave them backdoor access to the system using the account that JIRA runs under.”
By the morning of 9th April, the attackers had installed a JAR file that would collect all passwords on login and save them. They then sent password reset mails from JIRA to members of the Apache infrastructure team and these team members, thinking that JIRA had encountered an innocent bug, logged in using the temporary password sent in the mail, then changed the passwords on their accounts back to their usual passwords.
The attackers were thereby able to login to brutus.apache.org, and gain full root access to the machine. This machine hosted the Apache installs of JIRA, Confluence and Bugzilla, and once they had root on brutus.apache.org, the attackers found that several users had cached subversion authentication credentials, and used these passwords to log in to minotaur.apache.org (aka people.apache.org), its main shell server. On minotaur, they were unable to escalate privileges with the compromised accounts.
Finally, after six hours after they started resetting passwords, it noticed that the attackers had begun shutting down services. Apache said that it notified Atlassian of the previously unreported XSS attack in JIRA and contacted Slicehost.
It said: “Atlassian was responsive. Unfortunately, Slicehost did nothing and two days later, the very same virtual host (slice) attacked Atlassian directly. We started moving services to a different machine, thor.apache.org. The attackers had root access on brutus.apache.org for several hours, and we could no longer trust the operating system on the original machine. By 10th April, JIRA and Bugzilla were back online.”
It claimed that limited, specifically one-time passwords, were ‘a real lifesaver' as if JIRA passwords had been shared with other services/hosts, the attackers could have caused widespread damage to the Foundation's infrastructure.
It concluded that the same password should not have been used for a JIRA account as was used for sudo access on the host machine, while inconsistent application of one-time passwords were required on some machines but not on brutus, while SSH passwords should not have been enabled for login over the internet.
In its changes, it is making one-time-passwords mandatory for all super-users, on all of its Linux and FreeBSD hosts, and has disabled caching of SVN passwords, and removed all currently cached SVN passwords across all hosts. It said: “We hope our disclosure has been as open as possible and true to the ASF spirit. Hopefully others can learn from our mistakes.”
Toralv Dirro, McAfee Labs EMEA security strategist, claimed that Apache's advice will offer an example of how to document an attack should you ever become the unfortunate victim of one. He said: “So administrators - knowledgeable and security-minded users - with elevated privileges opened an unverified link that was supplied by an external (anonymous?) source. Worse: the link was clearly obfuscated. This is where all technical security measures fail. “Users worldwide are told again and again to be very careful with links in email and social networks, especially when they come from an untrusted source. Well, the fact that Koobface is alive and spreading makes it obvious that users still are too happy to click on any link they get. That experienced administrators fall for this makes the future look gloomy indeed.”
He said: “So administrators - knowledgeable and security-minded users - with elevated privileges opened an unverified link that was supplied by an external (anonymous?) source. Worse: the link was clearly obfuscated. This is where all technical security measures fail.
“Users worldwide are told again and again to be very careful with links in email and social networks, especially when they come from an untrusted source. Well, the fact that Koobface is alive and spreading makes it obvious that users still are too happy to click on any link they get. That experienced administrators fall for this makes the future look gloomy indeed.”
Commenting on the Atlassian attack, Amichai Shulman, CTO at Imperva, said: “This is an example of a database that was forgotten and left unprotected - something that happens more frequently that most would prefer to admit. In this case, the database contained sensitive information, but once it wasn't used as a production system it was forgotten. Unmanaged systems put sensitive data residing on them at a high risk - unmanaged systems are the top targeted systems.”
“In order to protect sensitive data, organisations must ensure that all their databases are managed and under control. It is imperative that organisations scan their networks to discover databases, including unmanaged databases, and follow with data discovery and classification, which provides the needed awareness. Access to databases hosting sensitive data should be tightly controlled and the data must be protected from both external threats (hackers) and malicious insiders.”