Like Dexter with a scalpel, the cyber thug behind the security incident below made quick work of destroying a company in just 12 hours.
Gone. Poof. No more shadow.
There are many lessons to be learned, most of which are not specific to cloud security (but rather IT security in general).
Article by Steve Ragan in CSO; original company message from the Code Spaces website.
Emphasis in red added by me.
Brian Wood, VP Marketing
——
Code Spaces forced to close its doors after security incident
It’s the nightmare scenario that organizations are warned about
Code Spaces, a Subversion and Git hosting provider, used by organizations for project management and development needs, has folded after an attacker compromised their internal systems.
The company, which was making a name for itself in the IaaS (Infrastructure as a Service) / DEVOPS community, says there’s just no way for them to resume operations.
It started with a DDoS attack on Tuesday. When Code Spaces reached out to the attacker, they were told to pay a ransom in order to stop the traffic flood.
However, the issue was much larger than a sustained DDoS. In fact, the reason they were able to contact the attacker in the first place is because they left contact details within Code Spaces’ Amazon EC2 control panel.
“It’s common for a DDoS to be a smokescreen for another attack that is aimed at gaining access to the target’s systems, and the Code Spaces attack appears to be a textbook case of this,” Trey Ford, global security strategist at Rapid7, said in a statement.
“Responding to a Denial of Service calls for all hands on deck, all available resources activated and focused on availability – diagnosing how the attack is being executed and how best to mitigate. In situations like this, responders would do well to re-task a separate ‘assurance team’ to verify systems, accounts and login activity.”
Code Spaces moved to regain control over their Amazon accounts, but the attacker had already taken steps to prevent this. According to a post on the incident, the intruder created backup log-ins on the EC2 panel and when recovery efforts were noticed, they started to delete artifacts at random.
“We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances. In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted,” the Code Spaces post explains.
It’s possible, Ford added, that having a runbook on how to do a lockdown (whether for locking out a rogue admin or to contain your environment) might have saved this team. Unfortunately, even if they did have such a plan, it didn’t work.
“Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.”
In a statement, Patrick Thomas, security consultant for Neohapsis, said that for companies using cloud services as part of their business, “this is the nightmare scenario.”
“This is a wakeup call to other organizations that have critical assets on cloud services. Two factor authentication and detailed event monitoring and alerting are essential components of any cloud strategy. Similarly, offline or warm-storage are critical business continuity measures,” he added.
Based on the limited data provided by Code Spaces and typical attacker behavior, the root cause of this disaster likely involved a Phishing attack against users with access to cloud service credentials. However, Code Spaces didn’t explain that aspect of the incident, but they promised a full report later, once their customers are taken care of.
While losing internal data is bad enough, the loss of off-site backups is a serious blow. For years, the use of off-site backups has been the standard operating procedure for organizations, especially where code is concerned.
“But in the age of cloud infrastructure many organizations think that they can simply pass the buck on backups, getting their geographic distribution and redundancy ‘for free’ as part of going to the cloud,” Thomas added.
“However, anything that’s vulnerable to the same threats isn’t fulfilling the original intent of offsite backups. Perhaps it makes more sense to start talking in terms of ‘diversified backups’ to emphasize the broad types of threats that a backup strategy must mitigate.”
In the end, the nightmare is real, and a small business might be forced to close for good due a single security incident. Code Spaces is the victim here, but many experts are in agreement that they share some of the blame.
Rob Ayoub, the Research Director of NSS Labs, recently authored a report on Amazon’s Web Services. He said that the Code Spaces attack illustrates some of the key challenges that companies need to consider when using any IaaS provider – not just Amazon:
- In an IaaS environment, once the controls are compromised, it’s very difficult to control or remediate quickly. Moreover, logs may require separate requests for support that may take days to receive, especially if extensive.
- Amazon only provides the infrastructure. The backing up of data is left entirely to the end user. There are several vendors that offer solutions to ease backup efforts from EC2, but those solutions cost extra. Amazon even offers Glacier, which is its own backup solution, but again the user has to implement the backup.
- Amazon continues to make improvements natively to EC2’s security and many vendors offer virtual appliance versions of their solutions, but many organizations have not ported the same controls from their on-premise infrastructure into services like Amazon. Would a virtual appliance version of a NGFW or UTM device have stopped this attack? Maybe.
“Again, I don’t have all the details on this attack. We may never know how the attacker got in, but in my discussions with customers and vendors, there are many false assumptions that organizations make when moving data and services to the cloud,” Ayoub said.
“It is sad that Code Spaces was potentially forced out of business by an attacker. I would hope that Amazon might offer some forensics help, because I feel ultimately there is a shared responsibility for security between Amazon and its customers.”
—–
Code Spaces : Is Down!
Dear Customers,
On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.
An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address
Reaching out to the address started a chain of events that revolved around the person trying to extort a large fee in order to resolve the DDOS.
Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.
At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances.
In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.
This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers needs.
Data Status
All svn repositories that had the following url structure have been deleted from our live EBS’s and all backups and snapshots have been deleted:
https://[ACCOUNT].codesapces.com/svn/[REPONAME]
All Svn repositoies using the following url format are still available for export but all backups and snapshots have been deleted:
https://svn.codespaces.com/[ACCOUNT]/[REPONAME]
All Git repositories are available for export but all backups and snapshots have been deleted
All Code Spaces machines have been deleted except some old svn nodes and one git node.
All EBS volumes containing database files have been deleted as have all snapshots and backups.
Code Spaces Status
Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.
As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us.
All that we can say at this point is how sorry we are to both our customers and to the people who make a living at Code Spaces for the chain of events that lead us here.
In order to get any remaining data exported please email us at support[at]codespaces.com with your account url and we will endeavour to process the request as soon as possible.
On behalf of everyone at Code Spaces, please accept our sincere apologies for the inconvenience this has caused to you, and ask for your understanding during this time! We hope that one day we will be able to and reinstate the service and credibility that Code Spaces once had!
