Stephen Schmidt, CISO, Amazon Web Services (AWS)
Stephen Schmidt, CISO, Amazon Web Services (AWS)

Stephen Schmidt, CISO, Amazon Web Services (AWS) explains how his organisation protects vast amounts of data spread across multiple systems - including use of machine learning and achieving GDPR compliance, balancing government security demands and individual privacy and  he describes his primary concern.

The still-expanding Amazon subsidiary AWS provides cloud-based web computing to millions of active customers each month - including 100,000 in the UK, achieving some US$ 18 billion (£13.3 billion) revenues last year (2017) from a truly global operation. AWS is a separate business from Amazon, the retail operation, which is a customer of AWS; each has a different customer base, different services, and a different leadership and security team, Among 90 services provided are compute, storage, networking, database, analytics, application services, deployment, management, developer, mobile, Internet of Things (IoT), Artificial Intelligence (AI), security, hybrid and enterprise applications.

Schmidt took on his current role a decade ago, moving from being a customer of Amazon - as an executive at the FBI where he handled intelligence analysis including reverse engineering of attackers'  code and understanding how foreign intelligence services broke into systems.

So how and why was the transition made? “(at the FBI) We kept filling up file systems with data we were trying to analyse and it wasn't working as well as we wanted them to. It was really exciting the find that Amazon S3 was precisely what we needed to solve our particular problem. They were building it for government contracts and  it ended up with a long conversation where they said, rather than being a customer, would you like to come and build these things?”  says Schmidt.

Amazon hired several of Schmidt's team to build the first development centre team. “My team built the networking virtualisation layer which sits underneath the virtual cloud above. I was helping to interview candidates for the CISO position when we realised we needed to separate Amazon Security from AWS security,” says Schmidt. For many of AWS' customers' their biggest competitor is Amazon retail and it was necessary to absolutely assure AWS customers that nobody in the retail organisation had any access to their data at all.  So two completely distinct security teams were built.

"‘how can we accelerate business while meeting the security requirements of our customers?”

“After searching for the CISO for a while, Andy Jassy, AWS CEO, pointed at me and said, “you're it.” I was surprised. It wasn't something I was looking for. I was asked why I hadn't been interested, and said, because security teams tend to be retarders of business, they slow things down, they're the house of ‘no'. Andy's advice was, “So go fix it.”  And it has turned into an absolutely wonderful job. I really enjoy it. It's all about, ‘how can we accelerate business while meeting the security requirements of our customers?”

Schmidt explains that achieving this outcome takes a lot of individual judgement on the part of staff: “... as opposed to doing the easy thing which is to say ‘no' and stop things. We have to apply real analytical judgement about the actual risks of any particular vulnerability at the point that it's revealed, and how our business is exposed and how our customers are exposed, the likelihood of detection, of us being able to respond rapidly, so its a huge calculus that you have to put together. Ultimately you have to come out of it with a decision at the other end [as to] what the company should be doing.”

GDPR is just one of the global sets of regulations that AWS has to comply with, and SC asked how can that be achieved when the regulations themselves can conflict?  Schmidt responded: “We comply with the law in every jurisdiction in which we operate. What that means for Europeans for example is, come May [2018] we  [AWS] will be GDPR compliant. The ‘Right to be forgotten' is something we can operate successfully from the US.  An advantage we have over others is that we've been operating in Europe for ten years, so we are very used to European interest in and focus on privacy.  Unlike some other folks, we don't have to bolt privacy controls onto our services afterwards - they're built from the beginning. Which means it's much easier for us to be compliant with things like GDPR.”

AWS has already obtained the EU data protection authorities' (Article 29 Working Party) approval of the AWS Data Protection Agreement (DPA) and Model Clauses to enable transfer of data outside Europe – including to the US. It has also completed ISO 27001, ISO 27017, ISO 27018, FISMA, SOC-1 (formerly SAS-70), and PCI DSS Level 1 and  is certified under the EU-US Privacy Shield.

Schmidt continues, “The other part is that we do respect the differences in regulations where appropriate, so for example The US Federal Government FedRAMP programme which is an accreditation programme for users of government services requires retention of video on Federal Services for a period of time - nine years or so. In Italy we can only retain video or 24 hours.  In Germany we can only retain it for 30 days, so we must respect those jurisdictions that have different limiting time factors and we just tell the people, “Sorry, we're not keeping video for more than 24 hours. That's the law.”

He also says AWS is: “Absolutely ready for 72 hour reporting. We have had the mechanisms in place for a long time.  We've had much tighter reporting requirements from some of our very highly regulated industries.  It's an area where we have had to build up the institutional processes that are required to do that kind of reporting. It's not something that has to be operationalised.”

AWS has also built GDPR compliance components into tools like Macie, which aims to help customers identify where they have data that they need to worry about GDPR compliance, described as often the most difficult first step.

"GDPR compliance is the fundamentally important part, and we're very confident [that we are compliant]."

But with ongoing objections to Peace Shield is it not working?  Schmidt disagrees:  “There were hiccups in the beginning, but the governments have come to an agreement on what a reasonable middle ground is.  From our perspective, that's just one piece of the overall puzzle.  GDPR compliance is the fundamentally important part, and we're very confident [that we are compliant].  We've been working on this for a long time and we're a little bit different from  lot of players as we build in privacy components into our services and have for a very long time. Things like, proper encryption, so I have no ability to see your data at all. For years we have had life -cycle monitoring of data so customers can say, ‘this must be disposed of within three years.'  That's one of the big pitfalls in GDPR - how long do you keep data for?  What do you do with it at the end of that period? If you are using our S3 for example you can set a retention policy that says, after one year, get rid of it and we automatically dispose of it for you.  The mechanical components allow us to be confident about it.  The other piece is that we have a lot of controls around who can access data, and where they can move it, and that allows us to be sure we are meeting GDPR requirements.”

And what about regulations that impinge on operations such as Germany or Russia requiring data on its citizens to be held in its territory?  Its a customer issue says Schmidt: “For every region in which we operate we give customers contractual obligations, that their data remains within the jurisdiction in which they place it, and it's at their discretion to move it somewhere else. Anyone can comply with their regulatory environment by choosing to do that.  For example, you cannot hand us an object and say, store this an Amazon S3 somewhere in the world - we won't do it. What you can do, is, you need to pick a region to put that data in. If you want to back your data up to another region - for example many of our Japanese customers choose to back up Japanese region data to the West coast of the US. Then if they suffer another catastrophic event, like Fukushima, their data is still accessible. But that's an intentional act that they take, and move the data from point A to point B. We won't move it out of a region unless asked by a company to do so.”

Then comes the issue of how do you reconcile legitimately helping the government with achieving the good things they want to do to protect us from terrorists and the like, with protecting the privacy of your customers and their customers, and maintaining that balance between legitimate privacy and security concerns?

"We recognise the legitimate needs of law enforcement to protect the populace.  However, those legitimate interests must be balanced with proper procedure."

“Use of the word balance is particularly apropos , its the right thing, checks and balances,” says Schmidt, continuing, “And we recognise the legitimate needs of law enforcement to protect the populace.  However, those legitimate interests must be balanced with proper procedure, proper judicial overview, and the proper judicial outcome before we would release any customer information. For example, you can't just come to us and say, ‘give me the data' on somebody. You have to say, ‘here is the court order that has been approved by a judge', that, we believe, is the least intrusive way to accomplish what we need to. And we will challenge things that we believe to be overly broad.    

“The guiding principle here is, our customers own their data.  It's something that we give them a lot of tools on how to protect. Its an area where we give them a lot of opportunity to encrypt, appropriately, and control their own encryption keys if they wish, and it's up to the customer then to choose ‘How do I want to manage my privacy?' and 'how do I want to manage access to information?'”

It was explained that many AWS customers - like Transport for London - put the information out there for everybody about what's going on with a tube strike or whatever, and then on the other side there are Hedge Funds who are very secretive about their core intellectual property, their analytic algorithms. “So we have to be able to meet the particular security requirements of their world as well. That means we have to build them a series of controls which encompass that whole space. That means we have to have a flexible platform that gives people the right controls to manage their data individually.  

“But we make some choices for them as well. For example, we believe that things like having a firewall on every virtual machine is just good internet hygiene. So we provide that, it's not an option, every machine gets it. Those firewalls are closed by default. So we give a customer a good, safe starting point and then they get to decide how they want to open those up.”

“We believe very strongly internally that people and data are best kept separate."

In addition, a lot of focus is put on things like reduction in human access to information. “We believe very strongly internally that people and data are best kept separate, and when they are in contact we should be extremely rigorous about logging who does what, when, where, why, how etc and that's reflected in the services we vend as well.  Customers can elect to use a lot of our logging services that help them provide insight into what their developers are doing using AWS. That way they know where their data is, they know how it's being accessed, they know the security controls around it, and they can set up rules around changes in security controls. One of the things we shipped about 18 months ago is called, ConfigRules, where you, as the owner of the information, can say for example, ‘This particular database could never have a firewall rule that permits access from the internet. And if it does, revert the change and the log. Those kind of proactive controls really change the way people can view security.  Its less responding to a fire alarm, and more preventing the problem in the first place.”

Schmidt also pointed out that AWS has also been very vocal about circumstances in which it believes the law has been over-reached the the authorities. “We have filed a couple of Amicus briefs in cases involving Apple as well as Microsoft.  Where we believe the government is over-reaching we will stand up and we will support court actions for what we believe to be the legitimate right of privacy.

“If we get a court order to provide your data we will inform you of that, unless we get prevented by law from doing so. The chances of prohibition are shrinking, there are a number of court cases in the US, compelling government disclosure when its looking for data - its actually reduced the amount of time that they are allowed to hide things.”

“The biggest concern I have is making sure the customers understand the division of responsibility in our shared responsibility model."

So what are the biggest risks or concerns faced by AWS?  “The biggest concern I have,” says Schmidt, is, “making sure the customers understand the division of responsibility in our shared responsibility model. If you look at a traditional hosting company they will do everything from start to finish for a customer's infrastructure.  In the cloud we give people the option of operating security controls that they choose. And making sure that customers choose the security controls that are right for their particular circumstances is critically important. So for example If you look at the payment card industry association rules around storing credit card information, you can be an approved payment card vendor running on AWS - there is this matrix of controls that you have to implement.  We do a number of those things, completely for the customer. There is a section in the middle where both we and the customer have responsibility. And then below that there are things the customer must do themselves. Getting that line right, where the customer knows exactly what they have to do and what we have to do is something that takes work, it take education, tooling, effort, and if done poorly, the worst case scenario would be, a customer says, “I thought you were doing that.” So we spend an enormous amount of time educating people and helping them build the tools that are right for their particular job.”

It was pointed out that under GDPR, while the client cannot subcontract responsibility for the security of its data, processors must also take responsibility too. Schmidt acknowledged that AWS does, elaborating: “A processor does take responsibility up to a point, for some things, and the data owner does as well, and there's that joint piece in the middle, so its a change in regulatory operation  - its really not a change for functional operation for us. We've always operated our controls the same way and GDPR doesn't change any of that.  The only thing it does change is the mandatory reporting requirements within a specified time frame.”


“ No, Amazon servers were not breached."

Another concern, SC asked, surely, must be the various reports of Amazon servers being breached over the past year? Schmidt responded with an emphatic, “ No, Amazon servers were not breached. Some customers changed our default security configuration from their closed position to open.  As a result, unfortunately, there was exposure to content of the customers' data.  It was not a failure of AWS security controls, it was an intentional choice on the part of the customer to open their controls up.”

It was acknowledged that DSoS attacks continue to be an issue for customers. “The number of them continues to rise.We mitigate several hundred DDoS attacks against our customers every day. Often the customer doesn't even know something has happened. AWS shield (a managed service)  is the protection service that every one of our customers gets automatically, and then there's Shield Advanced (AWS Shield Advanced is an optional extra for additional protections against larger and more sophisticated attacks for applications running on Elastic Load Balancing (ELB), Amazon CloudFront and Route 53.) which gives you a lot more control. The volume of attacks certainly increases over time, but so does the ability both to respond and to manage the attacks. The ability to distribute over larger and larger infrastructure estates for example.”

“A US AWS customer would have access to all the connectivity we have globally, including our ability to redirect traffic over the network providing active defence of the customer.

Ransomware was dismissed as not having much of an effect on AWS's business.


Avoiding breaches? “We do the same things that anybody else should be doing."

So what does AWS do to avoid breaches and spot if it has been breached?  “We do the same things that anybody else should be doing, that is, know your environment intimately, monitor it thoroughly, alarm when things exceed your normalcy thresholds, and most importantly, have a very narrowly confined long term blast radius so that if something does go wrong it can find the critical error. For example if you look at most customers' IT estate, you walk into their data centre, there are going to be many people in there who have administrative rights, or root access to servers. If you walk into an AWS data centre, we believe in separation of duties. So the human beings who can touch the machines in our data centre are not permitted to have logical access to their machines.

“For a long time the military has separated responsibilities for high value assets such as nuclear weapons,.  We do the same thing. So even if you can touch the physical things, you are not allowed to have logical access to them - and the reverse is true. So if I am on the S3 team and I build the storage platform, I may not have access to the data centre. Because it takes both ends of that equation for somebody to do something inappropriate. Where is the data and how to get access to the data?

“Especially when customers are using the encryption capabilities we give them. I am happiest when customers take advantage of all the encryption options that we have, when their data is sitting on the server encrypted and they hold the keys, that is the best place I can be in.  I do not know what is in their data, and I cannot see it because they control access.”

So what do you do to stop theft?  “Theft is something that has been around forever - its not new. Abnormality is the single biggest indicator that something is wrong - especially when you get a database as large as ours. The statistical distribution is such that we can very accurately predict what is a reasonable norm for somebody doing something. It allows us to then flag back to that cyber norm and act on it as appropriate. So we are trying to transfer some of those learnings to our customers so that they can do the same type of monitoring in their own infrastructure.”

And what's new at AWS to help improve this process?  “The hot new thing is probably the application of machine learning and Artificial Intelligence in the security space. Its an area that we have had a programme focussed on internally for quite a long time, and I was quite happy to be able to transfer some of that knowledge to the outside. My team launched a service called Amazon Macie in September (2017. Amazon Macie recognises sensitive data such as personally identifiable information (PII) or intellectual property, and provides dashboards and alerts that give visibility into how this data is being accessed or moved.); How are your employees interacting with that data? Who is accessing it, from where, when, and is that access pattern normal.?

“We can build machine-learning models of normalcy of behaviour - for staff, for example. And then alarm if the activity we see is not normal.  So for example if you have a file that contains all the credit information for your customers, and you see a sales person doing analytics on the marketplace those customers are using that may be normal, but if there is a software development engineer who all of a sudden tries to download that data, that's not a normal pattern. Conversely, if you have a sales person who starts downloading source code, that's not a normal thing either.   We've built this tool that allows us to do that and a lot of our larger customers who have very large data estates where they need to understand exactly what that data is and where it is stored and how it is protected, said ‘please, we want that thing you just built for your own use, can you make it available as a services outside?'

“Quite often we'll have conversations with customers, and they'll say - ‘how do you do this internally?  You've got all of this data, how do you protect it?'  Often those conversations will turn into ‘Can't you build that for me too please?'”

There's a lot of hype around AI so are we really just talking about logic-based machine learning?  “Logically, machine learning is simplistic.  What we are talking about is unstructured training. So you don't have to tell Macie what is good and what is not good. What it does is it takes in a flow of logs for a period of time and establishes baselines for what is normal. Then it will alarm when those norms are bypassed. Similarly, you can set boundaries - such as, this type of data should never be accessed from outside my corporate network, and there are alarms if it is; we should never be storing data which looks like this - so maybe you never want to store passport data that's not encrypted.  And then it learns over time from your interaction with the system. We don't know the structure of data for all companies. We have to learn that over time,” concludes Schmidt.