Demo

Cloud security has been a hot topic. As more and more organizations go to the cloud, security breaches and incidents hitting the news headlines have been catching the eyes of the general public, making some people skeptical to embracing the cloud. However, when we look at the security controls of the cloud services providers, we find that most of them have actually been doing quite a nice and decent job, but then what actually happened in those cases to cause such impression? Human learns best by examples, let’s look at a simulated attack to discover some common themes and patterns. We have created a demo illustrating a chained exploit, which takes advantage of different mistakes made by the customer in cloud deployment. It is also how a serious breach happened in US hitting the news headline last year. We picked AWS in our demo not because AWS has any problem, just because it is more popular and the real hacks we are simulating were on AWS.

In the first part of our demonstration, we can see a NodeJS web app running on AWS EC2 configured with IMDSv1 (Instance MetaData Service). IMDS provides a HTTP endpoint for the instance to query its metadata at an IP address not supposed to be routed outside i.e. 169.254.169.254, including the temporary credentials assigned to the role of the running instance. Sounds dangerous? Not quite given normally there is no way for an external attacker to access that endpoint. Then comes SSRF (Server Side Request Forgery), which is a software vulnerability that allows the attacker to pass in fabricated data or parameters, typically a URL, (blindly) trusted and used by the application to make HTTP requests and return the results. The application in the demonstration is vulnerable to SSRF. As a result, by passing in the IMDS endpoint address, the application makes an IMDS request on our behalf and returns us the metadata, including the temporary credentials, as if it is originated from the application legitimately. We can then use the stolen credentials to access the victim’s other AWS services, in this case and in many real world incidents, S3 is the target.

To help the customers mitigate the risk, AWS, as well as other major cloud service providers, attempt to make it more difficult for unintentional access to the metadata service. While Google (no more v1Beta!!) and Microsoft Azure require a static header when accessing the metadata service, AWS goes one step further by requiring a HTTP PUT request first to obtain a token that must be passed along with a second request in order to retrieve the metadata, and here we have IMDSv2, born since end of last year (how AWS made that design decision? Read the rationale behind from AWS).

Now, if the same thing happens, like what we demonstrate in the first part of the demo, it is a software bug plus a mistake in AWS instance configuration. However, if we think that we are safe by configuring IMDSv2 and our credentials can no longer be stolen, think again. The whole idea of IMDSv2 as well as other static header implementation is based on the fact that in most typical SSRF attacks, the attacker can only control the URL, but not the header or even the HTTP method. What if the attacker can control these other aspects as well? Or what if the application has a more serious bug that can lead to RCE?

In the second part of our demonstration, we put our application on an EC2 instance configured with IMDSv2. The problem is, the application is using an open source library to parse the input parameters, and this open source library has an insecure deserialization bug. So now in addition to SSRF, we have a second bug, and a more serious bug that allows us to inject arbitrary Javascript. The injected code will be dynamically evaluated by the vulnerable component, so that we can make any call we like. As a matter of fact, we can see in the later part of the demo that we have complete control of the instance by getting shell access (try curl the IMDSv2 endpoint if you like!!). The bug itself is serious enough, but don’t forget the extended impact (e.g. your data in S3 may get stolen).

As best effort only, the cloud service providers try to provide more mitigating controls to help, like IMDSv2 from AWS, and very often, it is not a fundamental flaw in the cloud services, but application bugs plus user mistakes, again. The cloud just exposes lots of people’s bad practice to the world, and amplifies it with bigger consequence. Remember some developers of a famous tech company checked in their AWS access keys in GitHub and got their GitHub repository compromised?

To conclude, when assessing the security risk of cloud deployment and its impact, do not just focus on security features or functions like firewalls, encryption, IAM, etc, and an attacker mindset is required. Most of the time, lots of security features and functions have been put in place in the architecture, but when we look at examples like this one and those in the real world, it has very little to do with the security software, it is more about software security, and security software is not software security. At the same time, like most air crash accidents, security incidents often involve a series of mistakes chained together to cause tremendous impact. A threat modeling exercise may be a good start to analyze potential attack paths, it would also help the penetration test team to test the deployment in a more efficient way. Next step: incident response, but before that, setup monitoring of your cloud logs, implement application logging on security events and stream your cloud and application logs to data analytics platform. Tools and data are the essential weapons for security operations and threat hunting.


Did you enjoy this post? Want to find out more about us? Contact us