Kubernetes 101 Security : Part 14

Let’s start with a sceneario

Assume that there is a voting between dog lover and cat lovers

And your friend “Nayeem” is a cat lover. She doesn’t want the dog’s team to win.

She wants to change the result. So using the terminal, she wants to make some changes.

She knows that there are two applications currently:

One for voting and another one for result. She doesn’t know where they are hosted or anything.

1. So, she tries to check the IP address using ping command

Both of the applications show same IP which indicates that they are part of the same infrastructure

2. She then looks for ports in the server.

She found a port where docker service listen to. Now, she is guessing the applications might be running as containers

3. Now she is hoping that there is no authentication enabled to the port (which is the default behavior)

Then she tries to list down all of the containers running on the voting application

Gladly, she finds them. She can see many things but can’t understand what they are meaning specially for the database.

4. She runs a privileged container based on ubuntu

5. She has successfully created a container on the server. She knows that there is a vulnerability named dirty cow that can be exploited to escape out of the container onto the underlying host.

She is now trying to download a script from the server

She can’t proceed as there are no curl or wget installed.

So, she installs curl there. She finds no restrictions while doing so.

once she is done, she downloads the script successfully.

6. Then she runs the script to get into the host

She now has the terminal to the underlying host. She runs different commands to verify.

She gets list of the containers using sudo docker ps

As she has control, she can show down the server or bring the entire application down

7. She learns that the name of the node as worker. But what worker? So,she lists the docker containers again

She finds our kubenetes mentioned in multiple files. So, she guesses that kubernetes cluster is used as here as well

Therefore, the node “worker” might be a worker node in the kubernetes cluster.

She is now guessing that one of the containers is running a kubernetes daskboard. It must be exposed on a port on the node “worker”. So, she lists the IP table rules on the node and looks for kubernetes dashboard.

She finds the port which exposes the kuberntes dashboard , which is 30080

Then she used the IP and port to look if the dashboard is public or not

What she did is, she looked into the db pod and found out the name and password for the db

She then tries to look for the container id and get into the container

She then uses the user and password for the db to get into it

She finds the vote table which stores the value for vote

Here is the voting list

Now what she wants is , she wants to change the vote to cat. She writes a script and update random votes and executes it

Now the election is overturned and cats are winning

Done! That’s how she hacked and changed the data.

Now, let’s learn how to prevent that

4 C’s of Cloud Native Security

• Security of the whole infrastructure hosted in Cloud/Private cloud/Data Center : The infrastructure that’s hosting Kubernetes cluster was not properly secured and enabled access to ports from anywhere. If network firewalls were placed, the hacking could have been prevented.

• Security of the Kubernetes Cluster : Nayeem was able to get access through the docker demon exposed publicly, kubernetes dashboard exposed publicly. There should have been some security enabled to prevent these.

• Security of the Container : Nayeem was able to run any container of her choice with no restrictions on what repository it is from or what tag was used. These could have been prevented if restrictions were put in place to only run images from a secure internal repository.

• Security on the Code: Hard coding applications with database credentials or passing critical information through environment variables or exposing without TLS are bad coding practices

  

Let’s learn more about them

Cloud Provider Security

Remember the attacker first tried to find the open port? She tried to break into the host server or the underlying infrastructure.

This could have been solved using “Cloud”

So, in the cloud infrastructure, we can use firewall to restrict unwanted access

Cloud platforms like AWS, GCP, Azure has their own platforms to secure it.

Let’s see what the platforms provide:

Azure provides Microsoft Sentinel which is a powerful tool that combines security information and event management, security, orchestration automation. So, it does not only detect security threats but also automatically deal with them.

AWS provides Amazon Guard duty , a service that uses machine learning to continuously monitor and detect potential security issues without needing specific rules set by users.

GCP on the other hand has has Security Command Center which monitors threats, manage assets, check security health of Google Cloud Services.

Web Application Firewalls (WAF)

All cloud service has a firewall solutions for web applications.

Azure WAF on its application gateway defend against top threats like SQL Injection, cross site scripting attacks.

AWS WAF allows users to create custom rules to stop these threats and manage traffic, which later can be linked to both AWS CloudFront and Load Balancer.

GCP has Google Cloud Armor which provides guard against DDOS attacks and other common web vulnerabilities.

Also, talking about Container level security,

Azure has AKS which ensures security from the start whereas AWS has EKS which uses Bottlerocket ( a special operating system) designed for container security and performance. AWS also uses kube-bench to meet CIS benchmarks.

Whereas GCP has GKE which secures container, enforces security policvies through intergration with Google Anthos and Open Policy Agent (OPA) aligning with security standards.

Shared Responsibility Model

In cloud, there are some parts which we don’t have control and that’s secured by the cloud organization. Also, there are some parts which we can control and manage. Those are our responsibilities.

For example, here Azure is dealing with the hardware (where we run our codes and all) security

Whereas we can control things like network settings, firewall etc. where we have to be cautious of threats.

In a diagram, you can see what the shared responsibilities are for Azure

For AWS, the shared responsibility diagram looks like this

Infrastructure Security

This involves everything related to server hardening and network configuration.

Remember, the hacker found out that both of the websites were hosted on the same IP.

So, we should have kept them in different IP address firstly. Moreover, removing the public IP address off the Kubernetes API server

or implement VPN access could have significantly reduce the risks.

Also, we saw how the hacker found out the docker port and controlled that remotely. This was done because there were not firewall enabled.

If there was a firewall enabled, attacker wouldn’t even get access to the server

Also, keep in mind that, using Network policies you can keep the worked Node VM safe.

We also saw, the hacker exploited the server using a privileged container .

This could have been solved if the the least privilege principle was applied, ensuring that only the permissions that are absolutely necessary are given.

We also saw how the hacker got access to the Kubernetes dashboard and exploited the database password etc.

To solve this, we could have used RBAC to allow access to the proper users.

Also, we saw the password in the db pod was in plain text. Thus it was easy for her to use it

If those were kept in secrets, this might not have happened even after the hacker got access to the db pod

Moreover, we can make things secured if we ensure etcd and it’s communication with the other services.

We can also use RBAC to allow access to the only authorized people

We can also keep backup and reliable recovery plan to ensure disaster recovery

These are the things we have to do if we host in the local system. But if we host it in cloud, we don’t need to secure etcd. The cloud service will do that for us

Kubernetes Isolation Techniques

Assume that we have Production, testing and development pods in our cluster

Now assume that,somehow the hacker got access to the dev pod. And as we can see , there are no safety in between the pods, the hacker can reach to the testing and production pod too. What can we do to prevent this?

We can create separate namespace so that test and prod pod remains safe.

Also we can keep separate namespace for our different teams. In this way, our internal teams can work on their designated pods safely without hampering our production pods

After that, we need to set proper Network policy because without that, pods can communicate within themselves.

For example, we can only traffic from pods within a namespace so that other pods from different namespace can’t reach each other.

Also we need to enable RBAC so that only assigned people can reach a namespace

Also in a multitenance setup, RBAC can be applied to give proper permissions

Also, we should be limiting the resources so that haters can’t use all of the resources if they get access to a pod

Also, if a user/hacker has root access , they can exploit everything. So, we need to limit them and mention who are root users and what capabilities a non-root user has

Workload and Application Code Security

We will start with learning SQL Injection Attack:

assume that a user has written query to valid user name and password from the login form. Now if an attacker inputs partial SQL query as a username , it will potentially log in the user without any valid password

To solve this issue, we have tools like sonaarqube, codacy, VERACODE etc which looks for the pattern in your codes that can be vulnerable for SQL Injection.

For example, here Sonarqube has given this recommendations to mitigate risks

Also, while working in a project, we might need multiple third party libraries. What if one of them has vulnerability?

To solve this issue, we can use OWASP which identifies vulnerabilities with severity ratings and CVE score. Then it shares a report that would list each library , it’s versions and any associated vulnerability with it.

What if we want to get updates about vulnerability on spot? Like when we are coding , we want to get alert about our dependencies?
Here Datadog Application Security Monitoring (ASM) can be very handy

Here you can see the code has Log4jExample used and it has a vulnerability called Log4Shell which allows attackers to execute code on servers using vulnerable versions of the log virtual library.

Also, sometimes we need to know how our code interacts with the system. May be certain pods of ours use resources more than it should have. In those cases, we need insights into containerized environments. How to do that?

Tools like Sysdig provide deep visibility into containerized environments which allow us to monitor resource usage, detect anomalies, troubleshoot issue in real time.

So, we can find our which container is causing high resource usage and all.