Let’s see a general architecture

Here the nginx server runs in its own pod within the frontend namespace, then we have our backend API . That’s basically a set of nodejs microservices that runs in separate pod within the backend namespace.Finally we have mysql database that stores userdata and application state.This database runs in a pod within the database namespace.

We need to ensure boundaries so that, if a hacker hacks one of the resources, that never hampers other one.

Also, we should have different clusters , so that hackers can’t impact another cluster. Specially the production one.

We also should have our nodes isolated

so that hacker access to one node can’t attack another.

Our namespaces should be isolated

so that hacker with one namespace access can’t impact another namespace

We also should isolate pods

so that hacker with access one pod can’t impact another

Again, within pods we should have multiple containers.

so that hacker with access to one, can’t impact another container.

Let’s explore where can we find issues?

Users get access to the service using website. There we have security using HTTPS. Once accessed it goes to the NGINX server. It then passes the request to backend services. Here proper API authentication is done.

Now, the backend services need to access the database to fetch or store user data.

Finally the backend services often need to communicate with each other to fulfill the user’s request and network policies. Here, kubernetes can help restrict which pods can communicate with each other making backend pods safer to communicate within themselves.

Persistence

Persistence refers to the ability of an attacker to maintain access to a compromise system even after reboots , updates or other interruptions.

So, in our architecture this is how a hacker can get in and remain persistent.

An attacker might gain initial access through a compromised application running in a container. SO once inside, their goal is to establish persistence allowing them to maintain control over the cluster despite efforts to remove their access.

Here you check the ways a hacker can have persistence

To solve that, we can enable Role based Access Controls

Then we need to restrict access to secrets

Then we should harden the pod security

We should have regular updates and patching

Finally, we should continuously monitor and keep auditing

Denial Of Services(DoS)

DoS attack is where attackers overwhelm the system with illegitimate requests or exhaust its resources making it unavailable for legitimate users.

These are the probables ways a hacker can do that

Checkout more

Assume that a hacker got access to the backend here

Using the service account token, the hacker can start many tokens which will drain resources.So, if proper quotas are not assigned, this can hamper later.

Again if a hacker get access to network, he can flood the kube-api server endpoint with numerous request.

So, proper firewall is needed to stop this hacking.

So, how to mitigate DoS attacks?
First of all, we must limit resources for pods etc. So that hacker can’t create various resources and drain them.

We also should limit service account’s permissions. For example, we can ensure that the services accounts used bythe nodejs Backend pods can’t create new pods or perform administrative actions

We also would like to use Network Policies to control traffic flow within the cluster and also configure traffic firewalls to protect the control plane.

We also need to monitor continuously. For example, here is an alert created which fires if the CPU usage access containers exceeds 80% for 5 minutes.

Malicious Code Execution

If a hacker gets access to a pod, he can then add a malicious code which can effect more pods.

These are the ways a hacker can add malicious code to our code

So, as mentioned earlier, the hacker can add malicious code once he gets access. Then the code might schedule multiple pods causing resource drains.

Also, another way the hacker can harm us is, if the hacker manages to know the image pull secrets, he can poison the container image repository by uploading malicious images and other parts of the cluster might then pull and run these compromised images and spread malicious code throughout the environment.

So, how to solve these issues?

We need to update patches and scan .

We need to restrict unauthorized access to the kube-api service.

Then we need to protect our container image repositories. We need to secure the image pull secrets and ensure that only trusted resources can upload images. We should store image pull secrets securely and we want to restrict their access to only necessary pods.

Here you can see that we have created a secret for the docker registry and then used the secret to image pull secret.

Also, we should set up alert for additional monitoring.

Here we have set two alerts. One is for the “SecretChangeDetected” and another one is for the CommandExecutionInContainer.

Finally we should regularly check audit logs.

Compromised applications in containers

Assume a hacker hacks a pod and using that it gets access to the whole cluster information. Then it’s a big issue.

Attackers on Network

Here we can see the attackers on the network

Let’s see how to mitigate these types of network based attacks.

We can configure firewalls to limit network access to the control plan and nodes by setup up firewall rules to restrict access to the kube-api server allowing only trusted IP addresses to the communicate with with it.

Also we need to update patches and monitor

We also should use network policies to control whom to contact whom

We also should have strong authentication and authorization techniques

We also should monitor and check logging. Here we haves set alerts for network alerts and highnetworktraffic

Access to sensitive Data

Here we can see the risks of attackers accessing sensitive data.

Assume that here, this nodejs pod has excessive permissions due to misconfigured RBAC, an attacker can exploit these permissions to read secrets stored in the cluster.

These secrets might include database credentials etc.

To solve this, we need to ensure that RBACs are configured properly and tightened for service accounts used by our backend

Also, if a hacker gets access to logs which includes pod passwords etc, that can be very harmful.

To solve that, we need to encrypt those

Privileged escalation

We also should set administrative and other access on the /etc/sudoers file

The sequence follows like this

Base vs Parent Image

Assume we want to build our own custom image and we are using http image to build it

Here httpd is our parent image.

And then if you explore, you can see the httpd image was built with debian:buster-slim image and that was built from scratch.

So, scratch was the base image.

In general parent image and base image almost as same.

Some of the best practices are:

Never build images with multiple applications like webserver, database and other services

Rather create different images with their dependencies. This is called modular practice

Also, we shouldn’t store data in containers because they may get deleted. Always store data on some volume externally.

Now, let’s see how can we build our own custom image. Initially we try to build an image with existing images.

Based on our need we may choose any image. Assuming that we need httpd server,

Then we need to make sure that the image is authentic

Images should also be updated frequently meaning vulnerabilities are solved.

We also should keep the image size minimum so that it can be pulled fast.

Also a minimal image is less vulnerable to attack.

Kubernetes 101 Security : Part 17

Table of contents