Remediating Kubernetes security issues discovered by GuardDuty - Amazon GuardDuty

Remediating Kubernetes security issues discovered by GuardDuty

Amazon GuardDuty generates findings that indicate potential Kubernetes security issues when Kubernetes protection is enabled for your account. For more information, see Kubernetes protection in Amazon GuardDuty. The following sections describe the recommended remediation steps for these scenarios. Specific remediation actions are described in the entry for that specific finding type. You can access the full information about a finding type by selecting it from the Active findings types table.

Different types of attacks and configuration issues can trigger GuardDuty Kubernetes findings. This guide helps you identify the root causes of GuardDuty findings against your cluster and outlines appropriate remediation guidance. The following are the primary root causes that lead to GuardDuty Kubernetes findings:

Note

Before Kubernetes version 1.14, the system:unauthenticated group was associated to system:discovery and system:basic-user ClusterRoles by default. This may allow unintended access from anonymous users. Cluster updates do not revoke these permissions, which means that even if you have updated your cluster to version 1.14 or later, these permissions may still be in place. We recommend that you disassociate these permissions from the system:unauthenticated group. For instructions on removing these permissions, see Review and revoke unnecessary anonymous access.

Configuration issues

If a finding indicates a configuration issue, see the remediation section of that finding for guidance on resolving that particular issue. For more information, see the following finding types that indicate configuration issues:

Remediating compromised Kubernetes users

A GuardDuty finding can indicate a compromised Kubernetes user when a user identified in the finding has performed an unexpected API action. You can identify the user in the Kubernetes user details section of a finding details in the console, or in the resources.eksClusterDetails.kubernetesDetails.kubernetesUserDetails of the findings JSON. These user details include user name, uid, and the Kubernetes groups that the user belongs to.

If the user was accessing the workload using an IAM entity, you can use the Access Key details section to identify the details of an IAM user or role. See the following user types and their remediation guidance.

Note

You can use Amazon Detective to further investigate the IAM user or role identified in the finding. While viewing the finding details in GuardDuty console, click on Investigate in Detective. Then click on the AWS user or role from the listed items to investigate it in Detective.

Built-in Kubernetes admin – The default user assigned by Amazon EKS to the IAM identity that created the cluster. This user type is identified by the user name kubernetes-admin.

To revoke access of a built-in Kubernetes admin:

OIDC authenticated user – A user granted access through an OIDC provider. Typically an OIDC user has an email address as a user name. You can check if your cluster uses OIDC with the following command: aws eks list-identity-provider-configs --cluster-name your-cluster-name

To revoke access of an OIDC authenticated user:

  1. Rotate the credentials of that user in the OIDC provider.

  2. Rotate any secrets that user had access to.

AWS-Auth ConfigMap defined user – An IAM user that was granted access through an AWS-auth ConfigMap. For more information see Managing users or IAM roles for your cluster in the &EKS; user guide. You can review their permissions using the following command: kubectl edit configmaps aws-auth --namespace kube-system

To revoke access of an AWS ConfigMap user:

  1. Use the following command to open the ConfigMap.

    kubectl edit configmaps aws-auth --namespace kube-system
  2. Identify the role or user entry under the mapRoles or mapUsers section with the same user name as the one reported in the Kubernetes user details section of your GuardDuty finding. See the following example, where the admin user has been identified in a finding.

    apiVersion: v1 data: mapRoles: | - rolearn: arn:aws:iam::444455556666:role/eksctl-my-cluster-nodegroup-standard-wo-NodeInstanceRole-1WP3NUE3O6UCF user name: system:node:EC2_PrivateDNSName groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam::123456789012:user/admin username: admin groups: - system:masters - userarn: arn:aws:iam::111122223333:user/ops-user username: ops-user groups: - system:masters
  3. Remove that user from the ConfigMap. See the following example where the admin user has been removed.

    apiVersion: v1 data: mapRoles: | - rolearn: arn:aws:iam::111122223333:role/eksctl-my-cluster-nodegroup-standard-wo-NodeInstanceRole-1WP3NUE3O6UCF username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam::111122223333:user/ops-user username: ops-user groups: - system:masters
  4. If the userType is User, or is a Role that was assumed by a user:

    1. Rotate the access key of that user.

    2. Rotate any secrets that user had access to.

    3. Review the information in My AWS account may be compromised for further details.

If the finding does not have a resource.accessKeyDetails section, the user is a Kubernetes service account.

Service account – The service account provides an identity for pods and can be identified by a user name with the following format: system:serviceaccount:namespace:service_account_name.

To revoke access to a service account:

  1. Rotate the service account credentials.

  2. Review the guidance for pod compromise in the following section.

Remediating compromised Kubernetes pods

When GuardDuty specifies details of a pod or workload resource inside the resource.kubernetesDetails.kubernetesWorkloadDetails section, that pod or workload resource is likely compromised. A GuardDuty finding can indicate a single pod has been compromised or that multiple pods have been compromised through a higher-level resource. See the following compromise scenarios for guidance on how to identify the pod or pods that have been compromised.

Single pod compromise

If the type field inside the resource.kubernetesDetails.kubernetesWorkloadDetails section is Pod, the finding identifies a single pod. The name field is the name of the pod and namespace field is its namespace. Use the instructions in Identify the offending Pod and worker node to identify the worker node running the pod.

Pods compromised through workload resource

If the type field inside the resource.kubernetesDetails.kubernetesWorkloadDetails section identifies a Workload Resource, such as a Deployment, it is likely that all of the pods within that workload resource have been compromised. Use the instructions in Identify the offending Pods and worker nodes using workload name to identify all the pods of the Workload Resource and the nodes they are running on.

Pods compromised through service account

If a GuardDuty finding identifies a Service Account in the resource.kubernetesDetails.kubernetesUserDetails section, it is likely that pods using the identified service account are compromised. The user name reported by a finding is a service account if it has the following format: system:serviceaccount:namespace:service_account_name. Use the instructions in Identify the offending Pods and worker nodes using service account name to identify all the pods using the service account and nodes they are running on.

After you have identified all the compromised pods and the nodes they are running on, use the following instructions in the EKS best practices guide to isolate the pod, rotate its credentials, and gather data for forensic analysis.

To remediate a compromised pod:

  1. Identify the vulnerability that compromised the pods.

  2. Implement the fix for that vulnerability and start new replacement pods.

  3. Delete the vulnerable pods. For more information see Redeploy compromised Pod or Workload Resource.

Remediating compromised container images

When a GuardDuty finding indicates a pod compromise, the image used to launch the pod could be malicious or compromised. GuardDuty findings identify the container image within the resource.kubernetesDetails.kubernetesWorkloadDetails.containers.image field. You can determine if the image is malicious by scanning it for malware.

To remediate a compromised container image:

  1. Stop using the image immediately and remove it from your image repository.

  2. Identify all pods using the image. For more information see Identify pods with vulnerable or compromised container images and worker nodes.

  3. Isolate the compromised pods, rotate credentials and gather data for analysis. For more information see the following instructions in the EKS best practices guide.

  4. Delete all pods using the compromised image.

Remediating compromised Kubernetes nodes

A GuardDuty finding can indicate a node compromise if the user identified in the finding represents a node identity, or if the finding indicates the use of a privileged container.

The user identity is a worker node if the username field has following format: system:node:node name. For example, system:node:ip-192-168-3-201.ec2.internal. This indicates that the adversary has gained access to the node and it is using the node’s credentials to talk to the Kubernetes API endpoint.

A finding indicates the use of a privileged container if one or more of the containers listed in the finding has the resource.kubernetesDetails.kubernetesWorkloadDetails.containers.securityContext.privileged finding field set to True.

To remediate a compromised node:

  1. Use the instructions in the EKS best practices guide to isolate the pod, rotate its credentials, and gather data for forensic analysis.

  2. Identify the service accounts used by all of the pods running on the node. Review their permissions and rotate the service accounts if needed.

  3. Terminate the node.