Use the AWS FIS aws:eks:pod actions - AWS Fault Injection Service
ActionsLimitationsRequirementsCreate a service role for the Kubernetes service accountConfigure the Kubernetes service accountMap your experiment role to the Kubernetes userPod container imagesExample experiment template

Use the AWS FIS aws:eks:pod actions

You can use the aws:eks:pod actions to inject faults into the Kubernetes pods running in your EKS clusters.

Actions

Limitations

  • The following actions do not work with AWS Fargate:

    • aws:eks:pod-network-blackhole-port

    • aws:eks:pod-network-latency

    • aws:eks:pod-network-packet-loss

  • The following actions do not support the bridge network mode:

    • aws:eks:pod-network-blackhole-port

    • aws:eks:pod-network-latency

    • aws:eks:pod-network-packet-loss

  • You can't identify targets of type aws:eks:pod in your experiment template using resource ARNs or resource tags. You must identify targets using the required resource parameters.

  • The actions aws:eks:pod-network-latency and aws:eks:pod-network-packet-loss should not be run in parallel and target the same pod. Depending on the value of the maxErrors parameter you specify, the action may end in completed or in failed state:

    • If maxErrorsPercent is 0 (default), the action will end in failed state.

    • Otherwise, the failure will add up to the maxErrorsPercent budget. If the number of failed injections do not reach the provided maxErrors, the action will end up in completed state.

    • You can identify these failures from the logs of the injected ephemeral container in the target pod. It will fail with Exit Code: 16.

  • The action aws:eks:pod-network-blackhole-port should not be run in parallel with other actions that target the same pod and use the same trafficType. Parallel actions using different traffic types are supported.

  • FIS can only moni†or the status of fault injection when the securityContext of the target pods is set to readOnlyRootFilesystem: false. Without this configuration, all EKS pod actions will fail.

Requirements

  • Install the AWS CLI on your computer. This is needed only if you'll use the AWS CLI to create IAM roles. For more information, see Installing or updating the AWS CLI.

  • Install kubectl on your computer. This is needed only to interact with the EKS cluster to configure or monitor the target application. For more information, see https://kubernetes.io/docs/tasks/tools/.

  • The minimum supported version of EKS is 1.23.

Create a service role for the Kubernetes service account

Create an IAM role to use as a service role. For more information, see IAM roles for AWS FIS experiments.

Configure the Kubernetes service account

Configure a Kubernetes service account to run experiments with targets in the specified Kubernetes namespace. In the following example, the service account is myserviceaccount and the namespace is default. Note that default is one of the standard Kubernetes namespaces.

To configure your Kubernetes service account

  1. Create a file named rbac.yaml and add the following.

    kind: ServiceAccount
apiVersion: v1
metadata:
  namespace: default
  name: myserviceaccount

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: role-experiments
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: [ "get", "create", "patch", "delete"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "list", "get", "delete", "deletecollection"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bind-role-experiments
  namespace: default
subjects:
- kind: ServiceAccount
  name: myserviceaccount
  namespace: default
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: fis-experiment
roleRef:
  kind: Role
  name: role-experiments
  apiGroup: rbac.authorization.k8s.io

  2. Run the following command.

    kubectl apply -f rbac.yaml

Map your experiment role to the Kubernetes user

Use the following command to create an identity mapping. For more information, see Manage IAM users and roles in the eksctl documentation.

eksctl create iamidentitymapping \
    --arn arn:aws:iam::123456789012:role/fis-experiment-role \
    --username fis-experiment \
    --cluster my-cluster

Pod container images

The pod container images provided by AWS FIS are hosted in Amazon ECR. When you reference an image from Amazon ECR, you must use the full image URI.

AWS Region Image URI
US East (Ohio) 051821878176.dkr.ecr.us-east-2.amazonaws.com/aws-fis-pod:0.1
US East (N. Virginia) 731367659002.dkr.ecr.us-east-1.amazonaws.com/aws-fis-pod:0.1
US West (N. California) 080694859247.dkr.ecr.us-west-1.amazonaws.com/aws-fis-pod:0.1
US West (Oregon) 864386544765.dkr.ecr.us-west-2.amazonaws.com/aws-fis-pod:0.1
Africa (Cape Town) 056821267933.dkr.ecr.af-south-1.amazonaws.com/aws-fis-pod:0.1
Asia Pacific (Hong Kong) 246405402639.dkr.ecr.ap-east-1.amazonaws.com/aws-fis-pod:0.1
Asia Pacific (Mumbai) 524781661239.dkr.ecr.ap-south-1.amazonaws.com/aws-fis-pod:0.1
Asia Pacific (Seoul) 526524659354.dkr.ecr.ap-northeast-2.amazonaws.com/aws-fis-pod:0.1
Asia Pacific (Singapore) 316401638346.dkr.ecr.ap-southeast-1.amazonaws.com/aws-fis-pod:0.1
Asia Pacific (Sydney) 488104106298.dkr.ecr.ap-southeast-2.amazonaws.com/aws-fis-pod:0.1
Asia Pacific (Tokyo) 635234321696.dkr.ecr.ap-northeast-1.amazonaws.com/aws-fis-pod:0.1
Canada (Central) 490658072207.dkr.ecr.ca-central-1.amazonaws.com/aws-fis-pod:0.1
Europe (Frankfurt) 713827034473.dkr.ecr.eu-central-1.amazonaws.com/aws-fis-pod:0.1
Europe (Ireland) 205866052826.dkr.ecr.eu-west-1.amazonaws.com/aws-fis-pod:0.1
Europe (London) 327424803546.dkr.ecr.eu-west-2.amazonaws.com/aws-fis-pod:0.1
Europe (Milan) 478809367036.dkr.ecr.eu-south-1.amazonaws.com/aws-fis-pod:0.1
Europe (Paris) 154605889247.dkr.ecr.eu-west-3.amazonaws.com/aws-fis-pod:0.1
Europe (Stockholm) 263175118295.dkr.ecr.eu-north-1.amazonaws.com/aws-fis-pod:0.1
Middle East (Bahrain) 065825543785.dkr.ecr.me-south-1.amazonaws.com/aws-fis-pod:0.1
South America (São Paulo) 767113787785.dkr.ecr.sa-east-1.amazonaws.com/aws-fis-pod:0.1
AWS GovCloud (US-East) 246533647532.dkr.ecr.us-gov-east-1.amazonaws.com/aws-fis-pod:0.1
AWS GovCloud (US-West) 246529956514.dkr.ecr.us-gov-west-1.amazonaws.com/aws-fis-pod:0.1

Example experiment template

The following is an example experiment template for the aws:eks:pod-network-latency action.

{
    "description": "Add latency and jitter to the network interface for the target EKS pods",
    "targets": {
        "myPods": {
            "resourceType": "aws:eks:pod",
            "parameters": {
                "clusterIdentifier": "mycluster",
                "namespace": "default",
                "selectorType": "labelSelector",
                "selectorValue": "mylabel=mytarget"
            },
            "selectionMode": "COUNT(3)"
        }
    },
    "actions": {
        "EksPod-latency": {
            "actionId": "aws:eks:pod-network-latency",
            "description": "Add latency",
            "parameters": {
                "kubernetesServiceAccount": "myserviceaccount",
                "duration": "PT5M",
                "delayMilliseconds": "200",
                "jitterMilliseconds": "10",
                "sources": "0.0.0.0/0"
            },
            "targets": {
                "Pods": "myPods"
            }
        }
    },
    "stopConditions": [
        {
            "source": "none",
        }
    ],
    "roleArn": "arn:aws:iam::111122223333:role/fis-experiment-role",
    "tags": {
        "Name": "EksPodNetworkLatency"
    }
}