AWS chaos engineering is the practice of breaking your own systems on purpose, under controlled conditions, to find the failure modes you’d otherwise discover at 3am. For EKS microservices, that means two complementary layers: killing pods inside the cluster, and injecting faults into the managed AWS services your pods depend on.
This guide covers both. Every action ID here was checked against the current AWS documentation, and every JSON template was validated with a parser before publication.
What You’ll Set Up
You’ll deploy kube-monkey for scheduled random pod kills inside EKS. Then you’ll build AWS FIS experiment templates for targeted pod deletion, RDS reboots, and ElastiCache availability-zone interruption. Finally, you’ll learn how to run these safely and automate them.
Two things are worth knowing before you start any AWS chaos engineering work. First, FIS charges per action-minute, so experiments cost money. Second, always run against staging before production, and always configure stop conditions.
Prerequisites
You’ll need an AWS account with permissions for EKS, RDS, ElastiCache, and FIS, plus a running EKS cluster on version 1.23 or later. Install kubectl, helm, and the AWS CLI, and configure your credentials.
You’ll also need an IAM role for FIS experiments, which we create in step two.
Step 1: Deploy kube-monkey for Random Pod Kills
Kube-monkey brings Netflix’s Chaos Monkey model to Kubernetes, and it’s the simplest entry point into AWS chaos engineering. It runs once each weekday morning, builds a schedule, then kills pods from opted-in deployments at random times during a configurable window.
Install the Chart
helm repo add kube-monkey https://asobti.github.io/kube-monkey/charts/repo
helm repo update
helm install kube-monkey kube-monkey/kube-monkey --namespace kube-system
kubectl get pods -n kube-system -l app=kube-monkeyStart with dry_run = true in your config so terminations are logged rather than executed. Verify the schedule looks sane before letting it kill anything.
Opt a Deployment In
This is where most guides go wrong. Kube-monkey needs the kube-monkey/identifier label to match pods back to their parent app, and the labels must appear on the pod template. Omit the identifier, and nothing will ever be killed.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-service
namespace: default
labels:
kube-monkey/enabled: "enabled"
kube-monkey/identifier: "orders-service"
kube-monkey/mtbf: "1"
kube-monkey/kill-mode: "fixed"
kube-monkey/kill-value: "1"
spec:
template:
metadata:
labels:
app: orders-service
kube-monkey/enabled: "enabled"
kube-monkey/identifier: "orders-service"
kube-monkey/mtbf: "1"
kube-monkey/kill-mode: "fixed"
kube-monkey/kill-value: "1"Apply it with kubectl apply -f orders-deployment.yaml. With mtbf: "1", this deployment becomes eligible for a pod kill roughly every weekday. The selection is a biased coin flip rather than a guarantee, and kills land inside the configured window, which defaults to 10am–4pm.
Step 2: Create the FIS IAM Role
FIS assumes a role to act on your resources, so the role needs both a trust policy and permissions. The trust policy is the part most tutorials skip.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "fis.amazonaws.com" },
"Action": "sts:AssumeRole"
}
]
}Then attach permissions covering the actions you actually intend to run.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["rds:RebootDBInstance", "rds:DescribeDBInstances"],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": ["elasticache:DescribeReplicationGroups", "elasticache:InterruptClusterAzPower"],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": ["eks:DescribeCluster", "ec2:DescribeSubnets"],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": ["cloudwatch:DescribeAlarms", "tag:GetResources"],
"Resource": "*"
}
]
}AWS also publishes managed policies such as AWSFaultInjectionSimulatorRDSAccess and AWSFaultInjectionSimulatorEKSAccess, which are simpler than hand-rolling permissions.
Step 3: Extra Setup for EKS Pod Actions
FIS pod actions need Kubernetes-side permissions as well as IAM. Create a service account with a Role and role binding granting access to pods, pods/ephemeralcontainers, pods/exec, and ConfigMaps in your target namespace.
Then map your IAM role to a Kubernetes user:
aws eks create-access-entry \
--principal-arn arn:aws:iam::123456789012:role/AWSFISExperimentRole \
--username fis-experiment \
--cluster-name orders-clusterOne easily-missed requirement: target pods must have readOnlyRootFilesystem: false in their security context, or every EKS pod action will fail.
Step 4: Build Your AWS Chaos Engineering Templates
Save each template as JSON, then create and run it:
aws fis create-experiment-template --cli-input-json file://eks-pod-delete.json
aws fis start-experiment --experiment-template-id EXT123abcDelete an EKS Pod
Note that aws:eks:pod targets cannot be selected by ARN or tag. You must use resource parameters.
{
"description": "Delete one pod from the orders-service deployment",
"targets": {
"OrdersPods": {
"resourceType": "aws:eks:pod",
"parameters": {
"clusterIdentifier": "orders-cluster",
"namespace": "default",
"selectorType": "labelSelector",
"selectorValue": "app=orders-service"
},
"selectionMode": "COUNT(1)"
}
},
"actions": {
"DeleteOrdersPod": {
"actionId": "aws:eks:pod-delete",
"description": "Delete one running pod",
"parameters": {
"kubernetesServiceAccount": "myserviceaccount",
"gracePeriodSeconds": "0"
},
"targets": { "Pods": "OrdersPods" }
}
},
"stopConditions": [
{
"source": "aws:cloudwatch:alarm",
"value": "arn:aws:cloudwatch:us-west-2:123456789012:alarm:OrdersApiHighErrorRate"
}
],
"roleArn": "arn:aws:iam::123456789012:role/AWSFISExperimentRole",
"tags": { "Name": "eks-pod-delete-orders" }
}FIS also offers pod-cpu-stress, pod-memory-stress, pod-io-stress, pod-network-latency, pod-network-packet-loss, and pod-network-blackhole-port for richer scenarios than deletion alone.
Reboot an RDS Instance
{
"description": "Reboot RDS instance with forced Multi-AZ failover",
"targets": {
"RDSInstance": {
"resourceType": "aws:rds:db",
"resourceArns": ["arn:aws:rds:us-west-2:123456789012:db:orders-db"],
"selectionMode": "ALL"
}
},
"actions": {
"RebootRDS": {
"actionId": "aws:rds:reboot-db-instances",
"parameters": { "forceFailover": "true" },
"targets": { "DBInstances": "RDSInstance" }
}
},
"stopConditions": [
{
"source": "aws:cloudwatch:alarm",
"value": "arn:aws:cloudwatch:us-west-2:123456789012:alarm:OrdersApiHighErrorRate"
}
],
"roleArn": "arn:aws:iam::123456789012:role/AWSFISExperimentRole",
"tags": { "Name": "rds-reboot-orders-db" }
}Interrupt an ElastiCache Availability Zone
FIS models ElastiCache failure as AZ power interruption rather than a direct failover call.
{
"description": "Interrupt power to ElastiCache replication group nodes in one AZ",
"targets": {
"RedisReplicationGroup": {
"resourceType": "aws:elasticache:replicationgroup",
"resourceArns": ["arn:aws:elasticache:us-west-2:123456789012:replicationgroup:orders-redis"],
"selectionMode": "ALL"
}
},
"actions": {
"InterruptRedisAZ": {
"actionId": "aws:elasticache:replicationgroup-interrupt-az-power",
"parameters": { "duration": "PT5M" },
"targets": { "ReplicationGroups": "RedisReplicationGroup" }
}
},
"stopConditions": [
{
"source": "aws:cloudwatch:alarm",
"value": "arn:aws:cloudwatch:us-west-2:123456789012:alarm:OrdersApiHighErrorRate"
}
],
"roleArn": "arn:aws:iam::123456789012:role/AWSFISExperimentRole",
"tags": { "Name": "elasticache-az-interrupt" }
}What FIS Cannot Do
Being clear about the gaps saves hours of wasted searching. FIS has no native actions for ACM, KMS, Route 53, or MSK, which limits how far pure AWS chaos engineering can reach. If a tutorial hands you aws:kms:disable-key or aws:route53:change-resource-record-sets, those action IDs are invented.
For MSK broker reboots, use aws:ssm:start-automation-execution to call an SSM document that invokes the MSK RebootBroker API. For certificate, key, or DNS failure testing, simulate the effect at the network layer with aws:network:disrupt-connectivity, or revoke access through IAM policy changes outside FIS.
Step 5: Run, Observe, and Automate
Watch pods with kubectl get pods --watch while the experiment runs. Track failovers and error rates in CloudWatch, and pull application logs from CloudWatch Logs or your EFK stack.
The metrics that matter in AWS chaos engineering are mean time to detect and mean time to recover. If a pod dies and your error rate spikes for ninety seconds, that’s a readiness-probe or connection-pool problem worth fixing.
Automating Your AWS Chaos Engineering Practice
Use EventBridge rules to start FIS experiments on a schedule. Invoke experiments from CodePipeline as a post-deployment gate. Push results to Slack through SNS subscriptions on your CloudWatch alarms.
Start with one experiment in staging, fix what it reveals, then expand. AWS chaos engineering earns its value through iteration, not through breaking everything at once.
Talk to our team about building a resilience testing practice on AWS → TALK
Frequently Asked Questions
Does AWS FIS support Route 53, KMS, or ACM? No. FIS has native actions for ARC, CloudWatch, Direct Connect, DSQL, DynamoDB, EBS, EC2, ECS, EKS, ElastiCache, Kinesis, Lambda, MemoryDB, Network, RDS, S3, and SSM only. Any tutorial offering aws:route53:, aws:kms:, or aws:acm: action IDs is using IDs that don’t exist.
Why isn’t kube-monkey killing any pods? Almost always a missing kube-monkey/identifier label, or labels applied to deployment metadata but not the pod template. Check that dry_run is false, and remember it only runs on weekdays within the configured hour window.
Do I need both kube-monkey and FIS? Not necessarily. FIS covers EKS pod chaos natively through aws:eks:pod-delete and its stress actions, with stop conditions and audit trails built in. Kube-monkey is useful when you want continuous, unattended randomness rather than discrete experiments.
Is stopConditions really required? Yes. create-experiment-template rejects templates without it. Use a CloudWatch alarm ARN in production. {"source": "none"} is valid for early testing but removes your safety net.
How do I run chaos experiments against MSK? Through aws:ssm:start-automation-execution pointing at an SSM document that calls the MSK RebootBroker API. There is no native FIS Kafka action.
Read More Here
