# Managing Secrets on EKS with External Secrets Operator and AWS Secrets Manager Applications running on EKS often need secrets such as database passwords or API keys. External Secrets Operator can automatically synchronize values from AWS Secrets Manager into Kubernetes Secrets, which containers consume through environment variables or mounted files. Imagine an online store where a customer submits an order at checkout. The checkout service calls a payment provider using an API key stored in AWS Secrets Manager. We will configure ESO to create and update a Kubernetes Secret containing that key, then make it available to the checkout container. Putting the value in a manifest makes deployment reproducible, but also puts a credential in source control. Passing it through Terraform can move that exposure into state and saved plans instead. Marking a Terraform value `sensitive` redacts ordinary output; it does not, by itself, exclude the value from state. [\[1\]](https://developer.hashicorp.com/terraform/language/manage-sensitive-data) The goal is one authoritative secret source, a narrow delivery path, and an explicit application-update procedure. We will build that path on an existing Amazon EKS cluster, using one namespace and one AWS secret. ## What does External Secrets Operator solve? External Secrets Operator (ESO) retrieves values from an external secret system and reconciles them into Kubernetes resources. Here, it creates an ordinary Kubernetes Secret. ESO keeps that Kubernetes Secret synchronized with the external value. Rotating the payment provider's credential and having the application use an updated value are separate steps. [\[2\]](https://external-secrets.io/latest/api/externalsecret/) ![General ESO architecture: an external provider supplies values to the controller inside Kubernetes. ExternalSecret defines mapping and refresh and references a SecretStore or ClusterSecretStore. ESO creates or updates a Kubernetes Secret consumed by a workload as environment variables or mounted files.](https://cdn.hashnode.com/uploads/gql/6ad858beca6b1f0a715e21e0/33cddfa5-538b-4e69-82d6-6762b2262e27.png align="center") *ESO synchronizes external values into a Kubernetes Secret; the workload consumes that copy, not the secret provider API.* External Secrets Operator can be used with Kubernetes on different cloud platforms and integrates with secret services such as AWS Secrets Manager, Azure Key Vault and Google Secret Manager. In this guide, we focus on AWS, using Amazon EKS and AWS Secrets Manager. The components have separate jobs: | Component | Responsibility in this example | | --- | --- | | AWS Secrets Manager | Stores the Terraform-managed demo JSON value and its versions. | | AWS STS and an IAM role | Exchange a trusted Kubernetes service-account identity for temporary credentials, limited to one secret. | | ESO controller | Authenticates, reads the selected value, and reconciles the Kubernetes Secret. | | `SecretStore` | Namespaced connection and authentication configuration for Secrets Manager. | | `ExternalSecret` | Selects the source property, destination key, refresh policy and ownership behavior. | | Kubernetes Secret | Holds the application-facing copy in `shop-secrets-demo`. | | Checkout consumer | Receives the copy as a mounted file and an environment variable; it has no AWS role. | A `SecretStore` holds connection and authentication configuration. An `ExternalSecret` identifies the source value and destination mapping through references, keeping the API key out of these definitions. Those definitions can be reviewed in Git. Terraform creates the demo value; ESO delivers it to Kubernetes. [\[3\]](https://external-secrets.io/latest/api/secretstore/) [\[4\]](https://external-secrets.io/latest/provider/aws-secrets-manager/) ![AWS Secrets Manager and STS outside an existing EKS cluster; a namespace-scoped ESO controller uses a reader service account to synchronize one secret into the checkout namespace.](https://cdn.hashnode.com/uploads/gql/6ad858beca6b1f0a715e21e0/d2c9c65d-4c91-45ee-82f9-c31e7696b183.png align="center") *The AWS services are outside the cluster. The controller and checkout consumer are inside it. Dashed identity connections and solid configuration/API connections are not application traffic. Terraform supplies the public dummy source value; ESO reads it from Secrets Manager.* ## Which identity reads the AWS secret? ESO reads the secret, not the checkout application. This guide uses ESO's supported `auth.jwt.serviceAccountRef` route: ESO requests a short-lived token for a designated reader service account, then uses that token with AWS STS to assume the reader's IAM role. [\[5\]](https://external-secrets.io/latest/provider/aws-access/) There are three Kubernetes service accounts worth keeping distinct: - `eso-shop-system/eso-shop` runs the controller and supplies its Kubernetes API permissions. It has no AWS role annotation. - `shop-secrets-demo/checkout-secret-reader` is the identity referenced by the SecretStore. Its annotation identifies the IAM role, and AWS trust binds that role to this exact namespace and service-account name. - `shop-secrets-demo/checkout-demo` runs the application-side demonstration. It has no AWS role and does not mount a Kubernetes API token. The reader service account does not need a Pod. ESO calls the Kubernetes TokenRequest API on its behalf. Setting `automountServiceAccountToken: false` prevents automatic token mounting; it does not prohibit this explicitly authorized token request. [\[6\]](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) This uses IAM Roles for Service Accounts (IRSA). ESO's AWS documentation specifies that `serviceAccountRef` cannot be used with EKS Pod Identity associations. We choose IRSA because it supports a distinct identity selected by this namespaced store without assigning the AWS reader role to the controller Pod. [\[5\]](https://external-secrets.io/latest/provider/aws-access/) No optional `provider.aws.role` is needed here. That field would add an `AssumeRole` hop after the selected authentication route; the reader service account already points to the final, narrowly scoped role. ### What does the namespace boundary protect? A namespaced SecretStore cannot be referenced by an ExternalSecret in a different namespace. That is useful containment, but not complete tenant isolation. Whoever can create Pods in the application namespace may be able to mount its Secrets; whoever controls a store or its reader service account may affect the authentication path. [\[7\]](https://external-secrets.io/latest/guides/security-best-practices/) [\[8\]](https://kubernetes.io/docs/concepts/security/secrets-good-practices/) For this example, the platform administrator owns the store, reader service account, token Role and RoleBinding. The application team must not have unrestricted edit access to those objects merely because they live beside its Deployment. Kubernetes RBAC is additive: a narrow Role does not cancel a broader existing grant. Workload-creation rights also need policy controls if developers must be prevented from selecting the reader service account for their own Pods. [\[9\]](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) The controller watches only `shop-secrets-demo`, and its ordinary secret-management Role is scoped there. Webhook, certificate-controller and CRD installation still involve cluster-level resources. A namespaced store keeps this example scoped to one namespace. `ClusterSecretStore` supports centrally managed sharing; its broader availability and namespace conditions require a separate access design. [\[7\]](https://external-secrets.io/latest/guides/security-best-practices/) The example files are available in the companion repository: %[https://github.com/ferhaty/external-secrets-eks] ## What do you need before starting? Use an existing, non-production EKS cluster and an approved AWS administrator/deployment context. This guide does not provision a cluster, establish a CI pipeline, or create an IAM OIDC provider. You need: - **Existing IRSA foundation:** the cluster's issuer registered as an IAM OIDC provider in the same AWS account, with the STS audience configured. Use the ARN of that registered IAM OIDC provider. [\[10\]](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) - **Permission to create the demo resources:** Terraform creates a dedicated Secrets Manager secret, its public dummy value, and the reader IAM role and policy in the same account and region. The deployment identity needs create, read (including `secretsmanager:GetSecretValue`), update and delete permissions for the demo resources; ESO receives read-only access to that one secret. - **Deployment tools:** Terraform 1.9 or later, Helm 3, AWS CLI v2, and a kubectl version compatible with your cluster. The configuration below pins the provider, chart and container image for reproducibility. [\[11\]](https://github.com/external-secrets/external-secrets/releases/tag/helm-chart-2.14.0) [\[12\]](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/iam_role.html.markdown) [\[13\]](https://raw.githubusercontent.com/docker-library/official-images/master/library/busybox) - **Kubernetes permissions:** a platform administrator able to install the chart and create the two example namespaces, service accounts and RBAC resources. The application identity does not need these permissions. - **Network access:** ESO must reach the Kubernetes API, DNS, STS and Secrets Manager. Image pulls and the admission webhook must also work. In a private cluster, supply the appropriate private endpoints or controlled egress; an IAM policy does not provide connectivity. [\[7\]](https://external-secrets.io/latest/guides/security-best-practices/) The chart installation below assumes no existing ESO installation or shared ESO CRD ownership. Check first. If your platform already runs ESO, let its administrator integrate the store, controller-class and RBAC design into that installation; do not install a second owner of its CRDs or disable another team's controllers. Run the following from a directory containing the files shown below. Replace the account, ARN and context placeholders before proceeding. Execute each stage separately and stop on errors. ```bash export AWS_REGION=us-east-1 export EKS_CONTEXT=YOUR_EXISTING_EKS_CONTEXT aws sts get-caller-identity kubectl --context "$EKS_CONTEXT" cluster-info helm list --all-namespaces --kube-context "$EKS_CONTEXT" kubectl --context "$EKS_CONTEXT" get namespaces shop-secrets-demo eso-shop-system ``` Confirm that the reported AWS identity and cluster match the intended deployment context. The two namespace lookups should report `NotFound` for a fresh exercise. If either exists, stop and establish ownership rather than reusing or deleting it. Also inspect the existing ESO release/CRDs with your platform administrator before installation. The intended configuration is one AWS account and region. Cross-account secrets, resource-policy sharing, custom STS audiences, application admission policies and an existing shared ESO installation are outside this walkthrough. ## 1\. Create the demo secret and grant one service account read access Terraform creates `shop/demo/checkout` with the JSON value `{"apiKey":"demo-only-v1"}` and grants the reader role access to its actual ARN. This deliberately public dummy value is stored in Terraform state; do not replace it with a real credential. ESO, not Terraform, manages the Kubernetes Secret. The source property `apiKey` will map to the Kubernetes key `api-key`. Save the following as `terraform/main.tf`. ```json terraform { required_version = ">= 1.9.0, < 2.0.0" required_providers { aws = { source = "hashicorp/aws" version = "6.71.0" } } } provider "aws" { region = var.aws_region allowed_account_ids = [var.aws_account_id] } variable "aws_region" { type = string default = "us-east-1" } variable "aws_account_id" { type = string validation { condition = can(regex("^[0-9]{12}$", var.aws_account_id)) error_message = "Use the intended 12-digit AWS account ID." } } variable "oidc_provider_arn" { description = "Existing IAM OIDC provider for this EKS cluster." type = string validation { condition = can(regex("^arn:aws:iam::[0-9]{12}:oidc-provider/oidc\\.eks\\.[a-z0-9-]+\\.amazonaws\\.com/id/[A-Za-z0-9]+$", var.oidc_provider_arn)) error_message = "Use the existing EKS OIDC provider ARN, not its HTTPS issuer URL." } } variable "kms_key_arn" { description = "Optional customer-managed key ARN; null for aws/secretsmanager." type = string default = null validation { condition = var.kms_key_arn == null ? true : can(regex("^arn:aws:kms:[a-z0-9-]+:[0-9]{12}:key/[A-Za-z0-9-]+$", var.kms_key_arn)) error_message = "Use a complete KMS key ARN, not an alias or wildcard." } } variable "demo_api_key" { description = "Public dummy value for the propagation exercise; never a real credential." type = string default = "demo-only-v1" } resource "aws_secretsmanager_secret" "demo" { name = "shop/demo/checkout" kms_key_id = var.kms_key_arn recovery_window_in_days = 30 } resource "aws_secretsmanager_secret_version" "demo" { secret_id = aws_secretsmanager_secret.demo.id secret_string = jsonencode({ apiKey = var.demo_api_key }) } locals { namespace = "shop-secrets-demo" service_account = "checkout-secret-reader" oidc_issuer = split("oidc-provider/", var.oidc_provider_arn)[1] } resource "aws_iam_role" "reader" { name = "shop-secrets-demo-reader" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Federated = var.oidc_provider_arn } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringEquals = { "${local.oidc_issuer}:aud" = "sts.amazonaws.com" "${local.oidc_issuer}:sub" = "system:serviceaccount:${local.namespace}:${local.service_account}" } } }] }) lifecycle { precondition { condition = ( split(":", var.oidc_provider_arn)[4] == var.aws_account_id && startswith(local.oidc_issuer, "oidc.eks.${var.aws_region}.amazonaws.com/") && (var.kms_key_arn == null ? true : startswith(var.kms_key_arn, "arn:aws:kms:${var.aws_region}:${var.aws_account_id}:key/")) ) error_message = "OIDC provider and optional KMS key must belong to the intended account and region." } } } resource "aws_iam_role_policy" "reader" { name = "read-checkout-demo-secret" role = aws_iam_role.reader.id policy = jsonencode({ Version = "2012-10-17" Statement = concat([ { Effect = "Allow" Action = ["secretsmanager:GetSecretValue"] Resource = aws_secretsmanager_secret.demo.arn } ], var.kms_key_arn == null ? [] : [{ Effect = "Allow" Action = ["kms:Decrypt"] Resource = var.kms_key_arn Condition = { StringEquals = { "kms:ViaService" = "secretsmanager.${var.aws_region}.amazonaws.com" "kms:EncryptionContext:SecretARN" = aws_secretsmanager_secret.demo.arn } } }]) }) } output "secret_arn" { value = aws_secretsmanager_secret.demo.arn } output "reader_role_arn" { value = aws_iam_role.reader.arn } ``` The trust policy has two exact conditions: the STS audience and the service-account subject. The exact service-account subject limits the trust relationship to the designated reader identity. The provider's allowed-account check and resource precondition also catch account/region mismatches. [\[10\]](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) The reader role's only Secrets Manager action is `GetSecretValue`, scoped to one complete ARN. This example uses an explicit `data` property mapping, not secret discovery, tag fetching, `dataFrom.find`, or PushSecret. The pinned AWS provider implementation retrieves this mapping through `GetSecretValue`; its store validation checks credential retrieval, rather than proving access to every referenced secret. Adding other ESO features can require additional permissions. [\[14\]](https://raw.githubusercontent.com/external-secrets/external-secrets/v2.14.0/providers/v1/aws/secretsmanager/secretsmanager.go) Leave `kms_key_arn` unset to use the AWS-managed `aws/secretsmanager` key. Terraform then omits a customer key from the secret configuration. For a customer-managed key, supply its exact ARN: Terraform uses it to encrypt the demo secret and scopes the reader's decrypt permission to it. Its key policy must allow the deployment identity to encrypt and decrypt through Secrets Manager, and the reader role to decrypt. The optional IAM statement limits decryption through regional Secrets Manager and to this secret's encryption context; it cannot override an explicit deny or a restrictive key policy. [\[15\]](https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access_iam-policies.html) [\[16\]](https://docs.aws.amazon.com/secretsmanager/latest/userguide/security-encryption.html) Save `terraform/terraform.tfvars.example`, then replace these illustrative identifiers in your working `terraform.tfvars`: ```json aws_region = "us-east-1" aws_account_id = "111122223333" oidc_provider_arn = "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE123" ``` ```bash cp terraform/terraform.tfvars.example terraform/terraform.tfvars # Edit terraform.tfvars with the existing account, region and OIDC provider; keep the public dummy value. terraform -chdir=terraform init terraform -chdir=terraform fmt -check terraform -chdir=terraform validate terraform -chdir=terraform plan -out=reader.tfplan ``` Inspect the plan before applying. It should create the demo secret, its value version, the reader role and its inline policy—not the existing OIDC provider, cluster or optional encryption key. The fixed secret and role names must be available. Keep the generated dependency lock file in your configuration repository. Keep local state, saved plans and working variable files out of it. ```bash terraform -chdir=terraform apply reader.tfplan terraform -chdir=terraform output -raw reader_role_arn export SECRET_ARN=$(terraform -chdir=terraform output -raw secret_arn) printf '%s\n' "$SECRET_ARN" ``` Use the returned role ARN in `store.yaml` below and the exported `SECRET_ARN` as the source ARN in `externalsecret.yaml`. The AWS provider supports managing a role's policy as a separate resource, which keeps the trust relationship and access policy explicit. [\[12\]](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/iam_role.html.markdown) ## 2\. Install a controller with a narrow reconciliation scope Save `namespaces.yaml`: ```yaml apiVersion: v1 kind: Namespace metadata: name: shop-secrets-demo --- apiVersion: v1 kind: Namespace metadata: name: eso-shop-system ``` Save `eso-values.yaml`: ```yaml fullnameOverride: eso-shop controllerClass: shop-demo scopedNamespace: shop-secrets-demo scopedRBAC: true installCRDs: true processClusterStore: false processClusterExternalSecret: false processClusterGenerator: false processClusterPushSecret: false processPushSecret: false crds: annotations: helm.sh/resource-policy: keep createClusterSecretStore: false createClusterExternalSecret: false createClusterGenerator: false createClusterPushSecret: false createPushSecret: false serviceAccount: create: true name: eso-shop annotations: {} rbac: serviceAccountTokenCreate: false servicebindings: create: false aggregateToView: false aggregateToEdit: false aggregateToAdmin: false extraEnv: - name: AWS_EC2_METADATA_DISABLED value: "true" ``` The important choices are the watched namespace, its scoped controller Role, an explicit controller class, and disabled cluster-wide/push reconciliation. The named class ties this store to this controller configuration; it is routing, not authorization. The chart normally permits service-account token creation within the controller's RBAC scope. We turn that blanket permission off and add one resource-name-constrained grant in the next stage. We also leave the controller without an AWS role annotation and disable AWS SDK instance-metadata credential discovery in its environment. Restrict Pod access to node credentials at the cluster level as well; this single environment setting is not an isolation boundary for arbitrary workloads. [\[7\]](https://external-secrets.io/latest/guides/security-best-practices/) ```bash kubectl --context "$EKS_CONTEXT" create -f namespaces.yaml helm repo add external-secrets https://charts.external-secrets.io helm repo update external-secrets helm install eso-shop external-secrets/external-secrets --version 2.14.0 --namespace eso-shop-system --kube-context "$EKS_CONTEXT" -f eso-values.yaml --wait --timeout 5m kubectl --context "$EKS_CONTEXT" wait --for=condition=Established --timeout=120s crd/secretstores.external-secrets.io crd/externalsecrets.external-secrets.io ``` The chart renders `external-secrets.io/v1` CRDs for the resources in this guide, so the manifests below use `v1` rather than `v1beta1`. When upgrading, check the chart and its release notes for API and lifecycle changes. This chart renders CRDs through templates, whose lifecycle behavior differs from Helm's `crds/` directory. [\[11\]](https://github.com/external-secrets/external-secrets/releases/tag/helm-chart-2.14.0) [\[18\]](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) `--wait` checks installation readiness, not AWS authentication. The first meaningful AWS-backed checks happen when the store and external secret reconcile. ## 3\. Connect the store and map the required property Save `store.yaml`. Replace the role ARN with Terraform's output and set the same region as the Terraform-created secret. ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: checkout-secret-reader namespace: shop-secrets-demo annotations: eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/shop-secrets-demo-reader automountServiceAccountToken: false --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: eso-checkout-token namespace: shop-secrets-demo rules: - apiGroups: [""] resources: ["serviceaccounts/token"] resourceNames: ["checkout-secret-reader"] verbs: ["create"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: eso-checkout-token namespace: shop-secrets-demo subjects: - kind: ServiceAccount name: eso-shop namespace: eso-shop-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: eso-checkout-token --- apiVersion: external-secrets.io/v1 kind: SecretStore metadata: name: checkout-aws namespace: shop-secrets-demo spec: controller: shop-demo provider: aws: service: SecretsManager region: us-east-1 auth: jwt: serviceAccountRef: name: checkout-secret-reader ``` The Role grants ESO permission to create a token only for `checkout-secret-reader`; the application receives no AWS access through this grant. ESO's chart supplies the separate namespaced permissions it needs to read service-account metadata and manage the target Secret. [\[7\]](https://external-secrets.io/latest/guides/security-best-practices/) Save `externalsecret.yaml`, replacing the illustrative `remoteRef.key` ARN with the full `SECRET_ARN` printed by Terraform, including its generated suffix: ```yaml apiVersion: external-secrets.io/v1 kind: ExternalSecret metadata: name: checkout-credentials namespace: shop-secrets-demo spec: refreshPolicy: Periodic refreshInterval: 1m secretStoreRef: name: checkout-aws kind: SecretStore target: name: checkout-credentials creationPolicy: Owner deletionPolicy: Retain data: - secretKey: api-key remoteRef: key: arn:aws:secretsmanager:us-east-1:111122223333:secret:shop/demo/checkout-AbCdEf property: apiKey version: AWSCURRENT decodingStrategy: None ``` These settings make the lifecycle explicit: - `Periodic` with `1m` asks ESO to reconcile periodically. It is not a one-minute end-to-end application-update guarantee. - `AWSCURRENT` follows the current Secrets Manager version label. A `uuid/...` version reference would pin a version instead of following that label. [\[4\]](https://external-secrets.io/latest/provider/aws-secrets-manager/) - `creationPolicy: Owner` creates the target Secret with an ownership relationship to this ExternalSecret. Deleting the ExternalSecret normally lets Kubernetes garbage-collect that target. - `deletionPolicy: Retain` retains the Kubernetes copy when source data disappears. It is not a request to retain the copy after deleting its owning ExternalSecret. [\[2\]](https://external-secrets.io/latest/api/externalsecret/) ```bash kubectl --context "$EKS_CONTEXT" apply -f store.yaml kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo wait --for=condition=Ready secretstore/checkout-aws --timeout=120s kubectl --context "$EKS_CONTEXT" apply -f externalsecret.yaml kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo wait --for=condition=Ready externalsecret/checkout-credentials --timeout=180s kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo get externalsecret checkout-credentials kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo get secret checkout-credentials ``` The store becoming Ready means its provider configuration and credentials passed the store check. The ExternalSecret becoming Ready is the more relevant signal that this particular mapping synchronized. Neither condition proves that the checkout service has reloaded or successfully authenticated with a payment provider. The plain `get secret` command shows metadata, type and key count, not the values. Avoid `-o yaml`, `-o json`, or decoding `.data` into shared terminal output. Anyone who can read the Kubernetes Secret can recover its value from the base64-encoded data. [\[19\]](https://kubernetes.io/docs/concepts/configuration/secret/) ## 4\. Consume the secret without giving the application an AWS role Use a standard BusyBox container as a small credential-consumption probe. It stands in for the checkout process to demonstrate Kubernetes delivery behavior, without requiring a custom application, container build, HTTP endpoint or payment account. Save `consumer.yaml`: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: checkout-demo namespace: shop-secrets-demo automountServiceAccountToken: false --- apiVersion: apps/v1 kind: Deployment metadata: name: checkout-demo namespace: shop-secrets-demo spec: replicas: 1 selector: matchLabels: app: checkout-demo template: metadata: labels: app: checkout-demo spec: serviceAccountName: checkout-demo automountServiceAccountToken: false securityContext: runAsNonRoot: true runAsUser: 10001 runAsGroup: 10001 fsGroup: 10001 seccompProfile: type: RuntimeDefault containers: - name: consumer image: busybox:1.37.0 command: ["sleep", "2147483647"] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] resources: requests: cpu: 10m memory: 16Mi limits: cpu: 100m memory: 32Mi env: - name: CHECKOUT_API_KEY valueFrom: secretKeyRef: name: checkout-credentials key: api-key volumeMounts: - name: credentials mountPath: /run/checkout readOnly: true volumes: - name: credentials secret: secretName: checkout-credentials defaultMode: 0440 ``` The two delivery forms intentionally use the same Secret. The environment variable is supplied when the container starts. The volume projects the key into `/run/checkout/api-key`; there is no `subPath` mount that would prevent automatic projected updates. A real application using the file must reopen it or implement a suitable reload mechanism rather than assuming a value cached in memory will change. [\[19\]](https://kubernetes.io/docs/concepts/configuration/secret/) [\[20\]](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/) ```bash kubectl --context "$EKS_CONTEXT" apply -f consumer.yaml kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo rollout status deployment/checkout-demo --timeout=120s kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo exec deployment/checkout-demo -- sh -c ' test -s /run/checkout/api-key && test -n "$CHECKOUT_API_KEY" && test "$CHECKOUT_API_KEY" = "$(cat /run/checkout/api-key)" && printf "File and environment match; values not printed.\n" ' ``` After initial synchronization, that command should report that the file and environment match without printing either value. A missing Secret or key should block this consumer from starting; it is not marked optional. If the command exits unsuccessfully, diagnose the reconciliation and mount before continuing. The consumer has no Kubernetes Secret-reading Role. Kubelet supplies the referenced Secret to the Pod; the application does not need Kubernetes API access merely to consume an injected value. Container execution, Pod creation, node access, broad Secret RBAC and backups remain relevant exposure paths. [\[8\]](https://kubernetes.io/docs/concepts/security/secrets-good-practices/) ## 5\. What happens when the source value changes? There are two separate transitions: ESO updates the Kubernetes Secret, then Kubernetes and the application make that value usable. A mounted Secret is updated eventually; an existing container's environment variables are not. ESO leaves the Deployment running; Pod replacement or application reload is a separate step. [\[19\]](https://kubernetes.io/docs/concepts/configuration/secret/) [\[20\]](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/) In `terraform/main.tf`, change the `demo_api_key` default from `"demo-only-v1"` to `"demo-only-v2"`, then plan the update: ```bash # In terraform/main.tf, change the demo_api_key default from "demo-only-v1" to "demo-only-v2". terraform -chdir=terraform plan -out=rotation.tfplan ``` Inspect the plan, then apply it separately: ```bash terraform -chdir=terraform apply rotation.tfplan ``` Changing `secret_string` creates a new Secrets Manager version; it does not overwrite the old version. With `version_stages` omitted, AWS makes the new version `AWSCURRENT`, which ESO follows. Keep this demo value managed through Terraform rather than changing it with the console or AWS CLI. This exercises propagation, not rotation of a credential in an external payment system. [\[17\]](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/secretsmanager_secret_version.html.markdown) Wait for the periodic reconciliation, or request an immediate reconciliation by updating the ExternalSecret annotation. Record the previous `refreshTime` before changing the source if you want to compare it with the next successful refresh. ```bash kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo annotate externalsecret checkout-credentials force-sync="$(date +%s)" --overwrite kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo get externalsecret checkout-credentials -o jsonpath='{.status.refreshTime}{"\n"}{.status.conditions}{"\n"}' kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo exec deployment/checkout-demo -- sh -c ' test "$(cat /run/checkout/api-key)" = "demo-only-v2" && test "$CHECKOUT_API_KEY" = "demo-only-v1" && printf "New file, old environment; no values printed.\n" ' ``` The final command should eventually report “New file, old environment.” Repeat that check after allowing time for reconciliation and kubelet's projected-volume update. A Ready condition that was already true can return immediately; it is not a barrier proving that this particular new version has reached the Pod. The explicit dummy-value comparison checks that distinction without revealing a real key. Now replace the consumer Pod: ```bash kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo rollout restart deployment/checkout-demo kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo rollout status deployment/checkout-demo --timeout=120s kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo exec deployment/checkout-demo -- sh -c ' test "$CHECKOUT_API_KEY" = "demo-only-v2" && test "$CHECKOUT_API_KEY" = "$(cat /run/checkout/api-key)" && printf "Replacement Pod uses the new value in both paths.\n" ' ``` After rollout, both paths should use the second value. In a real checkout service, also verify a safe authenticated request to the downstream system. A successful Secret sync, a running Pod, and a successful payment-provider request are three different acceptance checks. ### How does propagation fit into credential rotation? A credential must change in the system that validates it as well as in the store that distributes it. AWS describes rotation in terms of updating both the secret and the service's credentials. This walkthrough covers propagation; a full rotation workflow also coordinates a rotation function or the payment provider's credential lifecycle. [\[21\]](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html) Where supported, an overlap period in which old and new credentials are valid can accommodate reconciliation, rolling replacement and in-flight requests. Revoke the old credential only after consumers have switched and authentication has been checked. The overlap duration and rollback procedure depend on the actual provider. ## How do you diagnose a failed synchronization? Identify the failing layer while keeping secret values out of diagnostic output. Inspect the SecretStore, ExternalSecret conditions and controller logs, then check the consumer only after the destination Secret exists. ```bash kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo describe secretstore checkout-aws kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo describe externalsecret checkout-credentials kubectl --context "$EKS_CONTEXT" -n eso-shop-system logs deployment/eso-shop --tail=80 ``` Logs and events can contain resource identifiers or other operational detail. Review and redact them before sharing, and avoid debug logging that could expose credentials. | Symptom | First checks | | --- | --- | | Store cannot become Ready | Reader service account exists; role annotation is correct; ESO can request its token; IAM provider, subject and audience match; STS is reachable. | | Store Ready, ExternalSecret not Ready | Full secret ARN, region, `GetSecretValue` permission, KMS authorization if applicable, JSON property `apiKey`, and Secrets Manager connectivity. | | No controller activity | Store controller class, watched namespace, disabled reconciler settings, controller readiness and Kubernetes RBAC. | | Consumer cannot start | Target Secret exists in the same namespace and contains `api-key`; image pull and volume permissions work. | | Secret updated, process still uses old value | Environment variables, an application cache, a long-open file descriptor, or a `subPath` mount; synchronization is not reload. | | A previously working consumer still runs during an outage | It may be using the last synchronized copy. Continued availability is not proof of freshness. | For a bounded failure exercise, use only the dedicated demo mapping: change `remoteRef.property` in `externalsecret.yaml` to a nonexistent property, apply it, and inspect the ExternalSecret condition after reconciliation. Restore `apiKey`, reapply and confirm recovery. The pinned provider returns an error for a missing property. Do not alter production IAM policies or delete the AWS secret merely to test failure handling. [\[14\]](https://raw.githubusercontent.com/external-secrets/external-secrets/v2.14.0/providers/v1/aws/secretsmanager/secretsmanager.go) Expect an existing target to remain available under the selected retention behavior while failed refreshes need attention. A first-ever failed sync has no previous copy to retain. Monitor both reconciliation failures and freshness, and separately monitor the application's authentication failures. The existing [EKS observability walkthrough](https://blog.turingts.com/observability-on-amazon-eks-metrics-logs-and-traces) covers the surrounding metrics, logs and traces stack for optional further reading. ## What remains sensitive after synchronization? ESO reduces where people and delivery jobs have to handle the value. The resulting Kubernetes Secret still requires access controls. Limit Secret `get`, `list` and `watch` access; restrict workload creation and execution; protect node access, backups and diagnostic exports. Encryption at rest protects storage media, not access by an already authorized API caller. [\[8\]](https://kubernetes.io/docs/concepts/security/secrets-good-practices/) A compromised ESO controller can use the AWS permissions and Kubernetes permissions granted to it. The one-secret policy and single reader-token grant bound this example's intended access, but they do not protect against a cluster administrator or a broader existing grant. Keep the controller namespace platform-managed, review chart upgrades, and make admission and network policy part of your real tenant design. If the requirement is that the value must never exist as a Kubernetes Secret, this reconciliation pattern does not satisfy it. Evaluate a different consumption design, such as direct application retrieval or an appropriate mounted-secret integration, against that requirement. ## Cleanup without deleting shared dependencies Remove only the dedicated resources created by this walkthrough. Stop the consumer first, then delete the ExternalSecret and confirm that its owned Kubernetes Secret disappears. Removing the reader role must wait until this example no longer needs it. ```bash # Only for the dedicated example resources created by this walkthrough. kubectl --context "$EKS_CONTEXT" delete -f consumer.yaml kubectl --context "$EKS_CONTEXT" delete -f externalsecret.yaml kubectl --context "$EKS_CONTEXT" -n shop-secrets-demo wait --for=delete secret/checkout-credentials --timeout=120s kubectl --context "$EKS_CONTEXT" delete -f store.yaml helm uninstall eso-shop --namespace eso-shop-system --kube-context "$EKS_CONTEXT" terraform -chdir=terraform plan -destroy -out=cleanup.tfplan ``` Inspect the destroy plan. It should remove the demo secret and its managed version, plus the example IAM role and inline policy. Apply only after that review: ```bash terraform -chdir=terraform apply cleanup.tfplan ``` The demo secret is scheduled for deletion with a 30-day recovery window. Its name cannot be reused until AWS finishes deletion; choose a different demo name if you repeat the exercise sooner. Removing only the Terraform version resource would not delete the secret or its current value, so cleanup removes the secret resource too. [\[17\]](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/secretsmanager_secret_version.html.markdown) [\[24\]](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/secretsmanager_secret.html.markdown) [\[25\]](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_DeleteSecret.html) The explicit `helm.sh/resource-policy: keep` annotation in our values retains the CRDs on Helm uninstall; this walkthrough also leaves its namespaces in place for platform-admin inspection. Retained CRDs become an administrator-owned lifecycle concern—do not assume a later reinstall will automatically adopt them. [\[22\]](https://helm.sh/docs/howto/charts_tips_and_tricks/) Deleting these cluster-scoped CRDs can remove custom resources belonging to other users; leave their removal to the platform administrator. An owner can remove the empty example namespaces later after checking for anything added during the exercise. [\[18\]](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) ## Frequently asked questions ### Does ESO keep secret values out of Terraform state? No. ESO does not control what Terraform stores. This example deliberately provisions a public dummy value with Terraform; a real-secret workflow must account for how values are supplied and retained. [\[1\]](https://developer.hashicorp.com/terraform/language/manage-sensitive-data) ### Can the checkout application use the reader service account? It should not in this design. The application consumes a Kubernetes Secret and needs no AWS role. Giving its Pod the reader identity would create another path to AWS and undermine the intended separation between synchronization and consumption. ### Does changing Secrets Manager restart the application? No. ESO reconciles the target Secret. Projected files update eventually, while an existing environment variable remains unchanged. Use an application reload strategy for files or controlled Pod replacement for environment variables, then verify actual downstream authentication. [\[19\]](https://kubernetes.io/docs/concepts/configuration/secret/) [\[20\]](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/) ### Can several namespaces share the same SecretStore? Not a namespaced SecretStore. Use independent stores and appropriately scoped identities, or deliberately design a ClusterSecretStore with namespace restrictions and suitable RBAC. [\[3\]](https://external-secrets.io/latest/api/secretstore/) [\[7\]](https://external-secrets.io/latest/guides/security-best-practices/) ### How do you verify end-to-end secret delivery? Terraform mock-provider tests check configuration using simulated provider responses. Verifying AWS permissions and live secret synchronization requires an end-to-end check in the target environment. [\[23\]](https://developer.hashicorp.com/terraform/language/tests/mocking) Verify that the intended source value reaches the intended Pod, a changed value propagates as expected, and the application safely starts using it. ## Sources 1. [Terraform sensitive data](https://developer.hashicorp.com/terraform/language/manage-sensitive-data) 2. [ESO ExternalSecret API](https://external-secrets.io/latest/api/externalsecret/) 3. [ESO SecretStore API](https://external-secrets.io/latest/api/secretstore/) 4. [ESO AWS Secrets Manager provider](https://external-secrets.io/latest/provider/aws-secrets-manager/) 5. [ESO AWS authentication](https://external-secrets.io/latest/provider/aws-access/) 6. [Kubernetes service accounts and TokenRequest](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) 7. [ESO security best practices](https://external-secrets.io/latest/guides/security-best-practices/) 8. [Kubernetes Secret security](https://kubernetes.io/docs/concepts/security/secrets-good-practices/) 9. [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) 10. [AWS EKS service-account role association](https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html) 11. [ESO Helm chart 2.14.0 release](https://github.com/external-secrets/external-secrets/releases/tag/helm-chart-2.14.0) 12. [AWS Terraform provider 6.71.0 IAM role](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/iam_role.html.markdown) 13. [Docker Official Images BusyBox tags](https://raw.githubusercontent.com/docker-library/official-images/master/library/busybox) 14. [ESO 2.14.0 AWS provider implementation](https://raw.githubusercontent.com/external-secrets/external-secrets/v2.14.0/providers/v1/aws/secretsmanager/secretsmanager.go) 15. [AWS Secrets Manager identity policies](https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access_iam-policies.html) 16. [AWS Secrets Manager KMS encryption](https://docs.aws.amazon.com/secretsmanager/latest/userguide/security-encryption.html) 17. [AWS Terraform provider 6.71.0 secret version](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/secretsmanager_secret_version.html.markdown) 18. [Helm custom resource definitions](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) 19. [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) 20. [Kubernetes credential distribution](https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/) 21. [AWS Secrets Manager rotation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html) 22. [Helm retained-resource annotation](https://helm.sh/docs/howto/charts_tips_and_tricks/) 23. [Terraform mock-provider tests](https://developer.hashicorp.com/terraform/language/tests/mocking) 24. [AWS Terraform provider 6.71.0 secret](https://raw.githubusercontent.com/hashicorp/terraform-provider-aws/v6.71.0/website/docs/r/secretsmanager_secret.html.markdown) 25. [AWS DeleteSecret API](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_DeleteSecret.html) ## FAQ ### What is the role of External Secrets Operator in EKS? External Secrets Operator (ESO) synchronizes values from AWS Secrets Manager into Kubernetes Secrets. It ensures that Kubernetes Secrets are kept up-to-date with their external sources, allowing applications to consume them securely. ### How does ESO authenticate to AWS Secrets Manager? ESO authenticates using a service account with a specific IAM role. It requests a short-lived token via Kubernetes, which is used with AWS STS to assume the IAM role, allowing access to AWS Secrets Manager. ### Does changing a secret in AWS Secrets Manager restart the application? No, changing a secret in AWS Secrets Manager does not automatically restart the application. ESO updates the Kubernetes Secret, but the application needs a reload strategy or Pod replacement to use the new value. ### Can multiple namespaces share a SecretStore in ESO? A namespaced SecretStore cannot be shared across namespaces. For sharing, use a ClusterSecretStore with appropriate restrictions and RBAC settings. ### Does ESO keep secret values out of Terraform state? Yes, in this example, Terraform manages only IAM roles and policies, never reading or writing secret values. This configuration keeps secrets out of Terraform state, reducing exposure. ### How do you verify end-to-end secret delivery with ESO? Verify that the source value reaches the Pod as intended, changes propagate correctly, and the application starts using the new value safely. Mock-provider tests can simulate provider responses for configuration checks.