Secrets and Identity

Static credentials are a technical debt that compounds quietly until it doesn't. A long-lived AWS access key checked into a .env file in 2019. A database password that's been the same since the service launched. A CI pipeline secret that seventeen people have touched and nobody has rotated since the engineer who created it left the company.

The breach surface here is not exotic. Static credentials that leak — through logs, through git history, through a developer's compromised laptop — give attackers persistent access until someone notices and rotates. Which, on average, takes weeks.

Workload identity eliminates the credential. Instead of "here is a secret that proves who you are," the workload says "here is proof from my runtime environment that I am who I claim to be." Nothing to leak, nothing to rotate.

The Mental Model

Two separate problems get conflated: authentication (who is this workload?) and authorisation (what is this workload allowed to do?). Workload identity solves authentication. IAM policies and Vault policies solve authorisation. Keep them separate in your head or you'll build the wrong thing.

The credential is short-lived. Even if it leaks, the window of exploitation is minutes, not months.

IRSA: Kubernetes on AWS

If you're running Kubernetes on EKS, IRSA (IAM Roles for Service Accounts) is the baseline. A Kubernetes service account is mapped to an IAM role. Pods using that service account can assume the role via OIDC federation — no AWS credentials in environment variables.

# Terraform: create the IRSA binding
data "aws_iam_policy_document" "payments_assume" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.eks.arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:payments:payments-api"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}
 
resource "aws_iam_role" "payments_api" {
  name               = "payments-api-irsa"
  assume_role_policy = data.aws_iam_policy_document.payments_assume.json
}
 
resource "aws_iam_role_policy_attachment" "payments_s3" {
  role       = aws_iam_role.payments_api.name
  policy_arn = aws_iam_policy.payments_s3_read.arn
}

# Kubernetes: annotate the service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payments-api-irsa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  template:
    spec:
      serviceAccountName: payments-api   # this is all you need
      containers:
        - name: api
          image: payments-api:latest
          # No AWS_ACCESS_KEY_ID, no AWS_SECRET_ACCESS_KEY

The AWS SDK in the container automatically discovers credentials via the token file mounted by the EKS pod identity webhook. Zero code changes in the application.

Vault Agent for Non-AWS Secrets

Database passwords, third-party API keys, TLS certificates — these still need to exist somewhere, but they shouldn't live in Kubernetes Secrets as base64 plaintext (base64 is encoding, not encryption).

HashiCorp Vault with the Vault Agent Injector is the common solution:

# Pod annotation pattern — Vault Agent injects the secret as a file
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payments-api"
        vault.hashicorp.com/agent-inject-secret-db: "secret/payments/db"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "secret/payments/db" -}}
          DB_PASSWORD={{ .Data.data.password }}
          DB_HOST={{ .Data.data.host }}
          {{- end }}
    spec:
      containers:
        - name: api
          envFrom:
            - secretRef:
                name: payments-env   # written by vault-agent init container

The Vault policy for this role:

# vault/policies/payments-api.hcl
path "secret/data/payments/*" {
  capabilities = ["read"]
}
 
# Explicitly deny write — least privilege
path "secret/data/payments/*" {
  capabilities = ["deny"]
  denied_parameters = {}
}

Rotation and Blast Radius

Rotation is not enough on its own. The question after "how do we rotate?" is "when this secret leaks, what can an attacker do with it, for how long?"

Blast radius reduction checklist:

• Scope secrets minimally. A database credential that can only read the payments schema is better than one that can access all schemas.
• Set TTLs. Vault dynamic secrets generate a unique credential per request with a configurable TTL. When the TTL expires, Vault revokes it automatically.
• Audit all secret access. Vault audit logs every read. If a credential leaks and is used by an attacker, the access log tells you which secret, when, and from which IP.
• Separate environments strictly. A staging credential should never work in production. This sounds obvious; misconfigured sharing is common.

# Vault dynamic database credential — new password per request, auto-expires
resource "vault_database_secret_backend_role" "payments_api" {
  backend = vault_database_secrets_engine.postgres.path
  name    = "payments-api"
  db_name = "payments-postgres"
 
  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
    "GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA payments TO \"{{name}}\";",
  ]
 
  default_ttl = "1h"
  max_ttl     = "4h"
}

With dynamic credentials, there is no long-lived database password to leak. Each connection gets a unique credential that self-destructs.

Detecting Static Credentials in Code

Even with workload identity in place, someone will paste a secret into a config file. Run secret scanning in CI:

# .github/workflows/secret-scan.yml
name: Secret scan
on: [push, pull_request]
jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # full history — secrets hide in old commits too
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}

Block the merge. Don't just warn — secrets that make it to main are in git history forever.

Key Takeaways

• Static credentials are a persistent breach surface; workload identity eliminates the credential entirely by replacing it with cryptographically-verified runtime identity.
• IRSA on EKS binds a Kubernetes service account to an IAM role via OIDC federation — no AWS credentials in environment variables, no rotation ceremony.
• Vault Agent Injector handles non-AWS secrets by injecting them as files at runtime, avoiding plaintext secrets in Kubernetes Secrets objects.
• Dynamic database credentials (unique credential per connection, auto-expiring TTL) reduce blast radius from "permanent access" to "hours of access."
• Blast radius is the right mental model: scope secrets minimally, set TTLs, audit all access, and enforce strict environment separation.
• Secret scanning in CI must block merges, not just warn — secrets in git history are not recoverable by rotation alone.