Secrets at Rest and in Motion

Encrypting data is table stakes. The real question is who can decrypt it, under what conditions, and what happens when the key is compromised. Most teams encrypt a database column and call it done — without thinking about where the encryption key lives, how it rotates, and whether the system that holds it is more or less trustworthy than the system it is protecting.

This post covers the actual mechanics: KMS-backed envelope encryption for data at rest, and the in-transit controls that are frequently misconfigured or missing.

The Problem with "Just Encrypt It"

Encrypting data with a static application-level key stored in an environment variable solves the problem of someone reading the raw database file. It does not solve the problem of a compromised application server, a leaked .env file, or a malicious engineer with database access. The key and the ciphertext are both accessible from the same context.

Envelope encryption separates the key hierarchy: a master key that never leaves the KMS encrypts a data encryption key (DEK) that lives alongside the ciphertext. The application decrypts the DEK at startup or per-request; the master key is never in application memory.

Envelope Encryption

The DEK is generated by your application (or by the KMS via GenerateDataKey). It encrypts the actual data using a symmetric cipher (AES-256-GCM). The KMS encrypts the DEK using the CMK before storage. On read, the application calls KMS to decrypt the DEK, then uses it to decrypt the data.

AWS KMS: Concrete Implementation

import boto3, os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import secrets
 
kms = boto3.client("kms", region_name="us-east-1")
CMK_ID = os.environ["KMS_CMK_ID"]  # arn:aws:kms:...
 
def encrypt_field(plaintext: str) -> dict:
    """Returns a dict with encrypted_dek and ciphertext to store in DB."""
    # Ask KMS for a data key — returns plaintext and encrypted versions
    response = kms.generate_data_key(
        KeyId=CMK_ID,
        KeySpec="AES_256",
    )
    plaintext_dek: bytes = response["Plaintext"]
    encrypted_dek: bytes = response["CiphertextBlob"]  # store this
 
    # Encrypt the data locally with the plaintext DEK
    aesgcm = AESGCM(plaintext_dek)
    nonce = secrets.token_bytes(12)  # 96-bit nonce for GCM
    ciphertext = aesgcm.encrypt(nonce, plaintext.encode(), None)
 
    # Zero out the plaintext key from memory (best effort in Python)
    plaintext_dek = b"\x00" * len(plaintext_dek)
 
    return {
        "encrypted_dek": encrypted_dek.hex(),
        "nonce": nonce.hex(),
        "ciphertext": ciphertext.hex(),
    }
 
def decrypt_field(stored: dict) -> str:
    encrypted_dek = bytes.fromhex(stored["encrypted_dek"])
    nonce = bytes.fromhex(stored["nonce"])
    ciphertext = bytes.fromhex(stored["ciphertext"])
 
    # Ask KMS to decrypt the DEK — requires IAM permission
    response = kms.decrypt(CiphertextBlob=encrypted_dek)
    plaintext_dek: bytes = response["Plaintext"]
 
    aesgcm = AESGCM(plaintext_dek)
    return aesgcm.decrypt(nonce, ciphertext, None).decode()

IAM policy principle: the application role needs kms:GenerateDataKey and kms:Decrypt on the CMK. Database administrators have read access to the encrypted columns but not to the CMK — they cannot decrypt without going through the application's IAM role.

Key Rotation

Envelope encryption makes rotation cheap. To rotate the CMK:

1. Create a new CMK version.
2. Re-encrypt the stored DEKs with the new CMK (no need to re-encrypt the data).
3. Old CMK version remains available for decryption until all DEKs are rotated.

AWS KMS supports automatic annual rotation of symmetric keys, which re-encrypts all KMS-managed key material. For envelope encryption you control when to re-encrypt the DEKs.

Per-tenant DEKs (one DEK per tenant) let you revoke a single tenant's access by deleting their DEK without affecting any other tenant.

Secrets in Configuration: What Not to Do

Application secrets (database passwords, API keys, signing keys) should never live in:

• Source code or git history
• Unencrypted environment variable files committed to the repo
• Docker images (even in build args — these appear in layer history)
• Log output

Use a secrets manager: AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, or Doppler.

# Fetching a secret from AWS Secrets Manager at startup
import boto3, json
 
def get_secret(secret_name: str) -> dict:
    client = boto3.client("secretsmanager", region_name="us-east-1")
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])
 
# At app startup — not at every request
DB_CONFIG = get_secret("prod/myapp/database")

Secrets Manager supports automatic rotation with Lambda functions — the secret rotates in the manager and the new value is fetched on the next application startup or refresh cycle.

Secrets in Transit: TLS Configuration

HTTPS is not a checkbox. Common misconfigurations:

Mixed content: HTTPS page loads HTTP resources. The HTTP resource is visible and modifiable by a network attacker.

Certificate validation disabled: verify=False in Python requests, ssl: false in Node — often added during development and forgotten.

Weak cipher suites or TLS 1.0/1.1: Deprecated protocols with known weaknesses.

import httpx, ssl
 
# CORRECT — strict TLS for outbound service calls
ctx = ssl.create_default_context()
# Do not disable hostname checking or certificate verification
 
async with httpx.AsyncClient(verify=True) as client:  # verify=True is default
    resp = await client.get("https://internal-service.example.com/api/data")

For inbound TLS on your own services, configure the web server to reject TLS 1.0/1.1 and weak cipher suites:

# nginx — strong TLS configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:CHACHA20';
ssl_prefer_server_ciphers off;  # Let clients choose from the allowed list
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

HSTS (Strict-Transport-Security) tells browsers to only access your domain over HTTPS for the specified duration, even if the user types http://. The preload directive submits your domain to browser preload lists.

Internal Service Communication

Traffic between your own services — API to database, API to message queue, service to service — must also be encrypted. The assumption that internal network traffic is safe is wrong: cloud environments are multi-tenant, VPCs can be misconfigured, and internal attackers exist.

For service-to-service mTLS (mutual TLS), tools like SPIFFE/SPIRE or a service mesh (Istio, Linkerd) manage certificate issuance and rotation automatically.

# Kubernetes — enforce mTLS with Istio PeerAuthentication
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT  # All service-to-service traffic must use mTLS

Key Takeaways

• Encrypting data with a static application-level key is table stakes; envelope encryption with a KMS separates the key hierarchy so a compromised application server does not expose the master key.
• AES-256-GCM with a unique nonce per encryption operation is the correct symmetric cipher choice; never reuse nonces with the same key.
• Per-tenant DEKs enable surgical key revocation — you can cut off one tenant's data access without affecting others or re-encrypting the entire dataset.
• Application secrets belong in a secrets manager at startup, not in environment files, Docker images, source code, or log output.
• TLS configuration matters beyond enabling HTTPS — disable TLS 1.0/1.1, restrict cipher suites, and set HSTS with long max-age including subdomains.
• Internal service-to-service traffic must be encrypted; the internal network is not a trust boundary in cloud environments.