Encryption and Secrets

Almost everything you store in AWS is encrypted, and most of the time you never see it happen. This chapter is about the times you do see it: when a policy has to say who may use a key, when a key is rotated or scheduled for deletion, when an application needs a database password and you must decide where it lives, and when an auditor asks who used what. It covers AWS KMS (the key service), the defaults for S3, EBS and RDS, envelope encryption (the pattern that makes encrypting big data with a small key service possible), Secrets Manager, Parameter Store, certificates, and the CloudTrail records that prove what happened.

Encryption at Rest: What AWS Does Before You Ask

The defaults differ by service, and the differences matter when you are asked "is it encrypted?":

One RDS detail shows why key management is an availability matter, not only a security one. If the KMS key becomes unusable (for example it is disabled), the page says an encrypted instance with backups enabled "goes into the inaccessible-encryption-credentials-recoverable state 2 hours after detection", stays stopped for seven days while you can re-enable the key and restart it, and after that is no longer usable and can only be restored from a snapshot or backup.

KMS Keys

A KMS key is a logical key in the AWS Key Management Service; its secret material is kept inside AWS's hardware security modules and is "designed never to be exported from the HSM in plaintext". You ask the service to use it; you do not get it. There are three kinds, and the documentation's comparison is the one to remember:

The documentation's advice is "use customer managed keys when control is important, but use AWS owned keys when convenience is most important". It also says AWS managed keys are "a legacy key type that is no longer being created for new AWS services as of 2021". Prices from the KMS pricing page: $1 per month per key, $0.03 per 10,000 requests, a free tier of 20,000 requests per month across all Regions (asymmetric operations are excluded from it), and $1 per month extra for the first and second rotation of key material (later rotations are not billed). The page named no Region for the request price. You refer to a key by key ID, key ARN, alias name (alias/Name) or alias ARN; to use a key in another account you must use the key ARN or alias ARN.

Envelope Encryption

KMS will encrypt at most 4,096 bytes directly ("Encrypts plaintext of up to 4,096 bytes"). That is enough for a password and not enough for a file. The standard answer is envelope encryption: ask KMS for a data key (GenerateDataKey returns a plaintext copy and a copy encrypted under your KMS key), encrypt the big data locally with the plaintext key, throw the plaintext key away, and store the encrypted data key next to the data. To read it back, ask KMS to decrypt the stored data key, then decrypt locally. The key service sees only a 32-byte key per object; the bulk bytes never leave your process.

I built this by hand with a local stand-in for KMS (LocalKms: it keeps keys in memory and copies the documented behaviours listed at the top of the file, such as the 4,096-byte limit, the key and key version recorded inside the ciphertext blob, the encryption context as authenticated data, and rotation that adds key material without touching data). It is not KMS. What is real is the AES-256-GCM encryption, the sizes and the failures. The core:

Run here (level 1).

What the run shows. (1) 4,096 bytes go through directly and 4,097 do not. (2) 50 MB is encrypted with 16 bytes of overhead (the GCM tag), a 12-byte nonce and a 72-byte wrapped key in this stand-in's format; the timing is one run on this machine and says nothing about KMS, whose calls cross a network. (3) The encryption context is bound to the data: the wrong tenant, or no context, fails; one flipped bit fails; swapping in another object's wrapped key fails. (4) Rotation (as documented) adds key material and keeps the old, so old data still decrypts; "Key rotation has no effect on the data that the KMS key protects. It does not rotate the data keys ... or re-encrypt any data". Re-encrypting just the wrapped key (the ReEncrypt idea) moves an object to the current key version while the bulk ciphertext stays byte-for-byte the same (same hash). And if old key material were destroyed, anything still wrapped under it would be lost. (5) A key that is not enabled cannot unwrap or generate data keys, yet an application that already holds a plaintext data key in memory keeps working until it restarts. That matches the documentation's warning that scheduling deletion "might not immediately affect data keys encrypted by the KMS key", and it is why a key change can look harmless for hours and then break a restart.

The same pattern in a real library

The AWS Encryption SDK implements envelope encryption for you. It can use a KMS key, or a raw AES key you hold, and I could only use the second kind (no AWS account). So this run shows what the SDK does, not how a KMS keyring behaves. Run here (level 1): aws-encryption-sdk 4.0.7 with a raw AES-256 key.

Three findings. The encryption context is not secret: the words invoices and t-17 appear in clear text in the encrypted message (the KMS pages say the same: "Do not include confidential or sensitive information in this field"). The data itself does not appear. The SDK returns the context on decrypt, but you must check it: the "expect tenant t-99" line fails only because my code compared it. And damage anywhere fails: a flipped bit in the header or near the end of the message both fail (the library signs the message with its default algorithm suite; the algorithm name is in the output). A message can carry two encrypted data keys, so either of two keys (for example in two Regions) can decrypt it.

Key Policies, IAM Policies and Grants

KMS has an access model that surprises people who know IAM, and AWS Intermediate/Advanced 2 prepared you for it: a KMS key policy is a resource policy that must name its principals. The documentation:

• "Key policies are the primary way to control access to KMS keys. Every KMS key must have exactly one key policy."
• "No AWS principal, including the account root user or key creator, has any permissions to a KMS key unless they are explicitly allowed, and never denied, in a key policy, IAM policy, or grant."
• "Unless the key policy explicitly allows it, you cannot use IAM policies to allow access to a KMS key. ... (You can use an IAM policy to deny a permission to a KMS key without permission from a key policy.)"
• "Unlike IAM policies, which are global, key policies are Regional."

The default key policy for a key created through the API is one statement. The principal is the account root, and that does not give the root user the key: "When the principal in a key policy statement is the account principal, the policy statement doesn't give any IAM principal permission to use the KMS key. Instead, it allows the account to use IAM policies to delegate the permissions". Delete it and "IAM policies that allow access to the key are ineffective, although IAM policies that deny access to the key are still effective" (and you risk an unmanageable key).

The console adds more: a key administrators statement (manage the key, not use it), a key users statement (use it for cryptographic operations), and one that lets users create grants for AWS services, only when kms:GrantIsForAWSResource is true:

A warning on the page that is easy to miss: "Because key administrators have permission to change the key policy and create grants, they can give themselves and others AWS KMS permissions not specified in this policy." Separating administrators from users stops accidents, not a determined administrator.

Grants

A grant is a second, temporary route to a key. It gives a grantee (an IAM identity, or an AWS service principal) specific operations on exactly one key; it can allow but never deny; it needs no change to the key policy or any IAM policy; and "an authorized principal can delete the grant (retire or revoke it)", which removes everything it allowed. AWS services use them: they create a grant on your behalf, use it, and retire it. Details from the page: the KMS API is eventually consistent, so a new grant may not work at once ("typically less than a few seconds ... in some cases it can take several minutes"); a grant token lets you use it immediately; and there is a limit of 50,000 grants per key. Treat kms:CreateGrant like kms:PutKeyPolicy: "Permission to create grants has security implications".

The rules as tests

I ran these rules through the AWS Intermediate/Advanced 2 evaluator, using its "this resource policy must explicitly allow the principal" mode (explicit_resource_allow), which was built for exactly this case, and added grants and key state around it. As before, each expected answer comes from a sentence on the pages and was written first. (Level 1: a model of the documented rules, not AWS.)

All 31 match; the control with one flipped expectation reports 31 cases, mismatches: 1. Read it as a checklist. The root statement delegates (no IAM policy, no access). A key policy that names only one user makes everyone else's IAM allows useless. An administrator cannot decrypt, but can put a key policy, create a grant and schedule deletion, and the "warning" row shows the two-step path from administrator to user. A cross-account request needs both the key policy naming the other account and an IAM policy in that account. A grant works with no policy statement but cannot beat a Deny. A key pending deletion refuses cryptographic operations whatever allows them. Grants with constraints are refused by the model (not implemented).

The cross-account pair, as policy files. In the key's account, the key policy names the other account; in the other account, a role needs its own policy naming the key ARN:

One more control you can put in an identity policy: bind use of a key to an encryption context with the condition key form kms:EncryptionContext:<key> (the last three rows of the table above: the matching context is allowed, a different one is denied, and a request with no context at all is denied because a missing key makes the condition false):

Rotation and Deletion of Keys

Rotation (the Rotate keys page):

• Automatic rotation is optional for customer managed keys, on symmetric encryption keys whose key material AWS KMS generated. When you enable it, "by default ... AWS KMS generates new cryptographic material for the KMS key every year"; the default rotation period is 365 days, and you can set a custom period. On-demand rotation rotates immediately, even with automatic rotation off.
• Asymmetric keys, HMAC keys and keys in custom key stores cannot be rotated automatically or on demand; you create a new key and switch your applications or alias to it (manual rotation).
• Rotation changes only the current key material. "AWS KMS retains all key material for a KMS key with AWS_KMS origin, even if key rotation is disabled", the key ID and ARN do not change, and decryption "automatically chooses the correct version". It does not re-encrypt data or rotate data keys, and "will not mitigate the effect of a compromised data key".
• It is recorded: a RotateKey event in CloudTrail and a KMS CMK Rotation event in EventBridge. Cost: see above ($1 per month extra for the first two rotations).

The page is candid about why: KMS keys are mostly wrapping keys that are used rarely, so "they ... are almost never reused enough to risk key exhaustion", and you rotate mainly "due to business or contract rules or government regulations". The envelope-encryption run above is the reason it is cheap: rotation never touches the bulk data.

Deletion (the Deleting keys page) is the one operation you cannot undo:

• "After a KMS key is deleted, you can no longer decrypt the data that was encrypted under that KMS key, which means that data becomes unrecoverable." For a symmetric key this holds even if you have the same key material, because "metadata unique to each KMS key is cryptographically bound to each symmetric ciphertext".
• You schedule deletion with a waiting period of 7 to 30 days (default 30); the real wait "might be up to 24 hours longer". During it the key is Pending deletion: it cannot be used for cryptographic operations and is not rotated, and you can cancel. The page suggests disabling the key if you are not sure, and a CloudWatch alarm that warns if anything tries to use a key during the waiting period.
• You can delete only customer managed keys. Closing the AWS account makes keys inaccessible and stops the billing.

Secrets Manager and Parameter Store

The Secrets Manager introduction contains a useful decision table of its own: "AWS credentials: we recommend IAM. Encryption keys: we recommend KMS. SSH keys: we recommend Amazon EC2 Instance Connect. Private keys and certificates: we recommend AWS Certificate Manager." What is left for Secrets Manager is "database credentials, application credentials, OAuth tokens, API keys, and other secrets", and its pitch is rotation: "you replace hard-coded credentials with a runtime call to the Secrets Manager service". Systems Manager Parameter Store holds configuration and small secrets more cheaply. The facts, from the documentation:

Run here (level 1): will real key material fit? I generated RSA, EC and certificate PEM text and measured it. The costs below are arithmetic on the prices above (not a quote; no Region named; the free plan is not applied).

The fit table is a measurement, and it contains a trap: a 4,096-bit RSA key fits Parameter Store standard on its own (3,272 bytes), but the same key with a three-certificate chain does not (6,335 bytes). The cost table is the lesson about where you call: reading 25 secrets on every request at 10 requests per second is about $3,250 a month in API calls, while a five-minute cache in 20 instances is about $32. Cache secrets in memory and re-read on a schedule or on authentication failure. (How rotation interacts with a cache is the next section.)

The rotation protocol

A rotation Lambda function does four jobs, and Secrets Manager tracks them with staging labels: AWSCURRENT is the version applications read; AWSPENDING is the candidate; AWSPREVIOUS is the last good one. From the page:

1. createSecret checks whether a version with this request token exists; if not it makes a new value and stores it with the label AWSPENDING. "Storing the new secret value in AWSPENDING helps ensure idempotency."
2. setSecret changes the credential in the database or service to match AWSPENDING. The function is "a privileged deputy", so before it acts it must check that AWSCURRENT is valid, that both versions are for the same user, and that the target host is the same, otherwise someone who can edit a secret could point the function at another resource (the confused deputy problem from AWS Intermediate/Advanced 2).
3. testSecret uses AWSPENDING against the database (the templates use read access).
4. finishSecret moves AWSCURRENT to the new version; Secrets Manager adds AWSPREVIOUS to the old one. "Don't remove AWSPENDING before this point". If AWSPENDING is left on a different version than AWSCURRENT, "any later invocation of rotation assumes that a previous rotation request is still in progress and returns an error".

Other documented rules: only the rotation function can rotate the secret (it cannot call another Lambda function to do it); a provisioned concurrency below 10 "can cause throttling"; and be careful what you log, because logs go to CloudWatch. I turned these into a small model:

Run here (level 1): a model of the documented protocol, not AWS. "My reading" marks two things the page does not state outright: that a retry with the same token resumes, and that a put which cannot make room under the 100-version quota fails.

Four things to take from it. First, between setSecret and finishSecret the database already has the new password while AWSCURRENT still holds the old one, so an application that reads AWSCURRENT in that window cannot connect. That window is a consequence of the four steps as written (derived, not a quoted sentence); the documentation lists a second rotation strategy ("alternating users") and a managed rotation, which I did not read, so I do not describe how they avoid it. Second, a failure inside setSecret leaves the application untouched and a retry with the same token reuses the pending value. Third, a failure after the database change (the testSecret case) leaves AWSCURRENT wrong, and then even a retry is abandoned by the "is AWSCURRENT valid" check: a real operational hazard that the model makes visible, and why your application should retry with a fresh read. Fourth, the quota line: updating a secret more often than the page recommends (once every 10 minutes at most) eventually runs out of versions, because unlabelled versions younger than 24 hours are never removed; at one update every five minutes the model fails after 8.3 hours.

Certificates: ACM

AWS Certificate Manager "handles the complexity of creating, storing, and renewing public and private SSL/TLS X.509 certificates and keys". Facts from its overview page (level 3, not run): certificates can be issued by ACM or imported; wildcard certificates are supported; there is "no additional charge for SSL/TLS certificates that you manage with AWS Certificate Manager" (you pay for the resources that use them); certificates are regional and "you cannot copy a certificate between regions"; and to use one with CloudFront "you must request or import the certificate in the US East (N. Virginia) region". ACM certificates signed by AWS Private CA can be exported; for publicly trusted certificates on your own servers (EC2, say) the page points to the ACME protocol. I did not read the validity period or renewal rules, so I do not state them. HTTPS/TLS Fundamentals covers what the certificate itself is.

Secrets in Environment Variables

"Put it in an environment variable" is the common first step away from hard-coding. It is better than source code and worse than most people think. These are real experiments with child processes on this Windows machine (the password is made up):

What they show. Inheritance: every child and grandchild sees the variable unless you remove it deliberately. Dumps: a diagnostic that prints the environment leaks it, and filtering by variable name fails when the secret hides inside DATABASE_URL. Command lines: an argument is readable by another process of the same user while it runs (here through Get-CimInstance Win32_Process), so never pass a secret as an argument. Alternatives: passing a file path, or writing the secret to the child's standard input at start-up, keeps it out of the command line and the environment, but the file needs tight permissions and anything the program prints can still leak it. I did not run docker inspect (the engine was not running); I am not claiming what it shows. The documented pattern is to fetch the secret at run time, cache it in memory, and refresh it (see the cost table above).

Evidence: CloudTrail and KMS

CloudTrail "records all calls to AWS KMS by users, roles, and other AWS services", including read-only calls, key management and cryptographic operations such as GenerateDataKey and Decrypt, and internal operations such as RotateKey and DeleteKey. Details: failed calls are logged "in some scenarios"; cross-account operations are logged in both accounts, but cross-account requests rejected for access are logged "only in the caller's account"; some fields are omitted ("the Plaintext parameter of an Encrypt request, and the response to ... any cryptographic operation"); and the encryption context may appear in plain text. One warning to remember: a trail can exclude all KMS events (ExcludeManagementEventSources set to kms.amazonaws.com), and "excluding AWS KMS events from a CloudTrail Log can obscure actions that use your KMS keys", so guard cloudtrail:PutEventSelectors. Look for a PutEventSelectors event with that setting if KMS events seem to be missing.

Run here (level 1): I took the three example log entries the KMS documentation prints for Decrypt and GenerateDataKey, and added 18 invented events in the same layout (marked synthetic: I had no real CloudTrail data) so there is something to find. Note the first result: the documentation's own examples are not all valid JSON.

Two of the three printed examples are missing a comma, so any tool that loads them as given fails (see Working with JSON 1; I inserted one comma in each to analyse them). The check that compared the event's Region with the key ARN's Region flags the GenerateDataKey example as inconsistent (us-east-1 against a us-west-2 key): it is an example, but it shows the sort of mismatch worth alerting on in real data. The questions that matter are answerable from the fields: who used which key (userIdentity and resources), which calls failed and why (errorCode), what context they used, and, from the last section, whether anything touched a key that was pending deletion.

Offline Checks: the SDK and parliament

Two more things you can check before any AWS call. The first is how much of the KMS request limits the SDK enforces; the second is what a policy linter says about the files in this chapter. Run here (level 1):

botocore accepted a 0-byte and a 4,097-byte Plaintext and 11 grant tokens: here it checked types and required fields only, not the lengths the documentation states (AWS would reject those, according to the pages; I could not send a request to see). Consistent with AWS Intermediate/Advanced 2: do not treat an SDK's silence as validation. And parliament, which reads everything as an identity policy, flagged the Sid of the documented default key policy ("Enable IAM User Permissions" contains spaces) as INVALID_SID. That is its identity-policy rule applied to a key policy: key policies are a different policy type, and the linter does not know. The two identity policies (the cross-account role policy and the context-bound policy) had no findings.

Cost and Clean-Up

Nothing in this chapter creates anything in AWS, so there is nothing to delete and nothing to pay. When you do build it for real, the recurring costs to know are: $1 per month per customer managed key (plus $1 for each of the first two rotations), a per-request price, and for Secrets Manager $0.40 per secret per month plus calls (prices as read 2026-10-12, no Region stated, free plans not applied). A key you no longer need costs money until it is deleted, and deletion takes 7 to 30 days and cannot be undone; disable first. Put a budget alert in place before you start (AWS Fundamentals 9), and use the sandbox account from AWS Intermediate/Advanced 1.

Practice