Delivery and Operations

The earlier chapters described what to build. This one is about how a change gets from a commit to AWS without a person's access keys in the middle, how a new version is introduced without taking everyone with it when it is wrong, and how you find out afterwards who did what and whether anything has drifted. It leans on CI/CD Pipelines for the general ideas (stages, artifacts, approvals), on Observability for alarms and logs, on AWS Intermediate/Advanced 2 for the IAM evaluation model and on AWS Intermediate/Advanced 4, 5 and 8 for CloudFormation, CDK and SAM.

Two Ways to Reach AWS from a Pipeline

The outline's question for this chapter is "CodePipeline, CodeBuild and CodeDeploy, or GitHub Actions with OIDC?". They are not exclusive: a team that keeps its code on GitHub can run builds in GitHub Actions and deploy with a role it assumes over OIDC, or hand the commit to CodePipeline and let AWS run the stages. The picture:

What follows checks what can be checked of each, in this order: the identity link in A (workflow file, then the trust policy), the pipeline in B as a template, how new versions are shifted (CodeDeploy, ECS), and then the day-2 tools.

GitHub Actions and OIDC

What GitHub sends

From the documentation (GitHub, read 2026-10-13): a job that sets permissions: id-token: write can ask GitHub for a signed token. Its claims: iss is https://token.actions.githubusercontent.com; aud "defaults to the repository owner's URL" and a workflow can ask for another; sub depends on the context. The default sub formats:

A : inside a value (a branch called fix:thing) is written %3A. The GitHub page I fetched first did not define the claims or the formats (it only had examples); the reference page it pointed to did, and the table above is from that second page. It did not say what happens with forks or pull_request_target, and it gave no AWS trust policy or thumbprint advice.

What AWS needs

From the AWS documentation: an IAM OIDC provider has a URL (which must be https://, case-sensitive and unique in the account), at least one and at most 100 audiences, and thumbprints (1 to 5). AWS says it verifies the provider's JWKS endpoint using its library of trusted root CAs and falls back to the configured thumbprints only when the certificate is not signed by one of them (or the certificate cannot be retrieved, or TLS 1.3 is required). The aud in the token must match an audience registered on the provider, and an OIDC provider used in a role trust policy must be in the same account as the role. The action aws-actions/configure-aws-credentials (README read 2026-10-13) recommends a StringEquals condition on both token.actions.githubusercontent.com:aud (sts.amazonaws.com) and :sub, warns against ForAllValues: in an Allow statement, gives a default session of 1 hour (900 seconds to 12 hours allowed), and says that role chaining needs both sts:AssumeRole and sts:TagSession on the first role (sts:TagSession can be left out if session tagging is skipped).

A workflow file, checked

Run here: a deploy workflow that tests on every push and pull request and deploys from main using an environment and an assumed role. It has no stored AWS keys:

I then made 26 one-change mistakes and four harmless changes, and ran four checkers: PyYAML (does it parse?), the workflow JSON schema through check-jsonschema, actionlint, and wfcheck.py, a checker for the OIDC-specific rules. I wrote the rules in wfcheck.py from the pages above before this run (a role without id-token: write, write-all, stored keys, an action not pinned to a tag or SHA, pull_request_target with the pull request's code, a role ARN that is not a role ARN, untrusted event text in a run: step). One rule, R2, was meant to flag an id-token: write granted at workflow level, but I implemented it so that it never fired; the first run missed that change, and I fixed the rule afterwards. The result below is after the fix.

actionlint caught 14 of 26, the schema 7 and PyYAML none (they parse). actionlint's list includes things I did not expect from a linter: a typo in an action's input name (role-to-asume), a script-injection warning for a pull request title used in a run: step, and an action with no version. wfcheck.py caught 14, and 9 of them were missed by all three off-the-shelf checks: a missing id-token: write (which fails only at run time), write-all, a stored key beside the role, an action pinned to @main, a role ARN with an 11-digit account, a missing role-to-assume, and pull_request_target checking out the pull request. Three stay unseen by everything: a Region that does not exist (eu-west-9; I did not read the Region list), a role in the wrong account, and an environment name that no longer matches the trust policy, which only a check across the workflow and the policy can see (below). None of the four harmless changes raised an alarm.

The trust policy, modelled

The workflow is only half of the contract: the role's trust policy decides which tokens may use it. Run here: a model of sts:AssumeRoleWithWebIdentity for GitHub's tokens, built from the three pages above. It generates the documented sub formats and evaluates StringEquals and StringLike on :sub and :aud; any other operator makes it stop with NotModelled.

Then a matrix of eight trust policies against ten tokens (the intended repository on a branch, a feature branch, a pull request, two environments, a tag, a different repository of the same owner, a different owner with the same repository name, a lookalike name, and a correct token with the wrong audience). My expected answers came from the documented rules, so the matrix tests the model against my reading, not against AWS.

Read the rows. Policy A, with an audience condition and no sub condition, lets in all nine tokens that carry the right audience, including a repository owned by someone else. The audience is a fixed string that every GitHub workflow can request, so it does not identify you; the sub condition does. Policy E (repo:acme/*) lets in every repository of the organisation, including a lookalike such as acme/shop-evil (an organisation may well own that name), and F (repo:acme/shop*) admits the lookalike as well. Policy D (repo:acme/shop:*, with the colon) is the narrowest wildcard that is still one repository. The exact-match policies B and C admit one token each, and G lets a list of two values in. The last lines of the output show two further effects: a policy written for the old subject format stops matching a repository that uses the new, ID-based format (it fails closed, so your deployment breaks rather than your security), and the %3A encoding passes a wildcard that was written without thinking about it.

The workflow and the policy meet at the environment name. I derived the sub that the deploy job above would send, and evaluated it against the trust policy that the pipeline template (below) contains:

Renaming the environment to prod in the workflow makes AWS refuse the token while every file is individually valid, and removing the environment: line has the same effect. The second row is the one to think about: because a job that names an environment sends the environment sub whatever triggered it, a pull_request run of that job holds a token the policy accepts. In this workflow what stops a pull request from deploying is the job's if: github.ref == 'refs/heads/main', not AWS. GitHub's environment protection rules are a further control that I did not read.

CodePipeline and CodeBuild as a Template

From the documentation (pricing pages, 2026-10-13): CodePipeline V1 is $1.00 per active pipeline per month (first 30 days free, one free active pipeline); V2 is $0.002 per action execution minute rounded up, with 100 free minutes shared by the account, and manual approvals are not charged; an active pipeline is one that is older than 30 days and had a code change that month. CodeBuild's page gave $0.005 per minute for the small Linux on-demand type only in a worked example, with 100 free minutes a month. Neither page named a Region. Prices for GitHub Actions were not read.

Run here: one template with the GitHub provider, a deploy role whose trust policy matches the workflow above, a site bucket, an artifact bucket, a build role, a CodeBuild project, a pipeline role and a three-stage V2 pipeline (source from a GitHub connection, build, manual approval). The trust policy and the pipeline's wiring:

It linted clean with cfn-lint 1.57.2. Then 26 one-change mistakes and three harmless changes, with cfn-lint on one side and pipecheck.py on the other. pipecheck.py evaluates the template's own trust policy with the model above against eight tokens that should be refused and one that should be accepted, and checks permissions and wiring. As with the workflow checker, its rules were written before this run, from the pages above and the design's intent (only the production environment of this repository may assume the role).

cfn-lint reported an error for 8 of 26; pipecheck.py for 17; both for 3; neither for 4. cfn-lint found out-of-range values (a build timeout of 0, a session duration of 100, a pipeline type that does not exist), a dangling reference, an artifact name that no action produces or one produced twice, and a pattern that the parameter default breaks. It passed every other trust-policy change, a role with s3:* on every resource, a privileged build container, a bucket without a public-access block and a pipeline role without the right to start the build. pipecheck.py crashed on one change (the repository default that breaks the parameter pattern: my helper cannot split it) and I do not count that as a catch. The four that got through are: removing the audience condition (in my model the provider rejects tokens whose audience is not registered, so the condition adds nothing, but the README recommends both and I cannot say whether AWS treats them as equivalent), a typo in the build compute type, a lifecycle expiry of 0 days, and a missing branch name on the source action. The same caution as in chapter 10 applies: the checker is evidence that these rules can be written, not that it would catch a mistake it was not written for.

Deploying a New Version Safely

The predefined configurations

From the documentation (CodeDeploy, 2026-10-13): for Lambda there are four canary configurations (10 percent first, the rest after 5, 10, 15 or 30 minutes), four linear ones (10 percent every 1, 2, 3 or 10 minutes) and all-at-once. For ECS there are two linear ones (10 percent every 1 or 3 minutes), two canary ones (10 percent, rest after 5 or 15 minutes) and all-at-once; with a Network Load Balancer only all-at-once is supported. For EC2 and on-premises instances the predefined choices are all-at-once, half-at-a-time and one-at-a-time (the default), which count healthy hosts rather than traffic. You can define your own canary or linear configuration for Lambda and ECS, except that with CloudFormation-managed ECS blue/green you cannot.

What SAM generates

Run here: a SAM template with one function that has AutoPublishAlias: live and a DeploymentPreference (canary, an alarm and a pre-traffic hook), the hook function, and the alarm. SAM's page says this creates a CodeDeploy application and deployment group by itself, and that the first gradual deployment should be done in two steps (publish the alias first, add the preference second) because CodeDeploy needs a previous version to shift from. The translator, run offline, turned the three declared resources into 10:

The alias carries an UpdatePolicy that points at the deployment group; the group names the configuration (CodeDeployDefault.Lambda followed by Canary10Percent5Minutes), turns on rollback for failure, stop-on-alarm and stop-on-request, and is a blue/green deployment with traffic control. I changed one thing at a time (16 mistakes and two valid changes) and ran the SAM translator and cfn-lint:

The translator caught 1 of 16 (a deployment preference without AutoPublishAlias), cfn-lint 5, both 0, and 10 got past both. The two tools did catch alarm-property and reference problems, but not Canary10Percent7Minutes, which is not in the documented list of four canary options (I did not see what CodeDeploy does with a name that does not exist, so that miss is judged against the docs' list). They also did not notice a missing alarm resource, a hook function without the CodeDeployHook_ prefix that the SAM-created role policy expects (my understanding, not read this session), a hook role without the permission to report back, an alarm that watches the function instead of the alias, or an alarm with TreatMissingData: breaching. In my test, cfn-lint 1.57.2 also passed Runtime: python3.99 and a 901-second timeout in this template; chapter 8 saw the same weak result with 1.57.2 and a stronger one with 1.53.3, and I did not investigate why. I reinstalled the translator at the version pinned for chapter 8 to rule out a version mismatch before accepting that result.

Arithmetic: how much goes wrong before the stop?

The pages give the configurations; they do not say how bad a bad release is under each. Run here: a small model. Its assumptions, which are mine and not on the pages: a linear configuration sends its first 10 percent at minute 0 and finishes at nine times its interval; requests are spread evenly; the bug fails every request on the new version; and the alarm stops the deployment a fixed number of minutes after traffic first reaches the new version. The first block checks the model against the wording of the pages:

Then the failed requests at 100 requests a second, for an alarm that needs 2 minutes and then for one that needs 10:

The rows teach two things. With a quick alarm every gradual configuration holds the damage to 1,200 to 1,800 requests against 12,000 for all-at-once, whatever its length. With a slow alarm a short canary does not help: the 5-minute canary ends before the 10-minute alarm has decided, the remaining 90 percent arrives, and 33,000 requests fail, as many as the 1-minute linear. A 30-minute canary or a 10-minute linear step keeps it to 6,000. The window of the check has to be longer than the alarm's time to detect. And a canary only protects against what its alarm can see:

A bug that fails one request in a thousand, at 1 request a second, is invisible in a 5-minute canary 97 percent of the time (Poisson arithmetic, for a simple count of failures; the real alarm's threshold and evaluation periods decide what counts as seen). At 100 requests a second the same canary would almost always see it. Low-traffic services need a synthetic test (the pre-traffic hook) more than they need a long canary. A last check that the weighted split is a split: 13 percent of 100 requests went to the new version, 10.4 percent of 1,000 and 10.0 percent of 100,000 (random draws with a fixed seed, which says nothing about how Lambda chooses).

ECS rolling updates

From the documentation (ECS rolling update page): minimumHealthyPercent is "rounded up" and maximumPercent "rounded down"; the page's four examples (50 percent of 4 tasks lets the scheduler stop 2; 75 percent of 2 tasks lets it stop none; 200 percent of 4 lets it start 4; 125 percent of 3 lets it start none) are in the checks above. It also says to make sure the scheduler can stop or start at least one task, or the deployment is stuck. The circuit breaker and CloudWatch-alarm failure detection are named on the page and link to pages I did not read; the page says both can roll back and that when both are set, "the deployment is set to failed as soon as the failure criteria for either" is met. Blue/green and the newer linear and canary options for ECS were not read beyond the CodeDeploy table. My table of waves uses a simple model of the scheduler (start up to the maximum, stop down to the minimum), not the real one:

A setting of 100 and 100 percent is stuck for any task count, as is 100 and 125 percent with 3 tasks; with one task, 100 and 200 percent gives a start-first rollout (a second task is started before the first is stopped), while 0 and 100 stops the only task first, which means an outage for as long as the new task takes to become healthy. This is the arithmetic of the page's rule, not an observation of ECS.

Day-2 Operations

Systems Manager

From the documentation (Session Manager prerequisites): managed nodes need SSM Agent 2.3.68.0 or later; KMS encryption of session data needs 2.3.539.0; shell profiles 3.0.161.0; port forwarding and SSH sessions 3.0.222.0; streaming session data to CloudWatch Logs 3.0.284.0. The node must reach ec2messages, ssm and ssmmessages endpoints for its Region over HTTPS on port 443 (outbound), or through interface endpoints (AWS Fundamentals 5 covers VPC endpoints). Using the AWS CLI to start a session needs the Session Manager plugin and CLI 1.16.12 or later. The managed policy AmazonSSMManagedInstanceCore provides the permissions for a node. The page text I read did not discuss inbound ports. Session Manager and Patch Manager need real instances with the agent, so I could not run them here, and I did not read the Patch Manager pages.

Run here: Parameter Store through moto, against the limits read for chapter 3 (standard 4 KB, advanced 8 KB), then CloudTrail records (below):

moto agreed with the page on 3 of 6. It accepted a 4,097-byte standard parameter, an 8,193-byte advanced parameter and a 4,097-byte value with Tier given explicitly as Standard, and it rejected a type that does not exist. Two things I observed about its SecureString: reading it without decryption returned kms:alias/aws/ssm:hunter2, so the "encrypted" value is the plain text with a prefix, and a second put without Overwrite was refused with ParameterAlreadyExists (that last behaviour I believe matches the service, but I did not read a page for it). The emulator is fine for testing that your code calls the API; it cannot tell you that a value is too large.

CloudTrail

From the documentation: event history is "a viewable, searchable, downloadable, and immutable record of the past 90 days of CloudTrail management events in an AWS Region"; a trail delivers events to an S3 bucket, optionally to CloudWatch Logs and EventBridge; by default trails log management events but not data events or Insights events; and "CloudTrail log files aren't an ordered stack trace of the public API calls, so events don't appear in any specific order". The record-contents page lists the fields. The ones marked as always present (Optional: False) are eventTime, eventVersion, userIdentity, eventSource, eventName, awsRegion, sourceIPAddress, requestParameters, responseElements, eventID, eventType and eventCategory. The current eventVersion is 1.11; the page tells you to compare the major version for equality and the minor version with greater-or-equal. responseElements is null for read-only APIs, and some services put error details in responseElements rather than in errorCode and errorMessage. The page did not say how long delivery to a trail takes, and I did not read the pricing page.

Run here: a checker for those rules, run on ten synthetic records that I wrote (not real CloudTrail output), including the denied AssumeRoleWithWebIdentity event that a wrong trust policy would produce:

Use this as the shape of a real audit query, not as a test of CloudTrail: the question "who assumed the deploy role, from which repository?" is answered from the requestParameters and the token claims that appear in the event, and I did not read the page that lists the claims CloudTrail records for a web identity, so I wrote none.

Config, GuardDuty and Security Hub, in a paragraph each

AWS Config (from its overview page) "provides a detailed view of the configuration of AWS resources in your AWS account", including how they are related and how they were configured in the past; it needs an S3 bucket and an IAM role (and an SNS topic if you want notifications) and lets you choose which resource types to record. Config rules evaluate the recorded configuration and flag a resource as noncompliant; conformance packs deploy collections of rules; aggregators collect data from several accounts and Regions. Where CloudTrail answers "who called what", Config answers "what did this resource look like then, and does it comply now" (my summary of the two pages). Amazon GuardDuty "continuously monitors, analyzes, and processes AWS data sources and logs": foundational sources are CloudTrail management events, VPC flow logs and DNS logs; optional protection plans cover S3, EKS, Runtime Monitoring, RDS, Lambda, malware and others. A first-time enable includes a 30-day free trial and turns on all protection plans except Runtime Monitoring. AWS Security Hub CSPM aggregates findings (in the AWS Security Finding Format) from GuardDuty, Inspector, Macie and partners and runs checks against standards (AWS Foundational Security Best Practices, CIS, PCI DSS, NIST); most of its controls use AWS Config rules, so Config must be enabled and recording; it only sees findings generated after it is enabled, and only in the Region where it is enabled. I did not read pricing for any of the three, and none of them can be run without an account.

What a Real Run Involves (not run)

Level 2: needs an account and a repository. In outline: create the IAM OIDC provider (the AWS CLI command on the IAM page is aws iam create-open-id-connect-provider), create the role with a trust policy like the one in the template, set the repository's workflow to assume it, and look at the CloudTrail event for the assumption. I give no further commands because I did not read the CLI pages for them. Set a budget alert first (AWS Fundamentals 9), keep every test small, and delete the stack afterwards; the provider and role are free to hold, the pipeline and build minutes are not.

The one cost sketch I can make from the pages is for 10 builds a day on 22 working days at 3 minutes each:

Cost and Clean-Up

Nothing in this chapter created anything in AWS. On this machine: the scratch folder C:/lwaws/c11 (a Python environment, files and saved outputs). The moto emulator ran in-process, so no server was left running. Docker was not used.

Practice