aws2-4 Exercise 1: A Real Template, Linted, Broken Twenty-Two Ways, and the Documentation's Own Examples ======================================================================================================== Run with Python 3.14.0 on Windows 11 in a scratch venv: cfn-lint 1.57.2, cfn-flip 1.3.0, PyYAML 6.0.3. Setup: py -m venv .venv ; .venv/Scripts/python.exe -m pip install cfn-lint cfn-flip pyyaml. VERIFICATION LEVEL: 1 (run here; no AWS account, no AWS CLI, no stack created). cfn-resolve, the change set approximation, the failure simulator, the capability and quota checkers are MODELS of CloudFormation documentation pages read on 2026-10-12 and 2026-10-13 (through a page-fetch tool that summarises pages): they are not AWS, and AWS may change its rules and limits. --- app_stack.yaml --- AWSTemplateFormatVersion: "2010-09-09" Description: >- Report bucket encrypted with its own KMS key, a work queue with a dead-letter queue, a worker role, and a production-only alarm. Built for the AWS Intermediate/Advanced course, chapter 4. Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: { default: Deployment } Parameters: [Environment, RetentionDays] - Label: { default: Security } Parameters: [KeyAdminRoleArn] Parameters: Environment: Type: String Default: dev AllowedValues: [dev, staging, prod] Description: Which environment this stack is for RetentionDays: Type: Number Default: 30 MinValue: 1 MaxValue: 3650 Description: Days before report objects expire KeyAdminRoleArn: Type: String AllowedPattern: "^arn:aws:iam::[0-9]{12}:role/[A-Za-z0-9+=,.@/_-]+$" ConstraintDescription: Must be the ARN of an IAM role Description: Role that may administer (not use) the key Mappings: EnvSettings: dev: { VisibilityTimeout: 30, MaxReceive: 3 } staging: { VisibilityTimeout: 60, MaxReceive: 5 } prod: { VisibilityTimeout: 120, MaxReceive: 5 } Conditions: IsProd: !Equals [!Ref Environment, prod] Resources: DataKey: Type: AWS::KMS::Key DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: Description: !Sub "Data key for ${AWS::StackName} (${Environment})" EnableKeyRotation: true KeyPolicy: Version: "2012-10-17" Statement: - Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root" Action: "kms:*" Resource: "*" - Sid: Allow access for Key Administrators Effect: Allow Principal: AWS: !Ref KeyAdminRoleArn Action: - "kms:Create*" - "kms:Describe*" - "kms:Enable*" - "kms:List*" - "kms:Put*" - "kms:Update*" - "kms:Revoke*" - "kms:Disable*" - "kms:Get*" - "kms:Delete*" - "kms:TagResource" - "kms:UntagResource" - "kms:ScheduleKeyDeletion" - "kms:CancelKeyDeletion" Resource: "*" DataKeyAlias: Type: AWS::KMS::Alias Properties: AliasName: !Sub "alias/${AWS::StackName}-data" TargetKeyId: !Ref DataKey ReportBucket: Type: AWS::S3::Bucket DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: BucketEncryption: ServerSideEncryptionConfiguration: - BucketKeyEnabled: true ServerSideEncryptionByDefault: SSEAlgorithm: "aws:kms" KMSMasterKeyID: !GetAtt DataKey.Arn VersioningConfiguration: Status: Enabled PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true LifecycleConfiguration: Rules: - Id: expire-reports Status: Enabled ExpirationInDays: !Ref RetentionDays Tags: - Key: environment Value: !Ref Environment ReportBucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !Ref ReportBucket PolicyDocument: Version: "2012-10-17" Statement: - Sid: DenyInsecureTransport Effect: Deny Principal: "*" Action: "s3:*" Resource: - !GetAtt ReportBucket.Arn - !Sub "${ReportBucket.Arn}/*" Condition: Bool: "aws:SecureTransport": "false" WorkDeadLetterQueue: Type: AWS::SQS::Queue Properties: MessageRetentionPeriod: 1209600 KmsMasterKeyId: !Ref DataKey WorkQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: !FindInMap [EnvSettings, !Ref Environment, VisibilityTimeout] KmsMasterKeyId: !Ref DataKey RedrivePolicy: deadLetterTargetArn: !GetAtt WorkDeadLetterQueue.Arn maxReceiveCount: !FindInMap [EnvSettings, !Ref Environment, MaxReceive] WorkerRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: "sts:AssumeRole" Policies: - PolicyName: work PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "sqs:ReceiveMessage" - "sqs:DeleteMessage" - "sqs:GetQueueAttributes" Resource: !GetAtt WorkQueue.Arn - Effect: Allow Action: - "s3:GetObject" - "s3:PutObject" Resource: !Sub "${ReportBucket.Arn}/*" - Effect: Allow Action: - "kms:Decrypt" - "kms:GenerateDataKey" Resource: !GetAtt DataKey.Arn BacklogAlarm: Type: AWS::CloudWatch::Alarm Condition: IsProd Properties: AlarmDescription: Work queue backlog is growing Namespace: AWS/SQS MetricName: ApproximateNumberOfMessagesVisible Dimensions: - Name: QueueName Value: !GetAtt WorkQueue.QueueName Statistic: Maximum Period: 300 EvaluationPeriods: 1 Threshold: 1000 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching Outputs: BucketName: Description: Name of the report bucket Value: !Ref ReportBucket Export: Name: !Sub "${AWS::StackName}-BucketName" QueueUrl: Description: URL of the work queue Value: !Ref WorkQueue KeyArn: Description: ARN of the data key Value: !GetAtt DataKey.Arn AlarmName: Condition: IsProd Description: Backlog alarm (production only) Value: !Ref BacklogAlarm --- cfn-lint app_stack.yaml --- == app_stack.yaml exit status: 0 --- mutate_lint.py (22 text mutations of the template and six generated limit tests; the generator counts the template's real resources first) --- """mutate_lint.py - break app_stack.yaml in many specific ways and record exactly what cfn-lint (version printed) says about each. Every mutation is a plain text replacement, so a failure to be caught is a failure of the tool, not of the experiment. The last group generates templates that break the documented CloudFormation quotas to see which of them cfn-lint checks. """ import json, pathlib, subprocess, sys, tempfile B = pathlib.Path(r"C:/lwaws/c4") LINT = str(B / ".venv" / "Scripts" / "cfn-lint.exe") BASE = (B / "app_stack.yaml").read_text(encoding="utf-8") from cfn_flip import load as _load HAVE = len(_load(BASE)[0]["Resources"]) # the template's own resource count, not an assumed number def lint(text, suffix=".yaml"): f = pathlib.Path(tempfile.gettempdir()) / ("cfn_mut" + suffix) f.write_text(text, encoding="utf-8") r = subprocess.run([LINT, "--format", "json", str(f)], capture_output=True, text=True, encoding="utf-8") try: found = json.loads(r.stdout) if r.stdout.strip() else [] except json.JSONDecodeError: found = [{"Rule": {"Id": "?"}, "Level": "?", "Message": (r.stdout + r.stderr)[:120]}] return r.returncode, found def rep(old, new, count=1): assert old in BASE, "pattern not in template: " + old[:50] return BASE.replace(old, new, count) MUTANTS = [ ("unknown property name (EnableKeyRotations)", rep("EnableKeyRotation: true", "EnableKeyRotations: true")), ("wrong type (a word where a number belongs)", rep("MessageRetentionPeriod: 1209600", "MessageRetentionPeriod: two-weeks")), ("Ref to a resource that does not exist", rep("TargetKeyId: !Ref DataKey", "TargetKeyId: !Ref DataKeyy")), ("GetAtt of an attribute that does not exist", rep("Resource: !GetAtt WorkQueue.Arn", "Resource: !GetAtt WorkQueue.Arnn")), ("Default not among AllowedValues", rep("Default: dev\n AllowedValues", "Default: test\n AllowedValues")), ("required property removed (Alias TargetKeyId)", rep(" TargetKeyId: !Ref DataKey\n", "")), ("circular dependency (key policy names the role that uses the key)", rep('AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"', "AWS: !GetAtt WorkerRole.Arn")), ("Sub with a variable nothing defines", rep('!Sub "alias/${AWS::StackName}-data"', '!Sub "alias/${StackNam}-data"')), ("Fn::Sub inside Fn::Equals (the docs allow only Ref and FindInMap there)", rep("IsProd: !Equals [!Ref Environment, prod]", 'IsProd: !Equals [!Sub "${Environment}", prod]')), ("Condition name that is not defined", rep("Condition: IsProd\n Properties:\n AlarmDescription", "Condition: IsProduction\n Properties:\n AlarmDescription")), ("invalid DeletionPolicy value (Keep)", rep("DeletionPolicy: Retain", "DeletionPolicy: Keep")), ("DeletionPolicy written with !If", rep("DeletionPolicy: Retain", "DeletionPolicy: !If [IsProd, Retain, Delete]")), ("unused parameter", rep("Conditions:\n", " Unused:\n Type: String\n Default: x\n\nConditions:\n").replace("\n\nConditions:\n Unused", "\n Unused") if False else rep("Mappings:\n", " SpareParameter:\n Type: String\n Default: x\n\nMappings:\n").replace(" Description: Role that may administer (not use) the key\n\n SpareParameter", " Description: Role that may administer (not use) the key\n SpareParameter")), ("unused condition", rep("Resources:\n", " NeverUsed: !Equals [a, b]\n\nResources:\n").replace("IsProd: !Equals [!Ref Environment, prod]\n\n NeverUsed", "IsProd: !Equals [!Ref Environment, prod]\n NeverUsed")), ("output of a conditional resource without the condition", rep(" AlarmName:\n Condition: IsProd\n", " AlarmName:\n")), ("bucket name that breaks the naming rules", rep(" BucketEncryption:", " BucketName: Bad_Name\n BucketEncryption:")), ("IAM action without the service prefix", rep('"sqs:ReceiveMessage"', '"ReceiveMessage"')), ("IAM Effect misspelt (Allowed)", rep(" - Effect: Allow\n Action:\n - \"sqs:ReceiveMessage\"", " - Effect: Allow\n Action:\n - \"sqs:ReceiveMessage\"") if False else rep(" - Effect: Allow\n Action:\n - \"sqs:ReceiveMessage\"", " - Effect: Allowed\n Action:\n - \"sqs:ReceiveMessage\"")), ("FindInMap with a second-level key that does not exist", rep("!FindInMap [EnvSettings, !Ref Environment, MaxReceive]", "!FindInMap [EnvSettings, !Ref Environment, MaxReceives]")), ("Number parameter with a non-number default", rep("Default: 30\n MinValue", "Default: thirty\n MinValue")), ("KMS key policy with Principal missing", rep(" Principal:\n AWS: !Ref KeyAdminRoleArn\n", "")), ("SQS retention above the documented service maximum (1209600 -> 99999999)", rep("MessageRetentionPeriod: 1209600", "MessageRetentionPeriod: 99999999")), ] def pad_resources(n): """base template plus enough SSM parameters to reach n resources in total""" extra = "".join(f" P{i:04d}:\n Type: AWS::SSM::Parameter\n Properties:\n Type: String\n Value: x\n" for i in range(n - HAVE)) return BASE.replace("Outputs:\n", extra + "\nOutputs:\n") def pad_params(n): extra = "".join(f" Q{i:04d}:\n Type: String\n Default: x\n" for i in range(n - 3)) return BASE.replace("\nMappings:\n", extra + "\nMappings:\n") def pad_outputs(n): extra = "".join(f" O{i:04d}:\n Value: !Ref ReportBucket\n" for i in range(n - 4)) return BASE + extra LIMITS = [ ("501 resources (documented maximum is 500)", pad_resources(501)), ("500 resources (at the maximum)", pad_resources(500)), ("201 parameters (documented maximum is 200)", pad_params(201)), ("201 outputs (documented maximum is 200)", pad_outputs(201)), ("description of 1,100 bytes (documented maximum is 1,024)", BASE.replace("Description: >-\n Report bucket", "Description: >-\n " + "x" * 1050 + " Report bucket", 1)), ("template body of about 60,000 bytes (documented direct-request maximum is 51,200)", BASE + "\n# " + "padding " * 7000 + "\n"), ] def show(label, text): code, found = lint(text) ids = sorted({f"{f['Rule']['Id']}" for f in found}) first = found[0]["Message"].replace("\n", " ")[:100] if found else "" print(f"{label:<80} exit {code} {','.join(ids) or '-':<22} {first}") return code, found if __name__ == "__main__": import subprocess as sp print(sp.run([LINT, "--version"], capture_output=True, text=True).stdout.strip(), "\n") print(f"{'mutation of app_stack.yaml':<80} exit rule ids first message") code, _ = show("(the unchanged template)", BASE) missed = [] for label, text in MUTANTS: code, found = show(label, text) if code == 0: missed.append(label) print(f"\nmutations cfn-lint did NOT flag at all (exit status 0): {len(missed)} of {len(MUTANTS)}") for m in missed: print(" -", m) print("\nthe documented quotas") for label, text in LIMITS: show(label, text) --- py -X utf8 mutate_lint.py --- python.exe C:\lwaws\c4\.venv\Scripts\cfn-lint 1.57.2 mutation of app_stack.yaml exit rule ids first message (the unchanged template) exit 0 - unknown property name (EnableKeyRotations) exit 2 E3002 Additional properties are not allowed ('EnableKeyRotations' was unexpected. Did you mean 'EnableKeyR wrong type (a word where a number belongs) exit 2 E3012 'two-weeks' is not of type 'integer' Ref to a resource that does not exist exit 2 E1020 'DataKeyy' is not one of ['Environment', 'RetentionDays', 'KeyAdminRoleArn', 'DataKey', 'DataKeyAlia GetAtt of an attribute that does not exist exit 2 E1010 'Arnn' is not one of ['Arn', 'ContentBasedDeduplication', 'DeduplicationScope', 'DelaySeconds', 'Fif Default not among AllowedValues exit 2 E2015 Default should be a value within AllowedValues required property removed (Alias TargetKeyId) exit 2 E3003 'TargetKeyId' is a required property circular dependency (key policy names the role that uses the key) exit 2 E3004 Circular Dependencies for resource DataKey. Circular dependency with [WorkerRole] Sub with a variable nothing defines exit 2 E1019 'StackNam' is not one of ['Environment', 'RetentionDays', 'KeyAdminRoleArn', 'DataKey', 'DataKeyAlia Fn::Sub inside Fn::Equals (the docs allow only Ref and FindInMap there) exit 0 - Condition name that is not defined exit 2 E3015 'IsProduction' is not one of ['IsProd'] invalid DeletionPolicy value (Keep) exit 2 E3035 'Keep' is not one of ['Delete', 'Retain', 'RetainExceptOnCreate'] DeletionPolicy written with !If exit 0 - unused parameter exit 4 W2001 Parameter SpareParameter not used. unused condition exit 4 W8001,W8003 Condition NeverUsed not used output of a conditional resource without the condition exit 4 W1001 Ref to resource 'BacklogAlarm' that may not be available when condition 'IsProd' is False at Outputs bucket name that breaks the naming rules exit 2 E1161,E3031 'Bad_Name' is not a 'AWS::S3::Bucket.Name' with pattern '^(?![.\\-])(?!.*\\.\\.)(?!.*\\-\\.)(?!.*\\. IAM action without the service prefix exit 4 W3037 'ReceiveMessage' is not a valid action. Must be of the form service:action or '*' IAM Effect misspelt (Allowed) exit 2 E3510 'Allowed' is not one of ['Allow', 'Deny'] FindInMap with a second-level key that does not exist exit 0 - Number parameter with a non-number default exit 0 - KMS key policy with Principal missing exit 6 E3512,W2001 Parameter KeyAdminRoleArn not used. SQS retention above the documented service maximum (1209600 -> 99999999) exit 2 E3034 99999999 is greater than the maximum of 1209600 mutations cfn-lint did NOT flag at all (exit status 0): 4 of 22 - Fn::Sub inside Fn::Equals (the docs allow only Ref and FindInMap there) - DeletionPolicy written with !If - FindInMap with a second-level key that does not exist - Number parameter with a non-number default the documented quotas 501 resources (documented maximum is 500) exit 2 E3010 'Resources' has more than 500 properties 500 resources (at the maximum) exit 0 - 201 parameters (documented maximum is 200) exit 4 W2001 Parameter Q0000 not used. 201 outputs (documented maximum is 200) exit 2 E6010 'Outputs' has more than 200 properties description of 1,100 bytes (documented maximum is 1,024) exit 2 E1003 expected maximum length: 1024, found: 1238 template body of about 60,000 bytes (documented direct-request maximum is 51,200) exit 0 - --- doc/conditions_sample.yaml (copied from the Condition functions page) --- AWSTemplateFormatVersion: "2010-09-09" Parameters: EnvType: Description: Environment type. Default: test Type: String AllowedValues: [prod, dev, test] ConstraintDescription: must specify prod, dev, or test. Conditions: CreateProdResources: !Equals [!Ref EnvType, prod] CreateDevResources: !Equals [!Ref EnvType, "dev"] Resources: EC2Instance: Type: AWS::EC2::Instance Properties: ImageId: ami-1234567890abcdef0 InstanceType: !If [CreateProdResources, c5.xlarge, !If [CreateDevResources, t3.medium, t3.small]] MountPoint: Type: AWS::EC2::VolumeAttachment Condition: CreateProdResources Properties: InstanceId: !Ref EC2Instance VolumeId: !Ref NewVolume Device: /dev/sdh NewVolume: Type: AWS::EC2::Volume Condition: CreateProdResources Properties: Size: 100 AvailabilityZone: !GetAtt EC2Instance.AvailabilityZone --- doc/subnets_example.yaml (copied from the Parameters page) --- Parameters: VPC: Type: String Default: vpc-123456 VpcAzs: Type: CommaDelimitedList Default: us-west-2a, us-west-2b, us-west-2c DbSubnetIpBlocks: Type: CommaDelimitedList Default: 172.16.0.0/26, 172.16.0.64/26, 172.16.0.128/26 Resources: DbSubnet1: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Select - 0 - !Ref VpcAzs VpcId: !Ref VPC CidrBlock: !Select - 0 - !Ref DbSubnetIpBlocks DbSubnet2: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Sub - ${AWS::Region}${AZ} - AZ: !Select - 1 - !Ref VpcAzs VpcId: !Ref VPC CidrBlock: !Select - 1 - !Ref DbSubnetIpBlocks DbSubnet3: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Sub - ${AWS::Region}${AZ} - AZ: !Select - 2 - !Ref VpcAzs VpcId: !Ref VPC CidrBlock: !Select - 2 - !Ref DbSubnetIpBlocks --- doc/deletionpolicy.yaml (copied from the DeletionPolicy page) --- AWSTemplateFormatVersion: '2010-09-09' Resources: MyBucket: Type: AWS::S3::Bucket DeletionPolicy: Retain --- cfn-lint on the documentation's examples and on Chapter 1's SCP template (doc/ch1_scp_stack.yaml, the template from aws2-1 Exercise 1) --- == doc/conditions_sample.yaml exit status: 0 == doc/subnets_example.yaml W1030 {'Ref': 'VPC'} is not a 'AWS::EC2::VPC.Id' with pattern '^vpc-(([0-9A-Fa-f]{8})|([0-9A-Fa-f]{17}))$' when 'Ref' is resolved doc\subnets_example.yaml:18:7 W1030 {'Ref': 'VPC'} is not a 'AWS::EC2::VPC.Id' with pattern '^vpc-(([0-9A-Fa-f]{8})|([0-9A-Fa-f]{17}))$' when 'Ref' is resolved doc\subnets_example.yaml:30:7 W1030 {'Ref': 'VPC'} is not a 'AWS::EC2::VPC.Id' with pattern '^vpc-(([0-9A-Fa-f]{8})|([0-9A-Fa-f]{17}))$' when 'Ref' is resolved doc\subnets_example.yaml:42:7 exit status: 4 == doc/deletionpolicy.yaml W3011 Both 'UpdateReplacePolicy' and 'DeletionPolicy' are needed to protect resource from deletion doc\deletionpolicy.yaml:3:3 exit status: 4 == doc/ch1_scp_stack.yaml exit status: 0 WHY THIS WORKS AS AN ANSWER --------------------------- The template uses what the earlier chapters argued for: a key policy whose root statement delegates (so the worker role is NOT named in it, which avoids a circular dependency), Retain and UpdateReplacePolicy on the data stores, a Condition that creates the alarm only in production, and constrained parameters. It lints clean (exit status 0). The mutation experiment turns 'cfn-lint passes' into knowledge of what that means: of 22 deliberate mistakes, 18 produced a finding, four were caught only as warnings (exit status 4) and four were not flagged at all: Fn::Sub inside Fn::Equals, DeletionPolicy written with !If, a FindInMap key that exists in no mapping, and a Number parameter whose Default is a word. For the documented quotas cfn-lint flagged 501 resources (E3010), 201 outputs (E6010) and an over-long description (E1003), accepted 500 resources, and did not flag 201 parameters or a 62 KB template body. The first version of that experiment padded to 498/499 resources because it assumed the template had ten when it had eight; the quota checker in Exercise 3 disagreed, which exposed it, and the generator now counts. The documentation's own examples produce warnings: a default VPC value that is not a valid ID pattern, and a DeletionPolicy example that cfn-lint says also needs UpdateReplacePolicy. Not run: aws cloudformation validate-template, any stack operation, other cfn-lint versions or rule sets.