v3.0.0-beta

What's Changed

• Populated filename in the output by @akshayrane in #358
• Support for some function expressions / stateful rules by @joshfried-aws in #361
• Combined structured output and updated default rule clause name to include file name by @akshayrane in #360
• Clap auto-completions by @joshfried-aws in #340
• Added documentation for functions by @joshfried-aws in #362
• Deprecated migrate and previous engine by @akshayrane in #364
• 3.0.0-beta release changes by @akshayrane in #366
  • Bumped up version to 3.0.0-beta
  • Updates to README.md for 3.0.0
  • Added latest configuration for aws-guard-rules-registry integration tests
  • Added new deployment method for cfn-guard-lambda through AWS SAM-CLI
  • Deprecated -j as short flag for print-json
  • Updated names for artifacts to have v3 in them
  • Added rogue_one branch to docker workflow, this will publish an ECR image for all updates to this remote branch as well

Full Changelog: 3.0.0-alpha...3.0.0-beta

Table of Contents

1. Built-in Functions and Stateful Rules
2. Added alternative deployment method using SAM-CLI for cfn-guard-lambda
3. Updated the output to contain filename
4. Updated combined structured output & default anonymous rule name
5. Command auto-completions

Details

1. Built-in Functions and Stateful Rules

As of version 3.0.0 guard now supplies some builtin functions, allowing for stateful rules

NOTE: all examples are operating off the following yaml template

Data

(click to expand)

Resources:
  newServer:
    Type: AWS::New::Service
    Properties:
      Policy: |
        {
           "Principal": "*",
           "Actions": ["s3*", "ec2*"]
        }
      Arn: arn:aws:newservice:us-west-2:123456789012:Table/extracted
      Encoded: This%20string%20will%20be%20URL%20encoded
    Collection:
      - a
      - b
      - c
    BucketPolicy:
      PolicyText: '{"Version":"2012-10-17","Statement":[{"Sid":"DenyReducedReliabilityStorage","Effect":"Deny","Principal":"*","Action":"s3:*","Resource":"arn:aws:s3:::s3-test-123/*","Condition":{"StringEquals":{"s3:x-amz-storage-class-123":["ONEZONE_IA","REDUCED_REDUNDANCY"]}}}]}'

  s3:
    Type: AWS::S3::Bucket
    Properties:
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
  bucket:
    Type: AWS::S3::Bucket
    Properties:
      PublicAccessBlockConfiguration:
        BlockPublicAcls: false
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

String Manipulation Functions

The following functions all operate on queries that resolve to string values

1. json_parse()

The json_parse function adds support for parsing inline json strings from a given template. After parsing the string into an object,
you can now evaluate certain properties of this struct just like with a normal json/yaml object

This function accepts a single argument:

• this argument can either be a query that resolves to a string or a string literal.

The return value for this function is a query where each string that was resolved from the input is parsed into its json value

The following example shows how you could parse 2 fields on the above template and then write clauses on the results

Rule

(click to expand)

let template = Resources.*[ Type == 'AWS::New::Service']
rule TEST_JSON_PARSE when %template !empty {
    let policy = %template.Properties.Policy

    let res = json_parse(%policy)

    %res !empty
    %res == %expected

    let policy_text = %template.BucketPolicy.PolicyText
    let res2 = json_parse(%policy_text)

    %res2.Statement[*]
    {
            Effect == "Deny"
            Resource == "arn:aws:s3:::s3-test-123/*"
    }
}

2. regex_replace()

The regex_replace function adds support for replacing one regular expression with another

This function accepts 3 arguments:

• The first argument is a query, each string that is resolved from this query will be operated on
• The second argument is either a query that resolves to a string or a string literal, this is the expression we are looking for to extract
  • Note: if this string does not resolve to a valid regular expression an error will occur
• The third argument is either a query that resolves to a string or a string literal, this is the expression we are going to use replace the extracted part of the string

The return value for this function is a query where each string that was resolved from the input that contains the the regex from our 2nd argument is replaced with the regex in the 3rd argument

In this simple example, we will re-format an ARN by moving around some sections in it.
We will start with a normal ARN that has the following pattern: arn:<Partition>:<Service>:<Region>:<AccountID>:<ResourceType>/<ResourceID>
and we will try to convert it to: <Partition>/<AccountID>/<Region>/<Service>-<ResourceType>/<ResourceID>

Rule

(click to expand)

let template = Resources.*[ Type == 'AWS::New::Service']

rule TEST_REGEX_REPLACE when %template !empty {
    %template.Properties.Arn exists
    let arn = %template.Properties.Arn

    let arn_partition_regex = "^arn:(\w+):(\w+):([\w0-9-]+):(\d+):(.+)$"
    let capture_group_reordering = "${1}/${4}/${3}/${2}-${5}"
    let res = regex_replace(%arn, %arn_partition_regex, %capture_group_reordering)

    %res == "aws/123456789012/us-west-2/newservice-Table/extracted"
}

3. join()

The join function adds support to collect a query, and then join their values using the provided delimiter.

This function accepts 2 arguments:

• The first argument is a query, all string values resolved from this query will then be joined using the delimter argument
• The second argument is either a query that resolves to a string/character, or a literal value that is either a string or character

The return value for this function is query where each string that was resolved from the input is joined with the provided delimiter

The following example queries the template for a Collection field on a given resource, it then provides a join on ONLY the string values that this query resolves to with a , delimiter

Rule

(click to expand)

let template = Resources.*[ Type == 'AWS::New::Service']

rule TEST_COLLECTION when %template !empty {
    let collection = %template.Collection.*

    let res = join(%collection, ",")
    %res == "a,b,c"
}

4. to_lower() & 5. to_upper()

Both functions accept a single argument:

• This argument is a query that resolves to a string(s) - all strings resolved will have the operation applied on them

Both these functions are very similar, one manipulates all resolved strings from a query to lower case, and the other to upper case

Rule

(click to expand)

let type = Resources.newServer.Type

rule STRING_MANIPULATION when %type !empty {
    let lower = to_lower(%type)
    %lower == "aws::new::service"
    %lower == /aws::new::service/

    let upper = to_upper(%type)
    %upper == "AWS::NEW::SERVICE"
    %upper == /AWS::NEW::SERVICE/
}

6. substring()

The substring function adds support to collect a part of all strings resolved from a query

This function accepts 3 arguments:

• The first argument is a query, each string that is resolved from this query will be operated on
• The second argument is either a query that resolves to an int or a literal int, this is the starting index for the substring (inclusive)
• The third argument is either a query that resolves to an int or a literal int, this is the ending index for the substring (exclusive)

The return value for this function takes the strings resolved from the first argument, and returns a result of substrings for each one of them:
Note: Any string that would result in an index out of bounds from the 2nd or 3rd argument is skipped

Rule

(click to expand)

let template = Resources.*[ Type == 'AWS::New::Service']

rule TEST_SUBSTRING when %template !empty {
    %template.Properties.Arn exists
    let arn = %template.Properties.Arn

    let res = substring(%arn, 0, 3)

    %res == "arn"
}

7. url_decode()

This function accepts a single argument:

• this argument can either be a query that resolves to a string or a string literal.

The return value for this function is a query that contains each url decoded version of every string value from the input

The following rule shows how you could url_decode the string This%20string%20will%20be%20URL%20encoded

Rule

(click to expand)

let template = Resources.*[ Type == 'AWS::New::Service']

rule SOME_RULE when %template !empty {
    %template.Properties.Encoded exists
    let encoded = %template.Properties.Encoded

    let res = url_decode(%encoded)
    %res == "This string will be URL encoded"
}

Collection functions

8. count()

The count function adds support to count the number of items that a query resolves to

This function accepts a single argument:

• This argument is a query that can resolve to any type - the number of resolved values from this query is returned as the result

The following rules show different ways we can use the count function.

• One queries a struct, and counts the number of properties.
• The second queries a list object, and counts the elements in the list
• The third queries for all resources that are s3 buckets and have a PublicAcessBlockConfiguration property

Rule

(click to expand)

let template = Resources.*[ Type == 'AWS::New::Service' ]
rule SOME_RULE when %template !empty {
    let props = %template.Properties.*
    let res = count(%props)
    %res == 3

    let collection = %template.Collection.*
    let res2 = count(%collection)
    %res2 == 3

    let buckets = Resources.*[ Type == 'AWS::S3::Bucket' ]
    let b = %buckets[ Properties.PublicAccessBlockConfiguration exists ]
    let res3 = count(%b)
    %res3 == 2

}

2. Added alternative deployment method using SAM-CLI for cfn-guard-lambda

cfn-guard-lambda can now also be deployed using this new method, through AWS SAM CLI.

Dependencies

• SAM CLI installed
• AWS CLI installed and configured with permissions to deploy via CloudFormation. SAM CLI will internally use the credentials you setup AWS CLI with. You may use the following IAM policy as a reference for least privileged access.

IAM Policy for SAM CLI User

{  
"Version": "2012-10-17",  
"Statement":  
[  
{  
"Effect": "Allow",  
"Action":  
[  
"cloudformation:CreateChangeSet",  
"cloudformation:CreateStack",  
"cloudformation:DeleteChangeSet",  
"cloudformation:DeleteStack",  
"cloudformation:DescribeChangeSet",  
"cloudformation:DescribeStackEvents",  
"cloudformation:DescribeStackResource",  
"cloudformation:DescribeStackResources",  
"cloudformation:DescribeStacks",  
"cloudformation:ExecuteChangeSet",  
"cloudformation:GetTemplate",  
"cloudformation:GetTemplateSummary",  
"cloudformation:ListStackResources",  
"cloudformation:SetStackPolicy",  
"cloudformation:UpdateStack",  
"cloudformation:UpdateTerminationProtection",  
"iam:AttachRolePolicy",  
"iam:CreateRole",  
"iam:DeleteRole",  
"iam:DetachRolePolicy",  
"iam:GetRole",  
"iam:PassRole",  
"lambda:CreateFunction",  
"lambda:DeleteFunction",  
"lambda:GetFunction",  
"lambda:TagResource",  
"s3:GetObject",  
"s3:PutObject"  
],  
"Resource": "*"  
}  
]  
}  

• Docker installed

Building and deploying

1. Make sure docker is running
2. Navigate to guard-lambda directory and run sam build --use-container to build the code for the Lambda function
3. Run sam deploy --guided and complete the interactive workflow. This workflow will create a CloudFormation changeset and deploy it
4. Once it succeeds, the name of the function will be shown in the CloudFormationGuardLambdaFunctionName output
5. For subsequent updates, build the code again (step 2) and run sam deploy (without --guided)

3. Updated the output to contain filename

Output now shows the data template filename field populated under the field name.

Example:

Rule

(click to expand)

s3_bucket_server_side_encryption_enabled.guard

let s3_buckets_server_side_encryption = Resources.*[ Type == 'AWS::S3::Bucket']

rule S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED when %s3_buckets_server_side_encryption !empty {
  %s3_buckets_server_side_encryption.Properties.BucketEncryption exists
  %s3_buckets_server_side_encryption.Properties.BucketEncryption.ServerSideEncryptionConfiguration[*].ServerSideEncryptionByDefault.SSEAlgorithm in ["aws:kms","AES256"]
  <<
    Violation: S3 Bucket must enable server-side encryption.
    Fix: Set the S3 Bucket property BucketEncryption.ServerSideEncryptionConfiguration.ServerSideEncryptionByDefault.SSEAlgorithm to either "aws:kms" or "AES256"
  >>
}

Data

(click to expand)

s3-server-side-encryption-template.yaml

 Resources:
   MyBucket1:
     Type: AWS::S3::Bucket
     Properties:
       BucketEncryption:
         ServerSideEncryptionConfiguration:
           - ServerSideEncryptionByDefault:
               SSEAlgorithm: aws:kms

Command:

cfn-guard validate \
 -d s3-server-side-encryption-template.yaml \
 -r s3_bucket_server_side_encryption_enabled.guard \
 --show-summary none -o json 

Output

{
  "name": "s3-server-side-encryption-template.yaml",
  "metadata": {},
  "status": "PASS",
  "not_compliant": [],
  "not_applicable": [],
  "compliant": [
    "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
  ]
}

4. Updated combined structured output & default anonymous rule name

The output using the --structured flag with R rule files against D data files now gives a merged array of size D, all the related rules show up as R children of a single parent object per data template passed as the input. (see example 1 below). An anonymous rule will be fully-qualified and will contain the filename as well. (see example 2 below)

Example 1

Command

cfn-guard validate --structured --show-summary none --output-format json --payload

Payload input

(click to expand)

Note the rules only have conditions and no name.

{
    "data": ["{\"Resources\":{\"NewVolume\":{\"Type\":\"AWS::EC2::Volume\",\"Properties\":{\"Size\":500,\"Encrypted\":false,\"AvailabilityZone\":\"us-west-2b\"}},\"NewVolume2\":{\"Type\":\"AWS::EC2::Volume\",\"Properties\":{\"Size\":50,\"Encrypted\":false,\"AvailabilityZone\":\"us-west-2c\"}}},\"Parameters\":{\"InstanceName\":\"TestInstance\"}}", "{\"Resources\":{\"NewVolume\":{\"Type\":\"AWS::EC2::Volume\",\"Properties\":{\"Size\":500,\"Encrypted\":false,\"AvailabilityZone\":\"us-west-2b\"}},\"NewVolume2\":{\"Type\":\"AWS::EC2::Volume\",\"Properties\":{\"Size\":50,\"Encrypted\":false,\"AvailabilityZone\":\"us-west-2c\"}}},\"Parameters\":{\"InstanceName\":\"TestInstance\"}}"],
    "rules": ["Parameters.InstanceName == \"TestInstance\"", "Parameters.InstanceName == \"TestInstance\""]
}

Press <Ctrl+D> to register input

Output

[
  {
    "name": "DATA_STDIN[1]",
    "metadata": {},
    "status": "PASS",
    "not_compliant": [],
    "not_applicable": [],
    "compliant": [
      "RULES_STDIN[1]/default",
      "RULES_STDIN[2]/default"
    ]
  },
  {
    "name": "DATA_STDIN[2]",
    "metadata": {},
    "status": "PASS",
    "not_compliant": [],
    "not_applicable": [],
    "compliant": [
      "RULES_STDIN[1]/default",
      "RULES_STDIN[2]/default"
    ]
  }
]

Example 2

Command

cfn-guard validate \
   --show-summary all \
   --rules guard/resources/validate/rules-dir/advanced_regex_negative_lookbehind_rule.guard \
   --data guard/resources/validate/data-dir/advanced_regex_negative_lookbehind_non_compliant.yaml

Rule

(click to expand)

advanced_regex_negative_lookbehind_rule.guard

NotAwsAccessKey != /(?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9])/
NotSecretAccessKey != /(?<![A-Za-z0-9\\/+=])[A-Za-z0-9\\/+=]{40}(?![A-Za-z0-9\\/+=])/

Data

(click to expand)

advanced_regex_negative_lookbehind_non_compliant.yaml

NotAwsAccessKey: AKIAIOSFODNN7EXAMPLE
NotSecretAccessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Output

advanced_regex_negative_lookbehind_non_compliant.yaml Status = FAIL
FAILED rules
advanced_regex_negative_lookbehind_rule.guard/default    FAIL
---
Evaluation of rules advanced_regex_negative_lookbehind_rule.guard against data advanced_regex_negative_lookbehind_non_compliant.yaml
--
Property [/NotAwsAccessKey] in data [advanced_regex_negative_lookbehind_non_compliant.yaml] is not compliant with [advanced_regex_negative_lookbehind_rule.guard/default] because provided value ["AKIAIOSFODNN7EXAMPLE"] did match expected value ["/(?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9])/"]. Error Message []
Property [/NotSecretAccessKey] in data [advanced_regex_negative_lookbehind_non_compliant.yaml] is not compliant with [advanced_regex_negative_lookbehind_rule.guard/default] because provided value ["wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"] did match expected value ["/(?<![A-Za-z0-9\\/+=])[A-Za-z0-9\\/+=]{40}(?![A-Za-z0-9\\/+=])/"]. Error Message []
--

5. Command auto-completions

We now have auto-completions for all our commands in shell.

Setup

To setup auto-completions you will need to follow instructions for the specific shell your are running.

Currently guard only supports auto-completions for zsh, bash, and fish shells. If you would like autocompletions for a specific shell feel free to open up a new github issue.

Auto-completions are only something available for version >= 3.0

zsh

    cfn-guard completions --shell='zsh' > /usr/local/share/zsh/site-functions/_cfn-guard && compinit

bash

    cfn-guard completions --shell='bash' > ~/cfn-guard.bash && source ~/cfn-guard.bash

fish

    cfn-guard completions --shell='fish' > ~/cfn-guard.fish
    cd ~
    ./ ./cfn-guard.fish

NOTE: for both bash and fish shells you are able to output the completions script to any file in any location you would like, just make sure the file you output it to and the file you source are the same.
For bash shells if you dont want to do this everytime you open up a new terminal, once you have the script you can add source ~/cfn-guard.bash to your .bashrc