Build & Run Remediation Runbook

1. Go to the AWS Systems Manager console. Click Documents under Shared Resources on the left menu. Then click Create Automation as show in the screen shot below:

   

2. Enter Approval-Timer in the Name field and copy the notes shown below into the Document description field.

     # What does this automation do?
   
     Automatically trigger 'Approval' Signal to an execution, after a timer lapse
   
     ## Steps 
   
     1. Sleep for X time specified on the parameter input
     2. Automatically signal 'Approval' to the Execution specified in parameter input
   

3. In the Assume role field, enter the IAM role ARN we created in the previous section 3.0 Prepare Automation Document IAM Role.

4. Expand the Input Parameters section and enter Timer as the Parameter name. Set the type as String and Required as Yes.

5. Then add another parameter this time called AutomationExecutionId, of type String and set Required to Yes. Once you are done, your configuration should look like the screenshot below.

   

6. Under Step 1 section specify SleepTimer as Step name, select aws::sleep as the Action type.

7. Expand the Inputs section of the step, and specify {{Timer}} as the Duration

   

8. Click on Add step and specify ApproveExecution as Step name, select aws::executeAwsApi as the Action type.

9. Expand the Inputs section of the step, and specify ssm in the Service field and SendAutomationSignal in the API field.

10. Under Additional inputs specify below values.

    • Approve as the SignalType
    • {{AutomationExecutionId}} as the AutomationExecutionId.

    Once you are done, your configuration should look like the screenshot below.

    

    

6 . Click on Create automation once you are done.

Next, you will create the Approval-Gate runbook responsible for running the Approval-Timer runbook asynchronously. Follow below steps to complete the configuration:

1. From the AWS Systems Manager console, select Documents under Shared Resources on the left menu. Then click Create Automation as show in the screen shot below:

   

2. Next, enter Approval-Gate in the Name field and add the notes shown below to the Document description field.

     # What does this automation do?
   
     Place a gate before your desired step to create approval mechanism.
     Automation will trigger an asynchronously timer that will automatically approve once the time has lapsed.
     Automation will then send approval / deny request to the designated SNS Topic.
     When deny is triggered by approver, the step will fail and block the following step from executing.
   
     Note: Please ensure to have onFailure set to abort in your automation document.
   
     ## Steps 
   
     1. Trigger an asynchronously timer that will automatically approve once the time has lapsed.
     2. Send approval / deny request to the designated SNS Topic.
   
   

3. In the Assume role field, enter the IAM role ARN we created in the previous section 3.0 Prepare Automation Document IAM Role.

4. Expand the Input Parameters section and enter the following:

   • Timer as the Parameter name, set the type as String and Required as Yes.
   • NotificationMessage as the Parameter name, set the type as type String and Required is Yes.
   • NotificationTopicArn as the Parameter name, set the type as type String and Required is Yes.
   • ApproverRoleArn as the Parameter name, set the type as type String and Required is Yes.

5. Expand Step 1 create a step named executeAutoApproveTimer and action type aws:executeScript.

6. Expand Inputs, then set the Runtime as Python3.6 and paste in below code into the script section. Note that code snippet will execute the Approval-Timer runbook you created asyncronously.

   import boto3
   def script_handler(event, context):
     client = boto3.client('ssm')
     response = client.start_automation_execution(
         DocumentName='Approval-Timer',
         Parameters={
             'Timer': [ event['Timer'] ],
             'AutomationExecutionId' : [ event['AutomationExecutionId'] ]
         }
     )
     return None
   

7. Expand Additional Inputs, then select InputPayload under Input Name, and add the text shown below to Input Value:

   AutomationExecutionId: '{{automation:EXECUTION_ID}}'
   Timer: '{{Timer}}'
   

   Once you have completed this step, your Step 1 configuration should look like below screenshot.

   

8. Click Add step to create Step 2

9. Create a step named ApproveOrDeny and action type aws:approve.

10. Expand Inputs and specify below values under Approvers, replacing the AutomationRoleArn with the Arn of AutomationRole you took note of in section 3.0 Prepare Automation Document IAM Role.

    [ '{{ApproverRoleArn}}', 'AutomationRoleArn' ]
    

    Example:

    [ '{{ApproverRoleArn}}', 'arn:aws:iam::xxxxx:role/AutomationRole' ]
    

11. Expand Additional Inputs and specify the following values:

    • NotificationArn as the Input name, and {{NotificationTopicArn}} as the Input value
    • Message as the Input name, and {{NotificationMessage}} as the Input value
    • MinRequiredApprovals as the Input name, and 1 as the Input value

12. Expand Common properties and change the following properties to below values (keep the remaining as it is):

    • Continue for On failure
    • false for Is critical

    Once you have completed this step, your Step 2 configuration should look like below screenshot.

    

13. Click Add step to create Step 3

14. Create a step named getApprovalStatus and action type aws:executeAwsApi

15. Expand Inputs and specify ssm in the Service field, and DescribeAutomationStepExecutions in the API field.

16. Expand Additional Inputs and specify below values:

    • AutomationExecutionId as the Input Name, and {{automation:EXECUTION_ID}} as the Input value

    • Filters as the Input Name, and copy below values as the Input value

        - Key: StepName
          Values:
            - requestApproval
      

17. Expand Outputs and specify below values:

    • approvalStatusVariable as the Name
    • $.StepExecutions[0].Outputs.ApprovalStatus[0] as the Selector
    • String as the Type

    Once you have completed this step, your Step 3 configuration should look like below screenshot.

    

18. Click on Create automation to complete the configuation.

Download the template here.

If you decide to deploy the stack from the console, ensure that you follow below requirements & step:

1. Please follow this guide for information on how to deploy the CloudFormation template.
2. Use waopslab-runbook-approval-gate as the Stack Name, as this is referenced by other stacks later in the lab.

1. From the Cloud9 terminal, change to the templates folder as shown:

   cd ~/environment/aws-well-architected-labs/static/Operations/200_Automating_operations_with_playbooks_and_runbooks/Code/templates
   

2. Run the below commands, replacing the AutomationRoleArn with the Arn of AutomationRole you took note of in section 3.0 Prepare Automation Document IAM Role.

   aws cloudformation create-stack --stack-name waopslab-runbook-approval-gate \
                                   --parameters ParameterKey=PlaybookIAMRole,ParameterValue=AutomationRoleArn \
                                   --template-body file://runbook_approval_gate.yml 
   

   With your AutomationRole Arn in place your command will look similar to the following example:

   aws cloudformation create-stack --stack-name waopslab-runbook-approval-gate \
                                   --parameters ParameterKey=PlaybookIAMRole,ParameterValue=arn:aws:iam::000000000000:role/xxxx-runbook-role \
                                   --template-body file://runbook_approval_gate.yml 
   

3. Confirm that the stack has installed correctly. You can do this by running the describe-stacks command below, locate the StackStatus and confirm it is set to CREATE_COMPLETE.

aws cloudformation describe-stacks --stack-name waopslab-runbook-approval-gate

1. Go to the AWS Systems Manager console. Click Documents under Shared Resources on the left menu. Then click Create Automation as show in the screen shot below.

   

2. Next, enter Runbook-ECS-Scale-Up in the Name field and add the notes shown below to the Document description field:

     # What does this automation do?
   
     Scale up a given ECS service task desired count to certain number, with approval process.
     The automation will trigger Approval-Gate runbook, before executing.
   
     ## Steps 
   
     1. Trigger Approval-Gate
     2. Scale ECS Service by number of service
   

3. In the Assume role field, enter the IAM role ARN we created in the previous section 3.0 Prepare Automation Document IAM Role.

4. Expand the Input Parameters section and enter the following.

   • ECSDesiredCount as the Parameter name, set the type as Integer and Required as Yes.
   • ECSClusterName as the Parameter name, set the type as String and Required is Yes.
   • ECSServiceName, as the Parameter name, set the type as String and Required is Yes.
   • NotificationTopicArn, as the Parameter name, set the type as String and Required is Yes.
   • NotificationMessage, as the Parameter name, set the type as String and Required is Yes.
   • ApproverRoleArn, as the Parameter name, set the type as String and Required is Yes.
   • Timer, as the Parameter name, set the type as String and Required is Yes.

5. Expand Step 1 create a step named executeApprovalGate and action type aws:executeAutomation.

6. Expand Inputs, then set the Document name as Approval-Gate.

7. Expand Additional inputs and select RuntimeParameters as the Input Name

8. Paste in below as the Input Value

{
"Timer":'{{Timer}}',
"NotificationMessage":'{{NotificationMessage}}',
"NotificationTopicArn":'{{NotificationTopicArn}}',
"ApproverRoleArn":'{{ApproverRoleArn}}'
}

9. Click Add Step to create the second step.

10. Specify updateECSServiceDesiredCount as Step Name and select aws:executeAwsApi as Action type.

11. Expand Inputs and configure the following values:

    • ecs as Service
    • UpdateService as Api

12. Expand Additional inputs and configure the following values:

    • forceNewDeployment as the Input Name and true as Input Value
    • desiredCountas the Input Name and {{ECSDesiredCount}} as Input Value
    • service as the Input Name and {{ECSServiceName}} as Input Value
    • cluster as the Input Name and {{ECSClusterName}} as Input Value

13 . Click on Create automation once complete

Download the template here.

If you decide to deploy the stack from the console, ensure that you complete the following steps:

1. Please follow this guide for information on how to deploy the CloudFormation template.
2. Use waopslab-runbook-scale-ecs-service as the Stack Name, as this is referenced by other stacks later in the lab.

1. From the Cloud9 terminal, run the command to get into the working script folder.

   cd ~/environment/aws-well-architected-labs/static/Operations/200_Automating_operations_with_playbooks_and_runbooks/Code/templates
   

2. Then run below commands, replacing the ‘AutomationRoleArn’ with the Arn of AutomationRole you took note in previous step 3.0 Prepare Automation Document IAM Role.

   aws cloudformation create-stack --stack-name waopslab-runbook-scale-ecs-service \
                                   --parameters ParameterKey=PlaybookIAMRole,ParameterValue=AutomationRoleArn \
                                   --template-body file://runbook_scale_ecs_service.yml 
   

   Example:

   aws cloudformation create-stack --stack-name waopslab-runbook-scale-ecs-service \
                                   --parameters ParameterKey=PlaybookIAMRole,ParameterValue=arn:aws:iam::000000000000:role/AutomationRole \
                                   --template-body file://runbook_scale_ecs_service.yml 
   

3. Confirm that the stack has installed correctly. You can do this by running the describe-stacks command below, locate the StackStatus and confirm it is set to CREATE_COMPLETE.

aws cloudformation describe-stacks --stack-name waopslab-runbook-scale-ecs-service