Build & Run an Investigative Playbook

1. Download the template here.
2. Follow this guide for information on how to deploy the CloudFormation template.
3. Use waopslab-automation-role as the Stack Name, as this is referenced by other stacks later in the lab.

Note: To deploy from the command line, ensure that you have installed and configured AWS CLI with the appropriate credentials.

1. From the Cloud9 terminal change to the appropriate folder as shown:

cd ~/environment/aws-well-architected-labs/static/Operations/200_Automating_operations_with_playbooks_and_runbooks/Code/templates

2. Then run the command listed below:

aws cloudformation create-stack --stack-name waopslab-automation-role \
                                --capabilities CAPABILITY_NAMED_IAM \
                                --template-body file://automation_role.yml 

3. Confirm that the stack has installed correctly. You can do this by running the describe-stacks command:

aws cloudformation describe-stacks --stack-name waopslab-automation-role

Locate the StackStatus and confirm it is set to CREATE_COMPLETE

Download the template here.

If you decide to deploy the stack from the console, ensure that you follow below requirements & step:

1. Follow this guide for information on how to deploy the CloudFormation template.
2. Use waopslab-playbook-gather-resources as the Stack Name, as this is referenced by other stacks later in the lab.

Note: To deploy from the command line, ensure that you have installed and configured AWS CLI with the appropriate credentials.

1. From the Cloud9 terminal, run the command to get into the working script folder

cd ~/environment/aws-well-architected-labs/static/Operations/200_Automating_operations_with_playbooks_and_runbooks/Code/templates

2. Then run the below commands, replacing the ‘AutomationRoleArn’ with the Arn of AutomationRole you took note in previous step 3.0.

aws cloudformation create-stack --stack-name waopslab-playbook-gather-resources \
                                --parameters ParameterKey=PlaybookIAMRole,ParameterValue=AutomationRoleArn \
                                --template-body file://playbook_gather_resources.yml 

Example:

aws cloudformation create-stack --stack-name waopslab-playbook-gather-resources \
                                --parameters ParameterKey=PlaybookIAMRole,ParameterValue=arn:aws:iam::000000000000:role/AutomationRole \
                                --template-body file://playbook_gather_resources.yml 

Note: Please adjust your command-line if you are using profiles within your aws command line as required.

3. Confirm that the stack has installed correctly. You can do this by running the describe-stacks command below, locate the StackStatus and confirm it is set to CREATE_COMPLETE.

aws cloudformation describe-stacks --stack-name waopslab-playbook-gather-resources

1. Go to the AWS Systems Manager console. Click Documents under Shared Resources on the left menu. Then click Create Automation as show in the screen shot below:

2. Enter Playbook-Gather-Resources in the Name field and copy the notes shown below into the Document description field.

# What does this **playbook** do?

Query the CloudWatch Synthetics Canary and look for all resources related to the application based on it's Application Tag. This **playbook** takes an input of the CloudWatch Alarm ARN triggered by the canary

Note : Application resources must be deployed using CloudFormation and properly tagged accordingly.

## Actions taken in this playbook.
1. Describe CloudWatch Alarm ARN and identify the Canary resource.
2. Describe the Canary resource to gather the value of 'Application' tag
3. Gather CloudFormation Stack with the same value of 'Application' tag.
4. List all resources in CloudFormation Stack.
5. Parse list of resources into String Output.

3. In the Assume role field, enter the IAM role ARN we created in the previous section 3.0 Prepare Automation Document IAM Role.

4. Expand the Input Parameters section and enter AlarmARN as the Parameter name. Set the type as String and Required as Yes. This will define a Parameter within our playbook, so that the value of the CloudWatch Alarm ARN can be passed into the playbook to run the action.

5. Under Step 1 section specify Gather_Resources_For_Alarm Step name, select aws::executeScript as the Action type.

6. Under Inputs set Python3.6 as the Runtime and specify script_handler as the Handler.

7. Paste in below python codes into the Script section.

  import json
  import re
  from datetime import datetime
  import boto3
  import os

  def arn_deconstruct(arn):
  arnlist = arn.split(":")
  service=arnlist[2]
  region=arnlist[3]
  accountid=arnlist[4]
  servicetype=arnlist[5]
  name=arnlist[6]
  return {
    "Service": service,
    "Region": region,
    "AccountId": accountid,
    "Type": servicetype,
    "Name": name
  }

  def locate_alarm_source(alarm):
  cwclient = boto3.client('cloudwatch', region_name = alarm['Region'] )
  alarm_source = {}
  alarm_detail = cwclient.describe_alarms(AlarmNames=[alarm['Name']])  

  if len(alarm_detail['MetricAlarms']) > 0:
    metric_alarm = alarm_detail['MetricAlarms'][0]
    namespace = metric_alarm['Namespace']
    
    # Condition if NameSpace is CloudWatch Syntetics
    if namespace == 'CloudWatchSynthetics':
      if 'Dimensions' in metric_alarm:
        dimensions = metric_alarm['Dimensions']
        for i in dimensions:
          if i['Name'] == 'CanaryName':
            source_name = i['Value']
            alarm_source['Type'] = namespace
            alarm_source['Name'] = source_name
            alarm_source['Region'] = alarm['Region']
            alarm_source['AccountId'] = alarm['AccountId']

    result = alarm_source
    return result

  def locate_canary_endpoint(canaryname,region):
  result = None
  synclient = boto3.client('synthetics', region_name = region )
  res = synclient.get_canary(Name=canaryname)
  canary = res['Canary']
  if 'Tags' in canary:
    if 'TargetEndpoint' in canary['Tags']:
      target_endpoint = canary['Tags']['TargetEndpoint']
      result = target_endpoint
  return result


  def locate_app_tag_value(resource):
  result = None
  if resource['Type'] == 'CloudWatchSynthetics':
    synclient = boto3.client('synthetics', region_name = resource['Region'] )
    res = synclient.get_canary(Name=resource['Name'])
    canary = res['Canary']
    if 'Tags' in canary:
      if 'Application' in canary['Tags']:
        apptag_val = canary['Tags']['Application']
        result = apptag_val
  return result

  def locate_app_resources_by_tag(tag,region):
  result = None

  # Search CloufFormation Stacks for tag
  cfnclient = boto3.client('cloudformation', region_name = region )
  list = cfnclient.list_stacks(StackStatusFilter=['CREATE_COMPLETE','ROLLBACK_COMPLETE','UPDATE_COMPLETE','UPDATE_ROLLBACK_COMPLETE','IMPORT_COMPLETE','IMPORT_ROLLBACK_COMPLETE']  )
  for stack in list['StackSummaries']:
    app_resources_list = []
    stack_name = stack['StackName']
    stack_details = cfnclient.describe_stacks(StackName=stack_name)
    stack_info = stack_details['Stacks'][0]
    if 'Tags' in stack_info:
      for t in stack_info['Tags']:
        if t['Key'] == 'Application' and t['Value'] == tag:
          app_stack_name = stack_info['StackName']
          app_resources = cfnclient.describe_stack_resources(StackName=app_stack_name)
          for resource in app_resources['StackResources']:
            app_resources_list.append(
              { 
                'PhysicalResourceId' : resource['PhysicalResourceId'],
                'Type': resource['ResourceType']
              }
            )
          result =  app_resources_list

  return result
  def script_handler(event, context):
  result = {}
  arn = event['CloudWatchAlarmARN']
  alarm = arn_deconstruct(arn)
  # Locate tag from CloudWatch Alarm

  alarm_source = locate_alarm_source(alarm) # Identify Alarm Source
  tag_value = locate_app_tag_value(alarm_source) #Identify tag from source

  if alarm_source['Type'] == 'CloudWatchSynthetics':
    endpoint = locate_canary_endpoint(alarm_source['Name'],alarm_source['Region'])
    result['CanaryEndpoint'] = endpoint
    
  # Locate cloudformation with tag
  resources = locate_app_resources_by_tag(tag_value,alarm['Region'])
  result['ApplicationStackResources'] = json.dumps(resources) 

  return result

8. Under Additional inputs specify the input value to the step, passing in the parameter we created previously. To do this, specify below values:

   • InputPayload as the Input name
   • CloudWatchAlarmARN: '{{AlarmARN}}' as the Input Value.

9. Under Outputs specify below values:

   • Resources as Name
   • $.Payload.ApplicationStackResources as Selector
   • String as Type

10. Once your settings match the screenshot below, click on Create Automation

Download the template here.

If you decide to deploy the stack from the console, ensure that you follow below requirements & step:

1. Please follow this guide for information on how to deploy the CloudFormation template.
2. Use waopslab-playbook-investigate-resources as the Stack Name, as this is referenced by other stacks later in the lab.

1. From the Cloud9 terminal, change to the required folder as shown:

cd ~/environment/aws-well-architected-labs/static/Operations/200_Automating_operations_with_playbooks_and_runbooks/Code/templates

2. Run the command below, replacing the ‘AutomationRoleArn’ with the Arn of AutomationRole you took note in previous step 3.0 Prepare Automation Document IAM Role.

aws cloudformation create-stack --stack-name waopslab-playbook-investigate-resources \
                                --parameters ParameterKey=PlaybookIAMRole,ParameterValue=AutomationRoleArn \
                                --template-body file://playbook_investigate_application_resources.yml 

Example:

aws cloudformation create-stack --stack-name waopslab-playbook-investigate-resources \
                                --parameters ParameterKey=PlaybookIAMRole,ParameterValue=arn:aws:iam::000000000000:role/xxxx-playbook-role \
                                --template-body file://playbook_investigate_application_resources.yml 

3. Confirm that the stack has installed correctly. You can do this by running the describe-stacks command as follows:

aws cloudformation describe-stacks --stack-name waopslab-playbook-investigate-resources

4. Locate the StackStatus and confirm it is set to CREATE_COMPLETE

Download the template here.

If you decide to deploy the stack from the console, follow these steps:

1. Please follow this guide for information on how to deploy the CloudFormation template.
2. Use waopslab-playbook-investigate-application as the Stack Name, as this is referenced by other stacks later in the lab.
3. In the parameter input screen, under PlaybookIAMRole enter ARN of playbook IAM role (defined in previous step), under NotificationEmail enter your designated email for playbook notification

In the Cloud9 terminal go to the templates folder using the following command.

cd ~/environment/aws-well-architected-labs/static/Operations/200_Automating_operations_with_playbooks_and_runbooks/Code/templates

Then run below command :

aws cloudformation create-stack --stack-name waopslab-playbook-investigate-application \
                                --parameters ParameterKey=PlaybookIAMRole,ParameterValue=<ARN of **playbook** IAM role (defined in previous step)> \
                                --template-body file://playbook_investigate_application.yml 

Example:

aws cloudformation create-stack --stack-name waopslab-playbook-investigate-application \
                                --parameters ParameterKey=PlaybookIAMRole,ParameterValue=arn:aws:iam::000000000000:role/xxxx-playbook-role \
                                --template-body file://playbook_investigate_application.yml 

Note: Please adjust your command-line if you are using profiles within your aws command line as required.

Confirm that the stack has installed correctly. You can do this by running the describe-stacks command as follows:

aws cloudformation describe-stacks --stack-name waopslab-playbook-investigate-application 

Locate the StackStatus and confirm it is set to CREATE_COMPLETE

1. From the AWS Systems Manager console, click on documents as shown below. Once you are there, click on Create Automation

2. Next, enter in Playbook-Investigate-Application-From-Alarm in the Name and paste in the notes shown below into the Description box. This provides a description of the playbook. Systems Manager supports putting in notes as markdown, so feel free to format as required.

# What is does this **playbook** do?

This **playbook** will run **Playbook-Gather-Resources** to gather Application resources monitored by Canary.

Then subsequently run **Playbook-Investigate-Application-Resources** to Investigate the resources for issues. 

Outputs of the investigation will be sent to SNS Topic Subscriber
  

3. Under Assume role field, enter in the ARN of the IAM role we created in the previous step.

4. Under Input Parameters field, enter AlarmARN as the Parameter name. Set the type as String and Required as Yes. This will define a Parameter into our playbook, which allows the value of the CloudWatch Alarm to be passed to the main step that will run the action.

5. Add another parameter by clicking on the Add a parameter link. Enter SNSTopicARN as the Parameter name. Set the type as String and Required as Yes. This will define another Parameter into our playbook, so that we can send notification to the Owner and Developer.

6. Click Add Step and create the first step of aws:executeAutomation Action type with StepName PlaybookGatherAppResourcesCanaryCloudWatchAlarm

7. Specify Playbook-Gather-Resources as the Document name under Inputs and under Additional inputs specify RuntimeParameters with {"AlarmARN":'{{AlarmARN}}'} as it’s value (refer to screenshot below). This step we will be run the Gather-Resources playbook which we created previously.

8. Once this step is defined, add another step by clicking on Add Step at the bottom of the section.

9. For this second step, specify the Step name as PlaybookInvestigateAppResourcesELBECSRDS and an action type of aws:executeAutomation.

10. Specify Playbook-Investigate-Application-Resources as the Document name and RuntimeParameters as Resources: '{{PlaybookGatherAppResourcesCanaryCloudWatchAlarm.Output}}' This will take the output of the first step and pass to the second playbook to run the investigation of associated resources.

11. For the last step, take the output investigation from the second step and send that to the SNS topic where our owner, developers and admin are subscribed.

12. Specify the Step name as AWSPublishSNSNotification and the action type as aws:executeAutomation.

13. Specify AWS-PublishSNSNotification as the Document name and RuntimeParameters as shown below. This will take the output of the second step which contains summary data of the investigation and AWS-PublishSNSNotification which will send an email to the SNS we specified in the parameters.

TopicArn: '{{SNSTopicARN}}'
Message: '{{ PlaybookInvestigateAppResourcesELBECSRDS.Output }}'

14. Our playbook will run investigative tasks and send the result to an SNS topic where our Systems administrator / engineer will subscribe to. To do this we will need to create an SNS topic that our playbook will send notification to. Please follow the instructions specified in this link and create a Standard SNS topic and name it PlaybookNotificationSNSTopic

15. Once you’ve created the topic, go ahead and subscribe your an email using this instruction here