Many workloads depend on external resources or services for data or additional capabilities such as 3rd party data providers or service providers, DNS providers, etc. Functionality or outcomes of the workload may be at risk when dependent resources or services become degraded or unreachable.
Monitoring these dependencies will enable quick action to ensure business continuity is not affected. Setting up alerting and notifications will ensure that appropriate team members are aware of issues and can take action to address the situation.
This lab provides examples of how to implement Well-Architected Operational Excellence best practices such as “Implement dependency telemetry”, “Alert when workload outcomes are at risk”, and “Enable push notifications”.
In this lab there is an external service (3rd party data provider) that provides data which will be consumed by the workload. This has been emulated in this lab by using an EC2 instance which acts as the 3rd party data provider, and it writes data to an S3 bucket at 50 second intervals. Amazon S3 notification feature enables you to receive notifications when certain events happen in your bucket.
For this use-case the notification has been configured on the S3 bucket to invoke a lambda function after every write to the bucket, using the S3 PutObject API. The objective of this lab is to create awareness when an external service is experiencing downtime or is otherwise impaired. For this example the assumption is that the 3rd party data provider is experiencing downtime when data is no longer being written to the S3 bucket.
If you are attending an in-person workshop and were provided with an AWS account by the instructor:
If you are using your own AWS account:
You will use AWS CloudFormation to provision resources that will emulate the workload described in the use-case. AWS CloudFormation provides you a common language to model and provision AWS and third party application resources by applying Infrastructure as Code in your cloud environment.
Download the dependency_monitoring.yaml CloudFormation template (right-click on the link and select “Save Link As…")
Go to the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation and click Create Stack > With new resources (standard)
Leave Prepare template setting as-is since you already have a template ready (dependency_monitoring.yaml)
For Stack name use
wa-lab-<your last name>-<date><time>.
For Configure stack options click Next
On the Review page:
Note: The template creates 2 roles for Lambda as well as a role and instance profile for an EC2 instance. They are the minimum permissions necessary to read and write from an S3 bucket created as part of this lab and create an OpsItem in OpsCenter. These permissions can be reviewed in the CloudFormation template under “Resources” section - DataReadLambdaRole, OpsItemLambdaRole, and InstanceRole.
This will take you to the CloudFormation stack status page, showing the stack creation in progress.
Once stack creation starts, monitor the email address you entered. You should receive an email from SNS with the subject AWS Notification - Subscription Confirmation. Click on the link Confirm subscription to confirm the subscription of your email to the SNS Topic. This will allow SNS to send email notifications to the email address specified.
The stack takes about 3 mins to create all the resources. Periodically refresh the page until you see that the Stack Status is in CREATE_COMPLETE. The stack creates the following resources:
Once the stack is in CREATE_COMPLETE, visit the Outputs section for the stack and note down the Key and Value for each of the outputs. This information will be used later in the lab.