Level 300: Optimization Data Collection - BETA

Last Updated

July 2021

Authors

  • Stephanie Gooch, Commercial Architect (AWS)

Feedback

If you wish to provide feedback on this lab, there is an error, or you have a suggestion, please email: costoptimization@amazon.com

This lab is in BETA and your feedback is key to developing the lab. Please share any feedback, bugs, or ideas to help improve the lab

Introduction

Amazon Web Services offers a broad set of global cloud-based products including compute, storage, databases, analytics, networking, mobile, developer tools, management tools, IoT, security and enterprise applications. These services help organizations move faster, lower IT costs, and scale.

There are many ways to cost-optimize on AWS. Relevant data for identfying cost-optimization opportunities can come from different services and you may need to make apple-to-apples comparisons before making a business decision.

This lab is designed to enable you to collect utilization data from different services to help you identify optimization opportunities. This lab provides pre made modules to automate data collection and show you how to pull your additional data sets on your own. The CloudFormation modules in this lab follow the structure of using an AWS Lambda function to extract the data, then this is placed into Amazon S3. From there, Amazon Athena is able to read this data using an AWS Glue Crawler to produce a table that can be utilized for optimization analysis and even joined with your AWS Cost & Usage Report (CUR) to enrich it.

The three main styles of data are:

  • Optimization and rightsizing recommendations
  • Service Inventories
  • Resource utilization metrics

The services you will learn to pull data from in this lab are:

  • Cost Explorer Rightsizing Recommendations
  • Inventory Collector
  • Trusted Advisor
  • Compute Optimizer Collector
  • ECS Chargeback Data
  • RDS Utilization Data

Architecture

Images/Arc.png

Goals

  • Deploy main resources which will be used by the modules
  • Deploy modules to collect data
  • Retrieve optimization data

Prerequisites

Deployment Options

We suggest you do not deploy the main resources and collectors into your management account and instead use the cost account created in Account Setup Lab . However, Some IAM resources will be needed to read data from the management account.

Permissions required

Be able to create the below in the management account:

  • IAM role and policy
  • Deploy CloudFormation

Be able to create the below in a sub account where your CUR data is accessible:

  • Deploy CloudFormation
  • Amazon S3 Bucket
  • AWS Lambda function
  • IAM role and policy
  • Amazon CloudWatch trigger
  • Amazon Athena Table
  • AWS Glue Crawler

Costs

  • Estimated costs should be <$5 a month for small Organization

Time to complete

  • 30 minutes

Steps: