Create Pricing Data Sources

Create S3 Bucket and Folders

Create a single S3 bucket that contains two folders - od_pricedata and sp_pricedata, these will contain the on-demand pricing data and the Savings Plans pricing data.

1. Log into the console via SSO, go to the S3 service page:

2. Click Create bucket:

3. Enter a Bucket name starting with cost (we have used cost-sptool-pricingfiles, you will need to use a unique bucket name) and click Create bucket:

4. Click on the (bucket name):

5. Click Create folder:

6. Enter a folder name of od_pricedata, click Save:

7. Click Create folder:

8. Enter a folder name of sp_pricedata, click Save:

You have now setup the S3 bucket with the two folders that will contain the OnDemand and Savings Plans pricing data.

Create IAM Role and Policies

1. Go to the IAM Console

2. Select Policies and Create Policy

3. Edit the following policy and replace (S3 pricing bucket) with your bucket name, on the JSON tab, enter the following policy, click Review policy:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "S3SPTool",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:DeleteObjectVersion",
                   "s3:DeleteObject"
               ],
               "Resource": "arn:aws:s3:::(S3 pricing bucket)/*"
           }
       ]
   }
   

4. Policy Name S3PricingLambda, Description Access to S3 for Lambda SPTool function, click Create policy:

5. Select Roles, click Create role:

6. Select Lambda, click Next: Permissions:

7. Select the S3PricingLambda policy, click Next: Tags:

8. Click Next: Review

9. Role name SPToolS3Lambda, click Create role:

Setup On-Demand Pricing Lambda Function

Create the On-Demand Lambda function to get the pricing information, and extract the required parts from it.

1. Go to the Lambda service page:

2. Click Create function:

3. Enter the following details:

   • Select: Author from scratch
   • Function name: Cost_SPTool_ODPricing_Download
   • Runtime: Python (Latest)
   • Execution Role: Use an existing role
   • Role name: SPToolS3Lambda

4. Click Create function

5. Copy and paste the following code into the Function code section:

   # Lambda Function Code - SPTool_OD_pricing_Download
   # Function to download OnDemand pricing, get out the required lines & upload it to S3 as a zipped file
   # It will find 'OnDemand' and 'Compute Instance', and write to a file
   # Written by natbesh@amazon.com
   # Please reachout to costoptimization@amazon.com if there's any comments or suggestions
   import boto3
   import gzip
   import urllib3
   
   def lambda_handler(event, context):
   
       # Create the connection
       http = urllib3.PoolManager()
   
       try:
           # Get the EC2 OnDemand pricing file, its huge >1GB
           r = http.request('GET', 'https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.csv')
   
           # Put the response data into a variable & split it into an array line by line
           plaintext_content = r.data
           plaintext_lines = plaintext_content.splitlines()
   
           # Varaible to hold the OnDemand pricing data
           pricing_output = ""
   
           # Go through each of the pricing lines to find the ones we need
           for line in plaintext_lines:
   
               # If the line contains 'OnDemand' or 'Compute Instance' then add it to the output string
               if ((str(line).find('OnDemand') != -1) and (str(line).find('RunInstances') != -1)):
                   pricing_output += str(line.decode("utf-8")).replace('"', '')
                   pricing_output += "\n"
   
           # Add the output to a local temporary file & zip it
           with gzip.open('/tmp/od_pricedata.txt.gz', 'wb') as f:
               f.write(pricing_output.encode())
   
           # Upload the zipped file to S3
           s3 = boto3.resource('s3')
   
           # Specify the local file, the bucket, and the folder and object name - you MUST have a folder and object name
           s3.meta.client.upload_file('/tmp/od_pricedata.txt.gz', 'bucket_name', 'od_pricedata/od_pricedata.txt.gz')
   
       # Die if you cant get the pricing file                
       except Exception as e:
           print(e)
           raise e
   
       # Return happy
       return {
           'statusCode': 200
       }
   

6. Edit the pasted code, replacing bucket_name with the name of your bucket:

7. Click Deploy above the code

8. Scroll down and edit Basic settings:

   • Memory: 4096MB
   • Timeout: 2min
   • Click save

9. Scroll to the top and click Test

10. Enter an Event name of Test, click Create:

11. Click Test:

12. The function will run, it will take a minute or two given the size of the pricing files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in pricing file size over time:

13. Go to your S3 bucket and into the od_pricedata folder and you should see a gz file of non-zero size is in it:

Setup Savings Plan Pricing Lambda Function

Create the Savings Plan Lambda function to get the pricing information, and extract the required parts from it.

1. Go to the Lambda service page:

2. Click Create function:

3. Enter the following details:

   • Select: Author from scratch
   • Function name: Cost_SPTool_SPPricing_Download
   • Runtime: Python (Latest)
   • Execution Role: Use an existing role
   • Role name: SPToolS3Lambda

4. Click Create function

5. Copy and paste the following code into the Function code section:

   # Lambda Function Code - SPTool_SP_pricing_Download
   # Function to download SavingsPlans pricing, get out the required lines & upload it to S3 as a zipped file
   # It will get each regions pricing file in CSV, find 'Usage' and '1yr', and write to a file
   # Written by natbesh@amazon.com
   # Please reachout to costoptimization@amazon.com if there's any comments or suggestions
   import boto3
   import gzip
   import urllib3
   import json
   
   def lambda_handler(event, context):
   
       # Create the connection
       http = urllib3.PoolManager()
   
       try:
           # Get the SavingsPlans pricing index file, so you can get all the region files, which have the pricing in them
           r = http.request('GET', 'https://pricing.us-east-1.amazonaws.com/savingsPlan/v1.0/aws/AWSComputeSavingsPlan/current/region_index.json')
   
           # Load the json file into a variable, and parse it
           sp_regions = r.data
           sp_regions_json = (json.loads(sp_regions))
   
           # Variable to hold all of the pricing data, its large at over 150MB
           sp_pricing_data = ""
   
           # Cycle through each regions pricing file, to get the data we need
           for region in sp_regions_json['regions']:
   
               # Get the CSV URL
               url = "https://pricing.us-east-1.amazonaws.com" + region['versionUrl']
               url = url.replace('.json', '.csv')
   
               # Create a connection & get the regions pricing data CSV file
               http = urllib3.PoolManager()
               r = http.request('GET', url)
               spregion_content = r.data
   
               # Split the lines into an array
               spregion_lines = spregion_content.splitlines()
   
               # Go through each of the pricing lines
               for line in spregion_lines:
   
                   # If the line has 'Usage' then grab it for pricing data, exclude all others
                   if (str(line).find('Usage') != -1):
                       sp_pricing_data += str(line.decode("utf-8"))
                       sp_pricing_data += "\n"
   
           # Compress the text into a local temporary file
           with gzip.open('/tmp/sp_pricedata.txt.gz', 'wb') as f:
               f.write(sp_pricing_data.encode())
   
           # Upload the file to S3
           s3 = boto3.resource('s3')
   
           # Specify the local file, the bucket, and the folder and object name - you MUST have a folder and object name
           s3.meta.client.upload_file('/tmp/sp_pricedata.txt.gz', 'bucket_name', 'sp_pricedata/sp_pricedata.txt.gz')
   
       # Die if you cant get the file
       except Exception as e:
           print(e)
           raise e
   
       # Return happy
       return {
           'statusCode': 200
       }
   

6. Edit the pasted code, replacing bucket_name with the name of your bucket:

7. Click Deploy above the code

8. Edit Basic settings below:

   • Memory: 2048MB
   • Timeout: 2min
   • Click save

9. Scroll to the top and click Test

10. Enter an Event name of Test, click Create:

11. Click Test:

12. The function will run, it will take a minute or two given the size of the pricing files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in pricing file size over time:

13. Go to your S3 bucket and into the sp_pricedata folder and you should see a gz file of non-zero size is in it:

CloudWatch Events Setup

We will setup a CloudWatch Event to periodically run the Lambda functions, this will update the pricing and include any newly released instances.

1. Go to the CloudWatch service page:

2. Click on Events, then click Rules:

3. Click Create rule

4. For the Event Source select Schedule and set the required period, we have selected 5 days, click Add target:

5. Add the SPTool_ODPricing_Download Lambda function, and click Add target:

6. Add the SPTool_SPPricing_Download Lambda function, and click Configure details:

7. Add the name SPTool-pricing, optionally add a description and click Create rule:

Prepare the Pricing Data Source

We will prepare a pricing data source which we will use to join with the CUR. In this example we will take 1 year No Upfront Savings Plans rates and join them to On-Demand pricing. You can modify this part to select 3 year or Partial or All-Upfront rates.

1. Go to the Glue Service page:

2. Click Crawlers from the left menu:

3. Click Add crawler:

4. Enter a crawler name of OD_Pricing and click Next:

5. Ensure Data stores is the source type, click Next:

6. Click the folder icon to list the S3 folders in your account:

7. Expand the bucket which contains your pricing folders, and select the folder name od_pricedata, click Select:

8. Click Next:

9. Click Next:

10. Create an IAM role with a name of SPToolPricing, click Next:

11. Leave the frequency as Run on demand, and click Next:

12. Click on Add database:

13. Enter a database name of pricing, and click Create:

14. Click Next:

15. Click Finish:

16. Select the crawler OD_Pricing and click Run crawler:

17. Once its run, you should see tables created:

18. Repeat Steps 3 through to 17 with the following details:

    • Crawler name: SP_Pricing
    • Include path: s3://(pricing bucket)/sp_pricedata (replace pricing bucket)
    • IAM Role: Choose an existing IAM role and AWSGlueServiceRole-SPToolPricing
    • Database: pricing

19. Open the IAM Console in a new tab, click Policies:

20. Click on the AWSGlueServiceRole-SPToolPricing role:

21. Type in SPTool and click on the policy name AWSGlueServiceRole-SPTool:

22. Click Edit policy:

23. Click JSON:

24. Edit the Resource line by removing the od_pricedata folder to leave the bucket:

25. Click Review policy:

26. Click Save changes:

27. Go back to the Glue console, select the SP_Pricing crawler, click Run crawler:

28. Click on Databases:

29. Click on Pricing:

30. Click Tables in pricing:

31. Click od_pricedata:

32. Click Edit schema:

33. Click double next to col9:

34. Select string and click Update:

35. Click Save:

Create the Pricing Sources using Terraform

There is an AWS Github Repo which has a module to deploy all the resources needed in this lab. Please deploy using the instructions in the github repo then return to the step below.

Test Lambda Function

Now you have deployed the Terraform then you can test your lambda to get your first set of data in Amazon S3.

1. Go to the Lambda service page :

2. Search for your new function called Cost_SPTool_ODPricing_Download and click on it. To test your lambda function click Test

3. Enter an Event name of Test, click Create:

4. Click Test:

5. The function will run, it will take a minute or two given the size of the pricing files and processing required, then return success. Click Details and verify there is headroom in the configured resources and duration to allow any increases in pricing file size over time:

6. Do the same for the Cost_SPTool_SPPricing_Download function