Create Data Transfer Cost Analysis Dashboard

Images/quicksight_dashboard_dt_new_analysis.png

Authors

  • Chaitanya Shah, Sr. Technical Account Manager (AWS)

FAQ

The FAQ for this dashboard is here.

Introduction

The Data Transfer Dashboard is an interactive, customizable and accessible QuickSight dashboard to help customers gain insights into their data transfer. It will analyze any data transfer that incurs a cost such as outbound internet and regional data transfer from all services.

This dashboard contains data transfer breakdowns with the following visuals:

  • Amount and cost by service and region
  • Between regions
  • Internet data transfer
  • Regional Data transfer

Request Template Access

Ensure you have requested access to the Cost Intelligence template here.

Create Athena Views

The data source for the dashboard will be an Athena view of your existing Cost and Usage Report (CUR).

  1. Login via SSO in your Cost Optimization account, go into the Athena console:

  2. Create the Data Transfer view by modifying the following code, and executing it in Athena or using aws cli:

The Athena Views are updated to reflect any additions in the cost and usage report. If you created your dashboard prior to July 26, 2020 you will want to update to the latest views.

Create QuickSight Data Sets

We will now create the data sets in QuickSight from the Athena views.

  1. Go to the QuickSight service homepage

  2. Click on the Datasets and then click on New dataset Images/quicksight_dashboard_dt-2.png

  3. Click Athena Images/quicksight_dashboard_dt-3.png

  4. Enter a data source name of DataTransfer_Cost_Dashboard and click Create data source: Images/quicksight_dashboard_dt-4.png

  5. Select the costmaster database, and the data_transfer_view table, click Edit/Preview data: Images/quicksight_dashboard_dt-5.png

If you have large data for data transfer in CUR, we do NOT recommend using SPICE when setting up your data set in QuickSight, you can quickly use up the 10GB/user allocation and start to incur additional charges. Please use your judgment before enabling it.

  1. Select SPICE to change your Query mode: Images/quicksight_dataset_6.png

  2. Hover over linked_account_id to get the drop down arrow and click on it, then hover over Change data type then select # Int: Images/quicksight_dashboard_dt-7.png

  3. Repeat step 7 for:

    • payer_account_id
  4. Hover over region to get the drop down arrow and click on it, then hover over Change data type then select # String

  5. Hover over blended_rate to get the drop down arrow and click on it, then hover over Change data type then select # Decimal: Images/quicksight_dashboard_dt-7.1.png

  6. Ensure the following fields are all # Decimal, repeat step 10 if necessary for:

    • belnded_cost
    • unblended_cost
    • public_cost
    • unblended_rate
    • public_ondemand_rate
  7. Select Save: Images/quicksight_dashboard_dt-8.png

  8. Select the data_transfer_view Data Set: Images/quicksight_dashboard_dt-9.png

  9. Click Schedule refresh: Images/quicksight_dashboard_dt-10.png

  10. Click Create: Images/quicksight_dataset_13.png)

  11. Enter a schedule, it needs to be refreshed daily, and click Create: Images/quicksight_dashboard_dt-12.png

  12. Click Cancel to exit: Images/quicksight_dashboard_dt-12.png

  13. Click the x in the top corner: Images/qquicksight_dashboard_dt-13.png

You now have your data set setup ready to create a visualization.

Create the Dashboard

We will now use the CLI to create the dashboard from the Data Transfer Cost and Usage Analysis Dashboard template, then create an Analysis you can customize and modify in the next step.

  1. If you have not requested access, go to this we page to request access to the template: Template Access

  2. Edit the following command, replacing AccountID with your account ID, and region with the region you are working in, then using the CLI list the QuickSight datasets and copy the Name and Arn for the dataset: data_transfer_view:

     aws quicksight list-data-sets --aws-account-id (AccountID) --region (region)
    

     

     {
         "Arn": "arn:aws:quicksight:us-east-1:<your account id>:dataset/fc0cf1eb-173b-4aca-93b6-f58784637732",
         "DataSetId": "fc0cf1eb-173b-4aca-93b6-f58784637732",
         "Name": "data_transfer_view",
         "CreatedTime": "2020-08-09T23:06:41.666000-04:00",
         "LastUpdatedTime": "2020-08-11T23:15:35.438000-04:00",
         "ImportMode": "SPICE"
     }
    
  1. Get your users Arn by editing the following command, replacing AccountID with your account ID, and region with the region you are working in, then using the CLI run the command:

     aws quicksight list-users --aws-account-id (AccountID) --namespace default --region (region)
    

     

     {
             "Arn": "arn:aws:quicksight:us-east-1:<your account id>:user/default/<your user>",
             "UserName": "<your user>",
             "Email": "<your user email>",
             "Role": "ADMIN",
             "Active": true,
             "PrincipalId": "<principal id>"
     }
    
  1. Create a local file create-data-transfer-dashboard.json with the text below, replace the values (Account ID) with your account ID on line 2 and line 25, (User ARN) with your user ARN on line 7, and (DataTransfer view Dataset ARN) with your dataset ARN on line 25:

     {
         "AwsAccountId": "(Account ID)",
         "DashboardId": "data_transfer_cost_analysis_dashboard",
         "Name": "DataTransfer Cost Analysis Dashboard",
         "Permissions": [
             {
                 "Principal": "(User ARN)",
                 "Actions": [
                         "quicksight:DescribeDashboard",
                         "quicksight:ListDashboardVersions",
                         "quicksight:UpdateDashboardPermissions",
                         "quicksight:QueryDashboard",
                         "quicksight:UpdateDashboard",
                         "quicksight:DeleteDashboard",
                         "quicksight:DescribeDashboardPermissions",
                         "quicksight:UpdateDashboardPublishedVersion"
                 ]
             }
         ],
         "SourceEntity": {
             "SourceTemplate": {
                 "DataSetReferences": [
                     {
                         "DataSetPlaceholder": "data_transfer_view",
                         "DataSetArn": "arn:aws:quicksight:us-east-1:(Account ID):dataset/(DataTransfer view Dataset ARN)"
    
                     }
                 ],
                         "Arn": "arn:aws:quicksight:us-east-1:869004330191:template/data-transfer-cost-analysis-template"
             }
         },
         "VersionDescription": "1"
     }
    
  2. To create the dashboard from the template, edit then run the following command, replacing (region) with the region you are working in, and you should receive a 202 response:

     aws quicksight create-dashboard --cli-input-json file://create-data-transfer-dashboard.json --region (region)
    
    • Response: Images/quicksight_dashboard_dt_resp.png
  3. After a few minutes the dashboard will become available in QuickSight under All dashboard, click on the Dashboard name: Images/quicksight_dashboard_dt-14.png

  4. Click here - if you do not see your dashboard
  5. Click Share, click Share dashboard:, Images/quicksight_dashboard_dt_share_dashboard.png

  6. Click Manage dashboard access: Images/quicksight_dashboard_8.png

  7. Add the required users, or share with all users, ensure you check Save as for each user, then click the x to close the window: Images/quicksight_dashboard_dt_saveas.png

Images/quicksight_dashboard_9.png

  1. Click Save as: Images/quicksight_dashboard_dt_save_as_icon.png

  2. Enter an Analysis name and click Create: Images/quicksight_dashboard_dt_analysis.png

Perform steps 11 and 12 above to create additional analyses for other teams, this will allow each team to have their own customizable analysis.

  1. You will now have an analysis created from the template that you can edit and modify: Images/quicksight_dashboard_dt_new_analysis.png

  2. To view data transfer charts for desired period change From and To date filters. Dashboard will automatically refresh upon date changes. Images/quicksight_dashboard_dt_analysis_date.png

You have successfully created the analysis from a template. For a detailed description of the dashboard read the FAQ