Lab complete!
Now that you have completed this lab, make sure to update your Well-Architected review if you have implemented these changes in your workload.
Click here to access the Well-Architected Tool
We recommend deployment of CUDOS, KPI and Cost Intelligence Dashboard in a dedicated account, other than your Management (Payer) Account. This Lab provides a Cloud Formation template to move data from your Management Account to a dedicated one. You can use it deploy the dashboards on top of multiple Management (Payer) Accounts or multiple linked accounts.
If you do not want to use CloudFormation to setup your dashboards, you have an option of using our command line tool.
The Cost and Usage report (CUR) is generated in one or multiple Management (Payer) accounts. S3 replication copies CUR files from Source (Management Account) to the Destination (Data Collection Account).
This architecture supports aggregation of CURs from multiple Sources to one Data Collection Account. This is useful if you want have visibility across multiple Management (Payer) Accounts.
Also the same aggregation can be used when you do not have access to the Management (Payer) account and you need visibility across multiple Linked Accounts that belong to a single Business Unit.
There are 3 major steps in the Deployment:
If your Management(payer) / Source account is the same as your Destination account (where you want to deploy the dashboards) you can follow the steps for Destination Account only, and choose to activate Local CUR in the CFN parameter.
Each indivudal CUR has a prefix cur/<account>
so the aggregated CUR has the following structure:
s3://<prefix>-<destination-accountid>-shared/
cur/<src-account1>/cid/cid/year=XXXX/month=YY/*.parquet
cur/<src-account2>/cid/cid/year=XXXX/month=YY/*.parquet
cur/<src-account3>/cid/cid/year=XXXX/month=YY/*.parquet
This structure allows easily aggregate multiple CURs and allow a Glue crawler manage partitions: source_account_id, year, and month.
Using existing CUR is possible in a limited number of cases. See FAQ.
Here we will deploy the CFN template but setting the CFN parameters for a Destination Account.
Login to the Destination account in the region of your choice. It can be any account inside or outside your AWS Organization.
Click the Launch Stack button below to open the pre-populated stack template in your CloudFormation console.
NOTE: Please note this Account ID, we will need it later when we will deploy this same stack in your management (payer)/source accounts. |
---|
Disable CUR creation by entering False as the parameter value if you are replicating CURs from management (payer) accounts. You will only need to activate this if you are replicating CURs from linked accounts (not management payer accounts) and you want to have cost and usage data for this Destination account as well.
Enter your Source Account(s) IDs, using commas to separate multiple Account IDs.
Review the configuration, click I acknowledge that AWS CloudFormation might create IAM resources, and click Create stack.
You will see the stack will start with CREATE_IN_PROGRESS.
NOTE: This step can take 5-15mins |
---|
Once complete, the stack will show CREATE_COMPLETE.
You will be able to add or delete Source Account CURs later by updating this stack and adding or deleting Management (Payer) Account ID in a comma separated list of Source Accounts.
Login to your Source Account (can be management account or linked account if you’re using member CURs).
Click the Launch Stack button below to open the stack template in your CloudFormation console.
Enter a Stack name for your template such as CID-CUR-Replication.
Enter your Destination AWS Account ID as a parameter (Your Data Collection Account, where you will deploy dashboards).
Review the configuration, click I acknowledge that AWS CloudFormation might create IAM resources, and click Create stack.
You will see the stack will start with CREATE_IN_PROGRESS .
NOTE: This step can take 5-15mins |
---|
Once complete, the stack will show CREATE_COMPLETE.
It will take about 24 hours for your CUR to populate and replicate to your destination (data collection) account where you will deploy the dashboards. You can continue, but the dashboards will be empty. Or you can return to the next steps after 24 hours. We also recommend creating a Support Case in Service=Billing
and Category=Invoices and Reporting
, requesting a backfill of your CUR (name=cid
) with 12 months of data. Case must be created from your Source Account (Management/Payer account).
QuickSight is the AWS Business Intelligence tool that will allow you to not only view the Standard AWS provided insights into all of your accounts, but will also allow to produce new versions of the Dashboards we provide or create something entirely customized to you. If you are already a regular QuickSight user you can skip these steps and move on to the next step. If not, complete the steps below.
Log into your Destination Linked Account and search for QuickSight in the list of Services
You will be asked to sign up before you will be able to use it
After pressing the Sign up button you will be presented with 2 options, please ensure you select the Enterprise Edition during this step
Select continue and you will need to fill in a series of options in order to finish creating your account.
Ensure you select the region that is most appropriate based on where your S3 Bucket is located containing your Cost & Usage Report file.
Enable the Amazon S3 option and select the bucket where your Cost & Usage Report is stored, as well as your Athena query bucket
Click Finish and wait for the congratulations screen to display
Click Go to Amazon QuickSight
Click on the persona icon on the top right and select manage QuickSight.
Click on the SPICE Capacity option. Purchase enough SPICE capacity so that the total is roughly 40GB. If you get SPICE capacity errors later, you can come back here to purchase more. If you’ve purchased too much you can also release it after you’ve deployed the dashboards.
In this option we use guide you through using a CloudFormation template that will deploy all needed resources. You will cut and paste some parameters (An S3 path to CUR data, A QuickSight user that will be the owner of the QuickSight assets, and which dashboards you want to deploy) into the template and click run.
Warning: Ensure you use the same region as on the Step 1 (same region where your S3 Bucket with CUR is located) to avoid cross region Data Transfer costs. |
---|
All other resources are created automatically: Athena Workgroup and bucket, Glue table, Crawler, QS dataset, and finally the dashboards. The template uses a custom resource (a Lambda with this CLI tool) to create, delete, or update assets.
Login into your Linked (Data Collection) Account
Click the Launch Stack button below to open the pre-populated stack template in your CloudFormation.
Enter a Stack name for your template such as Cloud-Intelligence-Dashboards
Review Common Parameters and confirm prerequisites before specifying the other parameters. You must answer ‘yes’ to both prerequisites questions.
Copy and paste your QuicksightUserName into the parameter text box.
To find your QuickSight username:
- Open a new tab or window and navigate to the QuickSight console
- Find your username in the top right navigation bar
Update your CURBucketPath if needed.
s3://cid-1234567890123-shared/cur/
s3://cid-1234567890123-shared/prefix/name/name/
(double check this path, you must see /year=xxxx partitions in there).Please note that CURBucketPath parameter currently cannot be updated once the stack is created. If you need to change it you can delete and re-create the stack.
Select the Dashboards you want to install. We recommend deploying all three: Cost Intelligence Dashboard, CUDOS, and the KPI Dashboard.
Review the configuration, click I acknowledge that AWS CloudFormation might create IAM resources, and click Create stack.
You will see the stack will start in CREATE_IN_PROGRESS
NOTE: This step can take 5mins |
---|
Once complete, the stack will show CREATE_COMPLETE
While this is working, head back to QuickSight and click on manage Quicksight from the person icon on the top right.
aws-athena-query-results-cid-1234567890123-us-east-1
). If you do not see this bucket, please check if it is created by the CloudFormation stack.NOTE: This Output Section will be available once the Stack is Completed |
---|
If you see no data in QuickSight Dashboards after 24 hours, please check the following:
Failed
, you can click it to see more info).Billing
and Category=Invoices and Reporting
, requesting a backfill of your CUR (name=cid) with 12 months of data. Case must be created from the same account as CUR (Typically Management/Payer account).Any issue? Visit our FAQ.
If you can see data in your dashboards, you can continue to the post deployment steps for adding Account Names.
Now that you have completed this lab, make sure to update your Well-Architected review if you have implemented these changes in your workload.
Click here to access the Well-Architected Tool