Create the Visualizations

We will now create visualizations of our workload effiency. We will add the new dataset, and then build different visualizations to see what exactly impacts efficiency and where to look to improve it.

Basic Efficiency visualization

We will create a visualization from the application log files.

1. Go into QuickSight

2. Click Manage data:

3. Click New data set:

4. Click Athena:

5. Enter the Data source name: Efficiency, and click Create data source

6. Select the costusage Database, and the efficiency Table and click Select

7. Select Spice and click Edit/Preview data

8. Make sure there is data in the bottom pane:

9. Ensure you have data and click Save & visualize:

10. Create a Clustered bar combo chart. Place datetime (aggregate hour) on the x-axis, cost (sum) in the bars column, add request (count) to the lines field

11. Label the chart Requests vs Cost:

12. We can see that it roughly follows the same pattern, however there are times when the trends change. Below you can see the requests increase, but the cost decreases. Also the requests remain the same, and the cost decreases:

13. Maybe it can be explained through something other than request count, lets add mbytes (sum) to the lines field well to see if there is correlation there:

14. Again, similar trends and anomalies. MBytes remains constant and cost decreases:

15. Lets now create our efficiency visualition. Add a calculated field named efficiency with the formula below. Our efficiency metric will be requests per dollar:

    count(request) / sum(cost)
    

16. Add a visualization, select a Line chart. Place datetime (hour) in the x-axis, efficiency as the value,

17. We now have a chart showing our efficiency over time. Notice how the efficiency changes significantly at the end of the day:

18. You can now see increases and decreases in efficiency clearly, look when the output increases and cost remains the same, or the cost remains the same and the output decreases:

You now have a baseline efficiency metric. Use this to look for areas of low efficiency of your workload - this will provide areas to cost optimize.

Request visualization

Lets look deeper into the types of requests to see if we can get better insight into what is driving our costs and efficiency.

1. Lets look at a sample of a successful log request in our application log files:

    /index.php?name=Isabella,user=sponsored,work=26
   

2. We can see there are the fields:

   • Name
   • User
   • Work

3. Lets create a calculated field RequestType with the formula below. This will separate out the types of requests, health checks, image requests and other/errors from the request field:

    ifelse(locate(request,"index.html") > 0,split(request,',',2),ifelse(locate(request,"health.html") > 0,"HealthCheck",ifelse(locate(request,"image_file") > 0,"ImageFile","error")))
   

4. Create a Cluster bar combo chart visualization. Place datetime (HOUR) in the x-axis, add Request (count) in the bars, add RequestType in the Group/Color for bars, add efficiency to lines:

5. We can see a correlation between the user type and the efficiency, when paid users increase the efficiency goes down, this indicates its more costly to service paid customers. Also look at the efficiency increase when there are large amounts of errors, you may which to filter and exclude low value outputs from the measures of efficiency. These insights can be used to categorize different types of requests in your workload and understand how to charge them back appropriately.

You can use this to increase your efficiency by removing unwanted requests, for example you may use CloudFront to more efficiency handle errors - instead of processing them on the web servers.