Level 200: Remote Configuration, Installation, and Viewing of CloudWatch logs

Authors

  • Pavan Baloo, Solutions Architect Intern, Well-Architected

Introduction

Most applications require monitoring services that make up the workload to understand the workload state and performance. One way of collecting this data is through log files generated by the application and underlying services. Collecting and analyzing log files improves your security posture by creating a record of activity or audit trail in your workload, enabling you to detect and investigate potential threats.

Manually configuring logging on each instance is tedious and becomes difficult at scale. It increases your risk of human error and unintended access from accessing instances directly with a protocol like SSH. Manually processing the collected data is difficult to scale with large volumes of data.

Utilizing AWS services such as AWS Systems Manager, Amazon CloudWatch, Amazon Simple Storage Service (S3), Amazon Athena, and Amazon QuickSight, you can collect and store logs without having to directly access the instance, or accessing data directly. You minimize your threat surface area by removing SSH access on your instance and improve your threat detection by collecting valuable log data.

This lab illustrates the following Well-Architected Security Best Practices:

  1. “Configure service and application logging”: You will configure the CloudWatch agent on an EC2 instance. This enables you to collect logs from the instance used to host your application, such as Apache Web Server logs, SSH logs, boot logs, and more.
  2. “Configure services and resources centrally”: You will centrally configure your CloudWatch log agent by storing the configuration file in Systems Manager Parameter Store. Parameter Store enables you to maintain consistent, reusable configuration data.
  3. “Analyze logs centrally”: You will analyze logs centrally in this lab in two ways. Using the CloudWatch console, you can view all of your raw log data in one location. Through QuickSight, you can create visualizations from your logs that can be shared with others for central viewing of key data.
  4. “Enable people to perform actions at a distance”: You will use Systems Manager Run Command to install and start the CloudWatch agent on your EC2 instance. You will perform these actions “at a distance” through Run Command, as you will not need to SSH directly into the instance to perform these tasks.
  5. “Reduce attack surface”: Run Command removes the necessity to directly SSH into the EC2 instance. Because of this, you can close the SSH access port on your instance, reducing the attack surface of the workload.

In the lab, you will deploy an EC2 instance with Apache and PHP installed. The web server will host a very simple website. You will configure a CloudWatch Agent on the instance via Amazon Systems Manager (SSM). This agent will collect log files from services running on the EC2 instance, such as Apache access and error logs, yum logs, SSH logs, and CloudWatch agent logs. These logs are exported from the EC2 instance to the CloudWatch logs service for centralized storage. You will export these logs to an S3 bucket for long term storage and archival. These logs will then be queried via Athena, so people are kept away from accessing the log files directly. This data will be visually represented in a QuickSight dashboard.

datadistancingarch

Prerequisites

  • I have access to an AWS Account to use for testing, from which I can deploy EC2 instances, create S3 Buckets, access and export CloudWatch Logs, run Athena queries, and use QuickSight.
  • I am operating in a region in which I can use Amazon EC2, Amazon S3, Amazon CloudWatch, Amazon Athena, AWS Systems Manager, and Amazon QuickSight. To see if these services are available in your region, click to view the service availability page.

Files Used

Steps