Export Logs to S3

After collecting logs, you may want to export logs from CloudWatch to an S3 Bucket. This is useful as storing data in S3 is more cost effective and reliable than storing it in CloudWatch, making S3 a good option for long-term storage and archival of log files.

  1. Open up the CloudWatch console.
  2. On the left side menu, choose Log groups under Logs. On that screen, enter securitylablogs in the search bar. Click on the log group that appears in the results.
  3. Click Actions and Export data to Amazon S3 in the top menu.

export-s3-1

  1. You will have to fill out information about what data to export.

    1. In the From field, set the YYYY/MM/DD field to today’s date (the date you are doing this lab). This is the earliest creation date of logs you want to export.
    2. In the To field, set set the YYYY/MM/DD field to tomorrow’s date (the date after the day you are doing this lab). This is the latest creation date of logs you want to export.
    3. Leave the Stream prefix field blank, as we want to export all logs. This field allows you to select which logs you want to export.
    4. Set S3BucketName to the bucket name you entered in your CloudFormation stack, likely wa-lab-<your-account-id>-<date>. This is the bucket your logs will be exported to.
    5. Set S3 bucket prefix to lablogs. This is the subdirectory your exported logs will be stored in.

  2. Click Export

  3. Click on the View export tasks in the pop up box that appears. This will bring you to a list of Export tasks performed from CloudWatch

export-s3-2

  1. Click the radio bubble next to the most recent export. Click View in Amazon S3 to open these logs in the S3 bucket you created.

export-s3-3

  1. You should now see folders corresponding to all of the log streams you viewed earlier. You can explore these logs and download the .gz files if you’d like to see their contents.

Recap: In this portion of the lab, you exported logs from CloudWatch to S3, a good way to archive logs for long term storage. This demonstrates an important component of the security best practice of “configuring logging centrally” - the ability to extract meaningful insights from large volumes of log data. Compared with CloudWatch, storing log files in S3 is more cost-effective and allows you to use lifecycle policies on your stored logs. As the volume of logs generated by your workloads increase, so does the value of storing these data in S3. It also enables you to analyze logs from Athena, as you will see in the next section.