Level 300: VPC Flow Logs Analysis Dashboard

Author

  • Chaitanya Shah, Sr. Technical Account Manager, AWS

Introduction

VPC Flow Logs enables you to capture information about the IP traffic going to and from network interfaces in your VPC. The VPC Flow Logs Analysis Dashboard is an interactive, customizable and accessible QuickSight dashboard to help customers gain insights into traffic details of VPC in a graphical way.

The dashboard depends on all the fields below. Therefore all of these fields are required in the VPC Flow Logs that are stored in S3:

  • version, account-id, interface-id, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, start, end, action, log-status, vpc-id, az-id, instance-id, pkt-srcaddr, pkt-dstaddr, region, subnet-id, sublocation-id, sublocation-type, tcp-flags, type, flow-direction, pkt-dst-aws-service, pkt-src-aws-service, traffic-path

This dashboard contains breakdowns with the following visuals. Available views are Summary, Details by daily, Minutes level granularity, and Enhanced view:

  • By VPC, InterfaceIds
  • Between Source and Destination IPs
  • By Region, AZ and Instances
  • Source and destination AWS services paths (Enhanced view)

Supported flow log record formats:

  • Parquet

Supported Glue Partitions:

  • Non Hive-compatible S3 prefix
  • Hive-compatible S3 prefix

Note: We recommend creating Parquet file format with Hive-compatible S3 prefix for better performance and reducing cost for querying data from S3

Architecture

images/qs-vpcl-architecture_v3.png

Goals

  • This dashboard allows you to analyze and visualize vpc flow log data more flexibly, instead of focusing on the underlaying infrastructure, you can focus on investigating the logs.

Prerequisites

  • An AWS Account that you are able to use for testing. This account MUST NOT be used for production or other purposes.
  • An Amazon Enterprise Edition QuickSight Account
    • For supported QuickSight regions please visit link
  • Amazon QuickSight user has been already created

Costs

Note: Please refer to pricing page for current prices for below services

  • A QuickSight Enterprise license starts at $18 per month and Readers $0.30/session up to $5 max/month
  • AWS Athena $5.00 per TB of data scanned
  • AWS Glue Storage: Free for the first million objects stored and $1.00 per 100,000 objects stored above 1M, per . Requests: Free for the first million requests per month. $1.00 per million requests above 1M in a month
  • VPC Flow logs pricing(Example 5) to ingest data in S3
  • Data Transfer costs to store data coming from different accounts to central account bucket

Time to complete

  • The lab should take approximately 15-20 minutes to complete

Steps: