Automate Lifecycle rule creation using Lambda

In the previous section, we learnt how to enable S3 Intelligent-Tiering through a lifecycle rule for a single bucket. In real-world scenarios, customers may accumulate petabytes of objects in the S3 Standard storage class across tens to hundreds of buckets and in multiple accounts who look for an easier approach to apply a single S3 Lifecycle configuration across multiple buckets to transition data from S3 Standard tier to S3 Intelligent-Tiering.

In this lab, we are going to create a AWS CloudFormation stack which will deploy a Lambda function with IAM permissions to create S3 lifecycle policy rules and to put objects in S3 bucket. Following is a high level logic of the fuction:

IF (lifecycle policy already exist on the bucket)
    THEN Skip the bucket
    
ELSE Create a new policy to transition objects to S3 Intelligent Tiering and attach the policy to the bucket

ENDIF;

The lambda function does not modify Amazon S3 buckets with existing lifecycle policies. But you can customize this lambda function to modify existing S3 lifecycle policy by adding logic in the placeholder Additional customization.

Amazon S3 lifecycle policy defined in the the lambda function creates a rule to move all the existing objects in all the AWS S3 buckets in a given account to S3 Intelligent Tiering Storage Class on the day 0 of object creation/upload. You can customize this logic to define your own transition rule statements. You can refer to some customization examples here.

Deploy CloudFormation Template

  1. Download the s3lifecycle-automation.yaml CloudFormation template to your machine.

  2. Go to the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation and click Create Stack > With new resources Images/CFNCreateStackButton

  3. Leave Prepare template setting as-is

    • For Template source select Upload a template file
    • Click Choose file and supply the CloudFormation template you downloaded: s3lifecycle-automation.yaml CFNUploadTemplateFile

  4. Click Next

  5. For Stack name use S3TieringLifecycleAutomation

  6. Parameters

    • Look over the Parameters and their default values.

    • BucketNameParam - Name of the pre-existing bucket where you like to store automation results.

    • Click Next

  7. For Configure stack options we recommend configuring tags, which are key-value pairs, that can help you identify your stacks and the resources they create. For example, enter Owner in the left column which is the key, and your email address in the right column which is the value. We will not use additional permissions or advanced options so click Next. For more information, see Setting AWS CloudFormation Stack Options.

  8. For Review

    • Review the contents of the page
    • At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names
    • Click Create stack CFNIamCapabilities

  9. This will take you to the CloudFormation stack status page, showing the stack creation in progress.

    • Click on the Events tab
    • Scroll through the listing. It shows the activities performed by CloudFormation (newest events at top), such as starting to create a resource and then completing the resource creation.
    • Any errors encountered during the creation of the stack will be listed in this tab. StackCreationStarted

  10. When it shows status CREATE_COMPLETE, then you are finished with this step.

Trigger Lambda Function

  1. Once the stack is deployed successfully, Go to Resources tab on cloudformation stacks page and find AWS Lambda function deployed. Click on the Physical ID to navigate to the function in AWS Lambda console Images/S3IntelligentTiering14aa.png

  2. In the Lambda console, click on Test to create a test event. Images/S3IntelligentTiering14a.png

  3. Specify a name for your test event, keep everything else to default and click Save. Images/S3IntelligentTiering14b.png

  4. Before running the event, review the lambda function code for any customization as suggested above. Run Test. Images/S3IntelligentTiering14c.png

  5. Review the execution log to see if the function has been executed successfully. It will generate the output file in the Amazon S3 bucket you specified during cloudformation stack creation. Images/S3IntelligentTiering14d.png Images/S3IntelligentTiering14e.png

  6. Once the lambda function execution is successful, go back to S3 console and verify the bucket lifecycle policies created. Images/S3IntelligentTiering14f.png

You can also deploy this cloudformation template as a stack set if you wish to run this across multiple accounts within your organization. More information around AWS Clouformation Stack set deployment can be found here.

You can also run this exercise at scale via AWS CLI using S3 commands such as ListBuckets followed by PutLifecycleConfiguration to enable lifecycle policy on Amazon S3 buckets. For information on setting up AWS CLI on your machine please refer to the CLI documentation.

Refer to the following link for more details: https://github.com/aws-samples/automated-lifecycle-transition-rules-to-s3int/