New CloudFront Log Destinations

Accessing CloudFront access logs using Athena, Redshift, OpenSearch, and more

CloudFront is a content delivery network (CDN) service that leverages global edge locations to deliver data and APIs to users with low latency.

CloudFront can be configured to log all requests in access logs (also known as standard logs). These log entries include details such as date/time, request HTTP method, request protocol, request path, and server response (as well as other fields).

Up until recently, the only destination for CloudFront access logs was S3.

As of late 2024, AWS introduced “standard logging v2”, which makes it possible to have CloudFront access logs go directly to the following destinations:

• S3
• CloudWatch Logs
• Firehose

The now legacy standard logging already provided the S3 destination, however the addition of CloudWatch and Firehose make it much simpler to deliver the CloudFront logs to a location from where analysis tools can run, e.g. CloudWatch Logs Insights.

Why use the new destinations?

Previously, in order to analyse CloudFront logs delivered to S3, it was necessary to provision a Lambda function to copy the logs from S3 to another destination from which analysis tools (such as Athena or CloudWatch Logs Insights) could access the logs. It was common to use a Lambda to copy the logs from S3 into CloudWatch for convenient querying. Over time, this necessitates ongoing maintenance of the Lambda, things like updating the runtime.

This is often necessary in certain enterprises where service control policies prevent the use of tools used to query objects directly from S3, e.g. Athena.

Now it is possible to deliver the CloudFront logs directly to CloudWatch, which is where most other AWS services deliver their logs, and often, operations teams tend to have a high level of skill and experience in analysing logs using queries, alarms and dashboards.

Using the new Firehose destination, you can conveniently forward your CloudFront logs to any Firehose compatible destination, including RedShift and OpenSearch within AWS, as well as external targets such as Splunk or Snowflake.

As the diagram above illustrates, there’s no need to provision a lambda function to make use of the many possible log destinations.

How to use the new destinations

The CloudFront Developer Guide provides instructions to enable standard logging v2, from either the CloudFront console or the CloudWatch API. It does not provide complete instructions for doing this with CloudFormation (although it does state this is possible).

The following template snippet can be used to provision resources that stream CloudFront logs directly to CloudWatch. (Note the referenced CloudFrontDistribution resource is not shown for brevity.)

  # delivery source from where the logs originate, cloudfront distribution
  CloudFrontLogDeliverySource:
    Type: AWS::Logs::DeliverySource
    Properties:
      Name: myapp-prd-CloudFrontLogs
      LogType: ACCESS_LOGS
      ResourceArn: !Sub arn:aws:cloudfront::${AWS::AccountId}:distribution/${CloudFrontDistribution}

  # loggroup to which cloudfront logs will be delivered
  CloudFrontLogGroup:
    Type: AWS::Logs::LogGroup
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      LogGroupName: /aws/cloudfront/myapp-prd
      RetentionInDays: 30

  # wraps the loggroup in a destination that can be linked to the source below
  CloudFrontLogDeliveryDestination:
    Type: AWS::Logs::DeliveryDestination
    Properties:
      DestinationResourceArn: !GetAtt CloudFrontLogGroup.Arn
      Name: myapp-prd-CloudFrontLogDestination

  # links the source to the destination
  CloudFrontLogDelivery:
    Type: AWS::Logs::Delivery
    DependsOn: CloudFrontLogDeliverySource
    Properties:
      DeliverySourceName: myapp-prd-CloudFrontLogs
      DeliveryDestinationArn: !GetAtt CloudFrontLogDeliveryDestination.Arn

Credit to Chris Chen for figuring out the above CloudFormation.

The above stack of resources must be provisioned in the us-east-1 (N. Virginia) region (the developer guide does not provide a reason for this however I’m assuming this is because CloudFront is a global service).

Once the above stack has been deployed, you can go to CloudWatch Logs Insights and start querying CloudFront access logs as pictured below.

Cost implications

Additional charges will apply for using CloudWatch Logs.

• Data ingestion, based upon GB per month of data logged.
• Log storage, based upon GB per month.
• Log Insights queries, based upon the amount of data scanned by your queries.

Note that the free tier does cater for some of the costs up to a certain amount of GB per month.

Pricing varies per region, the following estimates relate to the us-east-1 (N. Virginia) region at time of writing. The figures provided are based upon 100GB of access logs per month.

Ingestion is charged at $0.50 per GB per month. There is the choice of Standard or Infrequent Access classes, the latter is roughly half the cost.

100GB x $0.50 = $50 USD per month

Storage is $0.03 per GB per month.

100GB x 0.15 Storage compression factor x 1 Logs retention factor x $0.03 = $0.45 USD per month.

Queries are $0.005 per GB of data scanned.

100GB x $0.005 = $0.50 USD

All up for 100GB of data per month, the expected cost is around $51 USD per month. Note that the number of queries run will affect this.

For more information and the latest prices, go to the CloudWatch Pricing page. To perform your own calculations, use the AWS Pricing Calculator.

Conclusion

The addition of CloudWatch and Firehose as destinations for CloudFront logs makes it easy to place the logs where they can be most conveniently analysed. It avoids the need to provision a Lambda function to copy the logs to alternative destinations. Just be mindful of additional costs when using any of the available destinations.