Transform and enrich CloudWatch Logs

Using log transformation and enrichment, you can normalize all your logs into a consistent and context-rich format during ingestion into CloudWatch Logs. You can add structure to your logs by using pre-configured templates for common AWS services such as AWS WAF and Amazon Route 53, or build custom transformers with native parsers such as Grok. You can also rename existing attributes and add additional metadata to your logs such as account ID, and Region.

Log transformation simplifies and shortens your log queries across applications and makes it easier to create alerts based on your logs. This feature provides transformation for common log types with out-of-the-box transformation templates for major AWS log sources like VPC Flow Logs, Route 53, and Amazon RDS for PostgreSQL. You can use pre-configured transformation templates or create custom transformers to suit your needs.

Log transformation helps you manage logs emitted from various sources that vary widely in format and attribute names.

What data will we use? This module uses logs generated by the PetSite application. While you can use your own log data, the format and query logic will differ. If you are using an AWS account provided to you as part of an organized event, this will already be set up for you. Otherwise, see getting started with your own AWS account for instructions to deploy the PetSite application.

A log transformer consists of one or more processors arranged in a logical pipeline. Processors are applied to log events sequentially in the order specified in the transformer configuration.