Amazon Lookout for Metrics is a service that uses machine learning (ML) to detect anomalies in your time series data.

Lookout for Metrics allows users to set up anomaly detectors in both continuous and backtest modes. Backtesting allows you to detect anomalies on historical data. This feature is helpful when you want to try out the service on past data or validate against known anomalies that occurred in the past. For this post, we use continuous mode, where you can detect anomalies on live data as they occur. In continuous mode, the detector monitors an input S3 bucket for continuous data and runs anomaly detection on new data at specified time intervals. For the live detector to consume continuous time series data from Amazon S3 correctly, it needs to know where to look for data for the current time interval, therefore, it requires continuous input data in S3 buckets organized by time interval.

Read my AWS Blog post here for more details. Included in this post is a sample streaming data generator to help you get started quickly. The included GitHub repo provides step-by-step deployment instructions, and uses the AWS Cloud Development Kit (AWS CDK) to simplify and automate the deployment.

The solution consists of the following components:

• Stream data generator that generates sample timeseries data real-time

Data generator python code in github repo.

• Kinesis stream connector to format real time time series data using Glue Spark streaming ETL

  Spark streaming ETL code in github repo.

• Anomaly detector using Amazon Lookout for Metrics

For explanation of the architecture, the generator and ETL code mentioned above, read my original blog post here.