Requirements
- Automatically deploy OpenTelemetry (OTEL) collectors whenever a new EC2 instance is created
- Track resource utilization metrics (CPU, memory, disk) across all developer instances
- Provide developers with a consistent OTLP endpoint they can use for custom metrics
- Maintain the same deployment configuration in both testing and production environments
Technologies
AWS EC2
Cloud computing platform that provides scalable compute capacity. We’ll use
EC2 instances as our deployment targets for running applications and
collecting metrics.
OpenTelemetry Collector
A vendor-agnostic way to receive, process and export telemetry data. The
collector will be automatically deployed to gather system metrics and
provide a standardized endpoint for custom application metrics.
Github Actions
CI/CD platform that will automate our deployment pipeline. It will handle
the provisioning of collectors and configuration management across all
environments.
Solution
1. Getting EC2 Instances into Ctrlplane
Before deploying OpenTelemetry collectors, EC2 instances must be registered as Resources in Ctrlplane. These Resources represent the external state of our EC2 instances and require consistent updates as the state of each instance changes. These are just some of the ways you can get EC2 instances into Ctrlplane.Using AWS Lambda
Using AWS Lambda
AWS Lambda can be used to automatically register EC2 instances in Ctrlplane
whenever they are launched. By setting up a Lambda function triggered by
CloudWatch Events, you can capture instance launch events and use the
Ctrlplane API to create or update resources representing these instances.
Using AWS Systems Manager
Using AWS Systems Manager
AWS Systems Manager provides a way to manage EC2 instances at scale. By
integrating Systems Manager with Ctrlplane, you can automatically register
instances as they are launched. This can be achieved by using Systems Manager
Run Command to execute scripts that call the Ctrlplane API for resource
registration.
Using Ctrlplane CLI
Using Ctrlplane CLI
AWS CloudFormation can be used to define and provision EC2 instances as
well as register them in Ctrlplane. By including a custom resource in your
CloudFormation template, you can trigger a Lambda-backed custom resource
that registers the instance in Ctrlplane upon creation.
2. Github Deployment Pipeline
.github/workflows/deploy-otel-agent.yml