Datadog Provider

The Datadog provider allows you to query metrics from Datadog’s Metrics API for verification checks. It uses the Datadog v2 Scalar Query API to fetch aggregated metric values.

Configuration

provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  site: us5.datadoghq.com
  aggregator: last
  intervalSeconds: 300
  queries:
    a: "avg:kubernetes.cpu.usage{cluster:{{.resource.name}}}"
  formula: "a"

Properties

provider.type

string

required

Must be "datadog".

provider.apiKey

string

required

Datadog API key. Supports Go templates (e.g., {{.variables.dd_api_key}}).

provider.appKey

string

required

Datadog Application key. Supports Go templates. Use the actual key value, NOT the Key ID.

provider.queries

map[string]string

required

Named queries map. Keys become accessible as result.queries.<name> in success conditions. Values support Go templates for dynamic filtering.

provider.formula

string

Formula to combine query results. Reference queries by their key names (e.g., "a / b * 100").

provider.aggregator

string

default:"last"

Aggregation method. Options: last, avg, min, max, sum, mean, percentile, l2norm, area.

provider.intervalSeconds

integer

default:"300"

Time window in seconds for the query. Determines how far back to look for metric data.

provider.site

string

default:"datadoghq.com"

Datadog site URL. Options: datadoghq.com, datadoghq.eu, us3.datadoghq.com, us5.datadoghq.com, ap1.datadoghq.com.

Aggregator Options

The aggregator property specifies how to aggregate metric values:

last (default) - Most recent value
avg - Average value
min - Minimum value
max - Maximum value
sum - Sum of values
mean - Mean value
percentile - Percentile value
l2norm - L2 norm
area - Area under the curve

Supported Sites

datadoghq.com (US1 - default)
datadoghq.eu (EU)
us3.datadoghq.com (US3)
us5.datadoghq.com (US5)
ap1.datadoghq.com (AP1)

Response Data Available in CEL

The Datadog provider makes the following data available in your CEL success conditions:

Field	Type	Description
`result.ok`	boolean	`true` if API call succeeded (2xx status)
`result.statusCode`	integer	HTTP status code from Datadog API
`result.queries.<name>`	float64 (or null)	Value for each named query
`result.json`	object	Full Datadog API response
`result.body`	string	Raw response body
`result.duration`	integer	Request duration in milliseconds

Accessing Query Values

Since queries are named, you access them by name in your success condition:

queries:
  cpu: "avg:kubernetes.cpu.usage{cluster:prod}"
  memory: "avg:kubernetes.memory.usage{cluster:prod}"
successCondition: result.queries.cpu < 80 && result.queries.memory < 90

Example Configurations

Single Query

provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    error_rate: "sum:requests.error.rate{service:api-service}"
successCondition: result.queries.error_rate < 0.01

Multiple Queries with Formula

Use multiple queries with a formula to calculate ratios or complex metrics:

provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:requests.errors{service:api}"
    total: "sum:requests.total{service:api}"
  formula: "errors / total * 100"
successCondition: result.queries.errors < 1

With Environment and Resource Tags

provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  site: us5.datadoghq.com
  intervalSeconds: 600
  queries:
    cpu: "avg:kubernetes.cpu.usage{cluster:{{.resource.name}},env:{{.environment.name}}}"
successCondition:
  result.ok && result.queries.cpu != null && result.queries.cpu < 80

Latency Percentile

provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  aggregator: percentile
  queries:
    p99: "trace.http.request.duration.by.service.99p{service:api-service}"
successCondition: result.queries.p99 < 200

Example Success Conditions

# Simple threshold check
successCondition: result.queries.value < 0.01

# Check if value exists and is under threshold
successCondition: result.ok && result.queries.cpu != null && result.queries.cpu < 80

# Multiple query conditions
successCondition: result.queries.errors < 10 && result.queries.latency < 500

# Check API success first
successCondition: result.ok && result.statusCode == 200 && result.queries.rate < 0.1

Template Variables

The Datadog provider supports Go templates in the apiKey, appKey, site, formula, and queries fields:

# Resource information
{{.resource.name}}
{{.resource.identifier}}
{{.resource.kind}}

# Environment information
{{.environment.name}}
{{.environment.id}}

# Deployment information
{{.deployment.name}}
{{.deployment.slug}}

# Version information
{{.version.tag}}
{{.version.id}}

# Custom variables (from deployment variables)
{{.variables.dd_api_key}}
{{.variables.dd_app_key}}

Storing Secrets in Variables

For sensitive values like API keys, use deployment variables:

Create deployment variables for your Datadog credentials
Reference them in the provider configuration using template syntax

provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:errors{service:{{.resource.name}}}"

Finding Your Datadog Keys

API Key

Go to Datadog console → Organization Settings → API Keys
Create or copy an existing API key (40-character hex string)

Application Key

Go to Datadog console → Organization Settings → Application Keys
Create or copy an existing Application key
Important: Use the actual key value, NOT the “Key ID” (which is a UUID)

Best Practices

Use deployment variables for API keys and application keys - never hardcode credentials
Name queries descriptively to make success conditions readable
Use appropriate intervals - shorter intervals provide faster feedback but may have less data
Tag your metrics with service, environment, and version information for better filtering
Test queries manually in Datadog before using them in verification
Handle missing data by checking result.ok and result.queries.<name> != null
Use formulas for calculated metrics like error rates or ratios

​Configuration

​Properties

​Aggregator Options

​Supported Sites

​Response Data Available in CEL

​Accessing Query Values

​Example Configurations

​Single Query

​Multiple Queries with Formula

​With Environment and Resource Tags

​Latency Percentile

​Example Success Conditions

​Template Variables

​Storing Secrets in Variables

​Finding Your Datadog Keys

​API Key

​Application Key

​Best Practices

Configuration

Properties

Aggregator Options

Supported Sites

Response Data Available in CEL

Accessing Query Values

Example Configurations

Single Query

Multiple Queries with Formula

With Environment and Resource Tags

Latency Percentile

Example Success Conditions

Template Variables

Storing Secrets in Variables

Finding Your Datadog Keys

API Key

Application Key

Best Practices