> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ctrlplane.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Datadog Provider

> Query metrics from Datadog's Metrics API for verification

The **Datadog provider** allows you to query metrics from Datadog's Metrics API
for verification checks. It uses the Datadog v2 Scalar Query API to fetch
aggregated metric values.

## Configuration

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  site: us5.datadoghq.com
  aggregator: last
  intervalSeconds: 300
  queries:
    a: "avg:kubernetes.cpu.usage{cluster:{{.resource.name}}}"
  formula: "a"
```

## Properties

<ParamField path="provider.type" type="string" required>
  Must be `"datadog"`.
</ParamField>

<ParamField path="provider.apiKey" type="string" required>
  Datadog API key. Supports Go templates (e.g., `{{.variables.dd_api_key}}`).
</ParamField>

<ParamField path="provider.appKey" type="string" required>
  Datadog Application key. Supports Go templates. Use the actual key value, NOT
  the Key ID.
</ParamField>

<ParamField path="provider.queries" type="map[string]string" required>
  Named queries map. Keys become accessible as `result.queries.<name>` in success conditions.
  Values support Go templates for dynamic filtering.
</ParamField>

<ParamField path="provider.formula" type="string">
  Formula to combine query results. Reference queries by their key names (e.g.,
  `"a / b * 100"`).
</ParamField>

<ParamField path="provider.aggregator" type="string" default="last">
  Aggregation method. Options: `last`, `avg`, `min`, `max`, `sum`, `mean`,
  `percentile`, `l2norm`, `area`.
</ParamField>

<ParamField path="provider.intervalSeconds" type="integer" default="300">
  Time window in seconds for the query. Determines how far back to look for
  metric data.
</ParamField>

<ParamField path="provider.site" type="string" default="datadoghq.com">
  Datadog site URL. Options: `datadoghq.com`, `datadoghq.eu`,
  `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`.
</ParamField>

## Aggregator Options

The `aggregator` property specifies how to aggregate metric values:

* `last` (default) - Most recent value
* `avg` - Average value
* `min` - Minimum value
* `max` - Maximum value
* `sum` - Sum of values
* `mean` - Mean value
* `percentile` - Percentile value
* `l2norm` - L2 norm
* `area` - Area under the curve

## Supported Sites

* `datadoghq.com` (US1 - default)
* `datadoghq.eu` (EU)
* `us3.datadoghq.com` (US3)
* `us5.datadoghq.com` (US5)
* `ap1.datadoghq.com` (AP1)

## Response Data Available in CEL

The Datadog provider makes the following data available in your CEL success
conditions:

| Field                   | Type              | Description                               |
| ----------------------- | ----------------- | ----------------------------------------- |
| `result.ok`             | boolean           | `true` if API call succeeded (2xx status) |
| `result.statusCode`     | integer           | HTTP status code from Datadog API         |
| `result.queries.<name>` | float64 (or null) | Value for each named query                |
| `result.json`           | object            | Full Datadog API response                 |
| `result.body`           | string            | Raw response body                         |
| `result.duration`       | integer           | Request duration in milliseconds          |

### Accessing Query Values

Since queries are named, you access them by name in your success condition:

```yaml theme={null}
queries:
  cpu: "avg:kubernetes.cpu.usage{cluster:prod}"
  memory: "avg:kubernetes.memory.usage{cluster:prod}"
successCondition: result.queries.cpu < 80 && result.queries.memory < 90
```

## Example Configurations

### Single Query

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    error_rate: "sum:requests.error.rate{service:api-service}"
successCondition: result.queries.error_rate < 0.01
```

### Multiple Queries with Formula

Use multiple queries with a formula to calculate ratios or complex metrics:

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:requests.errors{service:api}"
    total: "sum:requests.total{service:api}"
  formula: "errors / total * 100"
successCondition: result.queries.errors < 1
```

### With Environment and Resource Tags

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  site: us5.datadoghq.com
  intervalSeconds: 600
  queries:
    cpu: "avg:kubernetes.cpu.usage{cluster:{{.resource.name}},env:{{.environment.name}}}"
successCondition:
  result.ok && result.queries.cpu != null && result.queries.cpu < 80
```

### Latency Percentile

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  aggregator: percentile
  queries:
    p99: "trace.http.request.duration.by.service.99p{service:api-service}"
successCondition: result.queries.p99 < 200
```

## Example Success Conditions

```yaml theme={null}
# Simple threshold check
successCondition: result.queries.value < 0.01

# Check if value exists and is under threshold
successCondition: result.ok && result.queries.cpu != null && result.queries.cpu < 80

# Multiple query conditions
successCondition: result.queries.errors < 10 && result.queries.latency < 500

# Check API success first
successCondition: result.ok && result.statusCode == 200 && result.queries.rate < 0.1
```

## Template Variables

The Datadog provider supports Go templates in the `apiKey`, `appKey`, `site`,
`formula`, and `queries` fields:

```yaml theme={null}
# Resource information
{{.resource.name}}
{{.resource.identifier}}
{{.resource.kind}}

# Environment information
{{.environment.name}}
{{.environment.id}}

# Deployment information
{{.deployment.name}}
{{.deployment.slug}}

# Version information
{{.version.tag}}
{{.version.id}}

# Custom variables (from deployment variables)
{{.variables.dd_api_key}}
{{.variables.dd_app_key}}
```

## Storing Secrets in Variables

For sensitive values like API keys, use deployment variables:

1. **Create deployment variables** for your Datadog credentials
2. **Reference them in the provider configuration** using template syntax

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:errors{service:{{.resource.name}}}"
```

## Finding Your Datadog Keys

### API Key

1. Go to Datadog console → **Organization Settings** → **API Keys**
2. Create or copy an existing API key (40-character hex string)

### Application Key

1. Go to Datadog console → **Organization Settings** → **Application Keys**
2. Create or copy an existing Application key
3. **Important**: Use the actual key value, NOT the "Key ID" (which is a UUID)

## Best Practices

* **Use deployment variables** for API keys and application keys - never
  hardcode credentials
* **Name queries descriptively** to make success conditions readable
* **Use appropriate intervals** - shorter intervals provide faster feedback but
  may have less data
* **Tag your metrics** with service, environment, and version information for
  better filtering
* **Test queries manually** in Datadog before using them in verification
* **Handle missing data** by checking `result.ok` and
  `result.queries.<name> != null`
* **Use formulas** for calculated metrics like error rates or ratios