# System Overview
Source: https://docs.ctrlplane.dev/architecture/overview

Birds-eye view of the ctrlplane orchestration flow

This is the developer-facing entry point to the ctrlplane codebase. It shows
how the apps in this monorepo fit together when a deployment version moves
from creation to execution.

```mermaid theme={null}
flowchart TD
    CLI["CLI / curl"]:::ext
    Users["Users<br/>(browser)"]:::ext
    Web["apps/web<br/><i>React + tRPC client</i>"]
    API["apps/api<br/><i>Express + tRPC + webhooks</i>"]
    DB[("Postgres<br/><b>reconcile_work_scope</b>")]
    Engine["apps/workspace-engine<br/><i>Go controllers</i>"]
    Agents["Job agents<br/>GitHub Actions · ArgoCD ·<br/>Terraform Cloud · custom"]:::ext

    Users --> Web
    Web -->|tRPC| API
    CLI -->|"① register version"| API
    API -->|"② enqueue work"| DB
    DB <-->|"③ lease / requeue"| Engine
    Engine -->|"④ dispatch job"| Agents
    Agents -->|"⑤ result"| API
    API -->|"⑥ enqueue follow-up"| DB

    classDef ext fill:#444,stroke:#888,color:#ddd
```

## The orchestration loop

CLI or `curl` calls register a deployment version against `apps/api` (①). The
api persists the version and writes a work item into the `reconcile_work_scope`
table in Postgres (②) — **this is the only thing the api does to "start"
orchestration; it does not call the engine.**

`apps/workspace-engine` controllers continuously lease items from that queue
(③), and each controller's output enqueues work for the next controller
(planning → policy → dispatch). When dispatch fires, the engine reaches out to
a job agent over HTTPS (④).

Results come back through webhooks to the api (⑤), which writes the job update
plus any follow-up work into the queue (⑥). The engine picks it up again. The
loop ③↔⑥ is the whole orchestration model — every release phase is a trip
through the queue.


# Workspace Engine
Source: https://docs.ctrlplane.dev/architecture/workspace-engine

How apps/workspace-engine orchestrates the release lifecycle

The workspace-engine is the Go service that drives every release forward. It
polls a Postgres work queue (`reconcile_work_scope`), leases items by `kind`,
and runs the matching controller. Each controller's output is enqueueing more
work, so a single release moves through phases by chaining items through the
queue.

## The release-flow chain

When a release-target needs to be evaluated (a new version was created, a
policy changed, a job finished, a resource started matching), a
`desired-release` work item lands in the queue. From there:

```mermaid theme={null}
sequenceDiagram
    autonumber
    participant Q as reconcile_work_scope
    participant DR as desiredrelease
    participant JE as jobeligibility
    participant JD as jobdispatch
    participant JV as jobverificationmetric
    participant Ext as Job agent

    Note over Q: kind = desired-release<br/>scope = release-target
    Q->>DR: lease
    Note over DR: evaluate policies, pick the<br/>deployable version, resolve<br/>variables, persist release
    DR-->>Q: enqueue kind=job-eligibility
    Q->>JE: lease
    Note over JE: can this release run now?<br/>(concurrency, retry rules)
    JE-->>Q: enqueue kind=job-dispatch
    Q->>JD: lease
    Note over JD: create job, route to the<br/>right job agent
    JD->>Ext: dispatch
    Ext-->>Q: result via api, enqueue kind=job-verification-metric
    Q->>JV: lease
    Note over JV: poll metrics, on completion<br/>re-enqueue desired-release
    JV-->>Q: enqueue kind=desired-release (loop)
```

Four controllers, one queue between them. **No controller calls another
directly** — handoff is always via insert-then-lease. That means each phase is
independently retriable, leasable, and observable, and the engine can run as
multiple instances safely.

## How every controller works

Every controller is a `reconcile.Processor` registered for one `kind`. The
pattern is identical across all of them: lease an event, recompute the desired
state from current Postgres state, persist the result, enqueue follow-up.

```mermaid theme={null}
flowchart LR
    DB[(reconcile_work_scope)]
    C[Controller<br/>handles one kind]
    DB -->|lease event by kind| C
    C -->|persist results<br/>+ enqueue next kind| DB
```

Two things make this a reconciler rather than a job runner. First,
**controllers are stateless** — every invocation re-reads input from Postgres
rather than carrying state forward in memory. If the world changes between
events (a policy is disabled, an approval lands, a new version appears), the
next event picks up the change automatically. Second, **the loop closes back
to the start** — when a job finishes, `jobverificationmetric` enqueues another
`desired-release` event and `desiredrelease` recomputes from scratch.
Idempotent recomputation is the orchestration model.

## Inside `desiredrelease`

`desiredrelease` is the only controller in the chain that does meaningful
internal work — the other three are mostly routing or checking. Here is what
happens on a single lease:

```mermaid theme={null}
flowchart TD
    In([dequeued: desired-release work item])
    LP[load scope and policies]
    Iter[iterate candidate versions<br/>newest-first]
    Eval[evaluate policy rules<br/>inline via policyeval library]
    Decide{any version passes?}
    NoRel[persist 'no release']
    Resolve[resolve variables]
    Persist[persist release record]
    Out[enqueue job-eligibility]

    In --> LP --> Iter --> Eval --> Decide
    Decide -->|no| NoRel
    Decide -->|yes| Resolve --> Persist --> Out
```

Two things worth knowing:

1. **Policy evaluation is inline, not a separate controller.** A `policyeval`
   directory exists at `svc/controllers/policyeval/` but that's a different
   controller that writes per-version rule evaluations for the UI. The gating
   logic that decides whether a version can deploy lives in the `policyeval`
   *library subpackage* at `svc/controllers/desiredrelease/policyeval/` and is
   called as a function from inside `desiredrelease`.
2. **Versions are evaluated newest-first as a stream.** The controller doesn't
   load all candidate versions then filter — it iterates them and stops at the
   first one that passes all policy rules. That's what makes "skip blocked
   versions but deploy the newest passing one" cheap.

## Other release-flow controllers

**`jobeligibility`** — given a release record, decides whether a job can run
*right now*. Runs two evaluators: `releasetargetconcurrency` (under the
configured concurrency cap?) and `retry` (under the retry budget?). If both
pass, enqueue `job-dispatch`. If not, requeue with `notBefore`.

**`jobdispatch`** — given a job, picks the right job-agent adapter (GitHub
Actions, ArgoCD, Terraform Cloud, Argo Workflows, or the test runner) and
sends the job over HTTPS. The agent's `externalId` is recorded so results can
be correlated back later.

**`jobverificationmetric`** — given a finished job, polls verification
providers (Datadog, HTTP probes, Terraform Cloud run status, etc.) until they
return pass/fail. On completion, calls `EnqueueDesiredRelease` to close the
loop.

## Controllers outside the release-flow chain

The `svc/controllers/` directory contains several other controllers that exist
for UI surface or precomputed state, not for moving a release through phases:

* `policyeval` (top-level) — computes per-version rule evaluations so the UI
  can show "why isn't this version deploying yet."
* `deploymentplan` / `deploymentplanresult` — power plan previews and dry-run
  views.
* `deploymentresourceselectoreval` / `environmentresourceselectoreval` —
  precompute which resources currently match a deployment or environment
  selector.
* `relationshipeval` — evaluates resource relationship rules into the resource
  graph.
* `forcedeploy` — handles user-triggered manual deploys (a separate path from
  the policy-gated chain).

If you're trying to understand "what happens when I push a version," you can
safely ignore these and focus on the four chain controllers.


# ctrlc api upsert version
Source: https://docs.ctrlplane.dev/cli/api-upsert-version

Create or update a deployment version

The `ctrlc api upsert version` command creates or updates a deployment version
in Ctrlplane. This is typically called from CI/CD pipelines after a successful
build.

## Usage

```bash theme={null}
ctrlc api upsert version \
  --workspace <workspace> \
  --deployment <deployment-id> \
  --tag <version-tag> \
  [--name "<version-name>"] \
  [--metadata key=value ...]
```

## Flags

| Flag           | Required | Description                     |
| -------------- | -------- | ------------------------------- |
| `--workspace`  | Yes      | Workspace name or ID            |
| `--deployment` | Yes      | Deployment ID                   |
| `--tag`        | Yes      | Unique version identifier       |
| `--name`       | No       | Human-readable version name     |
| `--metadata`   | No       | Key-value metadata (repeatable) |

## Example

```bash theme={null}
ctrlc api upsert version \
  --workspace my-workspace \
  --deployment dep_abc123 \
  --tag v1.2.3 \
  --name "Release 1.2.3" \
  --metadata git/commit=abc123 \
  --metadata git/branch=main
```

The command is idempotent — running it again with the same `--tag` updates the
existing version rather than creating a duplicate.

## Next Steps

<CardGroup>
  <Card title="CI/CD Integration" icon="rotate" href="/integrations/cicd">
    Full CI/CD setup guide
  </Card>

  <Card title="CLI Overview" icon="terminal" href="/cli/overview">
    All CLI commands
  </Card>
</CardGroup>


# ctrlc apply
Source: https://docs.ctrlplane.dev/cli/apply

Apply resource definitions from YAML files

The `ctrlc apply` command creates or updates resources in Ctrlplane from YAML
definition files.

## Usage

```bash theme={null}
ctrlc apply -f <file>
```

## Flags

| Flag     | Short | Required | Description                               |
| -------- | ----- | -------- | ----------------------------------------- |
| `--file` | `-f`  | Yes      | Path to the YAML resource definition file |

## Example

```bash theme={null}
ctrlc apply -f resource.yaml
```

## Next Steps

<CardGroup>
  <Card title="CLI Overview" icon="terminal" href="/cli/overview">
    All CLI commands
  </Card>

  <Card title="Resources" icon="cube" href="/concepts/resources">
    Learn about resources
  </Card>
</CardGroup>


# CLI Reference
Source: https://docs.ctrlplane.dev/cli/overview

Install and use the ctrlc command-line interface

The `ctrlc` CLI is the primary tool for interacting with Ctrlplane from your
terminal, CI/CD pipelines, and automation scripts. It supports syncing
resources, managing deployments, and applying configuration.

## Installation

<CodeGroup>
  ```bash brew theme={null}
  brew tap ctrlplanedev/tap
  brew install ctrlplanedev/tap/ctrlc
  ```

  ```bash npm theme={null}
  npm install -g @ctrlplane/cli
  ```

  ```bash curl theme={null}
  curl -fsSL https://get.ctrlplane.dev | sh
  ```
</CodeGroup>

## Authentication

Set your API key and workspace as environment variables or pass them as flags:

```bash theme={null}
# Environment variables (recommended)
export CTRLPLANE_API_KEY="your-api-key"
export CTRLPLANE_WORKSPACE="your-workspace-id"
export CTRLPLANE_URL="https://your-ctrlplane-instance.com"

# Or pass as flags
ctrlc --api-key "your-api-key" --workspace "your-workspace-id" <command>
```

## Commands

### `ctrlc sync`

Sync infrastructure resources into Ctrlplane's inventory. Each subcommand
targets a specific provider or input method.

| Subcommand                                                           | Description                       |
| -------------------------------------------------------------------- | --------------------------------- |
| [`sync pipe`](/cli/sync-pipe)                                        | Read resources from stdin (JSON)  |
| [`sync kubernetes`](/integrations/resource-providers/kubernetes)     | Sync Kubernetes cluster resources |
| [`sync aws`](/integrations/resource-providers/aws)                   | Sync AWS resources                |
| [`sync google-cloud`](/integrations/resource-providers/google-cloud) | Sync Google Cloud resources       |
| [`sync azure`](/integrations/resource-providers/azure)               | Sync Azure resources              |
| [`sync terraform`](/integrations/resource-providers/terraform)       | Sync Terraform state resources    |
| [`sync helm`](/integrations/resource-providers/helm)                 | Sync Helm releases                |
| [`sync github`](/integrations/resource-providers/github)             | Sync GitHub repositories          |
| [`sync vcluster`](/integrations/resource-providers/vcluster)         | Sync virtual clusters             |

### `ctrlc api`

Interact directly with the Ctrlplane API.

| Subcommand           | Description                           |
| -------------------- | ------------------------------------- |
| `api upsert version` | Create or update a deployment version |

See [CI/CD Integration](/integrations/cicd) for detailed usage.

### `ctrlc apply`

Apply resource definitions from YAML files.

```bash theme={null}
ctrlc apply -f resource.yaml
```

## Global Flags

| Flag          | Environment Variable  | Description                |
| ------------- | --------------------- | -------------------------- |
| `--api-key`   | `CTRLPLANE_API_KEY`   | API key for authentication |
| `--workspace` | `CTRLPLANE_WORKSPACE` | Workspace name or ID       |
| `--url`       | `CTRLPLANE_URL`       | Ctrlplane API URL          |

## Next Steps

<CardGroup>
  <Card title="Sync Pipe" icon="pipe-section" href="/cli/sync-pipe">
    Sync resources from stdin
  </Card>

  <Card title="CI/CD Integration" icon="rotate" href="/integrations/cicd">
    Use the CLI in CI/CD pipelines
  </Card>

  <Card title="Resource Providers" icon="database" href="/integrations/resource-providers/overview">
    Built-in resource providers
  </Card>

  <Card title="Custom Provider" icon="code" href="/integrations/resource-providers/custom">
    Build your own provider
  </Card>
</CardGroup>


# ctrlc sync
Source: https://docs.ctrlplane.dev/cli/sync

Sync infrastructure resources into Ctrlplane's inventory

The `ctrlc sync` command discovers and syncs infrastructure resources into
Ctrlplane. Each subcommand targets a specific provider or input method.

## Usage

```bash theme={null}
ctrlc sync <subcommand> [flags]
```

## Subcommands

| Subcommand                                                      | Description                         |
| --------------------------------------------------------------- | ----------------------------------- |
| [`pipe`](/cli/sync-pipe)                                        | Read resources from stdin as JSON   |
| [`kubernetes`](/integrations/resource-providers/kubernetes)     | Sync Kubernetes cluster resources   |
| [`aws`](/integrations/resource-providers/aws)                   | Sync AWS resources (EKS, ECS, etc.) |
| [`google-cloud`](/integrations/resource-providers/google-cloud) | Sync Google Cloud resources         |
| [`azure`](/integrations/resource-providers/azure)               | Sync Azure resources                |
| [`terraform`](/integrations/resource-providers/terraform)       | Sync Terraform state resources      |
| [`helm`](/integrations/resource-providers/helm)                 | Sync Helm releases                  |
| [`github`](/integrations/resource-providers/github)             | Sync GitHub repositories            |
| [`vcluster`](/integrations/resource-providers/vcluster)         | Sync virtual clusters               |

## Common Flags

Most sync subcommands support these flags:

| Flag         | Description                                        |
| ------------ | -------------------------------------------------- |
| `--provider` | Resource provider name                             |
| `--interval` | Run continuously on an interval (e.g., `5m`, `1h`) |

## Examples

```bash theme={null}
# One-time sync
ctrlc sync kubernetes --cluster-name prod --cluster-identifier prod-us-east-1

# Continuous sync every 5 minutes
ctrlc sync kubernetes --cluster-name prod --cluster-identifier prod-us-east-1 --interval 5m

# Pipe custom resources from stdin
./discover.sh | ctrlc sync pipe --provider "custom"
```

## Next Steps

<CardGroup>
  <Card title="Sync Pipe" icon="pipe-section" href="/cli/sync-pipe">
    Sync from stdin
  </Card>

  <Card title="Resource Providers" icon="database" href="/integrations/resource-providers/overview">
    All resource providers
  </Card>
</CardGroup>


# ctrlc sync pipe
Source: https://docs.ctrlplane.dev/cli/sync-pipe

Sync resources into Ctrlplane by piping JSON through stdin

The `pipe` subcommand reads JSON resource data from stdin and upserts it into
Ctrlplane via a resource provider. This is the most flexible sync method — use
it to integrate any data source, script output, or API response without writing
a dedicated provider.

## Usage

```bash theme={null}
<source> | ctrlc sync pipe --provider <provider-name>
```

The command reads from stdin, parses the JSON into one or more resources, and
upserts them under the specified resource provider.

## Flags

| Flag         | Short | Required | Description                                                        |
| ------------ | ----- | -------- | ------------------------------------------------------------------ |
| `--provider` | `-p`  | Yes      | Resource provider name. Created automatically if it doesn't exist. |

[Global flags](/cli/overview#global-flags) (`--api-key`, `--workspace`, `--url`)
are also supported.

## Input Format

`ctrlc sync pipe` accepts JSON in two forms:

### Array of Resources

```json theme={null}
[
  {
    "name": "web-1",
    "identifier": "web-1-prod",
    "version": "custom/v1",
    "kind": "Server",
    "config": {},
    "metadata": {}
  },
  {
    "name": "web-2",
    "identifier": "web-2-prod",
    "version": "custom/v1",
    "kind": "Server",
    "config": {},
    "metadata": {}
  }
]
```

### Single Resource Object

A single object (without the array wrapper) is also accepted and is
automatically normalized to a one-element array:

```json theme={null}
{
  "name": "web-1",
  "identifier": "web-1-prod",
  "version": "custom/v1",
  "kind": "Server",
  "config": {},
  "metadata": {}
}
```

### Required Fields

Every resource must include these fields:

| Field        | Type   | Description                                           |
| ------------ | ------ | ----------------------------------------------------- |
| `name`       | string | Human-readable display name                           |
| `identifier` | string | Unique identifier for the resource                    |
| `version`    | string | Resource version or schema version                    |
| `kind`       | string | Resource type (e.g., `Server`, `Database/PostgreSQL`) |

### Optional Fields

| Field      | Type   | Description                                                  |
| ---------- | ------ | ------------------------------------------------------------ |
| `metadata` | object | Key-value pairs used for environment selectors and filtering |
| `config`   | object | Configuration data passed to job agents during deployment    |

For details on the distinction between metadata and config, see
[Resource Schema](/integrations/resource-providers/overview#resource-schema).

## Examples

### Pipe from a Discovery Script

Run a custom script that outputs JSON and pipe it directly:

```bash theme={null}
./discover-databases.sh | ctrlc sync pipe --provider "custom-db"
```

### Inline JSON

Quick one-liner to register resources:

```bash theme={null}
echo '[{"name":"web-1","identifier":"web-1-prod","version":"custom/v1","kind":"Server","config":{},"metadata":{}}]' \
  | ctrlc sync pipe --provider "my-servers"
```

### Single Resource

No array wrapper needed for a single resource:

```bash theme={null}
echo '{"name":"web-1","identifier":"web-1-prod","version":"custom/v1","kind":"Server"}' \
  | ctrlc sync pipe --provider "my-servers"
```

### Transform API Responses with jq

Fetch data from an internal API and reshape it into the expected schema:

```bash theme={null}
curl -s https://cmdb.internal/api/servers \
  | jq '[.[] | {
      name,
      identifier: .id,
      version: "cmdb/v1",
      kind: "Server",
      config: .,
      metadata: {}
    }]' \
  | ctrlc sync pipe --provider "cmdb"
```

### Sync from a CMDB with Metadata

Include metadata so resources can be targeted by environment selectors:

```bash theme={null}
curl -s https://cmdb.internal/api/servers \
  | jq '[.[] | {
      name: .hostname,
      identifier: .asset_id,
      version: "cmdb/v1",
      kind: "Server/Linux",
      config: {
        host: .ip_address,
        port: .ssh_port
      },
      metadata: {
        environment: .env,
        region: .datacenter,
        team: .owner_team,
        tier: .sla_tier
      }
    }]' \
  | ctrlc sync pipe --provider "cmdb-servers"
```

### Database Inventory from PostgreSQL

Query a database and pipe the results:

```bash theme={null}
psql -h localhost -U admin -d inventory -t -A -c "
  SELECT json_agg(json_build_object(
    'name', hostname,
    'identifier', instance_id,
    'version', 'inventory/v1',
    'kind', 'Database/' || engine,
    'metadata', json_build_object('environment', env, 'region', region),
    'config', json_build_object('host', endpoint, 'port', port)
  )) FROM databases;
" | ctrlc sync pipe --provider "db-inventory"
```

### Running on a Schedule with cron

Sync resources every 5 minutes:

```bash theme={null}
# crontab -e
*/5 * * * * /usr/local/bin/discover-servers.sh | /usr/local/bin/ctrlc sync pipe --provider "cron-servers" 2>&1 >> /var/log/ctrlc-sync.log
```

### Running in CI/CD

Use `sync pipe` in a GitHub Actions workflow to register build artifacts as
resources:

```yaml theme={null}
name: Sync Resources

on:
  schedule:
    - cron: "*/10 * * * *"
  workflow_dispatch:

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install ctrlc
        run: curl -fsSL https://get.ctrlplane.dev | sh

      - name: Discover and sync
        env:
          CTRLPLANE_API_KEY: ${{ secrets.CTRLPLANE_API_KEY }}
          CTRLPLANE_WORKSPACE: ${{ vars.CTRLPLANE_WORKSPACE }}
        run: |
          ./scripts/discover-resources.sh \
            | ctrlc sync pipe --provider "ci-discovered"
```

## Behavior

* **Provider auto-creation** — If the named provider doesn't exist, it is
  created automatically.
* **Upsert semantics** — Resources are matched by `identifier`. Existing
  resources are updated; new ones are created.
* **Stdin required** — The command exits with an error if no piped input is
  detected or if stdin is empty.
* **Validation** — Each resource is validated for the required fields (`name`,
  `identifier`, `version`, `kind`) before the API call. Missing fields produce a
  descriptive error message.

## Error Handling

| Error                       | Cause                                             | Fix                                                                  |
| --------------------------- | ------------------------------------------------- | -------------------------------------------------------------------- |
| `no piped input detected`   | Command was run interactively without piped input | Pipe JSON data to the command                                        |
| `stdin is empty`            | Piped input contained no data                     | Ensure the upstream command produces output                          |
| `invalid JSON input`        | Input is not valid JSON                           | Check the JSON syntax; a snippet of the input is shown for debugging |
| `missing required field(s)` | One or more resources are missing required fields | Add the missing fields to each resource object                       |

## Best Practices

### Use Stable Identifiers

Choose identifiers that won't change across syncs:

```json theme={null}
{ "identifier": "db-prod-primary" }
```

Avoid identifiers derived from volatile attributes like IP addresses.

### Include Rich Metadata

Metadata powers Ctrlplane's environment selectors. Include attributes that are
useful for targeting:

```json theme={null}
{
  "metadata": {
    "environment": "production",
    "region": "us-east-1",
    "team": "platform",
    "tier": "critical"
  }
}
```

### Follow the Kind Naming Convention

Use a `Category/Type` format for consistency:

```json theme={null}
{ "kind": "Server/Linux" }
{ "kind": "Database/PostgreSQL" }
{ "kind": "Cache/Redis" }
```

### Validate Before Syncing

Pipe through `jq` to catch malformed JSON early:

```bash theme={null}
./discover.sh | jq '.' | ctrlc sync pipe --provider "validated"
```

## Next Steps

<CardGroup>
  <Card title="Resource Providers" icon="database" href="/integrations/resource-providers/overview">
    Overview of all resource providers
  </Card>

  <Card title="Custom Provider" icon="code" href="/integrations/resource-providers/custom">
    Build a custom provider via API or SDK
  </Card>

  <Card title="Selectors" icon="filter" href="/concepts/selectors">
    Target resources with environment selectors
  </Card>

  <Card title="Environments" icon="layer-group" href="/concepts/environments">
    Create dynamic environments
  </Card>
</CardGroup>


# Ctrlplane vs Alternatives
Source: https://docs.ctrlplane.dev/comparisons

How Ctrlplane compares to other deployment and orchestration tools

Ctrlplane is often compared to CI/CD tools, GitOps engines, and deployment
platforms. This page clarifies where Ctrlplane fits and how it differs from
alternatives.

## The Key Difference

**Ctrlplane is an orchestration layer, not an execution layer.**

It doesn't build your code (that's CI). It doesn't apply manifests to clusters
(that's ArgoCD/Flux/kubectl). It decides *when* and *where* deployments should
happen, enforces policies, and coordinates the flow across environments.

## Ctrlplane vs ArgoCD

| Aspect              | ArgoCD                                         | Ctrlplane                                                |
| ------------------- | ---------------------------------------------- | -------------------------------------------------------- |
| **Primary purpose** | GitOps continuous delivery for Kubernetes      | Deployment orchestration across environments             |
| **What it does**    | Syncs Applications to K8s clusters             | Coordinates when/where deployments happen                |
| **Scope**           | Single cluster (or multi with ApplicationSets) | Multi-cluster, multi-environment, multi-region           |
| **Policies**        | Limited (sync waves, hooks)                    | Rich policy engine (approval, verification, progression) |
| **Verification**    | Health checks on K8s resources                 | External verification (Datadog, Prometheus, HTTP)        |
| **Inventory**       | No unified inventory                           | Centralized resource inventory                           |

**When to use together**: Ctrlplane orchestrates *when* to deploy to each
cluster; ArgoCD *executes* the deployment. Ctrlplane has a native ArgoCD job
agent that creates/syncs Applications.

**Example flow**:

1. CI creates a Version in Ctrlplane
2. Ctrlplane evaluates policies (approval needed for prod)
3. After approval, Ctrlplane tells ArgoCD to sync the Application
4. ArgoCD applies manifests to the cluster
5. Ctrlplane runs verification (checks Datadog metrics)
6. If verification passes, Ctrlplane promotes to the next environment

## Ctrlplane vs Flux

| Aspect              | Flux                                  | Ctrlplane                                    |
| ------------------- | ------------------------------------- | -------------------------------------------- |
| **Primary purpose** | GitOps toolkit for Kubernetes         | Deployment orchestration across environments |
| **What it does**    | Reconciles Git state to cluster state | Coordinates multi-environment rollouts       |
| **Multi-cluster**   | Via Flux controllers per cluster      | Centralized orchestration of all clusters    |
| **Policies**        | Kustomize overlays, dependencies      | Approval, verification, gradual rollout      |

**When to use together**: Flux handles the GitOps reconciliation; Ctrlplane
handles the higher-level orchestration of when each environment should receive
updates.

## Ctrlplane vs Spinnaker

| Aspect              | Spinnaker                                    | Ctrlplane                               |
| ------------------- | -------------------------------------------- | --------------------------------------- |
| **Primary purpose** | Multi-cloud continuous delivery              | Deployment orchestration with inventory |
| **Complexity**      | Complex, requires significant infrastructure | Lightweight, single binary or container |
| **Pipeline model**  | Visual pipeline builder                      | Policy-based (selectors + rules)        |
| **Cloud support**   | Deep cloud provider integrations             | Provider-agnostic via job agents        |
| **Inventory**       | Limited                                      | First-class resource inventory          |

**When to choose Ctrlplane**: You want simpler operations, policy-based
orchestration rather than complex pipelines, and a unified inventory.

**When to choose Spinnaker**: You need deep cloud provider integrations (AWS
CodeDeploy, GCP, etc.) and prefer visual pipeline building.

## Ctrlplane vs GitHub Actions (alone)

| Aspect           | GitHub Actions                    | Ctrlplane + GitHub Actions                     |
| ---------------- | --------------------------------- | ---------------------------------------------- |
| **Multi-env**    | Separate workflows or matrix jobs | Automatic fan-out to all matching resources    |
| **Approvals**    | Environment protection rules      | Flexible approval policies with selectors      |
| **Verification** | Custom scripts in workflow        | Built-in verification with Datadog, Prometheus |
| **Visibility**   | Per-workflow run logs             | Centralized view of all deployments            |
| **Rollback**     | Manual or custom scripting        | Automatic rollback on verification failure     |

**When to use together**: GitHub Actions builds your code and creates Versions.
Ctrlplane orchestrates the rollout. GitHub Actions can also be a job agent that
executes deployments triggered by Ctrlplane.

## Ctrlplane vs Harness / Octopus Deploy

| Aspect          | Harness / Octopus                        | Ctrlplane                               |
| --------------- | ---------------------------------------- | --------------------------------------- |
| **Model**       | Enterprise CD platforms with pipelines   | Open-source orchestration with policies |
| **Pricing**     | Commercial, often per-deployment pricing | Open-source (free) + cloud offering     |
| **Flexibility** | Opinionated pipeline structure           | Flexible policy engine                  |
| **Inventory**   | Varies                                   | First-class resource inventory          |

**When to choose Ctrlplane**: You prefer open-source, policy-based
orchestration, and want to avoid vendor lock-in.

## Ctrlplane vs Terraform Cloud

| Aspect              | Terraform Cloud                            | Ctrlplane                                       |
| ------------------- | ------------------------------------------ | ----------------------------------------------- |
| **Primary purpose** | Infrastructure provisioning                | Application deployment orchestration            |
| **What it manages** | Cloud resources (VMs, networks, databases) | Application releases to existing infrastructure |
| **Workspace model** | One workspace per environment/component    | Dynamic release targets from inventory          |

**When to use together**: Terraform Cloud provisions infrastructure; Ctrlplane
orchestrates application deployments to that infrastructure. Ctrlplane has a
native Terraform Cloud job agent.

## Summary: When to Use Ctrlplane

Use Ctrlplane when you need:

| Need                                       | Ctrlplane Helps By                                |
| ------------------------------------------ | ------------------------------------------------- |
| Coordinated multi-environment rollouts     | Policy-based environment progression              |
| Approval workflows for production          | Flexible approval policies with selectors         |
| Verification before promoting releases     | Built-in Datadog, Prometheus, HTTP verification   |
| Visibility into what's deployed where      | Centralized resource inventory                    |
| Gradual rollouts across many clusters      | Gradual rollout policies with configurable timing |
| Consistent deployment process across teams | Centralized policy engine                         |

## Integration Architecture

Here's how Ctrlplane typically fits with other tools:

```mermaid theme={null}
flowchart TB
    subgraph CI["CI/CD"]
        GHA["GitHub Actions"]
        GitLab["GitLab CI"]
        Jenkins["Jenkins"]
    end

    subgraph Ctrl["Ctrlplane"]
        Orch["Orchestration"]
        Inv["Inventory"]
        Pol["Policies"]
    end

    subgraph Exec["Execution"]
        Argo["ArgoCD"]
        TFC["Terraform Cloud"]
        K8s["Kubernetes Jobs"]
    end

    subgraph Monitor["Monitoring"]
        DD["Datadog"]
        Prom["Prometheus"]
    end

    CI -->|"create version"| Ctrl
    Ctrl -->|"dispatch jobs"| Exec
    Monitor -->|"verification"| Ctrl
```

## Next Steps

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="./quickstart">
    Set up your first deployment pipeline
  </Card>

  <Card title="How It Works" icon="diagram-project" href="./how-it-works">
    Understand the architecture
  </Card>
</CardGroup>


# Deployments
Source: https://docs.ctrlplane.dev/concepts/deployments

Define what to deploy and how — services, versions, and execution

A **Deployment** defines **what** you want to deploy and **how** it gets
deployed. It represents a service or application along with its job agent
configuration — connecting the *what* (your service and its versions) with the
*how* (the agent and workflow that executes the deployment).

<Tip>
  In Ctrlplane's mental model: **Deployments** = what & how, [**Environments**](./environments) = where, [**Policies**](../policies/overview) = when.
</Tip>

## What is a Deployment?

A deployment is a logical unit of software that you want to orchestrate:

* **API Service** - Your backend API
* **Frontend Application** - Web or mobile frontend
* **Background Worker** - Async job processor
* **Database Migration** - Schema changes
* **Configuration Update** - Infrastructure configuration

Each deployment can have multiple **versions** (builds, releases) that get deployed to different environments and resources.

## Deployment Properties

| Property           | Type   | Required | Description                                             |
| ------------------ | ------ | -------- | ------------------------------------------------------- |
| `id`               | string | Auto     | Unique identifier                                       |
| `name`             | string | Yes      | Human-readable display name                             |
| `slug`             | string | Yes      | URL-friendly identifier, unique within the workspace    |
| `description`      | string | No       | What this deployment does                               |
| `resourceSelector` | string | No       | CEL expression limiting which resources can be targeted |
| `jobAgents`        | array  | No       | Job agent configurations (see below)                    |
| `metadata`         | object | No       | Key-value pairs for classification                      |

### name

Human-readable display name for the deployment.

**Examples**: "API Service", "Frontend Application", "Payment Processor"

### slug

URL-friendly identifier, unique within the workspace. In the Terraform provider,
the slug is auto-generated from the `name` field.

**Examples**: `api-service`, `frontend-app`, `payment-processor`

### resourceSelector

A CEL expression that limits which resources this deployment can target. If
specified, only resources matching this expression will have release targets
created.

**Examples**:

```
resource.kind == "Kubernetes"
```

```
resource.version == 'ctrlplane.dev/kubernetes/cluster/v1' && resource.metadata['kubernetes/status'] == 'running'
```

```
resource.metadata['region'] == 'us-east-1'
```

If not specified, the deployment can target any resource in its environments.

### jobAgents

An array of job agent configurations that execute deployment jobs. Each entry
specifies which agent to use and how to configure it, with optional routing
via selectors.

| Field      | Type   | Required | Description                                      |
| ---------- | ------ | -------- | ------------------------------------------------ |
| `ref`      | string | Yes      | Job agent reference identifier                   |
| `config`   | object | Yes      | Agent-specific configuration                     |
| `selector` | string | No       | CEL expression for routing to specific resources |

When multiple job agents are configured, the `selector` field determines which
agent handles which resources. This enables patterns like using ArgoCD for
some clusters and GitHub Actions for others within the same deployment.

### metadata

Optional key-value pairs for classification and policy matching:

```json theme={null}
{
  "team": "backend",
  "language": "nodejs",
  "tier": "critical"
}
```

Metadata can be referenced in policy selectors (e.g.,
`deployment.metadata['tier'] == 'critical'`).

## Creating a Deployment

### Via Terraform

<Tabs>
  <Tab title="GitHub Actions Agent">
    ```hcl theme={null}
    resource "ctrlplane_deployment" "api" {
      name              = "API Service"
      resource_selector = "resource.version == 'ctrlplane.dev/kubernetes/cluster/v1' && resource.metadata['kubernetes/status'] == 'running'"

      metadata = {
        team    = "backend"
        service = "api"
      }

      job_agent {
        id = ctrlplane_job_agent.github.id

        github {
          owner       = "my-org"
          repo        = "api-service"
          workflow_id = 12345678
        }
      }
    }

    resource "ctrlplane_deployment_system_link" "api" {
      deployment_id = ctrlplane_deployment.api.id
      system_id     = ctrlplane_system.main.id
    }
    ```
  </Tab>

  <Tab title="ArgoCD Agent">
    ```hcl theme={null}
    resource "ctrlplane_deployment" "operator" {
      name              = "Weights & Biases Operator"
      resource_selector = "resource.version == 'ctrlplane.dev/kubernetes/cluster/v1' && resource.metadata['kubernetes/status'] == 'running'"
      metadata          = {}

      job_agent {
        id = ctrlplane_job_agent.argocd.id

        argocd {
          template = file("${path.module}/templates/application.yaml")
        }
      }
    }

    resource "ctrlplane_deployment_system_link" "operator" {
      deployment_id = ctrlplane_deployment.operator.id
      system_id     = ctrlplane_system.main.id
    }
    ```
  </Tab>
</Tabs>

<Note>
  Deployments must be linked to a system using `ctrlplane_deployment_system_link`.
  This is a separate resource that associates a deployment with a system.
</Note>

### Via REST API

```bash theme={null}
curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "API Service",
    "slug": "api-service",
    "description": "Main backend API service",
    "resourceSelector": "resource.kind == '\''Kubernetes'\''",
    "jobAgents": [
      {
        "ref": "github-actions-agent",
        "config": {
          "workflow": "deploy.yml",
          "owner": "my-org",
          "repo": "api-service"
        }
      }
    ],
    "metadata": {
      "team": "backend",
      "language": "nodejs"
    }
  }'
```

The API returns `202 Accepted` with the deployment `id`. Reads are available
immediately via `GET /v1/workspaces/{workspaceId}/deployments/{deploymentId}`.

### Via CLI (YAML)

```yaml theme={null}
# deployment.yaml
type: Deployment
name: API Service
slug: api-service
description: Main backend API service
jobAgent:
  ref: github-actions-agent
jobAgentConfig:
  workflow: deploy.yml
  owner: my-org
  repo: api-service
resourceSelector: resource.kind == "Kubernetes"
```

```bash theme={null}
ctrlc apply -f deployment.yaml
```

### Multiple Deployments

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_deployment" "api" {
      name              = "API Service"
      resource_selector = "resource.kind == 'Kubernetes'"
      metadata          = { service = "api" }

      job_agent {
        id = ctrlplane_job_agent.github.id
        github {
          owner       = "my-org"
          repo        = "api-service"
          workflow_id = 12345678
        }
      }
    }

    resource "ctrlplane_deployment" "frontend" {
      name              = "Frontend App"
      resource_selector = "resource.kind == 'Kubernetes'"
      metadata          = { service = "frontend" }

      job_agent {
        id = ctrlplane_job_agent.github.id
        github {
          owner       = "my-org"
          repo        = "frontend-app"
          workflow_id = 87654321
        }
      }
    }

    resource "ctrlplane_deployment_system_link" "api" {
      deployment_id = ctrlplane_deployment.api.id
      system_id     = ctrlplane_system.main.id
    }

    resource "ctrlplane_deployment_system_link" "frontend" {
      deployment_id = ctrlplane_deployment.frontend.id
      system_id     = ctrlplane_system.main.id
    }
    ```
  </Tab>

  <Tab title="CLI">
    ```yaml theme={null}
    # deployments.yaml
    ---
    type: Deployment
    name: API Service
    slug: api-service
    description: Main backend API service
    jobAgent:
      ref: github-actions-agent
    jobAgentConfig:
      workflow: deploy.yml
      owner: my-org
      repo: api-service
    ---
    type: Deployment
    name: Frontend App
    slug: frontend-app
    description: Customer-facing web application
    jobAgent:
      ref: kubernetes-agent
    jobAgentConfig:
      namespace: default
    ```

    ```bash theme={null}
    ctrlc apply -f deployments.yaml
    ```
  </Tab>
</Tabs>

## Job Agent Configuration

Each `job_agent` block (Terraform) or entry in the `jobAgents` array (API)
configures how a specific agent executes deployments. The Terraform provider
supports typed provider blocks for each agent type.

### GitHub Actions

```hcl theme={null}
job_agent {
  id = ctrlplane_job_agent.github.id

  github {
    owner       = "my-org"
    repo        = "api-service"
    workflow_id = 12345678
    ref         = "main"
  }
}
```

| Attribute         | Type   | Description                              |
| ----------------- | ------ | ---------------------------------------- |
| `owner`           | string | GitHub repository owner                  |
| `repo`            | string | GitHub repository name                   |
| `workflow_id`     | int    | GitHub Actions workflow ID               |
| `ref`             | string | Git ref to run on (defaults to `"main"`) |
| `installation_id` | int    | GitHub App installation ID               |

### ArgoCD

```hcl theme={null}
job_agent {
  id = ctrlplane_job_agent.argocd.id

  argocd {
    template   = file("${path.module}/application.yaml")
    server_url = "https://argocd.example.com"
  }
}
```

| Attribute    | Type   | Description                  |
| ------------ | ------ | ---------------------------- |
| `template`   | string | ArgoCD Application template  |
| `server_url` | string | ArgoCD server address        |
| `api_key`    | string | ArgoCD API token (sensitive) |

### Terraform Cloud

```hcl theme={null}
job_agent {
  id = ctrlplane_job_agent.tfc.id

  terraform_cloud {
    organization = "my-org"
    template     = file("${path.module}/workspace.json")
    token        = var.tfc_token
  }
}
```

| Attribute      | Type   | Description                       |
| -------------- | ------ | --------------------------------- |
| `organization` | string | Terraform Cloud organization name |
| `template`     | string | Workspace template                |
| `address`      | string | Terraform Cloud address           |
| `token`        | string | API token (sensitive)             |

### Agent Routing with Selectors

When a deployment uses multiple job agents, use `selector` to route to specific
resources:

```hcl theme={null}
resource "ctrlplane_deployment" "app" {
  name              = "Application"
  resource_selector = "resource.kind == 'Kubernetes'"

  job_agent {
    id       = ctrlplane_job_agent.argocd.id
    selector = "resource.metadata['cluster-type'] == 'managed'"

    argocd {
      template = file("${path.module}/managed-app.yaml")
    }
  }

  job_agent {
    id       = ctrlplane_job_agent.github.id
    selector = "resource.metadata['cluster-type'] == 'self-hosted'"

    github {
      owner       = "my-org"
      repo        = "app-deploy"
      workflow_id = 12345678
    }
  }
}
```

## Deployment Versions

Versions represent specific builds or releases of your deployment. They are
typically created by your CI system after building.

### Version Properties

| Property         | Type   | Required | Description                                      |
| ---------------- | ------ | -------- | ------------------------------------------------ |
| `id`             | string | Auto     | Unique identifier                                |
| `deploymentId`   | string | Auto     | Parent deployment                                |
| `tag`            | string | Yes      | Version tag (e.g., `"v1.2.3"`, git SHA)          |
| `name`           | string | No       | Human-readable name                              |
| `status`         | string | No       | `building`, `ready`, or `failed`                 |
| `config`         | object | No       | General version configuration, visible in UI     |
| `jobAgentConfig` | object | No       | Execution config (overrides deployment's config) |
| `metadata`       | object | No       | Custom key-value pairs                           |
| `createdAt`      | string | Auto     | Creation timestamp                               |

### Creating Versions (from CI)

After your CI builds an artifact, create a version in Ctrlplane:

```yaml theme={null}
# GitHub Actions example
- name: Create Ctrlplane version
  env:
    CTRLPLANE_API_KEY: ${{ secrets.CTRLPLANE_API_KEY }}
  run: |
    ctrlc api upsert version \
      --workspace ${{ vars.CTRLPLANE_WORKSPACE }} \
      --deployment ${{ vars.CTRLPLANE_DEPLOYMENT_ID }} \
      --tag ${{ github.sha }} \
      --name "Build #${{ github.run_number }}" \
      --metadata git/commit=${{ github.sha }} \
      --metadata git/branch=${{ github.ref_name }} \
      --metadata build/number=${{ github.run_number }}
```

Or via the REST API:

```bash theme={null}
curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments/{deploymentId}/versions \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tag": "v1.2.3",
    "name": "Release 1.2.3",
    "status": "ready",
    "config": {
      "buildNumber": "456"
    },
    "jobAgentConfig": {
      "imageTag": "my-org/api-service:v1.2.3"
    },
    "metadata": {
      "git_commit": "abc123def",
      "git_branch": "main"
    }
  }'
```

### Version Status

Versions have a status field:

* **`building`** - Version is being built (won't be deployed yet)
* **`ready`** - Version is ready for deployment (default for policies)
* **`failed`** - Build failed (won't be deployed)

**Workflow**:

1. CI starts building → Create version with status `building`
2. Build succeeds → Update status to `ready`
3. Ctrlplane creates releases/jobs for `ready` versions

Or simpler:

1. Build completes → Create version with status `ready` immediately

### Version Config vs Job Agent Config

**config**: General version configuration, visible in UI

```json theme={null}
{
  "buildNumber": "456",
  "gitCommit": "abc123"
}
```

**jobAgentConfig**: Specific configuration for job execution (merged with the
deployment's job agent config at runtime)

```json theme={null}
{
  "imageTag": "my-org/api-service:v1.2.3",
  "helmValues": {
    "replicas": 3
  }
}
```

## Deployment Variables

Variables allow environment-specific or resource-specific configuration that
gets resolved at job creation time and passed to the job agent.

### Via Terraform

The Terraform provider has dedicated resources for deployment variables and
their values:

```hcl theme={null}
resource "ctrlplane_deployment_variable" "replica_count" {
  deployment_id = ctrlplane_deployment.api.id
  key           = "REPLICA_COUNT"
  description   = "Number of replicas to run"
  default_value = "1"
}

resource "ctrlplane_deployment_variable_value" "replica_count_prod" {
  deployment_id     = ctrlplane_deployment.api.id
  variable_id       = ctrlplane_deployment_variable.replica_count.id
  priority          = 0
  resource_selector = "resource.metadata['environment'] == 'production'"
  literal_value     = "5"
}
```

#### Reference Values

Variable values can reference data from the workspace or resource metadata
instead of using literal values:

```hcl theme={null}
resource "ctrlplane_deployment_variable" "size" {
  deployment_id = ctrlplane_deployment.redis.id
  key           = "size"
  description   = "The size of the Redis deployment"
  default_value = "small"
}

resource "ctrlplane_deployment_variable_value" "size_from_workspace" {
  deployment_id     = ctrlplane_deployment.redis.id
  variable_id       = ctrlplane_deployment_variable.size.id
  priority          = 0
  resource_selector = "resource.version == 'ctrlplane.dev/kubernetes/cluster/v1'"

  reference_value = {
    reference = "workspace"
    path      = ["metadata", "size"]
  }
}
```

### Via REST API

**Create a variable:**

```bash theme={null}
curl -X PUT https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments/{deploymentId}/variables/{variableId} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "REPLICA_COUNT",
    "description": "Number of replicas to run"
  }'
```

**Set an environment-specific value:**

```bash theme={null}
curl -X PUT https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments/{deploymentId}/variables/{variableId}/values/{valueId} \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "priority": 0,
    "value": "5"
  }'
```

### Using Variables in Jobs

Variables are resolved during job creation and passed to the job agent:

```yaml theme={null}
# In GitHub Actions workflow
- name: Get job inputs
  uses: ctrlplanedev/get-job-inputs@v1
  id: job
  with:
    job_id: ${{ inputs.job_id }}

- name: Deploy with variables
  run: |
    echo "Replicas: ${{ steps.job.outputs.variable_REPLICA_COUNT }}"
    helm upgrade my-app ./chart \
      --set replicaCount=${{ steps.job.outputs.variable_REPLICA_COUNT }}
```

## Release Targets

When you create a deployment, Ctrlplane automatically creates **release targets** by crossing the deployment with environments and resources.

**Formula**: `Deployment × Environment × Resource = Release Targets`

**Example**:

Given:

* Deployment: "API Service"
* Environments: Development, Staging, Production
* Resources in Production: 3 clusters

Release Targets Created:

1. API Service → Development → dev-cluster
2. API Service → Staging → staging-cluster
3. API Service → Production → prod-cluster-1
4. API Service → Production → prod-cluster-2
5. API Service → Production → prod-cluster-3

Each release target can receive deployment versions independently.

### Filtering Release Targets

Use the deployment's `resourceSelector` to limit which targets are created:

```
resource.version == 'ctrlplane.dev/kubernetes/cluster/v1' && resource.metadata['kubernetes/status'] == 'running'
```

Only resources matching this CEL expression will have release targets for this
deployment.

## Deployment Lifecycle

### 1. Create Deployment

Define the deployment in Ctrlplane with job agent configuration.

### 2. CI Builds and Creates Version

Your CI pipeline builds the artifact and creates a deployment version.

### 3. Ctrlplane Evaluates Policies

Ctrlplane checks policies (approvals, environment progression, etc.).

### 4. Jobs Created

For each release target that should receive the version, a job is created.

### 5. Job Agent Executes

The configured job agent picks up the job and executes the deployment.

### 6. Status Updated

The job reports status back to Ctrlplane.

## REST API Reference

### Deployment CRUD

**Create:**

```
POST /v1/workspaces/{workspaceId}/deployments
```

**Get:**

```
GET /v1/workspaces/{workspaceId}/deployments/{deploymentId}
```

**Update:**

```
PUT /v1/workspaces/{workspaceId}/deployments/{deploymentId}
```

**Delete:**

```
DELETE /v1/workspaces/{workspaceId}/deployments/{deploymentId}
```

**List:**

```
GET /v1/workspaces/{workspaceId}/deployments
```

### Versions

**Create version:**

```
POST /v1/workspaces/{workspaceId}/deployments/{deploymentId}/versions
```

**List versions:**

```
GET /v1/workspaces/{workspaceId}/deployments/{deploymentId}/versions
```

**Update version:**

```
PATCH /v1/workspaces/{workspaceId}/deployment-versions/{deploymentVersionId}
```

### Variables

**Upsert variable:**

```
PUT /v1/workspaces/{workspaceId}/deployments/{deploymentId}/variables/{variableId}
```

**Upsert variable value:**

```
PUT /v1/workspaces/{workspaceId}/deployments/{deploymentId}/variables/{variableId}/values/{valueId}
```

## Terraform Provider Reference

| Resource                              | Description                                  |
| ------------------------------------- | -------------------------------------------- |
| `ctrlplane_deployment`                | The deployment itself                        |
| `ctrlplane_deployment_system_link`    | Links a deployment to a system               |
| `ctrlplane_deployment_variable`       | Defines a deployment variable                |
| `ctrlplane_deployment_variable_value` | Sets a value for a variable (with selectors) |

## Viewing Deployment Status

### Via Web UI

1. Navigate to the deployment
2. See tabs:
   * **Versions**: All versions created
   * **Releases**: Active releases across targets
   * **Jobs**: Execution history
   * **Variables**: Configured variables

### Via API

**Get deployment details:**

```bash theme={null}
curl https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments/{deploymentId} \
  -H "Authorization: Bearer $TOKEN"
```

**List versions:**

```bash theme={null}
curl https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments/{deploymentId}/versions \
  -H "Authorization: Bearer $TOKEN"
```

## Common Patterns

### Microservices

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_deployment" "user_service" {
      name              = "User Service"
      resource_selector = "resource.kind == 'Kubernetes'"
      metadata          = { service = "user" }

      job_agent {
        id = ctrlplane_job_agent.argocd.id
        argocd { template = file("${path.module}/user-service.yaml") }
      }
    }

    resource "ctrlplane_deployment" "order_service" {
      name              = "Order Service"
      resource_selector = "resource.kind == 'Kubernetes'"
      metadata          = { service = "order" }

      job_agent {
        id = ctrlplane_job_agent.argocd.id
        argocd { template = file("${path.module}/order-service.yaml") }
      }
    }

    resource "ctrlplane_deployment" "payment_service" {
      name              = "Payment Service"
      resource_selector = "resource.kind == 'Kubernetes'"
      metadata          = { service = "payment" }

      job_agent {
        id = ctrlplane_job_agent.argocd.id
        argocd { template = file("${path.module}/payment-service.yaml") }
      }
    }
    ```
  </Tab>

  <Tab title="CLI">
    ```yaml theme={null}
    # microservices.yaml
    ---
    type: Deployment
    name: User Service
    slug: user-service
    jobAgent:
      ref: argocd-agent
    ---
    type: Deployment
    name: Order Service
    slug: order-service
    jobAgent:
      ref: argocd-agent
    ---
    type: Deployment
    name: Payment Service
    slug: payment-service
    jobAgent:
      ref: argocd-agent
    ```

    ```bash theme={null}
    ctrlc apply -f microservices.yaml
    ```
  </Tab>
</Tabs>

### Database + Application

```hcl theme={null}
resource "ctrlplane_deployment" "database_migration" {
  name              = "Database Migration"
  resource_selector = "resource.kind == 'Kubernetes'"

  job_agent {
    id = ctrlplane_job_agent.github.id
    github {
      owner       = "my-org"
      repo        = "migrations"
      workflow_id = 11111111
    }
  }
}

resource "ctrlplane_deployment" "api_service" {
  name              = "API Service"
  resource_selector = "resource.kind == 'Kubernetes'"

  job_agent {
    id = ctrlplane_job_agent.github.id
    github {
      owner       = "my-org"
      repo        = "api-service"
      workflow_id = 22222222
    }
  }
}
```

Use a [Deployment Dependency](../policies/deployment-dependency) policy to
ensure migrations complete before the API deploys.

### Multi-Platform

```hcl theme={null}
resource "ctrlplane_deployment" "web_service" {
  name              = "Web Service"
  resource_selector = "resource.kind == 'Kubernetes'"

  job_agent {
    id = ctrlplane_job_agent.argocd.id
    argocd { template = file("${path.module}/web.yaml") }
  }
}

resource "ctrlplane_deployment" "lambda_function" {
  name              = "Lambda Function"
  resource_selector = "resource.kind == 'AWSLambda'"

  job_agent {
    id = ctrlplane_job_agent.github.id
    github {
      owner       = "my-org"
      repo        = "lambda-deploy"
      workflow_id = 33333333
    }
  }
}
```

## Best Practices

### Naming

**Good Names**:

* ✅ "API Service"
* ✅ "Frontend Application"
* ✅ "Payment Processor"

**Good Slugs**:

* ✅ `api-service`
* ✅ `frontend-app`
* ✅ `payment-processor`

**Avoid**:

* ❌ "Service" (too generic)
* ❌ "api\_service" (use hyphens)
* ❌ "API-SERVICE" (use lowercase)

### Job Agent Configuration

**Do**:

* ✅ Use version's `jobAgentConfig` for version-specific values (like image tags)
* ✅ Use deployment's job agent config for stable configuration
* ✅ Keep sensitive data in secrets (not in config)
* ✅ Use agent `selector` when routing to different resource types

### Version Tags

**Good Tags**:

* ✅ `v1.2.3` (semantic version)
* ✅ `2024-01-15-prod` (date-based)
* ✅ `abc123def` (git commit)

**Avoid**:

* ❌ `latest` (not specific)
* ❌ `prod` (not a version)
* ❌ `test` (too vague)

### Resource Selectors

Use CEL resource selectors to:

* Limit Kubernetes deployments to Kubernetes resources
* Filter by resource status (e.g., `resource.metadata['status'] == 'running'`)
* Separate platform-specific deployments
* Control which resources can receive a deployment

## Troubleshooting

### No release targets created

* Check `resourceSelector` CEL expression matches some resources
* Verify environments have matching resources
* Ensure deployment is linked to a system (via `ctrlplane_deployment_system_link`
  in Terraform)

### Jobs not being created for new versions

* Check version status is `ready`
* Review policies for denials/pending approvals
* Verify job agent is configured
* Check release target exists

### Job agent config not working

* Verify job agent block matches the agent type (e.g., `github {}` for GitHub
  Actions agents)
* Check version's `jobAgentConfig` overrides correctly
* Review job agent logs for configuration errors

## Next Steps

* [Releases and Jobs](./releases-and-jobs) - Understand version lifecycle
* [Release Targets](./release-targets) - How the deployment matrix works
* [CI/CD Integration](../integrations/cicd) - Integrate with your CI system
* [Job Agents](../integrations/job-agents/github) - Configure deployment executors
* [Policies](../policies/overview) - Control when and how deployments proceed


# Environments
Source: https://docs.ctrlplane.dev/concepts/environments

Define where to deploy — logical stages that group resources

An **Environment** defines **where** to deploy. It represents a logical
deployment stage in your pipeline, such as development, staging, or production.
Environments use selectors to dynamically determine which resources belong to
them — answering the question of *where* a version should be released.

<Tip>
  In Ctrlplane's mental model: [**Deployments**](./deployments) = what & how, **Environments** = where, [**Policies**](../policies/overview) = when.
</Tip>

## What is an Environment?

Environments organize your deployment pipeline into stages:

* **Development** - Where developers test their changes
* **QA** - Quality assurance and testing
* **Staging** - Pre-production validation
* **Production** - Live customer-facing environment

Environments are **not** static groups of resources. Instead, they use **selectors** to dynamically match resources based on their metadata. This means:

* New resources matching the selector automatically join the environment
* Resources can belong to multiple environments
* No manual resource assignment needed

## Environment Properties

```yaml theme={null}
id: env_abc123
systemId: sys_xyz789
name: Production
directory: environments/prod
description: Production environment for customer-facing services
resourceSelector: resource.metadata["environment"] == "production"
createdAt: "2024-01-15T10:00:00Z"
metadata:
  owner: platform-team
  alert-channel: "#prod-alerts"
```

### name

Display name for the environment.

**Examples**:

* "Production"
* "Staging"
* "Development"
* "QA US East"

### systemId

The system this environment belongs to. Environments are scoped to systems.

### resourceSelector

The selector that determines which resources belong to this environment. This is the heart of how environments work.

**Example Selectors**:

**Simple metadata match** (CEL expression):

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

**Multiple conditions (AND)**:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

**OR logic**:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["region"] == "us-east-1" ||
  resource.metadata["region"] == "us-west-2"
```

### directory

Optional path for hierarchical organization of environments.

**Examples**:

* `""` - Root level
* `"regions"` - First level grouping
* `"regions/us-east"` - Nested grouping

**Use Cases**:

* Organizing by region: `regions/us-east`, `regions/eu-west`
* Organizing by type: `permanent/production`, `temporary/feature-1`
* Organizing by team: `teams/platform`, `teams/product`

### description

Optional description of the environment's purpose and usage.

### metadata

Key-value pairs with environment information (not used for resource matching).

**Common Metadata**:

```yaml theme={null}
metadata:
  owner: platform-team
  alert-channel: "#prod-alerts"
  cost-center: engineering
  sla: "99.99%"
```

## Creating Environments

### Via Web UI

1. Navigate to your system
2. Click "Environments" tab
3. Click "Create Environment"
4. Fill in:
   * Name: "Production"
   * Description: "Production environment"
   * Resource Selector: Add conditions
5. Click "Create"

### Via CLI

```yaml theme={null}
# environment.yaml
type: Environment
name: Production
description: Production environment for customer-facing services
resourceSelector: resource.metadata["environment"] == "production"
metadata:
  owner: platform-team
```

```bash theme={null}
ctrlc apply -f environment.yaml
```

### Multiple Environments

```yaml theme={null}
# environments.yaml
---
type: Environment
name: Development
description: Development environment
resourceSelector: resource.metadata["environment"] == "development"
---
type: Environment
name: Staging
description: Staging environment
resourceSelector: resource.metadata["environment"] == "staging"
---
type: Environment
name: Production
description: Production environment
resourceSelector: resource.metadata["environment"] == "production"
```

```bash theme={null}
ctrlc apply -f environments.yaml
```

## How Resource Selectors Work

Selectors match resources based on their metadata, kind, identifier, or other properties.

### Selector Types

Ctrlplane uses CEL (Common Expression Language) for resource selectors.

#### Metadata Selector

Match resources by metadata key-value pairs.

**Single condition**:

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

**Operators**:

* `==` - Exact match
* `!=` - Not equals
* `in` - Value in list
* `has()` - Key exists

**Multiple conditions (AND)**:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

**Multiple conditions (OR)**:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["region"] == "us-east-1" ||
  resource.metadata["region"] == "us-west-2"
```

#### Kind Selector

Match resources by their `kind` field.

```yaml theme={null}
resourceSelector: resource.kind == "KubernetesCluster"
```

#### Identifier Selector

Match resources by their identifier.

```yaml theme={null}
resourceSelector: resource.identifier.contains("prod")
```

#### Complex CEL Expressions

Use CEL for complex matching logic.

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["tier"] == "critical"
```

### Selector Evaluation

When determining which resources belong to an environment:

1. Ctrlplane evaluates the `resourceSelector` against all resources
2. Resources that match are included in the environment
3. As resources are added/updated, they're re-evaluated
4. Resources are automatically added/removed as they match/unmatch

**Example**:

Environment selector:

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

Resources:

* ✅ `{name: "Cluster A", metadata: {environment: "production"}}` - **Matched**
* ❌ `{name: "Cluster B", metadata: {environment: "staging"}}` - Not matched
* ✅ `{name: "Cluster C", metadata: {environment: "production", region: "us-east"}}` - **Matched**

Result: Environment contains Cluster A and Cluster C

## Release Targets

When you create a deployment and an environment, Ctrlplane automatically creates **release targets** for each matching resource.

**Formula**: `Deployment × Environment × Resource = Release Target`

**Example**:

Given:

* Deployment: "API Service"
* Environment: "Production" (matches 3 resources)
  * Resource A: k8s-prod-use1
  * Resource B: k8s-prod-usw2
  * Resource C: k8s-prod-euw1

Result: 3 release targets created:

1. API Service → Production → k8s-prod-use1
2. API Service → Production → k8s-prod-usw2
3. API Service → Production → k8s-prod-euw1

When you create a new deployment version, jobs can be created for each release target (subject to policies).

## Hierarchical Environments

Use the `directory` field to organize environments hierarchically.

### Example Structure

```
Root
├── regions/
│   ├── us-east-1/
│   │   ├── Production
│   │   └── Staging
│   └── eu-west-1/
│       ├── Production
│       └── Staging
└── temporary/
    ├── feature-branch-1
    └── feature-branch-2
```

### Creating Hierarchical Environments

```yaml theme={null}
# hierarchical-environments.yaml
---
type: Environment
name: Global Production
directory: ""
resourceSelector: resource.metadata["environment"] == "production"
---
type: Environment
name: US East Production
directory: regions/us-east
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
---
type: Environment
name: US East 1a Production
directory: regions/us-east/availability-zones
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["zone"] == "us-east-1a"
```

```bash theme={null}
ctrlc apply -f hierarchical-environments.yaml
```

### Benefits

* **Visual organization** in UI
* **Easier navigation** with many environments
* **Logical grouping** by region, team, or purpose
* **Permission scoping** (future feature)

## Common Environment Patterns

### Standard Pipeline

```yaml theme={null}
# environments.yaml
---
type: Environment
name: Development
resourceSelector: resource.metadata["environment"] == "development"
---
type: Environment
name: Staging
resourceSelector: resource.metadata["environment"] == "staging"
---
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```bash theme={null}
ctrlc apply -f environments.yaml
```

### Multi-Region Environments

```yaml theme={null}
# multi-region-environments.yaml
---
type: Environment
name: Production US East
directory: production/us-east
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
---
type: Environment
name: Production US West
directory: production/us-west
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-west-2"
```

```bash theme={null}
ctrlc apply -f multi-region-environments.yaml
```

### Canary Environments

```yaml theme={null}
# canary-environments.yaml
---
type: Environment
name: Production Canary
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["canary"] == "true"
---
type: Environment
name: Production Stable
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["canary"] != "true"
```

```bash theme={null}
ctrlc apply -f canary-environments.yaml
```

### Team-Specific Environments

```yaml theme={null}
# team-environments.yaml
---
type: Environment
name: Platform Team Development
resourceSelector: >-
  resource.metadata["team"] == "platform" &&
  resource.metadata["environment"] == "development"
---
type: Environment
name: Product Team Development
resourceSelector: >-
  resource.metadata["team"] == "product" &&
  resource.metadata["environment"] == "development"
```

```bash theme={null}
ctrlc apply -f team-environments.yaml
```

## Viewing Environment Resources

### Via CLI

```bash theme={null}
ctrlc api get resources --environment {environmentId}
```

Returns all resources matching the environment's selector.

### Via Web UI

1. Navigate to the environment
2. Click "Resources" tab
3. See all matched resources

The UI shows:

* Resource name and kind
* Metadata
* Current deployed versions
* Job status

## Environment Variables

Environments can have variables that differ per deployment.

### Deployment Variables

Define a variable at the deployment level, then set values per environment. Variables are typically configured via the Web UI, but can also be managed via API.

See the [Deployments documentation](./deployments) for details on creating and setting deployment variables.

During job execution, the appropriate value is resolved based on the environment.

## Policies and Environments

Policies often target specific environments using selectors.

### Approval Required for Production

```yaml theme={null}
# policy.yaml
type: Policy
name: Production Approval Required
selectors:
  - environments: environment.name == "Production"
rules:
  - anyApproval:
      minApprovals: 2
```

```bash theme={null}
ctrlc apply -f policy.yaml
```

### Environment Progression

Ensure deployments go to staging before production:

```yaml theme={null}
# progression-policy.yaml
type: Policy
name: Staging Before Production
selectors:
  - environments: environment.name == "Production"
rules:
  - environmentProgression:
      fromEnvironment: Staging
      toEnvironment: Production
```

```bash theme={null}
ctrlc apply -f progression-policy.yaml
```

## Best Practices

### Naming

**Good Names**:

* ✅ "Production"
* ✅ "Staging"
* ✅ "Development"
* ✅ "QA US East"

**Avoid**:

* ❌ "Prod" (use full names)
* ❌ "Environment 1" (not descriptive)
* ❌ "John's Test" (not permanent)

### Selector Design

**Do**:

* ✅ Use consistent metadata keys across resources
* ✅ Make selectors explicit and clear
* ✅ Test selectors match expected resources
* ✅ Document complex selectors

**Don't**:

* ❌ Overly complex selectors that are hard to understand
* ❌ Selectors that might accidentally match wrong resources
* ❌ Selectors based on frequently changing metadata

### Environment Count

**Start Simple**:

```
- Development
- Production
```

**Grow as Needed**:

```
- Development
- Staging
- Production
- Production Canary
```

**Don't Over-Engineer**:

* Avoid creating environments you won't use
* Consolidate similar environments
* Use resource metadata to differentiate within an environment

## Troubleshooting

### Environment shows no resources

* Check the resource selector syntax
* Verify resources exist with matching metadata
* Test selector with "Query Resources" feature
* Check resources aren't deleted

### Wrong resources in environment

* Review resource selector logic
* Check for OR vs AND conditions
* Verify resource metadata is correct
* Test selector in isolation

### Release targets not created

* Verify environment has matching resources
* Check deployment has resource selector (if any)
* Ensure deployment and environment are in same system
* Review logs for errors

## Next Steps

* [Deployments](./deployments) - Create deployments
* [Selectors](./selectors) - Deep dive into selector syntax
* [Release Targets](./release-targets) - Understand the deployment matrix
* [Policies](../policies/overview) - Control deployments with policies


# Concepts Overview
Source: https://docs.ctrlplane.dev/concepts/overview

Overview of the core concepts in Ctrlplane

This page provides an overview of the core concepts in Ctrlplane.

The three pillars of Ctrlplane answer the fundamental questions of deployment:

* **Deployments** define **what** to deploy and **how** — the service, its versions, and the job agent that executes the deployment.
* **Environments** define **where** to deploy — logical stages that dynamically group resources via selectors.
* **Policies** define **when** to deploy — rules like approvals, verification, progression gates, and deployment windows that control timing.

```
System
  ├── Deployments (what & how)
  │     └── Versions (specific builds)
  ├── Environments (where)
  │     └── Resources (deployment targets)
  └── Policies (when)

Release Target = Deployment × Environment × Resource

Release = Version deployed to a Release Target
  └── Job (executed by Job Agent)
```

## System

A **System** is a logical grouping of related deployments, environments, and
resources. Think of it as a workspace for a product or team.

| Property      | Description                  |
| ------------- | ---------------------------- |
| `name`        | Display name                 |
| `slug`        | URL-friendly identifier      |
| `description` | What this system encompasses |

**Example**: "E-commerce Platform" system containing API, Frontend, and Payment
deployments.

**When to create a System**: One per product, platform, or team boundary.

## Resource

A **Resource** is a deployment target—the actual infrastructure where your code
runs.

| Property     | Description                                    |
| ------------ | ---------------------------------------------- |
| `name`       | Human-readable name                            |
| `kind`       | Type (e.g., `KubernetesCluster`, `AWS/Lambda`) |
| `identifier` | Unique identifier                              |
| `metadata`   | Key-value pairs for classification             |
| `config`     | Resource-specific configuration                |
| `version`    | Current version/state                          |

**Examples**: Kubernetes cluster, EC2 instance, Lambda function, VM.

**How created**: Via Resource Providers (auto-sync from K8s, AWS, GCP) or API.

```yaml theme={null}
name: prod-us-east-1
kind: KubernetesCluster
identifier: k8s-prod-use1
metadata:
  region: us-east-1
  environment: production
  tier: critical
config:
  server: https://k8s.example.com
```

## Environment

An **Environment** defines **where** to deploy. It represents a logical
deployment stage (dev, staging, prod) that groups resources using selectors.

| Property           | Description                                 |
| ------------------ | ------------------------------------------- |
| `name`             | Environment name                            |
| `systemId`         | Parent system                               |
| `resourceSelector` | Selector determining which resources belong |
| `directory`        | Optional path for hierarchical organization |

**Key concept**: Environments are *dynamic*. When you add a new resource
matching the selector, it automatically joins the environment.

```yaml theme={null}
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

## Deployment

A **Deployment** defines **what** to deploy and **how**. It represents a service
or application you want to deploy, along with the job agent configuration that
determines how the deployment is executed.

| Property           | Description                                      |
| ------------------ | ------------------------------------------------ |
| `name`             | Deployment name                                  |
| `slug`             | URL-friendly identifier                          |
| `description`      | What this deployment does                        |
| `systemId`         | Parent system                                    |
| `resourceSelector` | Optional filter for which resources can run this |
| `jobAgentId`       | Which job agent executes deployments             |
| `jobAgentConfig`   | Configuration passed to the job agent            |

**Examples**: "API Service", "Frontend App", "Payment Processor".

## Version

A **Version** is a specific build or release of a deployment, typically created
by your CI pipeline.

| Property         | Description                                       |
| ---------------- | ------------------------------------------------- |
| `deploymentId`   | Parent deployment                                 |
| `tag`            | Version identifier (e.g., `v1.2.3`, `sha-abc123`) |
| `name`           | Optional human-readable name                      |
| `status`         | `building`, `ready`, or `failed`                  |
| `metadata`       | Arbitrary metadata (git commit, build number)     |
| `config`         | Version-specific configuration                    |
| `jobAgentConfig` | Overrides deployment's job agent config           |

**Status meanings**:

* `building` — Still being built, won't be deployed
* `ready` — Ready for deployment (default for policies)
* `failed` — Build failed, won't be deployed

```bash theme={null}
# CI creates a version after building
curl -X POST ".../deployments/{id}/versions" \
  -d '{
    "tag": "v1.2.3",
    "status": "ready",
    "metadata": {"commit": "abc123"}
  }'
```

## Release Target

A **Release Target** is the combination of a Deployment, Environment, and
Resource. It represents a specific place where a deployment can be released.

```
Release Target = Deployment × Environment × Resource
```

**Example**:

* Deployment: "API Service"
* Environment: "Production"
* Resource: "us-east-1 cluster"
* **Release Target**: "API Service on Production/us-east-1"

**Automatic creation**: Release targets are computed from the intersection of:

1. Environment's resource selector → which resources
2. Deployment's resource selector (if any) → further filtering

## Release

A **Release** is an instance of deploying a specific Version to a Release
Target.

| Property        | Description                  |
| --------------- | ---------------------------- |
| `versionId`     | Which version to deploy      |
| `deploymentId`  | Which deployment             |
| `environmentId` | Which environment            |
| `resourceId`    | Which resource               |
| `createdAt`     | When the release was created |

Releases do not carry their own status. The state of a release is inferred from
the **Release Target State** — which tracks the desired release, current release,
and latest job for each target.

## Job

A **Job** is the actual deployment task executed by a Job Agent.

| Property     | Description                                     |
| ------------ | ----------------------------------------------- |
| `releaseId`  | The release this job deploys                    |
| `jobAgentId` | Which agent executes this job                   |
| `status`     | `pending`, `in_progress`, `completed`, `failed` |
| `externalId` | External identifier (e.g., GitHub run ID)       |
| `message`    | Status message or error details                 |

**Job lifecycle**:

1. Ctrlplane creates job for approved release
2. Job agent polls and receives the job
3. Agent acknowledges and executes
4. Agent updates status as it progresses
5. Agent marks completed or failed

## Job Agent

A **Job Agent** is the executor that performs deployments. It bridges Ctrlplane
to your infrastructure.

## Policy

A **Policy** defines **when** a version is allowed to deploy. Policies are the
rules governing deployment timing—approvals, gates, windows, and verification
that control when releases progress.

| Component   | Description                                  |
| ----------- | -------------------------------------------- |
| `name`      | Policy name                                  |
| `selectors` | Which release targets this policy applies to |
| `rules`     | The specific rules to enforce                |

**Policy types**:

| Type                    | Description                                   |
| ----------------------- | --------------------------------------------- |
| Approval                | Requires manual sign-off                      |
| Environment Progression | Wait for another environment to succeed first |
| Gradual Rollout         | Deploy to targets sequentially with delays    |
| Deployment Window       | Only deploy during certain hours              |
| Version Selector        | Filter which versions can deploy              |
| Version Cooldown        | Minimum time between deployments              |
| Deployment Dependency   | Wait for another deployment to complete       |
| Verification            | Check metrics after deployment                |

**Policy evaluation**: When a release target needs deployment:

1. Find all policies matching the target
2. Evaluate each rule
3. All pass → create job
4. Any requires action → release is pending
5. Any denies → release is blocked

## Selector

**Selectors** are query expressions used to match resources, environments, or
deployments.

```yaml theme={null}
# Match resources in production
resource.metadata["environment"] == "production"

# Match critical deployments in any region
deployment.metadata["tier"] == "critical" &&
  resource.metadata["region"] in ["us-east-1", "eu-west-1"]
```

Used in:

* Environment resource selectors
* Deployment resource selectors
* Policy target selectors

See [Selectors](/concepts/selectors) for full syntax.

## Variables

**Variables** provide dynamic configuration for deployments.

| Type                 | Description                                 |
| -------------------- | ------------------------------------------- |
| Deployment Variables | Defined per deployment, vary by environment |
| Resource Variables   | Defined on resources                        |

```json theme={null}
{
  "deploymentVariable": "replicas",
  "values": [
    { "environmentId": "dev", "value": "1" },
    { "environmentId": "prod", "value": "3" }
  ]
}
```

## Quick Reference Table

| Entity         | What It Is                                                  |
| -------------- | ----------------------------------------------------------- |
| System         | Workspace grouping deployments and environments             |
| Resource       | Deployment target (cluster, VM, function)                   |
| Deployment     | **What & how** — service/app to deploy and its agent config |
| Environment    | **Where** — logical stage grouping resources via selectors  |
| Version        | Specific build of a deployment                              |
| Release Target | Deployment × Environment × Resource                         |
| Release        | Version being deployed to a release target                  |
| Job            | Execution task sent to a job agent                          |
| Job Agent      | Executor (ArgoCD, GitHub Actions, etc.)                     |
| Policy         | **When** — rules controlling deployment timing              |
| Selector       | Query expression matching entities                          |

## Next Steps

* [How It Works](/how-it-works) — Understand the flow
* [Quickstart](/quickstart) — Hands-on tutorial
* [Selectors](/concepts/selectors) — Deep dive into selector syntax
* [Policies](/policies/overview) — Configure deployment rules


# Release Targets
Source: https://docs.ctrlplane.dev/concepts/release-targets

The combination of a Deployment, Environment, and Resource

A **Release Target** is the combination of a Deployment, Environment, and Resource. It represents a specific place where a deployment version can be released.

## Understanding Release Targets

The core formula:

```
Deployment × Environment × Resource = Release Target
```

### Example

Given:

* **Deployment**: API Service
* **Environment**: Production (matches 2 resources)
  * Resource 1: prod-cluster-us-east
  * Resource 2: prod-cluster-us-west

**Release Targets Created**:

1. API Service → Production → prod-cluster-us-east
2. API Service → Production → prod-cluster-us-west

Each release target can independently receive deployment versions.

## How Release Targets are Created

Release targets are created automatically based on selectors:

### 1. Environment Selector Matches Resources

Environment defines a resource selector:

```yaml theme={null}
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

This matches all resources with `metadata.environment = "production"`.

### 2. Deployment Selector Further Filters (Optional)

Deployment can have an additional resource selector:

```yaml theme={null}
type: Deployment
name: API Service
resourceSelector: resource.kind == "KubernetesCluster"
```

This limits the deployment to only Kubernetes clusters.

### 3. Intersection Creates Release Targets

The final set of resources is the intersection of:

* Resources matched by environment selector
* Resources matched by deployment selector (if present)

**Example Calculation**:

```
Environment "Production" matches:
  - prod-k8s-cluster-1 (kind: kubernetes-cluster, env: production)
  - prod-k8s-cluster-2 (kind: kubernetes-cluster, env: production)
  - prod-vm-server-1 (kind: vm, env: production)

Deployment "API Service" selector (kind: kubernetes-cluster):
  - prod-k8s-cluster-1 ✓
  - prod-k8s-cluster-2 ✓
  - prod-vm-server-1 ✗ (not a kubernetes-cluster)

Release Targets Created:
  1. API Service → Production → prod-k8s-cluster-1
  2. API Service → Production → prod-k8s-cluster-2
```

## Release Target Properties

```yaml theme={null}
id: rt_abc123
deploymentId: dep_api
environmentId: env_prod
resourceId: res_cluster1
key: dep_api:env_prod:res_cluster1
createdAt: "2024-01-15T10:00:00Z"
```

### key

Unique identifier combining deployment, environment, and resource IDs.

Format: `{deploymentId}:{environmentId}:{resourceId}`

This key is used to track which version is currently deployed to each target.

## Release Target Lifecycle

### Creation

Release targets are created when:

* A new deployment is created (targets created for all matching environments and resources)
* A new environment is created (targets created for all deployments and matching resources)
* A new resource is created that matches existing environment selectors (targets created for all deployments)

### Deletion

Release targets are deleted when:

* The deployment is deleted
* The environment is deleted
* The resource is deleted or no longer matches the selectors

### Dynamic Updates

As resources are added, updated, or removed:

* Resource metadata changes → Release targets re-evaluated
* Resource matches environment selector → New release targets created
* Resource stops matching → Release targets removed

## Viewing Release Targets

### Via Web UI

1. Navigate to your deployment
2. Click "Release Targets" tab
3. See all targets with current deployed versions

### Via CLI

```bash theme={null}
# List release targets for a deployment
ctrlc api get release-targets --deployment {deploymentId}
```

**Response**:

```yaml theme={null}
releaseTargets:
  - id: rt_abc123
    deployment:
      id: dep_api
      name: API Service
    environment:
      id: env_prod
      name: Production
    resource:
      id: res_cluster1
      name: Production US East Cluster
      identifier: k8s-prod-use1
    currentRelease:
      versionTag: v1.2.3
      status: completed
```

## Release Target Matrix

Think of release targets as a 3D matrix:

```
                    Resources
                 ┌─────┬─────┬─────┐
                 │ R1  │ R2  │ R3  │
    ┌────────────┼─────┼─────┼─────┤
    │ Dev        │ RT  │ RT  │ RT  │ Environments
    │ Staging    │ RT  │ RT  │ RT  │
    │ Production │ RT  │ RT  │ RT  │
    └────────────┴─────┴─────┴─────┘
                    ↑
               Deployment
```

Each cell "RT" is a release target.

For multiple deployments, this becomes a 3D cube:

```
          Deployments (layers)
               ↓
        [API] [Frontend] [Worker]
         ↓       ↓          ↓
    Environment × Resource matrices
```

## Release Target States

While release targets don't have an explicit "status" field, their state is determined by their current release:

### No Release

Target exists but no version has been deployed yet.

### Active Release

A version is currently being deployed (job in progress).

### Completed Release

A version is successfully deployed (job completed).

### Failed Release

Latest deployment attempt failed.

### Pending Release

A release exists but is waiting for approval/policies.

## Deployment Targeting Strategies

### Strategy 1: Broad Matching

Environment matches many resources, deployment doesn't filter:

```yaml theme={null}
# Environment matches all production resources
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```yaml theme={null}
# Deployment (no selector)
# → Deploys to ALL production resources
type: Deployment
name: Monitoring Agent
# resourceSelector: not specified
```

**Result**: Deployment goes to every production resource.

**Use Case**: Services that should run everywhere (monitoring agents, logging).

### Strategy 2: Deployment-Level Filtering

Deployment limits which resources it can target:

```yaml theme={null}
# Environment
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```yaml theme={null}
# Deployment filters to Kubernetes only
type: Deployment
name: API Service
resourceSelector: resource.kind == "KubernetesCluster"
```

**Result**: Deployment only goes to production Kubernetes clusters.

**Use Case**: Platform-specific deployments (containers vs. VMs).

### Strategy 3: Fine-Grained Environments

Create specific environments for precise targeting:

```yaml theme={null}
# Environment: Production Kubernetes US East
type: Environment
name: Production Kubernetes US East
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1" &&
  resource.kind == "KubernetesCluster"
```

**Result**: Very specific set of release targets.

**Use Case**: Region-specific deployments, gradual rollouts.

## Release Target Locking

Release targets can be locked to prevent new deployments:

```bash theme={null}
ctrlc api lock release-target {releaseTargetId} --reason "Maintenance window"
```

To unlock:

```bash theme={null}
ctrlc api unlock release-target {releaseTargetId}
```

**Use Cases**:

* Maintenance windows
* Incident response (freeze deployments)
* Testing/debugging (keep specific version)

Locked targets won't receive new deployments until unlocked.

## Querying Release Targets

### By Deployment

```bash theme={null}
ctrlc api get release-targets --deployment {deploymentId}
```

Shows all targets for a specific deployment.

### By Environment

```bash theme={null}
ctrlc api get release-targets --environment {environmentId}
```

Shows all targets in a specific environment.

### By Resource

```bash theme={null}
ctrlc api get release-targets --resource {resourceId}
```

Shows all targets for a specific resource (all deployments).

### With Filters

```bash theme={null}
ctrlc api get release-targets \
  --deployment dep_api \
  --environment env_prod \
  --status completed
```

## Release Target Count Estimates

Before creating a deployment, estimate how many release targets will be created:

```bash theme={null}
ctrlc api estimate-targets \
  --system sys_abc \
  --selector 'resource.kind == "KubernetesCluster"'
```

**Response**:

```yaml theme={null}
environments:
  - name: Development
    matchingResources: 1
    releaseTargets: 1
  - name: Production
    matchingResources: 3
    releaseTargets: 3
totalReleaseTargets: 4
```

## Common Patterns

### Pattern 1: Environment-Per-Region

```yaml theme={null}
# regional-setup.yaml
---
type: Deployment
name: API Service
slug: api-service
jobAgent:
  ref: kubernetes-agent
---
type: Environment
name: Production US East
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
---
type: Environment
name: Production US West
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-west-2"
```

```bash theme={null}
ctrlc apply -f regional-setup.yaml
```

**Release Targets**: API Service gets separate targets per region.

**Benefit**: Independent regional deployments, gradual rollout by region.

### Pattern 2: Tiered Resources

```yaml theme={null}
# tiered-setup.yaml
---
type: Deployment
name: Critical Service
slug: critical-service
resourceSelector: resource.metadata["tier"] == "critical"
jobAgent:
  ref: kubernetes-agent
---
type: Deployment
name: Standard Service
slug: standard-service
resourceSelector: resource.metadata["tier"] == "standard"
jobAgent:
  ref: kubernetes-agent
---
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```bash theme={null}
ctrlc apply -f tiered-setup.yaml
```

**Release Targets**: Services only deploy to appropriate tier resources.

**Benefit**: Resource isolation, cost optimization.

### Pattern 3: Canary Deployments

```yaml theme={null}
# canary-setup.yaml
---
type: Deployment
name: API Service
slug: api-service
jobAgent:
  ref: kubernetes-agent
---
type: Environment
name: Production Canary
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["canary"] == "true"
---
type: Environment
name: Production Stable
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["canary"] != "true"
```

```bash theme={null}
ctrlc apply -f canary-setup.yaml
```

**Release Targets**: Separate targets for canary vs. stable.

**Benefit**: Test new versions on canary before rolling out to stable.

## Best Practices

### Resource Metadata Design

Design resource metadata with targeting in mind:

```yaml theme={null}
# Good resource metadata
metadata:
  environment: production
  region: us-east-1
  zone: us-east-1a
  tier: critical
  team: platform
  canary: "false"
```

This allows flexible targeting across multiple dimensions.

### Selector Simplicity

**Prefer Simple Selectors**:

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

**Over Complex Ones**:

```yaml theme={null}
resourceSelector: >-
  (resource.metadata["environment"] == "production" ||
   resource.metadata["environment"] == "prod") &&
  resource.metadata["tier"] != "deprecated" && ...
```

Complex selectors are harder to understand and maintain.

### Validate Target Count

Before deploying:

1. Check estimated release target count
2. Verify targets match expectations
3. Test in lower environment first

### Monitor Release Targets

* Track release target creation/deletion events
* Alert on unexpected target count changes
* Review targets when resources are added/updated

## Troubleshooting

### Too many release targets created

* Review environment resource selectors
* Check if deployment needs a resource selector to filter
* Verify resource metadata is correct

### Expected targets not created

* Check environment selector matches resources
* Verify deployment selector (if present) isn't too restrictive
* Ensure deployment and environment are in same system
* Check resources exist and aren't deleted

### Targets created for wrong resources

* Review selector logic (AND vs. OR)
* Check resource metadata matches expectations
* Test selectors with query API before applying

### Release target disappeared

* Check if resource was deleted
* Verify resource still matches environment selector
* Check if resource metadata changed

## Next Steps

* [Releases and Jobs](./releases-and-jobs) - How versions deploy to targets
* [Selectors](./selectors) - Master the selector syntax
* [Environments](./environments) - Create and configure environments
* [Policies](../policies/overview) - Control which targets receive deployments


# Releases and Jobs
Source: https://docs.ctrlplane.dev/concepts/releases-and-jobs

How Ctrlplane turns deployment versions into actual deployments

This guide explains how Ctrlplane turns deployment versions into actual
deployments through **releases** and **jobs**.

## Overview

The flow from version to execution:

```
Deployment Version (what to deploy)
    ↓
Release (intent to deploy to a specific target)
    ↓
Job (actual deployment task)
    ↓
Job Agent (executes the deployment)
```

## Release

A **Release** represents the intent to deploy a
specific version to a specific release target
(Deployment × Environment × Resource).

Ctrlplane creates one release per target. A target is a single resource that
matches the deployment's resource selector and is also included in an
environment (via the environment's selector). When a release is approved,
Ctrlplane creates a job for that same target.

### Release Properties

```yaml theme={null}
id: rel_abc123
deploymentId: dep_xyz
environmentId: env_prod
resourceId: res_cluster1
versionId: ver_v123
createdAt: "2024-01-15T12:00:00Z"
```

Releases do not carry their own status field. Instead, the state of a release
is determined by the **Release Target State**, which tracks:

* **Desired release** — the release that policies have selected for deployment
* **Current release** — the release whose job completed successfully (with passing verifications)
* **Latest job** — the most recent job created for this target, along with its status and verifications

### How Releases are Created

Releases are created automatically by Ctrlplane when:

1. A new deployment version is created with status `ready`
2. Ctrlplane evaluates which release targets should receive this version
3. For each target, a release is created
4. Policies are evaluated to determine if the release can proceed

**Example Flow**:

```
1. CI creates version: API Service v1.2.3 (status: ready)
   ↓
2. Ctrlplane finds release targets for "API Service":
   - API Service → Production → cluster-1
   - API Service → Production → cluster-2
   - API Service → Staging → staging-cluster
   ↓
3. Ctrlplane creates 3 releases, one for each target
   ↓
4. Ctrlplane evaluates policies:
   - Production requires approval → Releases pending
   - Staging auto-deploys → Release approved
   ↓
5. Jobs created for approved releases
```

### Release Lifecycle

```
          [Version Created]
                 ↓
       [Policy Evaluation]
         ↓               ↓
    [Blocked]      [Passes] ←→ [User Action]
                        ↓
              [Release Created]
                        ↓
                 [Job Created]
                        ↓
               [Job Executes]
         ↓                        ↓
    [Completed]              [Failed]
```

### Viewing Releases

**Via Web UI**:

1. Navigate to deployment
2. Click "Releases" tab
3. See all releases with their current and desired state

**Via CLI**:

```bash theme={null}
# List releases for a deployment
ctrlc api get releases --deployment {deploymentId}

# Get release details
ctrlc api get release {releaseId}
```

### Release Variables

Releases have resolved variables from:

* Deployment variables (with environment-specific values)
* Resource variables
* Version config

These are passed to the job for execution.

### Cancelling Releases

Cancel a pending or active release:

```bash theme={null}
ctrlc api cancel release {releaseId}
```

Active releases will have their jobs cancelled.

## Job

A **Job** is the actual deployment task that gets executed by a job agent. Jobs are created from approved releases.

### Job Properties

```yaml theme={null}
id: job_abc123
releaseId: rel_xyz789
jobAgentId: agent_123
status: in_progress
externalId: "12345"
message: Deploying version v1.2.3
createdAt: "2024-01-15T12:00:00Z"
startedAt: "2024-01-15T12:01:00Z"
completedAt: null
```

### Job Status

* **`pending`** - Job created, waiting for agent to pick up
* **`triggered`** - Job dispatched to external system (e.g., GitHub Actions)
* **`in_progress`** - Job is currently executing
* **`completed`** - Job finished successfully
* **`failed`** - Job failed
* **`cancelled`** - Job was cancelled
* **`skipped`** - Job was skipped
* **`invalid_job_agent`** - Job agent configuration invalid

### Job Lifecycle

```
     [Release Approved]
             ↓
      [Job Created]
             ↓
       [Pending] ──────→ Agent polls /queue/next
             ↓
      [Agent Gets Job]
             ↓
   [Agent Acknowledges] ──→ Job status: in_progress
             ↓
  [Agent Executes Deployment]
             ↓
    [Agent Reports Status]
             ↓
    [Completed / Failed]
```

### Job Configuration

Jobs receive configuration from multiple sources (in priority order):

1. **Version's jobAgentConfig** (highest priority)
2. **Deployment's jobAgentConfig**
3. **Resolved variables**
4. **Release context** (environment, resource info)

**Example merged config**:

```yaml theme={null}
# From deployment
workflow: deploy.yml
owner: my-org
repo: api-service

# From version (overrides)
imageTag: my-org/api-service:v1.2.3

# From variables
replicaCount: "5"

# From release context
environmentName: Production
resourceIdentifier: k8s-prod-use1
```

### Job Agents

Job agents are responsible for:

1. Polling for new jobs (`GET /job-agents/{agentId}/queue/next`)
2. Acknowledging jobs
3. Executing the deployment
4. Reporting status updates
5. Marking job as completed or failed

**Built-in Agent Types**:

* **GitHub Actions** - Triggers GitHub workflow
* **Kubernetes** - Creates Kubernetes Job
* **ArgoCD** - Syncs ArgoCD Application

### Job Execution Flow

#### 1. Job Agent Polls

```bash theme={null}
GET /api/v1/job-agents/{agentId}/queue/next
```

**Response**:

```yaml theme={null}
jobs:
  - id: job_abc123
    release:
      id: rel_xyz
      version:
        tag: v1.2.3
        config: {}
      environment:
        name: Production
      resource:
        identifier: k8s-prod-use1
        config: {}
      variables:
        REPLICA_COUNT: "5"
    jobAgentConfig:
      imageTag: my-org/api-service:v1.2.3
```

#### 2. Agent Acknowledges Job

```bash theme={null}
PATCH /api/v1/jobs/{jobId}
```

```yaml theme={null}
status: in_progress
message: Starting deployment
```

#### 3. Agent Executes Deployment

The agent uses the job configuration to perform the deployment:

**Kubernetes Agent Example**:

```typescript theme={null}
const manifest = renderManifest(job.jobAgentConfig.manifest, {
  version: job.release.version.tag,
  replicas: job.release.variables.REPLICA_COUNT,
  image: job.jobAgentConfig.imageTag
});

await kubectl.apply(manifest, job.release.resource.config.namespace);
```

**GitHub Actions Example**:

```typescript theme={null}
await github.actions.createWorkflowDispatch({
  owner: job.jobAgentConfig.owner,
  repo: job.jobAgentConfig.repo,
  workflow_id: job.jobAgentConfig.workflow,
  ref: job.jobAgentConfig.ref || 'main',
  inputs: {
    job_id: job.id,
    deployment_version: job.release.version.tag,
    environment: job.release.environment.name,
    resource_identifier: job.release.resource.identifier
  }
});
```

#### 4. Agent Updates Status

As the deployment progresses:

```bash theme={null}
# Update with progress message
PATCH /api/v1/jobs/{jobId}
# Body: { "message": "Applying Kubernetes manifests" }

# Mark as completed
PATCH /api/v1/jobs/{jobId}
# Body: { "status": "completed", "message": "Deployment successful" }

# Or mark as failed
PATCH /api/v1/jobs/{jobId}
# Body: { "status": "failed", "message": "Deployment failed: connection timeout" }
```

### External Job IDs

When job agents trigger external systems (like GitHub Actions), they can store the external job ID:

```bash theme={null}
PATCH /api/v1/jobs/{jobId}
# Body: { "externalId": "github-workflow-run-12345" }
```

This allows linking to the external system's UI for detailed logs.

### Job Retry

If a job fails, you can create a new job for the same release:

```bash theme={null}
ctrlc api redeploy release {releaseId}
```

This creates a new job for the release.

### Viewing Jobs

**Via Web UI**:

1. Navigate to deployment
2. Click "Jobs" tab
3. See execution history with status

**Via CLI**:

```bash theme={null}
# List jobs for a deployment
ctrlc api get jobs --deployment {deploymentId}

# Get job details
ctrlc api get job {jobId}
```

## Version → Release → Job Example

Let's walk through a complete example:

### 1. CI Creates Version

```bash theme={null}
ctrlc api upsert version \
  --workspace my-workspace \
  --deployment dep_api \
  --tag v1.2.3 \
  --metadata image/tag=my-org/api:v1.2.3
```

### 2. Ctrlplane Creates Releases

Ctrlplane finds release targets:

* API Service → Production → cluster-1
* API Service → Production → cluster-2

Creates 2 releases. Production releases are blocked by the approval policy.

### 3. User Approves

```bash theme={null}
ctrlc api approve release rel_123
```

Both releases are now eligible for job creation.

### 4. Ctrlplane Creates Jobs

Two jobs created:

* `job_1` for cluster-1
* `job_2` for cluster-2

Both jobs start with status: `pending`

### 5. Job Agent Polls

```bash theme={null}
GET /api/v1/job-agents/agent_k8s/queue/next
```

Receives both jobs.

### 6. Agent Processes Jobs

For each job:

```typescript theme={null}
// Acknowledge
await job.acknowledge();

// Execute
await kubectl.apply(manifest);

// Complete
await job.update({ status: 'completed' });
```

### 7. Jobs Complete

Both jobs complete successfully. The release target state updates: the release
becomes the **current release** for each target.

## Release Sequencing

By default, when a new version is created for a deployment:

* **Existing jobs** for older versions may be cancelled
* **New jobs** are created for the new version

This behavior is controlled by policies and can be configured.

### Sequential Releases

Ensure releases happen one at a time:

```yaml theme={null}
releaseSequencing:
  mode: sequential
```

### Parallel Releases

Allow multiple versions to deploy concurrently:

```yaml theme={null}
releaseSequencing:
  mode: parallel
```

## Deployment Tracing

For detailed observability, use the trace API to report execution steps:

```bash theme={null}
# In your deployment script
ctrlplane trace start "Deploy to Kubernetes"
ctrlplane trace step "Pull image" "completed"
ctrlplane trace step "Apply manifests" "completed"
ctrlplane trace end "completed" "Deployment successful"
```

Traces are linked to jobs and provide fine-grained visibility into deployment execution.

## Best Practices

### Job Status Updates

**Do**:

* ✅ Acknowledge jobs immediately when picked up
* ✅ Update status regularly during execution
* ✅ Provide descriptive messages
* ✅ Mark as completed or failed explicitly

**Don't**:

* ❌ Leave jobs in pending state indefinitely
* ❌ Skip acknowledging jobs
* ❌ Provide generic error messages

### Error Handling

When jobs fail:

1. Set status to `failed`
2. Include detailed error message
3. Store external job ID for log access
4. Consider automatic retry logic in agent

### Release Management

* Monitor pending releases (awaiting approval)
* Set up notifications for failed jobs
* Regularly review release history
* Clean up old completed releases (retention policy)

## Troubleshooting

### Releases not progressing

* Check if policies require approval
* Review policy evaluation results
* Verify environment progression requirements met
* Check for blocking policy rules

### Jobs not being picked up by agent

* Verify agent is running and polling
* Check agent ID matches deployment configuration
* Review job agent logs for errors
* Confirm agent has network access to API

### Jobs fail immediately

* Review job agent configuration
* Check external system (GitHub, Kubernetes) availability
* Verify credentials/permissions
* Review job logs and error messages

### Multiple jobs created for same release

* Check release sequencing configuration
* Review job creation logic
* May be intentional for retry scenarios

## Next Steps

* [Job Agents](../integrations/job-agents/github) - Set up and configure job agents
* [Policies](../policies/overview) - Control release approval and progression
* [CI/CD Integration](../integrations/cicd) - Integrate version creation into your CI
* [Verification](../policies/verification/overview) - Validate deployment health


# Resources
Source: https://docs.ctrlplane.dev/concepts/resources

Deployment targets in your infrastructure

A **Resource** represents a deployment target - the actual infrastructure where your code runs. Resources can be Kubernetes clusters, VMs, cloud functions, or any other compute environment.

## What is a Resource?

Resources are the "where" in your deployment pipeline. They represent:

* **Kubernetes clusters** (e.g., prod-us-east-1-cluster)
* **Virtual machines** (e.g., web-server-01)
* **Cloud functions** (e.g., lambda-handler-prod)
* **Containers** (e.g., ECS service)
* **Custom infrastructure** (anything you can deploy to)

## Resource Properties

### Core Fields

```yaml theme={null}
id: res_abc123
name: Production US East Cluster
kind: KubernetesCluster
identifier: k8s-prod-use1
version: "1.28.0"
workspaceId: ws_xyz789
providerId: provider_123
config:
  endpoint: https://k8s.prod.example.com
  region: us-east-1
metadata:
  environment: production
  region: us-east-1
  team: platform
  cost-center: engineering
createdAt: "2024-01-15T10:00:00Z"
updatedAt: "2024-01-15T10:00:00Z"
```

### name

Human-readable display name for the resource.

**Examples**:

* "Production US East Cluster"
* "Web Server 01"
* "Lambda Production Handler"

### kind

Classification of the resource type. Used for filtering and grouping.

**Common Values**:

* `kubernetes-cluster`
* `vm`
* `lambda-function`
* `ecs-service`
* `cloud-run-service`
* `server` (generic)

### identifier

Unique identifier for this resource. This is how external systems reference the resource.

**Examples**:

* `k8s-prod-use1`
* `i-0abc123def456789` (EC2 instance ID)
* `arn:aws:lambda:us-east-1:123456789:function:my-function`

**Requirements**:

* Must be unique within the workspace
* Should be stable (don't change frequently)
* Often matches the infrastructure's native identifier

### metadata

Key-value pairs used for classification and selector matching. This is crucial for resource targeting.

**Common Metadata Keys**:

```yaml theme={null}
metadata:
  environment: production
  region: us-east-1
  zone: us-east-1a
  team: platform
  cost-center: engineering
  tier: high-availability
  version: "1.28.0"
  managed-by: terraform
```

**Best Practices**:

* Use consistent key names across resources
* Use lowercase with hyphens: `cost-center` not `CostCenter`
* Include classification useful for targeting
* Don't put sensitive data in metadata

### config

Resource-specific configuration. Unlike metadata (which is for selection), config contains operational details.

**Examples**:

**Kubernetes Cluster**:

```yaml theme={null}
config:
  endpoint: https://k8s.prod.example.com
  certificateAuthority: "..."
  namespace: default
```

**VM**:

```yaml theme={null}
config:
  ipAddress: 10.0.1.50
  sshUser: deploy
  port: 22
```

**Lambda**:

```yaml theme={null}
config:
  functionName: my-function
  region: us-east-1
  runtime: nodejs20.x
```

### version

The current version or state of the resource itself (not the deployed application).

**Examples**:

* `1.28.0` (Kubernetes version)
* `20.04` (Ubuntu version)
* `nodejs20.x` (Lambda runtime)

### providerId

Reference to the Resource Provider that created/manages this resource. Optional for manually created resources.

## Creating Resources

### Via CLI

```yaml theme={null}
# resource.yaml
type: Resource
name: Production US East Cluster
kind: KubernetesCluster
identifier: k8s-prod-use1
version: "1.28.0"
metadata:
  environment: production
  region: us-east-1
  team: platform
config:
  endpoint: https://k8s.prod.example.com
```

```bash theme={null}
ctrlc apply -f resource.yaml
```

### Via Web UI

1. Navigate to your system
2. Go to "Resources" tab
3. Click "Create Resource"
4. Fill in the form:
   * Name, Kind, Identifier
   * Add metadata key-value pairs
   * Add config (JSON)
5. Click "Create"

### Automated Creation (Resource Providers)

**Recommended approach for production**: Use Resource Providers to automatically sync resources from your infrastructure.

Resource Providers continuously sync resources from external sources:

* **Kubernetes Provider**: Discovers clusters from kubeconfig
* **AWS Provider**: Syncs EC2 instances, ECS services, Lambda functions
* **GCP Provider**: Syncs GCE instances, Cloud Run services
* **Azure Provider**: Syncs VMs, container instances
* **Custom Provider**: Your own integration

**Example using Node SDK**:

```typescript theme={null}
import { createClient } from "@ctrlplane/node-sdk";

const client = createClient({
  baseUrl: "https://your-ctrlplane-instance.com",
  apiKey: process.env.CTRLPLANE_API_KEY,
});

const provider = client.resourceProvider({
  name: "AWS EC2 Provider",
  workspaceId: "ws_xyz789",
});

// Sync resources from AWS
const instances = await getEC2Instances(); // Your AWS SDK call

await provider.set(
  instances.map(instance => ({
    name: instance.Tags.Name,
    kind: "ec2-instance",
    identifier: instance.InstanceId,
    version: instance.ImageId,
    metadata: {
      environment: instance.Tags.Environment,
      region: instance.Placement.AvailabilityZone,
      instanceType: instance.InstanceType,
    },
    config: {
      privateIp: instance.PrivateIpAddress,
      publicIp: instance.PublicIpAddress,
    },
  }))
);
```

The provider automatically:

* Creates new resources
* Updates existing resources
* Removes resources no longer in the source

## Resource Metadata for Targeting

Metadata is how resources get matched to environments and deployments. Design your metadata schema carefully.

### Example Metadata Schema

```yaml theme={null}
# Metadata schema for consistent targeting
metadata:
  environment: production | staging | development
  region: us-east-1 | us-west-2 | eu-west-1
  zone: us-east-1a | us-east-1b | ...
  team: platform | product | data
  cluster-tier: high-availability | standard
  cost-center: engineering | sales | ...
  managed-by: terraform | manual
```

### Selector Matching Example

**Environment Configuration**:

```yaml theme={null}
type: Environment
name: Production US East
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

**Matched Resources**:

* ✅ Resource A: `{environment: "production", region: "us-east-1"}`
* ✅ Resource B: `{environment: "production", region: "us-east-1", team: "platform"}`
* ❌ Resource C: `{environment: "production", region: "us-west-2"}`
* ❌ Resource D: `{environment: "staging", region: "us-east-1"}`

## Resource Variables

Resources can have variables attached to them, which are available during job execution.

### Creating Resource Variables

Variables are typically configured via the Web UI. You can also use the API:

```bash theme={null}
ctrlc api create resource-variable \
  --resource {resourceId} \
  --key KUBERNETES_NAMESPACE \
  --value production
```

### Using in Job Execution

When a job executes on a resource, it receives all resource variables:

```yaml theme={null}
# In GitHub Actions workflow
- name: Deploy
  run: |
    kubectl apply -f manifest.yaml \
      --namespace ${{ steps.job.outputs.resource_variables_KUBERNETES_NAMESPACE }}
```

### Common Resource Variables

**Kubernetes Resources**:

* `KUBERNETES_NAMESPACE` - Target namespace
* `KUBERNETES_CONTEXT` - Kubectl context
* `HELM_RELEASE_NAME` - Helm release name

**VM Resources**:

* `SSH_HOST` - Host to SSH into
* `SSH_USER` - SSH username
* `DEPLOY_PATH` - Where to deploy files

**Lambda Resources**:

* `FUNCTION_NAME` - Lambda function name
* `AWS_REGION` - AWS region
* `AWS_ACCOUNT_ID` - AWS account

## Resource Lifecycle

### States

Resources don't have explicit states in Ctrlplane, but they can be:

1. **Active** - `deletedAt` is null, available for deployments
2. **Deleted** - `deletedAt` is set, excluded from deployments

### Updating Resources

Update resource metadata with a YAML file:

```yaml theme={null}
# resource-update.yaml
type: Resource
identifier: k8s-prod-use1
metadata:
  environment: production
  region: us-east-1
  updated: "2024-01-15"
```

```bash theme={null}
ctrlc apply -f resource-update.yaml
```

### Deleting Resources

Soft delete (recommended):

```bash theme={null}
ctrlc api delete resource {resourceId}
```

This sets `deletedAt` timestamp. The resource is excluded from new deployments but historical data is preserved.

## Querying Resources

### List All Resources

```bash theme={null}
ctrlc api get resources --workspace {workspaceId}
```

### Filter by Selector

```bash theme={null}
ctrlc api get resources \
  --workspace {workspaceId} \
  --selector 'resource.metadata["environment"] == "production"'
```

### Get Resource Details

```bash theme={null}
ctrlc api get resource {resourceId}
```

## Resource Providers

### Creating a Resource Provider

```typescript theme={null}
import { createClient } from "@ctrlplane/node-sdk";

const client = createClient({
  baseUrl: "https://your-ctrlplane-instance.com",
  apiKey: process.env.CTRLPLANE_API_KEY,
});

const provider = client.resourceProvider({
  name: "My Infrastructure Provider",
  workspaceId: "ws_xyz789",
});

await provider.get(); // Registers provider
```

### Syncing Resources

```typescript theme={null}
// Fetch resources from your infrastructure
const resources = await fetchInfrastructure();

// Sync to Ctrlplane
await provider.set(
  resources.map(r => ({
    name: r.name,
    kind: r.type,
    identifier: r.id,
    metadata: r.tags,
    config: r.config,
  }))
);
```

The provider:

* Creates resources that don't exist
* Updates resources that changed
* **Removes** resources not in the provided list (careful!)

### Provider Sync Strategy

**Full Sync** (default):

```typescript theme={null}
// All resources currently in infrastructure
await provider.set(allResources);
```

This removes resources not in the list. Good for keeping Ctrlplane in sync with source of truth.

**Incremental Updates**:
If you want to preserve manually created resources, filter by provider:

```typescript theme={null}
// Only sync resources managed by this provider
const managedResources = await fetchManagedResources();
await provider.set(managedResources);
```

## Best Practices

### Metadata Design

**Do**:

* ✅ Use consistent key names across all resources
* ✅ Include classification useful for targeting
* ✅ Use hierarchical values when appropriate (`region/zone`)
* ✅ Include ownership information (`team`, `cost-center`)

**Don't**:

* ❌ Put sensitive data in metadata (use config or variables)
* ❌ Use inconsistent naming (`env` vs `environment`)
* ❌ Include data that changes frequently
* ❌ Duplicate information already in other fields

### Resource Identifiers

**Good Identifiers**:

* ✅ `k8s-prod-us-east-1` - Descriptive and stable
* ✅ `i-0abc123def456789` - Native AWS instance ID
* ✅ `arn:aws:...` - Full ARN for AWS resources

**Poor Identifiers**:

* ❌ `cluster-1` - Not descriptive
* ❌ `10.0.1.50` - IP addresses can change
* ❌ `temp-cluster` - Suggests instability

### Resource Variables

* Use for environment-specific configuration
* Mark sensitive variables as `sensitive: true`
* Prefer resource variables over hardcoding in deployment config
* Use consistent variable naming across similar resources

### Provider Usage

* Use providers for production (keeps resources in sync)
* Run provider sync on a schedule (cron job, CI pipeline)
* Test provider sync in staging first
* Monitor provider sync failures

## Common Patterns

### Multi-Region Kubernetes

```yaml theme={null}
# multi-region-resources.yaml
---
type: Resource
name: Production US East
kind: KubernetesCluster
identifier: k8s-prod-use1
metadata:
  environment: production
  region: us-east-1
---
type: Resource
name: Production EU West
kind: KubernetesCluster
identifier: k8s-prod-euw1
metadata:
  environment: production
  region: eu-west-1
```

```bash theme={null}
ctrlc apply -f multi-region-resources.yaml
```

### Tiered Resources

```yaml theme={null}
# tiered-resources.yaml
---
type: Resource
name: Critical Production Cluster
kind: KubernetesCluster
identifier: k8s-prod-critical
metadata:
  environment: production
  tier: critical
---
type: Resource
name: Standard Production Cluster
kind: KubernetesCluster
identifier: k8s-prod-standard
metadata:
  environment: production
  tier: standard
```

```bash theme={null}
ctrlc apply -f tiered-resources.yaml
```

### Team-Based Resources

```yaml theme={null}
# team-resources.yaml
---
type: Resource
name: Platform Team Cluster
kind: KubernetesCluster
identifier: k8s-platform-team
metadata:
  team: platform
  environment: shared
---
type: Resource
name: Product Team Cluster
kind: KubernetesCluster
identifier: k8s-product-team
metadata:
  team: product
  environment: shared
```

```bash theme={null}
ctrlc apply -f team-resources.yaml
```

## Troubleshooting

### Resource not appearing in environment

* Check the environment's resource selector
* Verify resource metadata matches the selector
* Confirm resource is not deleted (`deletedAt` is null)

### Provider sync not working

* Verify API key has correct permissions
* Check provider name matches
* Review provider sync logs for errors

### Duplicate resources created

* Ensure identifier is truly unique
* Check if provider is creating duplicates
* Review provider logic for identifier generation

## Next Steps

* [Environments](./environments) - Group resources into environments
* [Selectors](./selectors) - Learn selector syntax for targeting resources
* [Resource Providers](../integrations/resource-providers/overview) - Set up automated resource sync


# Selectors
Source: https://docs.ctrlplane.dev/concepts/selectors

Powerful filtering system for targeting resources, environments, and deployments

**Selectors** are Ctrlplane's powerful filtering system for targeting resources, environments, and deployments. They allow dynamic, rules-based matching instead of static assignments.

## Why Selectors?

Traditional deployment tools require manually assigning resources to environments. Ctrlplane uses selectors for dynamic matching:

### Without Selectors (Traditional)

```
Production Environment:
  ├─ Manually add: cluster-1
  ├─ Manually add: cluster-2
  └─ Manually add: cluster-3

New cluster added? Manually add it to production.
```

### With Selectors (Ctrlplane)

```
Production Environment:
  └─ Selector: metadata.environment == "production"

New cluster with metadata.environment = "production"?
Automatically included in production!
```

## Selector Types

Ctrlplane supports several types of selectors:

1. **Metadata Selector** - Match on resource metadata
2. **Kind Selector** - Match on resource kind
3. **Identifier Selector** - Match on resource identifier
4. **Name Selector** - Match on resource/environment name
5. **CEL Selector** - Complex expressions using CEL (Common Expression Language)
6. **Composite Selectors** - Combine selectors with AND/OR logic

## Metadata Selector

Match resources based on metadata key-value pairs using CEL expressions.

### Simple Equality

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

Matches resources where `metadata.environment === "production"`.

### Supported Operators

#### equals

Exact match:

```yaml theme={null}
resourceSelector: resource.metadata["region"] == "us-east-1"
```

#### not\_equals

Does not match:

```yaml theme={null}
resourceSelector: resource.metadata["deprecated"] != "true"
```

#### contains

Value contains substring:

```yaml theme={null}
resourceSelector: resource.identifier.contains("prod")
```

Matches: `prod-cluster-1`, `k8s-prod`, `production-server`.

#### matches (Regex)

Regular expression match:

```yaml theme={null}
resourceSelector: resource.identifier.matches("^prod-.*-[0-9]+$")
```

Matches: `prod-cluster-1`, `prod-server-42`.

#### exists

Key exists with any value:

```yaml theme={null}
resourceSelector: has(resource.metadata["team"])
```

Matches any resource with a `team` metadata key.

#### not\_exists

Key does not exist:

```yaml theme={null}
resourceSelector: "!has(resource.metadata[\"deprecated\"])"
```

#### in

Value is in a list:

```yaml theme={null}
resourceSelector: resource.metadata["region"] in ["us-east-1", "us-east-2", "us-west-1"]
```

#### not\_in

Value is not in a list:

```yaml theme={null}
resourceSelector: "!(resource.metadata[\"status\"] in [\"deprecated\", \"decommissioned\"])"
```

## Kind Selector

Match resources by their `kind` field.

```yaml theme={null}
resourceSelector: resource.kind == "KubernetesCluster"
```

**Common Use Cases**:

* Kubernetes-only deployments: `resource.kind == "KubernetesCluster"`
* VM-only deployments: `resource.kind == "VirtualMachine"`
* Exclude certain kinds: `resource.kind != "DeprecatedResource"`

## Identifier Selector

Match resources by their identifier.

```yaml theme={null}
resourceSelector: resource.identifier.contains("prod")
```

**Use Cases**:

* Match specific naming patterns
* Filter by identifier prefix/suffix
* Regex matching on identifiers

## Name Selector

Match by the resource or environment name field.

```yaml theme={null}
# In policy selectors
environments: environment.name == "Production"
```

**Use Cases**:

* Policy selectors targeting specific environments
* Matching resources by name patterns

## Composite Selectors (AND/OR)

Combine multiple conditions with boolean logic using CEL operators.

### AND Logic

All conditions must match:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

Matches: Resources with BOTH `environment=production` AND `region=us-east-1`.

### OR Logic

Any condition can match:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["region"] == "us-east-1" ||
  resource.metadata["region"] == "us-west-2"
```

Matches: Resources in EITHER `us-east-1` OR `us-west-2`.

### Nested Logic

Combine AND and OR:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  (resource.metadata["region"] == "us-east-1" ||
   resource.metadata["region"] == "us-west-2")
```

Matches: `environment=production` AND (`region=us-east-1` OR `region=us-west-2`).

## CEL Expressions

Ctrlplane uses CEL (Common Expression Language) for all selectors:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["tier"] == "critical"
```

### CEL Basics

**Field Access**:

* `resource.metadata["environment"]` - Access metadata field
* `resource.kind` - Access kind field
* `resource.name` - Access name field
* `resource.identifier` - Access identifier field

**Operators**:

* `==`, `!=` - Equality
* `&&`, `||` - Logical AND/OR
* `!` - Logical NOT
* `<`, `>`, `<=`, `>=` - Comparison
* `in` - List membership
* `.matches()` - Regex match

**Examples**:

```yaml theme={null}
# Production OR Staging
resourceSelector: resource.metadata["environment"] in ["production", "staging"]

# Critical tier in production
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["tier"] == "critical"

# Not deprecated
resourceSelector: >-
  !has(resource.metadata["deprecated"]) ||
  resource.metadata["deprecated"] != "true"

# Regex match on identifier
resourceSelector: resource.identifier.matches("^prod-cluster-[0-9]+$")

# Complex nested logic
resourceSelector: >-
  (resource.metadata["environment"] == "production" &&
   resource.metadata["region"] in ["us-east-1", "us-west-2"]) ||
  (resource.metadata["environment"] == "staging" &&
   resource.metadata["region"] == "us-east-1")
```

### CEL Best Practices

**Do**:

* ✅ Use CEL for complex nested logic
* ✅ Keep expressions readable with line breaks
* ✅ Test selectors before applying
* ✅ Reference the [CEL Reference](../reference/cel) for syntax

**Avoid**:

* ❌ Overly complex expressions that are hard to understand
* ❌ Expressions that might accidentally match wrong resources

## Selector Use Cases

### Environment Resource Selectors

Define which resources belong to an environment:

```yaml theme={null}
# Production environment
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```yaml theme={null}
# Multi-region production
type: Environment
name: Production US East
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

### Deployment Resource Selectors

Limit which resources a deployment can target:

```yaml theme={null}
# Kubernetes-only deployment
type: Deployment
name: Container Service
resourceSelector: resource.kind == "KubernetesCluster"
```

```yaml theme={null}
# Region-specific deployment
type: Deployment
name: US-Only Service
resourceSelector: resource.metadata["region"].matches("^us-.*")
```

### Policy Selectors

Target policies to specific deployments/environments/resources:

```yaml theme={null}
type: Policy
name: Production Approval Required
selectors:
  - environments: environment.name == "Production"
    deployments: deployment.metadata["critical"] == "true"
rules:
  - anyApproval:
      minApprovals: 2
```

This policy applies to critical deployments going to production.

## Testing Selectors

### Via CLI

Test a selector against your resources:

```bash theme={null}
ctrlc api get resources \
  --workspace {workspaceId} \
  --selector 'resource.metadata["environment"] == "production"'
```

Returns all resources matching the selector.

### Via Web UI

Most selector configuration screens have a "Test Selector" or "Preview Resources" button that shows which resources match.

## Common Selector Patterns

### Pattern 1: Environment-Based

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

**Use Case**: Standard environment separation (dev/staging/prod).

### Pattern 2: Region-Based

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

**Use Case**: Multi-region deployments, gradual regional rollout.

### Pattern 3: Team-Based

```yaml theme={null}
resourceSelector: resource.metadata["team"] == "platform"
```

**Use Case**: Team-specific resources and deployments.

### Pattern 4: Tier-Based

```yaml theme={null}
resourceSelector: resource.metadata["tier"] in ["critical", "high-priority"]
```

**Use Case**: SLA-based resource grouping.

### Pattern 5: Canary

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["canary"] == "true"
```

**Use Case**: Canary deployment pattern.

### Pattern 6: Percentage-Based (using identifier)

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.identifier.matches(".*[02468]$")
```

**Use Case**: Target resources with even-numbered identifiers (50% rollout).

## Best Practices

### Metadata Schema Design

Design your resource metadata with selectors in mind:

**Good Metadata Schema**:

```yaml theme={null}
# Recommended metadata keys
metadata:
  environment: production | staging | development
  region: us-east-1 | us-west-2 | eu-west-1
  zone: us-east-1a | us-east-1b | ...
  tier: critical | high | standard | low
  team: platform | product | data
  canary: "true" | "false"
```

**Why Good**:

* Consistent key names
* Enumerated values (not free-form)
* Multiple dimensions for targeting
* Clear, hierarchical organization

### Selector Simplicity

**Prefer Simple Selectors**:

```yaml theme={null}
resourceSelector: resource.metadata["environment"] == "production"
```

**Over Complex Ones**:

```yaml theme={null}
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"].startsWith("us-") &&
  resource.metadata["tier"] == "critical" &&
  !has(resource.metadata["deprecated"])
```

Simple selectors are:

* Easier to understand
* Less error-prone
* More performant

### Document Complex Selectors

If you must use complex selectors, document them:

```yaml theme={null}
type: Environment
name: Production Critical
description: >-
  Targets production resources in US regions that are marked 
  critical tier and not deprecated
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"].startsWith("us-") &&
  resource.metadata["tier"] == "critical" &&
  !has(resource.metadata["deprecated"])
```

### Test Before Applying

Always test selectors before using them:

1. Use the query API to see matched resources
2. Verify count matches expectations
3. Check for unexpected matches
4. Test in lower environment first

### Avoid Over-Matching

Be specific to avoid accidentally matching wrong resources:

**Too Broad**:

```yaml theme={null}
resourceSelector: resource.identifier.contains("prod")
```

Might match: `prod-cluster`, `product-server`, `reproduction-env`.

**Better**:

```yaml theme={null}
resourceSelector: resource.identifier.matches("^prod-.*")
```

Only matches identifiers starting with `prod-`.

## Troubleshooting

### Selector matches too many resources

* Make selector more specific (add conditions)
* Use AND instead of OR
* Add negation conditions
* Check resource metadata for unexpected values

### Selector matches too few resources

* Check metadata key names match exactly (case-sensitive)
* Verify resources have the expected metadata
* Use OR logic if resources vary
* Test with simpler selector first

### Selector matches nothing

* Verify selector syntax is correct
* Check resources exist with expected metadata
* Test each condition independently
* Review resource metadata in UI

### CEL expression errors

* Check syntax (use CEL validator)
* Verify field names are correct
* Test expression step-by-step
* Consult [CEL Reference](../reference/cel) or [CEL specification](https://github.com/google/cel-spec)

## Advanced Topics

### Selector Evaluation Order

When multiple selectors apply (e.g., environment + deployment):

1. Environment selector filters resources
2. Deployment selector (if present) filters further
3. Final set is intersection of both

### Selector Performance

Selectors are evaluated:

* When resources are created/updated
* When environments/deployments are created/updated
* When querying resources

**Performance Tips**:

* Simple selectors are faster than complex ones
* Metadata equality checks are very fast
* Regex matching is slower
* CEL expressions have some overhead

### Selector Caching

Ctrlplane caches selector evaluation results:

* Cache invalidated on resource/metadata changes
* Improves performance for repeated queries
* No manual cache management needed

## Next Steps

* [CEL Reference](../reference/cel) - Full CEL expression language reference
* [Environments](./environments) - Use selectors in environments
* [Deployments](./deployments) - Filter deployment targets
* [Resources](./resources) - Design resource metadata
* [Policies](../policies/overview) - Target policies with selectors


# Systems
Source: https://docs.ctrlplane.dev/concepts/systems

Logical groupings of related deployments and environments

A **System** is the top-level organizational unit in Ctrlplane. It represents a logical grouping of related deployments, environments, and resources.

## What is a System?

Think of a system as a workspace for a product, platform, or major component of your infrastructure. It contains everything related to deploying that product:

* **Deployments** - The services/applications you deploy
* **Environments** - Where you deploy (dev, staging, prod)
* **Resources** - The infrastructure targets
* **Policies** - Rules governing deployments
* **Variables** - Configuration values

## When to Create a System

Create a system when you have:

✅ **A product or platform** - E.g., "E-commerce Platform", "Data Pipeline"\
✅ **Multiple related services** - Microservices that work together\
✅ **Shared deployment policies** - Common approval/progression rules\
✅ **Logical separation** - Different teams, security boundaries

## System Structure

```txt theme={null}
System: E-commerce Platform
├── Deployments
│   ├── API Service
│   ├── Frontend App
│   ├── Payment Service
│   └── Notification Service
├── Environments
│   ├── Development
│   ├── Staging
│   └── Production
├── Resources
│   ├── dev-cluster-1
│   ├── staging-cluster-1
│   ├── prod-cluster-us-east
│   └── prod-cluster-us-west
└── Policies
    ├── Production Approval Required
    └── Environment Progression (Staging → Production)
```

## Creating a System

### Via Web UI

1. Navigate to your workspace
2. Click "Create System"
3. Fill in the form:
   * **Name**: Display name (e.g., "E-commerce Platform")
   * **Slug**: URL-friendly identifier (e.g., "ecommerce-platform")
   * **Description**: What this system encompasses

### Via CLI

```yaml theme={null}
# system.yaml
type: System
name: E-commerce Platform
slug: ecommerce-platform
description: Complete e-commerce system including API, frontend, and backend services
```

```bash theme={null}
ctrlc apply -f system.yaml
```

### Full System Configuration

Create a system with deployments and environments in one file:

```yaml theme={null}
# ecommerce-system.yaml
---
type: System
name: E-commerce Platform
slug: ecommerce-platform
description: Complete e-commerce system
---
type: Deployment
name: API Service
slug: api-service
description: Backend API
jobAgent:
  ref: github-actions
---
type: Deployment
name: Frontend App
slug: frontend-app
description: Customer-facing web application
jobAgent:
  ref: github-actions
---
type: Environment
name: Development
description: Development environment
resourceSelector: resource.metadata["environment"] == "development"
---
type: Environment
name: Production
description: Production environment
resourceSelector: resource.metadata["environment"] == "production"
```

```bash theme={null}
ctrlc apply -f ecommerce-system.yaml
```

## System Properties

### Name

* Display name shown in the UI
* Can contain spaces and special characters
* Can be changed without affecting existing deployments

### Slug

* URL-friendly identifier
* Must be unique within workspace
* Used in API paths and CLI commands
* Cannot be changed after creation
* Format: lowercase letters, numbers, hyphens

### Description

* Optional text describing the system's purpose
* Supports markdown formatting
* Displayed in the UI

### Workspace ID

* The parent workspace this system belongs to
* Automatically set when creating a system
* Cannot be changed

## Organizing with Multiple Systems

### When to Use Multiple Systems

Use separate systems when:

1. **Team Boundaries**

   ```txt theme={null}
   System: Platform Team Services
   System: Product Team Services
   ```

2. **Security Boundaries**

   ```txt theme={null}
   System: Public Services
   System: Internal Services
   System: PCI Compliant Services
   ```

3. **Deployment Independence**

   ```txt theme={null}
   System: Core Platform
   System: Analytics Platform
   System: ML Platform
   ```

4. **Different Policies**

   ```txt theme={null}
   System: Experimental Features (auto-deploy)
   System: Production Services (requires approval)
   ```

### When to Use One System

Keep things in one system when:

1. **Tightly Coupled Services**
   * Microservices that must deploy together
   * Services with shared dependencies

2. **Shared Deployment Pipeline**
   * Same approval requirements
   * Same environment progression

3. **Single Team/Product**
   * One team owns all services
   * Deployed as a unit

## System-Level Operations

### Viewing System Status

Get an overview of all deployments in a system:

```bash theme={null}
ctrlc api get system {systemId} --status
```

**Response**:

```yaml theme={null}
systemId: sys_abc123
name: E-commerce Platform
deployments: 4
environments: 3
resources: 8
activeReleases: 12
pendingApprovals: 2
failedJobs: 1
```

### Listing All Entities

**List Deployments**:

```bash theme={null}
ctrlc api get deployments --system {systemId}
```

**List Environments**:

```bash theme={null}
ctrlc api get environments --system {systemId}
```

**List Policies**:

```bash theme={null}
ctrlc api get policies --system {systemId}
```

### Deleting a System

⚠️ **Warning**: Deleting a system removes all associated deployments, releases, and jobs. This cannot be undone.

```bash theme={null}
ctrlc api delete system {systemId}
```

## Best Practices

### Naming Conventions

**Good Names**:

* ✅ "E-commerce Platform"
* ✅ "Analytics Pipeline"
* ✅ "Customer Portal"

**Good Slugs**:

* ✅ `ecommerce-platform`
* ✅ `analytics-pipeline`
* ✅ `customer-portal`

**Avoid**:

* ❌ "System 1", "Test", "Prod" (too generic)
* ❌ `My_System`, `PROD-SYSTEM` (inconsistent formatting)

### Start Simple

Begin with one system for your product:

```
System: My Application
  ├── API Deployment
  ├── Frontend Deployment
  ├── Dev Environment
  └── Prod Environment
```

Split into multiple systems as you grow:

```
System: Customer-Facing Services
  ├── API Deployment
  └── Frontend Deployment

System: Internal Services
  ├── Admin API Deployment
  └── Analytics Deployment
```

### Documentation

Use the description field to document:

* Purpose of the system
* Ownership/team responsible
* Key dependencies
* Links to external documentation

Example:

```
E-commerce platform for online sales.

Owner: Platform Team
Slack: #platform-team
Docs: https://wiki.company.com/ecommerce
```

### Consistent Structure

Maintain consistent structure across systems:

* Standard environment names (dev, staging, prod)
* Similar deployment naming patterns
* Common policy approaches

## Common Patterns

### Microservices System

```yaml theme={null}
# microservices-system.yaml
---
type: System
name: Microservices Platform
slug: microservices-platform
---
type: Deployment
name: User Service
slug: user-service
jobAgent:
  ref: kubernetes-agent
---
type: Deployment
name: Order Service
slug: order-service
jobAgent:
  ref: kubernetes-agent
---
type: Deployment
name: Payment Service
slug: payment-service
jobAgent:
  ref: kubernetes-agent
---
type: Environment
name: Development
resourceSelector: resource.metadata["environment"] == "development"
---
type: Environment
name: Staging
resourceSelector: resource.metadata["environment"] == "staging"
---
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```bash theme={null}
ctrlc apply -f microservices-system.yaml
```

### Multi-Region System

```yaml theme={null}
# global-system.yaml
---
type: System
name: Global Application
slug: global-application
---
type: Deployment
name: API Service
slug: api-service
jobAgent:
  ref: kubernetes-agent
---
type: Environment
name: Production US East
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
---
type: Environment
name: Production EU West
resourceSelector: >-
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "eu-west-1"
```

```bash theme={null}
ctrlc apply -f global-system.yaml
```

### Monorepo with Multiple Apps

```yaml theme={null}
# monorepo-system.yaml
---
type: System
name: Monorepo Applications
slug: monorepo-applications
---
type: Deployment
name: Web App
slug: web-app
jobAgent:
  ref: github-actions
---
type: Deployment
name: Mobile API
slug: mobile-api
jobAgent:
  ref: github-actions
---
type: Deployment
name: Admin Dashboard
slug: admin-dashboard
jobAgent:
  ref: github-actions
---
type: Environment
name: Staging
resourceSelector: resource.metadata["environment"] == "staging"
---
type: Environment
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

```bash theme={null}
ctrlc apply -f monorepo-system.yaml
```

## Troubleshooting

### System not appearing in UI

* Verify system was created successfully
* Check you're in the correct workspace
* Refresh the page

### Cannot create deployment in system

* Verify you have permissions for the system
* Check system ID is correct
* Ensure deployment slug is unique within system

### System deletion fails

* Active releases may prevent deletion
* Cancel or complete all active releases first
* Or force delete via API with `force=true` parameter

## Next Steps

* [Resources](./resources) - Define deployment targets
* [Environments](./environments) - Create deployment stages
* [Deployments](./deployments) - Set up your first deployment


# Deployment Overview
Source: https://docs.ctrlplane.dev/deployment/overview

Progressive delivery with policy-driven orchestration

Ctrlplane's deployment system orchestrates releases across environments with
configurable policies, verification, and approval workflows.

## What is Deployment Orchestration?

Deployment orchestration manages how releases flow through your environments —
from build to production — with:

* **Gradual rollouts** — Deploy to targets sequentially with verification
  between each
* **Policy-driven gates** — Approvals, verification, and dependencies
* **Environment promotion** — Automated staging → production progression
* **Rollback & recovery** — Automatic rollback on verification failure

```mermaid theme={null}
flowchart LR
    Build["Build (CI)"] --> Version["Version"]

    subgraph Staging["Staging"]
        direction TB
        S1["Deploy"] --> S2["Verify"]
    end

    subgraph Production["Production"]
        direction TB
        P0["Approval"] --> P1["Deploy"] --> P2["Verify"]
    end

    Version --> Staging
    Staging -->|"policy gates"| Production
```

## Core Concepts

<CardGroup>
  <Card title="Systems" icon="sitemap" href="../concepts/systems">
    Logical groupings of related deployments
  </Card>

  <Card title="Deployments" icon="rocket" href="../concepts/deployments">
    Services or applications to deploy
  </Card>

  <Card title="Releases & Jobs" icon="play" href="../concepts/releases-and-jobs">
    Deployment execution units
  </Card>

  <Card title="Release Targets" icon="bullseye" href="../concepts/release-targets">
    Deployment × Environment × Resource
  </Card>
</CardGroup>

## How It Works

### 1. CI Creates Versions

Your CI pipeline creates a version after successful builds:

```yaml theme={null}
- name: Create Version
  env:
    CTRLPLANE_API_KEY: ${{ secrets.CTRLPLANE_API_KEY }}
  run: |
    ctrlc api upsert version \
      --workspace ${{ vars.CTRLPLANE_WORKSPACE }} \
      --deployment ${{ vars.CTRLPLANE_DEPLOYMENT_ID }} \
      --tag ${{ github.sha }} \
      --name "Build #${{ github.run_number }}" \
      --metadata git/commit=${{ github.sha }} \
      --metadata git/branch=${{ github.ref_name }}
```

### 2. Ctrlplane Creates Releases

For each release target (deployment × environment × resource), Ctrlplane:

1. Evaluates policies (approvals, gates, dependencies)
2. Creates a release with the new version
3. Dispatches a job to the job agent

### 3. Job Agents Execute

Job agents perform the actual deployment:

* **GitHub Actions** — Trigger workflows
* **ArgoCD** — Create/sync Applications
* **Terraform Cloud** — Create workspaces and runs

### 4. Verification Validates

After deployment, verification checks health metrics (error rates, latency,
etc.) and automatically determines whether the release should proceed or roll
back. See [Verification](../policies/verification/overview) for details.

## Defining a Deployment

A deployment connects *what* (your service) with *how* (the job agent that
executes it). You can define deployments via Terraform, CLI, or API.

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_deployment" "api" {
      name              = "API Service"
      resource_selector = "resource.kind == 'Kubernetes' && resource.metadata['status'] == 'running'"

      metadata = {
        team    = "backend"
        service = "api"
      }

      job_agent {
        id = ctrlplane_job_agent.github.id

        github {
          owner       = "my-org"
          repo        = "api-service"
          workflow_id = 12345678
        }
      }
    }

    resource "ctrlplane_deployment_system_link" "api" {
      deployment_id = ctrlplane_deployment.api.id
      system_id     = ctrlplane_system.example.id
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/deployments \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "API Service",
        "slug": "api-service",
        "resourceSelector": "resource.kind == '\''Kubernetes'\''",
        "jobAgents": [
          {
            "ref": "github-actions-agent",
            "config": {
              "owner": "my-org",
              "repo": "api-service",
              "workflow": "deploy.yml"
            }
          }
        ],
        "metadata": { "team": "backend" }
      }'
    ```
  </Tab>
</Tabs>

See [Deployments](../concepts/deployments) for the full reference on
properties, versions, variables, and more.

## Policies

Policies control how releases flow through environments:

<CardGroup>
  <Card title="Approval" icon="check" href="../policies/approval">
    Require sign-off before deployment
  </Card>

  <Card title="Environment Progression" icon="arrow-right" href="../policies/environment-progression">
    Gate production on staging success
  </Card>

  <Card title="Gradual Rollouts" icon="chart-line" href="../policies/gradual-rollouts">
    Stagger deployments across targets
  </Card>

  <Card title="Verification" icon="shield" href="../policies/verification/overview">
    Validate deployment health
  </Card>

  <Card title="Deployment Dependency" icon="link" href="../policies/deployment-dependency">
    Enforce ordering between deployments
  </Card>

  <Card title="Deployment Window" icon="clock" href="../policies/deployment-window">
    Time-based deployment scheduling
  </Card>

  <Card title="Version Cooldown" icon="hourglass" href="../policies/version-cooldown">
    Batch frequent releases
  </Card>

  <Card title="Version Selector" icon="filter" href="../policies/version-selector">
    Control which versions can deploy
  </Card>

  <Card title="Retry" icon="rotate-right" href="../policies/retry">
    Automatic retry on failure
  </Card>
</CardGroup>

## Job Agents

Execute deployments on your infrastructure:

<CardGroup>
  <Card title="GitHub Actions" icon="github" href="../integrations/job-agents/github">
    Trigger workflow dispatch
  </Card>

  <Card title="ArgoCD" icon="rotate" href="../integrations/job-agents/argocd">
    GitOps deployments
  </Card>

  <Card title="Terraform Cloud" icon="cloud" href="../integrations/job-agents/terraform-cloud">
    Infrastructure as code
  </Card>
</CardGroup>

## Key Benefits

| Benefit                | Description                           |
| ---------------------- | ------------------------------------- |
| **Consistent process** | Same workflow for all deployments     |
| **Policy enforcement** | Gates prevent unauthorized releases   |
| **Visibility**         | Track what's deployed where           |
| **Automatic rollback** | Failed verification triggers rollback |

## Next Steps

* [Systems](../concepts/systems) — Organize your deployments
* [CI/CD Integration](../integrations/cicd) — Connect your build pipeline
* [Policies](../policies/overview) — Configure deployment rules
* [Job Agents](../integrations/job-agents/github) — Set up execution


# Version Dependencies
Source: https://docs.ctrlplane.dev/deployment/version-dependencies

Declare per-version dependencies between deployments so a version only deploys to a resource when an upstream deployment is running an acceptable version on that same resource.

A **version dependency** is a hard, per-version edge from a deployment version
to another deployment. The version will only roll out to a resource when the
upstream deployment is *currently deployed* on that resource and its current
version satisfies a CEL `versionSelector`.

Use version dependencies to pin compatibility against a specific upstream cut
(e.g. `frontend v3.4.0` requires `api ≥ v2.1.0`) — without baking that rule
into a workspace-wide policy.

## How It Works

```mermaid theme={null}
flowchart LR
    V[frontend v3.4.0] -- requires --> D[api deployment]
    D -. current version on resource .-> S{satisfies<br/>versionSelector?}
    S -- yes --> Allow[Allow deploy]
    S -- no / not deployed --> Block[Block]
```

1. **Resolution is per-resource.** For each resource the version targets,
   Ctrlplane looks up the upstream deployment's current release on that same
   resource.
2. **Current = last successful release.** If the upstream has no successful
   release on the resource, the dependency is **unsatisfied** and the version
   is blocked on that target.
3. **CEL evaluation.** The `versionSelector` is evaluated against the
   upstream's current `version.*`. All declared edges must pass.
4. **Reconciliation.** When an upstream deployment successfully deploys a new
   version on a resource, every downstream version with an edge to it is
   re-evaluated for that resource — so a previously-blocked version can
   automatically unblock.

## Version Dependencies vs. Deployment-Dependency Policy

Both gate a deployment on another, but they're different tools:

|                     | Version Dependency                                               | [Deployment Dependency Policy](../policies/deployment-dependency)                 |
| ------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Lives on            | A specific **version**                                           | A **policy** matched by CEL `selector`                                            |
| Pin granularity     | Per-version (different versions can require different upstreams) | Per-deployment (uniform across all versions matching the policy)                  |
| CEL scope           | `version.*` of upstream's current release                        | `deployment.*` and `version.*` of any successful upstream release on the resource |
| Typical author      | CI / the build that produced the version                         | Platform / SRE author authoring a workspace rule                                  |
| Edit after creation | Yes, via the dependency endpoints                                | Edit the policy                                                                   |

Rule of thumb: if the requirement comes from the **build** itself
("this build of frontend needs api ≥ v2"), use a version dependency. If it
comes from an **operational rule** ("api always waits for db migrations"),
use a deployment-dependency policy.

## Creating Version Dependencies

### Inline at version creation

The `POST .../deployments/{deploymentId}/versions` endpoint accepts a
`dependencies` map keyed by **upstream deployment ID**. The dependency edges
are inserted in the same transaction as the version itself, so reconciliation
never sees a version with a missing edge.

```bash theme={null}
curl -X POST "https://api.ctrlplane.com/v1/workspaces/$WORKSPACE_ID/deployments/$DEPLOYMENT_ID/versions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tag": "v3.4.0",
    "name": "Release 3.4.0",
    "status": "ready",
    "dependencies": {
      "dep_api_uuid": {
        "versionSelector": "version.tag.startsWith(\"v2.\")"
      },
      "dep_auth_uuid": {
        "versionSelector": "version.metadata.channel == \"stable\""
      }
    }
  }'
```

If any selector is malformed or any dependency deployment doesn't exist in the
workspace, the whole request 4xxs and no version is created.

### Upsert a single dependency

To add or change one edge after the version exists:

```bash theme={null}
curl -X PUT "https://api.ctrlplane.com/v1/workspaces/$WORKSPACE_ID/deployment-versions/$VERSION_ID/dependencies/$DEPENDENCY_DEPLOYMENT_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "versionSelector": "version.tag.startsWith(\"v2.\")"
  }'
```

Returns `202 Accepted`; downstream release targets are re-queued for
re-evaluation.

### List dependencies for a version

```bash theme={null}
curl "https://api.ctrlplane.com/v1/workspaces/$WORKSPACE_ID/deployment-versions/$VERSION_ID/dependencies" \
  -H "Authorization: Bearer $TOKEN"
```

### Delete a dependency

```bash theme={null}
curl -X DELETE "https://api.ctrlplane.com/v1/workspaces/$WORKSPACE_ID/deployment-versions/$VERSION_ID/dependencies/$DEPENDENCY_DEPLOYMENT_ID" \
  -H "Authorization: Bearer $TOKEN"
```

## `versionSelector` Reference

The selector is a CEL expression evaluated against the upstream deployment's
**current** version on the resource:

| Variable            | Type      | Description                                      |
| ------------------- | --------- | ------------------------------------------------ |
| `version.id`        | string    | Upstream version ID                              |
| `version.tag`       | string    | Upstream version tag (e.g. `v2.1.0`)             |
| `version.name`      | string    | Upstream version name                            |
| `version.status`    | string    | Upstream version status                          |
| `version.metadata`  | map       | Upstream version metadata (`map<string,string>`) |
| `version.createdAt` | timestamp | When the upstream version was created            |

`deployment.*` and `environment.*` are **not** in scope — this selector only
filters on the upstream version. Use a deployment-dependency policy if you
need to gate on `deployment.*` or `environment.*`.

### Common Selectors

```cel theme={null}
// Require any v2.x release of the upstream
version.tag.startsWith("v2.")

// Require a specific minimum tag (lexicographic)
version.tag >= "v2.1.0"

// Require a stable channel
version.metadata.channel == "stable"

// Always require *some* successful upstream release (any version)
true

// Pin to a specific upstream version
version.tag == "v2.4.7"
```

## Constraints

* A version cannot depend on **its own deployment** (`400`).
* Both the version and every dependency deployment must live in the same
  workspace (`404` otherwise).
* The `(deploymentVersionId, dependencyDeploymentId)` pair is unique — `PUT`
  upserts the `versionSelector` for that pair.
* Edges cascade-delete with the version or the dependency deployment.

## Reconciliation Behavior

Version dependencies hook into the release-policy evaluator and the
job-dispatch downstream trigger:

* **At evaluation time**, every declared edge must pass; the first failing
  edge denies the release.
* **On upstream success**, each downstream deployment that has *any* version
  declaring an edge to the upstream is re-queued so previously-blocked
  versions can flip to allowed without manual intervention.

This means you can ship a frontend version that requires `api v2.x`
*before* the API has actually rolled out — it will sit blocked, and unblock
itself the moment the API reaches `v2.x` on each resource.

## Next Steps

* [Deployment Dependency Policy](../policies/deployment-dependency) — the
  policy-based, deployment-wide variant
* [Deployment Overview](./overview) — how versions become releases
* [CEL Reference](../reference/cel) — the expression language


# Introduction
Source: https://docs.ctrlplane.dev/index

Deployment orchestration and infrastructure inventory for platform teams

## What is Ctrlplane?

**Ctrlplane is the orchestration layer between your CI/CD pipelines and your
infrastructure.**

Your CI builds code. Your infrastructure runs it. Ctrlplane orchestrates the
rest through three core concepts:

| Concept         | Question It Answers            | Example                                        |
| --------------- | ------------------------------ | ---------------------------------------------- |
| **Deployment**  | **What** to deploy and **how** | "API Service" deployed via ArgoCD              |
| **Environment** | **Where** to deploy            | "Production" = all clusters with `env: prod`   |
| **Policy**      | **When** to deploy             | "Only after staging succeeds and SRE approves" |

```
Your CI/CD    ──►    Ctrlplane    ──►    Your Infrastructure
 (builds)         (what, where, when)         (deploys)
```

## Start Here

<CardGroup>
  <Card title="Why Ctrlplane?" icon="lightbulb" href="./why-ctrlplane">
    See the problems Ctrlplane solves—you'll recognize them
  </Card>

  <Card title="5-Minute Overview" icon="clock" href="./overview">
    Understand the mental model quickly
  </Card>

  <Card title="Quickstart" icon="rocket" href="./quickstart">
    Set up your first deployment pipeline in 15 minutes
  </Card>

  <Card title="Comparisons" icon="scale-balanced" href="./comparisons">
    How Ctrlplane fits with ArgoCD, Spinnaker, GitHub Actions
  </Card>
</CardGroup>

## Two Core Systems

<CardGroup>
  <Card title="Deployment Orchestration" icon="rocket" href="./use-cases/environment-promotion">
    Control **when** and **where** releases happen:

    * Auto-promote after verification passes
    * Approval gates for production
    * Gradual rollouts across regions
    * Automatic rollback on failure
  </Card>

  <Card title="Infrastructure Inventory" icon="server" href="./use-cases/infrastructure-inventory">
    Track **what exists** and **what's running**:

    * Real-time resource inventory
    * Dynamic environment membership
    * Version tracking across all targets
    * Works with K8s, AWS, GCP, custom
  </Card>
</CardGroup>

## Common Use Cases

| Scenario                                                         | What Ctrlplane Does                                               |
| ---------------------------------------------------------------- | ----------------------------------------------------------------- |
| [Multi-region deployments](./use-cases/multi-region)             | Deploy to 10 clusters sequentially with verification between each |
| [Environment promotion](./use-cases/environment-promotion)       | Auto-promote staging → prod when verification passes              |
| [Deployment verification](./use-cases/deployment-verification)   | Check Datadog metrics before marking a deploy successful          |
| [Infrastructure inventory](./use-cases/infrastructure-inventory) | Answer "what version is running where?" instantly                 |
| [Dynamic environments](./use-cases/dynamic-environments)         | New clusters auto-join environments via selectors                 |

## Who Uses Ctrlplane?

| Team                     | Use Case                                                                    |
| ------------------------ | --------------------------------------------------------------------------- |
| **Platform Engineering** | Building an IDP with deployment orchestration and infrastructure visibility |
| **DevOps / SRE**         | Enforcing deployment policies and tracking what's running where             |
| **Scaling Startups**     | Moving from "we manually deploy" to automated, policy-driven releases       |
| **Multi-region Teams**   | Coordinating deployments across clusters/regions with a unified inventory   |

## How It Fits Your Stack

Ctrlplane doesn't replace your CI or deployment tooling—it coordinates them:

| Layer           | Tools                                       | What They Do                                  |
| --------------- | ------------------------------------------- | --------------------------------------------- |
| **Build**       | GitHub Actions, GitLab CI, Jenkins          | Build artifacts, create versions in Ctrlplane |
| **Orchestrate** | **Ctrlplane**                               | Decide when/where to deploy, enforce policies |
| **Execute**     | ArgoCD, K8s Jobs, GitHub Actions, Terraform | Perform the actual deployment                 |
| **Monitor**     | Datadog, Prometheus                         | Provide metrics for verification              |

## Next Steps

<CardGroup>
  <Card title="Why Ctrlplane?" icon="lightbulb" href="./why-ctrlplane">
    Problems you'll recognize
  </Card>

  <Card title="Quickstart" icon="rocket" href="./quickstart">
    Build your first pipeline
  </Card>
</CardGroup>


# Installation
Source: https://docs.ctrlplane.dev/installation

Deploy Ctrlplane to your infrastructure.

Ctrlplane is self-hosted and can be deployed on your infrastructure. This guide
covers production-ready deployment options for platform teams.

## Deployment Options

| Option         | Best For                           | Maintenance  |
| -------------- | ---------------------------------- | ------------ |
| Kubernetes     | Production self-hosted deployments | Self-managed |
| Docker Compose | Development and evaluation         | Self-managed |

## Self-Hosted Options

Choose the deployment method that best fits your infrastructure:

### Docker Compose (Development & Testing)

Best for: Local development, testing, small teams

**Prerequisites**:

* Docker Engine 20.10+
* Docker Compose v2.0+
* 2GB RAM minimum

**Quick Start**:

1. Clone the repository:

```bash theme={null}
git clone https://github.com/ctrlplanedev/ctrlplane.git
cd ctrlplane
```

2. Copy environment files:

```bash theme={null}
cp .env.example .env
# Edit .env with your configuration
```

3. Start services:

```bash theme={null}
docker compose up -d
```

4. Run database migrations:

```bash theme={null}
docker compose exec api pnpm --filter @ctrlplane/db migrate
```

5. Access Ctrlplane:

* Web UI: [http://localhost:3000](http://localhost:3000)
* API: [http://localhost:4000](http://localhost:4000)

**Services Started**:

* Web UI (port 3000)
* API Server (port 4000)
* Workspace Engine (port 50051)
* PostgreSQL (port 5432)
* Redis (optional, for job queue)

**Configuration**:

Key environment variables in `.env`:

```bash theme={null}
# Database
DATABASE_URL=postgresql://user:password@postgres:5432/ctrlplane

# Authentication
AUTH_SECRET=your-secret-key-change-this
NEXTAUTH_URL=http://localhost:3000

# API Keys
API_KEY_SALT=your-api-key-salt

# GitHub OAuth (optional)
GITHUB_CLIENT_ID=your-github-client-id
GITHUB_CLIENT_SECRET=your-github-client-secret
```

**Updating**:

```bash theme={null}
git pull
docker compose pull
docker compose up -d
```

### Kubernetes (Production)

Best for: Production deployments, high availability

**Prerequisites**:

* Kubernetes cluster 1.24+
* kubectl configured
* Helm 3.8+
* PostgreSQL database (managed or in-cluster)

**Installation with Helm**:

1. Add the Ctrlplane Helm repository:

```bash theme={null}
helm repo add ctrlplane https://charts.ctrlplane.dev
helm repo update
```

2. Create a namespace:

```bash theme={null}
kubectl create namespace ctrlplane
```

3. Create a values file `values.yaml`:

```yaml theme={null}
# Basic configuration
ingress:
  enabled: true
  hostname: ctrlplane.example.com
  tls:
    enabled: true
    secretName: ctrlplane-tls

# Database configuration (use managed DB in production)
postgresql:
  enabled: false # Set to true for embedded PostgreSQL
  external:
    host: your-postgres-host
    port: 5432
    database: ctrlplane
    username: ctrlplane
    existingSecret: ctrlplane-db-secret # Contains password

# API Server
api:
  replicaCount: 2
  resources:
    requests:
      memory: "512Mi"
      cpu: "500m"
    limits:
      memory: "1Gi"
      cpu: "1000m"

# Workspace Engine
workspaceEngine:
  replicaCount: 2
  resources:
    requests:
      memory: "1Gi"
      cpu: "1000m"
    limits:
      memory: "2Gi"
      cpu: "2000m"

# Web UI
web:
  replicaCount: 2
  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"
    limits:
      memory: "512Mi"
      cpu: "500m"

# Authentication
auth:
  secret: your-auth-secret
  providers:
    github:
      enabled: true
      clientId: your-github-client-id
      existingSecret: github-oauth-secret

# Redis (for job queue)
redis:
  enabled: true
  auth:
    enabled: true
    existingSecret: redis-secret
```

4. Create secrets:

```bash theme={null}
# Database password
kubectl create secret generic ctrlplane-db-secret \
  --from-literal=password=your-db-password \
  -n ctrlplane

# GitHub OAuth (if enabled)
kubectl create secret generic github-oauth-secret \
  --from-literal=client-secret=your-github-client-secret \
  -n ctrlplane

# Redis password
kubectl create secret generic redis-secret \
  --from-literal=password=your-redis-password \
  -n ctrlplane
```

5. Install the chart:

```bash theme={null}
helm install ctrlplane ctrlplane/ctrlplane \
  --namespace ctrlplane \
  --values values.yaml
```

6. Run database migrations:

```bash theme={null}
kubectl run migrate --rm -it --restart=Never \
  --image=ctrlplane/api:latest \
  --namespace ctrlplane \
  --env="DATABASE_URL=$DATABASE_URL" \
  -- pnpm --filter @ctrlplane/db migrate
```

7. Verify installation:

```bash theme={null}
kubectl get pods -n ctrlplane
kubectl get svc -n ctrlplane
```

**Accessing Ctrlplane**:

If you configured ingress:

```bash theme={null}
# Should be accessible at your configured hostname
https://ctrlplane.example.com
```

If using port-forward for testing:

```bash theme={null}
kubectl port-forward -n ctrlplane svc/ctrlplane-web 3000:80
# Access at http://localhost:3000
```

**Updating**:

```bash theme={null}
helm repo update
helm upgrade ctrlplane ctrlplane/ctrlplane \
  --namespace ctrlplane \
  --values values.yaml
```

### Manual Installation

For custom deployments, you can run each component separately.

**Components**:

1. **PostgreSQL Database** (required)
   * Version: 14+
   * Extensions: uuid-ossp, pgcrypto

2. **API Server**

   ```bash theme={null}
   docker run -d \
     --name ctrlplane-api \
     -p 4000:4000 \
     -e DATABASE_URL=postgresql://... \
     -e AUTH_SECRET=your-secret \
     ctrlplane/api:latest
   ```

3. **Workspace Engine**

   ```bash theme={null}
   docker run -d \
     --name ctrlplane-workspace-engine \
     -p 50051:50051 \
     -e DATABASE_URL=postgresql://... \
     ctrlplane/workspace-engine:latest
   ```

4. **Web UI**

   ```bash theme={null}
   docker run -d \
     --name ctrlplane-web \
     -p 3000:3000 \
     -e API_URL=http://localhost:4000 \
     ctrlplane/web:latest
   ```

5. **Run Migrations**
   ```bash theme={null}
   docker run --rm \
     -e DATABASE_URL=postgresql://... \
     ctrlplane/api:latest \
     pnpm --filter @ctrlplane/db migrate
   ```

## Configuration

### Environment Variables

#### API Server

```bash theme={null}
# Database
DATABASE_URL=postgresql://user:pass@host:5432/ctrlplane

# Authentication
AUTH_SECRET=random-secret-key
API_KEY_SALT=random-salt-for-api-keys

# GitHub OAuth (optional)
GITHUB_CLIENT_ID=your-client-id
GITHUB_CLIENT_SECRET=your-client-secret

# Logging
LOG_LEVEL=info

# OpenTelemetry (optional) - To fully disable OTEL, set OTEL_SDK_DISABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
```

#### Workspace Engine

```bash theme={null}
# Database
DATABASE_URL=postgresql://user:pass@host:5432/ctrlplane

# gRPC Server
WORKSPACE_ENGINE_HOST=0.0.0.0
WORKSPACE_ENGINE_PORT=50051

# Logging
LOG_LEVEL=info
```

#### Web UI

```bash theme={null}
# API endpoint
NEXT_PUBLIC_API_URL=https://api.ctrlplane.example.com

# Authentication
NEXTAUTH_URL=https://ctrlplane.example.com
NEXTAUTH_SECRET=same-as-auth-secret
```

### Database Setup

**Create Database**:

```sql theme={null}
CREATE DATABASE ctrlplane;
CREATE USER ctrlplane WITH PASSWORD 'your-password';
GRANT ALL PRIVILEGES ON DATABASE ctrlplane TO ctrlplane;
```

**Enable Extensions**:

```sql theme={null}
\c ctrlplane
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
```

**Run Migrations**:

```bash theme={null}
DATABASE_URL=postgresql://ctrlplane:password@localhost:5432/ctrlplane \
  pnpm --filter @ctrlplane/db migrate
```

### Authentication Setup

#### GitHub OAuth

1. Create a GitHub OAuth App:
   * Go to GitHub Settings → Developer settings → OAuth Apps
   * New OAuth App
   * Homepage URL: `https://ctrlplane.example.com`
   * Callback URL: `https://ctrlplane.example.com/api/auth/callback/github`

2. Configure environment variables:
   ```bash theme={null}
   GITHUB_CLIENT_ID=your-client-id
   GITHUB_CLIENT_SECRET=your-client-secret
   ```

#### API Keys

API keys are used for programmatic access (CI/CD integration).

Generate salt for API keys:

```bash theme={null}
openssl rand -base64 32
```

Set in environment:

```bash theme={null}
API_KEY_SALT=generated-salt
```

Users can generate API keys in the UI under Settings → API Keys.

## Resource Requirements

### Minimum (Development/Testing)

* CPU: 2 cores
* RAM: 4GB
* Disk: 20GB
* Database: Shared PostgreSQL

### Recommended (Small Production)

* CPU: 4-8 cores
* RAM: 8-16GB
* Disk: 50GB SSD
* Database: Managed PostgreSQL (2 vCPU, 8GB RAM)

### Scaling (Large Production)

* API Server: 2-4 replicas (2 vCPU, 2GB RAM each)
* Workspace Engine: 2-4 replicas (2 vCPU, 4GB RAM each)
* Web UI: 2+ replicas (1 vCPU, 1GB RAM each)
* Database: Managed PostgreSQL (4+ vCPU, 16+ GB RAM)
* Redis: Managed Redis (optional, for job queue)

## High Availability

For production deployments:

1. **Multiple Replicas**: Run 2+ replicas of each service
2. **Managed Database**: Use managed PostgreSQL with automated backups
3. **Load Balancing**: Use Kubernetes service or external load balancer
4. **Health Checks**: Configure liveness and readiness probes
5. **Monitoring**: Set up observability with metrics and logs
6. **Backups**: Regular database backups

## Upgrades

### Before Upgrading

1. **Backup Database**: Always backup before upgrading

   ```bash theme={null}
   pg_dump -h localhost -U ctrlplane ctrlplane > backup.sql
   ```

2. **Check Release Notes**: Review breaking changes

3. **Test in Staging**: Test upgrade in non-production environment

### Upgrade Process

**Docker Compose**:

```bash theme={null}
docker compose down
git pull
docker compose pull
docker compose up -d
```

**Kubernetes**:

```bash theme={null}
helm repo update
helm upgrade ctrlplane ctrlplane/ctrlplane \
  --namespace ctrlplane \
  --values values.yaml
```

### Rollback

**Docker Compose**:

```bash theme={null}
git checkout previous-tag
docker compose up -d
psql < backup.sql  # Restore database if needed
```

**Kubernetes**:

```bash theme={null}
helm rollback ctrlplane -n ctrlplane
```

## Troubleshooting

### Cannot connect to database

* Verify DATABASE\_URL is correct
* Check database is running and accessible
* Verify credentials and database exists

### Migrations fail

* Ensure database user has proper permissions
* Check for schema conflicts
* Review migration logs

### Services won't start

* Check logs: `docker compose logs` or `kubectl logs`
* Verify all environment variables are set
* Ensure ports are not already in use

### Authentication not working

* Verify AUTH\_SECRET is set and consistent across services
* Check OAuth configuration if using GitHub
* Review callback URLs match your domain

## Next Steps

* [Quickstart Guide](./quickstart.md) - Set up your first deployment
* [Core Concepts](./core-concepts.md) - Understand Ctrlplane fundamentals
* [Operations](../operations/monitoring.md) - Set up monitoring and
  observability


# CI/CD Integration
Source: https://docs.ctrlplane.dev/integrations/cicd

Create deployment versions from your CI/CD pipeline

Connect your CI/CD pipeline to Ctrlplane to automatically create deployment
versions after successful builds. This triggers the deployment orchestration
flow.

## How It Works

```mermaid theme={null}
sequenceDiagram
    participant CI as CI/CD Pipeline
    participant C as Ctrlplane
    participant A as Job Agents

    CI->>CI: Build & Test
    CI->>C: Create Version
    C->>C: Evaluate Policies
    C->>C: Create Releases
    C->>A: Dispatch Jobs
```

1. Your CI/CD pipeline builds and tests your code
2. After success, CI creates a version in Ctrlplane
3. Ctrlplane evaluates policies for each release target
4. Jobs are dispatched to job agents
5. Deployments execute across your environments

## GitHub Actions

### Basic Integration

```yaml theme={null}
name: Build and Deploy

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build
        run: |
          # Your build steps
          docker build -t myapp:${{ github.sha }} .
          docker push myapp:${{ github.sha }}

      - name: Create Ctrlplane Version
        env:
          CTRLPLANE_API_KEY: ${{ secrets.CTRLPLANE_API_KEY }}
        run: |
          ctrlc api upsert version \
            --workspace ${{ vars.CTRLPLANE_WORKSPACE }} \
            --deployment ${{ vars.CTRLPLANE_DEPLOYMENT_ID }} \
            --tag ${{ github.sha }} \
            --name "Build #${{ github.run_number }}" \
            --metadata git/commit=${{ github.sha }} \
            --metadata git/branch=${{ github.ref_name }} \
            --metadata github/actor=${{ github.actor }} \
            --metadata github/run-url=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
```

### Using the Ctrlplane Action

```yaml theme={null}
- name: Create Ctrlplane Version
  uses: ctrlplanedev/create-version@v1
  with:
    api_key: ${{ secrets.CTRLPLANE_API_KEY }}
    deployment_id: ${{ vars.DEPLOYMENT_ID }}
    tag: ${{ github.sha }}
    name: "Build #${{ github.run_number }}"
    metadata: |
      commit: ${{ github.sha }}
      branch: ${{ github.ref_name }}
```

## GitLab CI

```yaml theme={null}
stages:
  - build
  - deploy

build:
  stage: build
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker push myapp:$CI_COMMIT_SHA

create-version:
  stage: deploy
  script:
    - |
      ctrlc api upsert version \
        --workspace ${CTRLPLANE_WORKSPACE} \
        --deployment ${CTRLPLANE_DEPLOYMENT_ID} \
        --tag ${CI_COMMIT_SHA} \
        --name "Pipeline #${CI_PIPELINE_ID}" \
        --metadata git/commit=${CI_COMMIT_SHA} \
        --metadata git/branch=${CI_COMMIT_REF_NAME} \
        --metadata gitlab/pipeline-url=${CI_PIPELINE_URL}
  only:
    - main
```

## Jenkins

```groovy theme={null}
pipeline {
    agent any

    environment {
        CTRLPLANE_API_KEY = credentials('ctrlplane-api-key')
        CTRLPLANE_WORKSPACE = 'your-workspace'
        CTRLPLANE_DEPLOYMENT_ID = 'your-deployment-id'
    }

    stages {
        stage('Build') {
            steps {
                sh 'docker build -t myapp:${GIT_COMMIT} .'
                sh 'docker push myapp:${GIT_COMMIT}'
            }
        }

        stage('Create Version') {
            steps {
                sh '''
                    ctrlc api upsert version \
                        --workspace ${CTRLPLANE_WORKSPACE} \
                        --deployment ${CTRLPLANE_DEPLOYMENT_ID} \
                        --tag ${GIT_COMMIT} \
                        --name "Build #${BUILD_NUMBER}" \
                        --metadata git/commit=${GIT_COMMIT} \
                        --metadata git/branch=${GIT_BRANCH} \
                        --metadata jenkins/build-url=${BUILD_URL}
                '''
            }
        }
    }
}
```

## CircleCI

```yaml theme={null}
version: 2.1

jobs:
  build:
    docker:
      - image: cimg/base:stable
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Build and Push
          command: |
            docker build -t myapp:${CIRCLE_SHA1} .
            docker push myapp:${CIRCLE_SHA1}
      - run:
          name: Create Ctrlplane Version
          command: |
            ctrlc api upsert version \
              --workspace ${CTRLPLANE_WORKSPACE} \
              --deployment ${CTRLPLANE_DEPLOYMENT_ID} \
              --tag ${CIRCLE_SHA1} \
              --name "Build #${CIRCLE_BUILD_NUM}" \
              --metadata git/commit=${CIRCLE_SHA1} \
              --metadata git/branch=${CIRCLE_BRANCH} \
              --metadata circleci/build-url=${CIRCLE_BUILD_URL}

workflows:
  build-deploy:
    jobs:
      - build:
          filters:
            branches:
              only: main
```

## Installing ctrlc

The `ctrlc` CLI is the recommended way to interact with Ctrlplane from CI/CD pipelines.

```bash theme={null}
# Download and install
curl -fsSL https://get.ctrlplane.dev | sh

# Or using npm
npm install -g @ctrlplane/cli
```

Set the `CTRLPLANE_API_KEY` environment variable in your CI secrets.

## CLI Reference

### Create Version

```bash theme={null}
ctrlc api upsert version \
  --workspace <workspace> \
  --deployment <deployment-id> \
  --tag <version-tag> \
  --name "<version-name>" \
  --metadata key=value \
  --metadata another/key=value
```

| Flag           | Required | Description                     |
| -------------- | -------- | ------------------------------- |
| `--workspace`  | Yes      | Workspace name or ID            |
| `--deployment` | Yes      | Deployment ID                   |
| `--tag`        | Yes      | Unique version identifier       |
| `--name`       | No       | Human-readable version name     |
| `--metadata`   | No       | Key-value metadata (repeatable) |

## REST API

For environments where the CLI isn't available, use the REST API:

```bash theme={null}
POST /api/v1/deployments/{deploymentId}/versions

curl -X POST \
  "https://your-ctrlplane-instance.com/api/v1/deployments/{deploymentId}/versions" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "tag": "v1.2.3",
    "name": "Release 1.2.3",
    "status": "ready",
    "metadata": {
      "commit": "abc123",
      "branch": "main"
    }
  }'
```

## Best Practices

### Semantic Versioning

Use semantic versions for release branches:

```yaml theme={null}
- name: Create Version
  run: |
    VERSION=$(cat version.txt)
    ctrlc api upsert version \
      --workspace $CTRLPLANE_WORKSPACE \
      --deployment $CTRLPLANE_DEPLOYMENT_ID \
      --tag $VERSION
```

### Include Build Metadata

Add useful context to help with debugging:

```yaml theme={null}
- name: Create Version
  run: |
    ctrlc api upsert version \
      --workspace $CTRLPLANE_WORKSPACE \
      --deployment $CTRLPLANE_DEPLOYMENT_ID \
      --tag ${{ github.sha }} \
      --metadata git/commit=${{ github.sha }} \
      --metadata git/branch=${{ github.ref_name }} \
      --metadata build/number=${{ github.run_number }} \
      --metadata build/url=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} \
      --metadata build/actor=${{ github.actor }} \
      --metadata build/trigger=${{ github.event_name }}
```

### Idempotency

The `ctrlc api upsert version` command is idempotent — it creates or updates the
version if it already exists. This makes it safe to retry failed CI runs without
worrying about duplicate version errors.


# Ansible
Source: https://docs.ctrlplane.dev/integrations/job-agents/ansible

Run Ansible playbooks via GitHub Actions

Use the GitHub Actions job agent to dispatch workflows that execute Ansible
playbooks. This lets you keep your existing Ansible automation while Ctrlplane
orchestrates environments, approvals, and verification.

## How It Works

```mermaid theme={null}
sequenceDiagram
    participant C as Ctrlplane
    participant G as GitHub API
    participant W as Workflow
    participant A as Ansible

    C->>G: Dispatch workflow (job_id)
    G->>W: Trigger workflow_dispatch
    W->>W: Get job context from Ctrlplane
    W->>A: Run ansible-playbook
    W->>C: Update job status
```

1. Ctrlplane creates a job and dispatches it to GitHub
2. GitHub triggers your workflow with `workflow_dispatch`
3. The workflow fetches job context (version, environment, resource)
4. Ansible runs the playbook against the target inventory
5. Job status is reported back to Ctrlplane

## Prerequisites

* GitHub App installed in your organization
* Workflow file with `workflow_dispatch` trigger
* Repository permissions for the GitHub App
* Ansible available on the runner (or install via pip)

## Configuration

### Job Agent Setup

Create a job agent with type `github-app`:

```yaml theme={null}
type: JobAgent
name: ansible
agentType: github-app
```

### Deployment Configuration

Configure the deployment to dispatch the workflow:

```yaml theme={null}
type: Deployment
name: ansible-playbook
jobAgent: ansible
jobAgentConfig:
  installationId: 12345678
  owner: your-org
  repo: your-repo
  workflowId: 12345678
  ref: main # optional, defaults to main
```

| Field            | Required | Description                                |
| ---------------- | -------- | ------------------------------------------ |
| `installationId` | Yes      | GitHub App installation ID                 |
| `owner`          | Yes      | Repository owner (org or user)             |
| `repo`           | Yes      | Repository name                            |
| `workflowId`     | Yes      | Workflow ID (numeric)                      |
| `ref`            | No       | Git ref to run workflow on (default: main) |

## Workflow Setup

Create a workflow file in your repository:

```yaml theme={null}
# .github/workflows/ansible-deploy.yml
name: Ansible Deploy

on:
  workflow_dispatch:
    inputs:
      job_id:
        description: "Ctrlplane Job ID"
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install Ansible
        run: pip install ansible

      - name: Get job context
        uses: ctrlplanedev/get-job-inputs@v1
        id: job
        with:
          base_url: ${{ secrets.CTRLPLANE_BASE_URL }}
          job_id: ${{ inputs.job_id }}
          api_key: ${{ secrets.CTRLPLANE_API_KEY }}

      - name: Build inventory
        run: |
          echo '${{ steps.job.outputs.resource_config }}' | jq -r '.hosts[]' > inventory.txt

      - name: Run playbook
        run: |
          ansible-playbook \
            -i inventory.txt \
            playbooks/deploy.yml \
            --extra-vars "version=${{ steps.job.outputs.version_tag }} env=${{ steps.job.outputs.environment_name }}"
```

## Resource Config Example

Store inventory targets on the resource so each release target can map to a
different Ansible inventory:

```yaml theme={null}
type: Resource
identifier: prod-web-1
kind: Server
name: prod-web-1
version: infra/servers/v1
config:
  hosts:
    - web-01.example.com
    - web-02.example.com
```

## Templating

You can use Go templates in your job agent config to select repositories,
workflows, or refs dynamically:

```yaml theme={null}
jobAgentConfig:
  installationId: "{{.variables.github_installation_id}}"
  owner: "{{.variables.github_org}}"
  repo: "{{.deployment.slug}}"
  workflowId: "{{.variables.workflow_id}}"
  ref: "{{.version.tag}}"
```

## Status Reporting

The workflow should update job status. You can use the Ctrlplane API:

```yaml theme={null}
- name: Mark job successful
  if: success()
  run: |
    curl -X PATCH "https://your-ctrlplane-instance.com/api/v1/jobs/${{ inputs.job_id }}" \
      -H "Authorization: Bearer ${{ secrets.CTRLPLANE_API_KEY }}" \
      -H "Content-Type: application/json" \
      -d '{"status": "successful"}'

- name: Mark job failed
  if: failure()
  run: |
    curl -X PATCH "https://your-ctrlplane-instance.com/api/v1/jobs/${{ inputs.job_id }}" \
      -H "Authorization: Bearer ${{ secrets.CTRLPLANE_API_KEY }}" \
      -H "Content-Type: application/json" \
      -d '{"status": "failure"}'
```


# ArgoCD
Source: https://docs.ctrlplane.dev/integrations/job-agents/argocd

Deploy applications using ArgoCD GitOps

The ArgoCD job agent creates and syncs ArgoCD Applications, enabling
GitOps-style deployments with automatic health verification.

## How It Works

```mermaid theme={null}
sequenceDiagram
    participant C as Ctrlplane
    participant A as ArgoCD
    participant K as Kubernetes

    C->>A: Create/Update Application
    A->>K: Sync resources
    C->>A: Poll health status
    A-->>C: Healthy + Synced
    C->>C: Mark job successful
```

1. Ctrlplane renders an ArgoCD Application from your template
2. The Application is created or updated via ArgoCD API
3. ArgoCD syncs the application to Kubernetes
4. Ctrlplane verifies the application reaches `Healthy` + `Synced` status
5. Job is marked successful when verification passes

## Prerequisites

* ArgoCD server with API access
* API token with application create/update permissions
* Network connectivity from Ctrlplane to ArgoCD

## Configuration

### Job Agent Setup

Create a job agent with type `argo-cd`:

```yaml theme={null}
type: JobAgent
name: argocd
agentType: argo-cd
```

### Deployment Configuration

```yaml theme={null}
type: Deployment
name: api-service
jobAgent: argocd
jobAgentConfig:
  serverUrl: argocd.example.com:443
  apiKey: "{{.variables.argocd_token}}"
  template: |
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: {{.deployment.slug}}-{{.environment.name}}
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/your-repo
        targetRevision: {{.version.tag}}
        path: k8s/{{.environment.name}}
      destination:
        server: {{.resource.config.server}}
        namespace: {{.resource.config.namespace}}
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
```

| Field       | Required | Description                        |
| ----------- | -------- | ---------------------------------- |
| `serverUrl` | Yes      | ArgoCD server URL (host:port)      |
| `apiKey`    | Yes      | ArgoCD API token                   |
| `template`  | Yes      | Go template for ArgoCD Application |

## Template Context

The template has access to all job context. Variables are accessed using Go template
syntax: `{{.variable.property}}`. The job context is derived from the resources
returned by the deployment selector (and narrowed by the environment selector),
so the data available to the template reflects that specific target.

### Top-Level Variables

| Variable       | Description                                                 |
| -------------- | ----------------------------------------------------------- |
| `.job`         | Current job details                                         |
| `.version`     | Version being deployed (shortcut)                           |
| `.deployment`  | User-defined deployment configuration and properties        |
| `.environment` | User-defined environment you deploy into and its properties |
| `.resource`    | User-defined resource you deploy against and its properties |
| `.variables`   | Merged deployment variables (key-value strings)             |

### Resource Properties

Each job invocation is tied to a specific resource instance returned by
your selector, so the `.resource` values can differ for every ArgoCD
template invocation.

| Property                | Type               | Description                             |
| ----------------------- | ------------------ | --------------------------------------- |
| `.resource.id`          | string             | Unique resource ID                      |
| `.resource.name`        | string             | Display name                            |
| `.resource.identifier`  | string             | Unique identifier within workspace      |
| `.resource.kind`        | string             | Resource kind (e.g., KubernetesCluster) |
| `.resource.version`     | string             | Resource schema version                 |
| `.resource.config`      | object             | Arbitrary configuration data            |
| `.resource.metadata`    | map\[string]string | Key-value metadata labels               |
| `.resource.workspaceId` | string             | Parent workspace ID                     |
| `.resource.providerId`  | string             | Resource provider ID (if any)           |
| `.resource.createdAt`   | timestamp          | Creation timestamp                      |
| `.resource.updatedAt`   | timestamp          | Last update timestamp                   |
| `.resource.lockedAt`    | timestamp          | Lock timestamp (if locked)              |

### Deployment Properties

Each job invocation is tied to a specific deployment, so `.deployment` values can differ
for every ArgoCD template invocation.

| Property                       | Type   | Description                     |
| ------------------------------ | ------ | ------------------------------- |
| `.deployment.id`               | string | Unique deployment ID            |
| `.deployment.name`             | string | Display name                    |
| `.deployment.slug`             | string | URL-friendly identifier         |
| `.deployment.description`      | string | Optional description            |
| `.deployment.systemId`         | string | Parent system ID                |
| `.deployment.jobAgentId`       | string | Associated job agent ID         |
| `.deployment.jobAgentConfig`   | object | Job agent configuration         |
| `.deployment.resourceSelector` | object | Resource selector for targeting |

### Environment Properties

Each job invocation is tied to a specific environment, so `.environment` values can differ
for every ArgoCD template invocation.

| Property                        | Type      | Description           |
| ------------------------------- | --------- | --------------------- |
| `.environment.id`               | string    | Unique environment ID |
| `.environment.name`             | string    | Display name          |
| `.environment.description`      | string    | Optional description  |
| `.environment.systemId`         | string    | Parent system ID      |
| `.environment.resourceSelector` | object    | Resource selector     |
| `.environment.createdAt`        | timestamp | Creation timestamp    |

### Version Properties

Access via `.version`:

| Property                  | Type               | Description                     |
| ------------------------- | ------------------ | ------------------------------- |
| `.version.id`             | string             | Unique version ID               |
| `.version.tag`            | string             | Version tag (e.g., v1.2.3)      |
| `.version.name`           | string             | Display name                    |
| `.version.message`        | string             | Optional commit/release message |
| `.version.status`         | string             | Version status                  |
| `.version.config`         | object             | Version-specific configuration  |
| `.version.metadata`       | map\[string]string | Version metadata                |
| `.version.jobAgentConfig` | object             | Version-level job agent config  |
| `.version.deploymentId`   | string             | Parent deployment ID            |
| `.version.createdAt`      | timestamp          | Creation timestamp              |

### Job Properties

| Property           | Type               | Description               |
| ------------------ | ------------------ | ------------------------- |
| `.job.id`          | string             | Unique job ID             |
| `.job.status`      | string             | Current status            |
| `.job.message`     | string             | Status message            |
| `.job.externalId`  | string             | External system reference |
| `.job.metadata`    | map\[string]string | Job metadata              |
| `.job.jobAgentId`  | string             | Executing job agent ID    |
| `.job.releaseId`   | string             | Associated release ID     |
| `.job.createdAt`   | timestamp          | Creation timestamp        |
| `.job.startedAt`   | timestamp          | Execution start timestamp |
| `.job.completedAt` | timestamp          | Completion timestamp      |
| `.job.updatedAt`   | timestamp          | Last update timestamp     |

### Variables

The `.variables` map contains all resolved deployment variables as strings:

```yaml theme={null}
template: |
  metadata:
    annotations:
      replicas: "{{.variables.replica_count}}"
      db-host: "{{.variables.database_host}}"
```

### Accessing Nested Config

Resource and version configs are arbitrary objects. Access nested properties:

```yaml theme={null}
template: |
  spec:
    destination:
      server: {{.resource.config.cluster_url}}
      namespace: {{.resource.config.namespaces.app}}
    source:
      helm:
        parameters:
          - name: image.tag
            value: {{.version.config.imageTag}}
```

## Template Functions

The template supports [Sprig](http://masterminds.github.io/sprig/) functions:

```yaml theme={null}
template: |
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: {{ .deployment.slug | lower | replace "_" "-" }}
    labels:
      version: {{ .version.tag | trunc 63 }}
      deployed-at: {{ now | date "2006-01-02" }}
```

## Automatic Verification

When an ArgoCD Application is created, Ctrlplane automatically starts a
verification that checks:

* Application sync status is `Synced`
* Application health status is `Healthy` or `Progressing`

The verification polls the ArgoCD API every 10 seconds for 5 iterations.

## Example: Multi-Environment Setup

```yaml theme={null}
type: Deployment
name: frontend
jobAgent: argocd
jobAgentConfig:
  serverUrl: "{{.variables.argocd_server}}"
  apiKey: "{{.variables.argocd_token}}"
  template: |
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: frontend-{{.environment.name | lower}}
      namespace: argocd
      labels:
        app: frontend
        env: {{.environment.name | lower}}
        version: {{.version.tag}}
      finalizers:
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: {{.environment.name | lower}}
      source:
        repoURL: https://github.com/your-org/frontend
        targetRevision: {{.version.tag}}
        path: deploy/{{.environment.name | lower}}
        helm:
          valueFiles:
            - values.yaml
            - values-{{.environment.name | lower}}.yaml
          parameters:
            - name: image.tag
              value: {{.version.tag}}
            - name: replicas
              value: "{{.resource.config.replicas | default 2}}"
      destination:
        server: {{.resource.config.cluster_url}}
        namespace: frontend
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
```

## Example: Kustomize Overlay

```yaml theme={null}
template: |
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: {{.deployment.slug}}-{{.resource.identifier}}
    namespace: argocd
  spec:
    project: default
    source:
      repoURL: https://github.com/your-org/{{.deployment.slug}}
      targetRevision: {{.version.tag}}
      path: overlays/{{.environment.name}}
      kustomize:
        images:
          - {{.deployment.slug}}={{.variables.registry}}/{{.deployment.slug}}:{{.version.tag}}
    destination:
      server: {{.resource.config.server}}
      namespace: {{.deployment.slug}}
```

## Troubleshooting

### Application not syncing

* Check ArgoCD server logs
* Verify the repository is accessible from ArgoCD
* Check the target revision exists

### Authentication errors

* Verify the API token is valid
* Check token has correct permissions
* Ensure `serverUrl` includes the correct port

### Verification failing

* Check ArgoCD Application status in the UI
* Review sync errors in ArgoCD
* Verify destination cluster is accessible


# GitHub Actions
Source: https://docs.ctrlplane.dev/integrations/job-agents/github

Execute deployments using GitHub Actions workflows

The GitHub Actions job agent triggers workflow dispatch events to execute your
deployments. This is ideal for teams already using GitHub Actions for CI/CD.

## How It Works

```mermaid theme={null}
sequenceDiagram
    participant C as Ctrlplane
    participant G as GitHub API
    participant W as Workflow

    C->>G: Dispatch workflow (job_id)
    G->>W: Trigger workflow_dispatch
    W->>W: Get job context from Ctrlplane
    W->>W: Execute deployment
    G->>C: Webhook: workflow_run event
    C->>C: Update job status automatically
```

1. Ctrlplane creates a job and dispatches it to GitHub
2. GitHub triggers your workflow with `workflow_dispatch`
3. Your workflow fetches job context (version, environment, resource)
4. Your workflow executes the deployment
5. GitHub sends a `workflow_run` webhook event to Ctrlplane as the run progresses
6. Ctrlplane automatically updates the job status (in progress, successful, failure, etc.)

## Prerequisites

* A **GitHub App** registered in your organization (see [GitHub App Setup](#github-app-setup))
* The GitHub App **installed** on the repositories you want to dispatch workflows to
* A workflow file with `workflow_dispatch` trigger in each target repository
* Ctrlplane's workspace engine configured with the GitHub App credentials (see [Server Configuration](#server-configuration))

## GitHub App Setup

Ctrlplane authenticates with GitHub using a GitHub App. The workspace engine
generates a JWT from the App's private key and exchanges it for a short-lived
installation token to dispatch workflows.

### Creating a GitHub App

1. Go to **GitHub → Settings → Developer settings → GitHub Apps → New GitHub App**
2. Fill in the required fields:
   * **GitHub App name**: e.g. `ctrlplane-bot`
   * **Homepage URL**: your Ctrlplane instance URL
3. Under **Webhook**:
   * **Webhook URL**: `https://<your-ctrlplane-domain>/api/github/webhook`
   * **Webhook secret**: generate a strong secret (e.g. `openssl rand -hex 32`)
   * Subscribe to the **Workflow runs** event
4. Under **Permissions**, grant the following **Repository permissions**:
   * **Actions**: Read and write (required to dispatch workflow events)
   * **Contents**: Read-only (required to access workflow files)
   * **Metadata**: Read-only (granted by default)
5. Click **Create GitHub App**

### Gathering Credentials

After creating the App, collect the following values:

| Credential         | Where to find it                                                                    |
| ------------------ | ----------------------------------------------------------------------------------- |
| **App ID**         | GitHub App settings page → **App ID** (numeric)                                     |
| **Private Key**    | GitHub App settings page → **Private keys** → **Generate a private key** (PEM file) |
| **Client ID**      | GitHub App settings page → **Client ID**                                            |
| **Client Secret**  | GitHub App settings page → **Generate a new client secret**                         |
| **Webhook Secret** | The secret you entered when creating the App (step 3 above)                         |

### Installing the App

1. From the GitHub App settings page, click **Install App** in the left sidebar
2. Choose the organization or account to install on
3. Select the repositories the App should have access to
4. Note the **Installation ID** — you can find it in the URL after installation:
   `https://github.com/settings/installations/INSTALLATION_ID`

You can also retrieve it via the GitHub API:

```bash theme={null}
curl -H "Authorization: Bearer $GITHUB_TOKEN" \
  https://api.github.com/orgs/YOUR_ORG/installations
```

## Server Configuration

The workspace engine (and the API server) need the GitHub App credentials to
authenticate with the GitHub API. These are provided via environment variables.

### Environment Variables

| Variable                 | Service          | Required | Description                                                        |
| ------------------------ | ---------------- | -------- | ------------------------------------------------------------------ |
| `GITHUB_BOT_APP_ID`      | Workspace Engine | Yes      | The numeric App ID from your GitHub App settings                   |
| `GITHUB_BOT_PRIVATE_KEY` | Workspace Engine | Yes      | The PEM-encoded private key generated for the GitHub App           |
| `GITHUB_WEBHOOK_SECRET`  | API Server       | Yes      | The webhook secret used to verify incoming GitHub webhook payloads |

The **workspace engine** uses `GITHUB_BOT_APP_ID` and `GITHUB_BOT_PRIVATE_KEY`
to generate JWTs for authenticating with the GitHub API and dispatching
workflows. If either is missing, dispatching will fail with a
`GitHub bot not configured` error.

The **API server** uses `GITHUB_WEBHOOK_SECRET` to verify incoming
`workflow_run` webhook events from GitHub. When configured, Ctrlplane
automatically updates job status as workflows progress — no manual status
reporting is needed in your workflow.

### Helm Chart Configuration

When deploying with the Ctrlplane Helm chart, configure the GitHub App
credentials in your `values.yaml` under `global.integrations.github.bot`:

```yaml theme={null}
global:
  integrations:
    github:
      url: "https://github.com"
      bot:
        appId: "123456"
        privateKey: |
          -----BEGIN RSA PRIVATE KEY-----
          ...
          -----END RSA PRIVATE KEY-----
        webhookSecret: "your-webhook-secret"
```

All bot values support either inline strings or Kubernetes `valueFrom`
references for secrets:

```yaml theme={null}
global:
  integrations:
    github:
      url: "https://github.com"
      bot:
        appId:
          valueFrom:
            secretKeyRef:
              name: github-bot-secret
              key: app-id
        privateKey:
          valueFrom:
            secretKeyRef:
              name: github-bot-secret
              key: private-key
        webhookSecret:
          valueFrom:
            secretKeyRef:
              name: github-bot-secret
              key: webhook-secret
```

The full set of Helm values for the GitHub bot:

| Helm Value                                     | Env Variable               | Required for Actions | Description                                         |
| ---------------------------------------------- | -------------------------- | -------------------- | --------------------------------------------------- |
| `global.integrations.github.bot.appId`         | `GITHUB_BOT_APP_ID`        | Yes                  | GitHub App numeric ID                               |
| `global.integrations.github.bot.privateKey`    | `GITHUB_BOT_PRIVATE_KEY`   | Yes                  | PEM private key                                     |
| `global.integrations.github.bot.name`          | `GITHUB_BOT_NAME`          | No                   | Display name for the bot                            |
| `global.integrations.github.bot.clientId`      | `GITHUB_BOT_CLIENT_ID`     | No                   | GitHub App client ID                                |
| `global.integrations.github.bot.clientSecret`  | `GITHUB_BOT_CLIENT_SECRET` | No                   | GitHub App client secret                            |
| `global.integrations.github.bot.webhookSecret` | `GITHUB_WEBHOOK_SECRET`    | Yes                  | Webhook secret for verifying incoming GitHub events |

### Docker / Manual Deployment

When running the workspace engine directly, pass the environment variables:

```bash theme={null}
docker run -d \
  --name ctrlplane-workspace-engine \
  -e GITHUB_BOT_APP_ID=123456 \
  -e GITHUB_BOT_PRIVATE_KEY="$(cat private-key.pem)" \
  # ... other env vars ...
  ctrlplane/workspace-engine:latest
```

## Configuration

### Job Agent Setup

Create a job agent with type `github-app`:

```yaml theme={null}
type: JobAgent
name: github-actions
agentType: github-app
```

### Deployment Configuration

Configure the deployment to use GitHub Actions:

```yaml theme={null}
type: Deployment
name: api-service
jobAgent: github-actions
jobAgentConfig:
  installationId: 12345678
  owner: your-org
  repo: your-repo
  workflowId: 12345678
  ref: main # optional, defaults to main
```

| Field            | Required | Description                                |
| ---------------- | -------- | ------------------------------------------ |
| `installationId` | Yes      | GitHub App installation ID                 |
| `owner`          | Yes      | Repository owner (org or user)             |
| `repo`           | Yes      | Repository name                            |
| `workflowId`     | Yes      | Workflow ID (numeric)                      |
| `ref`            | No       | Git ref to run workflow on (default: main) |

### Finding Your Installation ID

The installation ID is the numeric ID assigned when the GitHub App is installed
on an organization or user account. You can find it:

1. **From the URL**: after installing the App, the URL will be
   `https://github.com/settings/installations/INSTALLATION_ID`
2. **From the API**:

```bash theme={null}
curl -H "Authorization: Bearer $GITHUB_TOKEN" \
  https://api.github.com/orgs/YOUR_ORG/installations
```

Look for the `id` field in the response.

### Finding Your Workflow ID

Use the GitHub API to find your workflow ID:

```bash theme={null}
curl -H "Authorization: Bearer $GITHUB_TOKEN" \
  https://api.github.com/repos/OWNER/REPO/actions/workflows
```

The `id` field in each workflow object is the numeric workflow ID you need.

## Workflow Setup

Create a workflow file in your repository:

```yaml theme={null}
# .github/workflows/deploy.yml
name: Deploy

on:
  workflow_dispatch:
    inputs:
      job_id:
        description: "Ctrlplane Job ID"
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Get job context
        uses: ctrlplanedev/get-job-inputs@v1
        id: job
        with:
          base_url: ${{ secrets.CTRLPLANE_BASE_URL }}
          job_id: ${{ inputs.job_id }}
          api_key: ${{ secrets.CTRLPLANE_API_KEY }}

      - name: Deploy
        run: |
          echo "Deploying ${{ steps.job.outputs.version_tag }}"
          echo "Environment: ${{ steps.job.outputs.environment_name }}"
          echo "Resource: ${{ steps.job.outputs.resource_identifier }}"

          # Your deployment commands here
```

## Available Job Context

The `get-job-inputs` action provides these outputs:

| Output                 | Description                 |
| ---------------------- | --------------------------- |
| `job_id`               | The Ctrlplane job ID        |
| `version_tag`          | Version tag being deployed  |
| `version_name`         | Version name                |
| `environment_name`     | Target environment name     |
| `environment_id`       | Target environment ID       |
| `resource_identifier`  | Target resource identifier  |
| `resource_name`        | Target resource name        |
| `resource_kind`        | Target resource kind        |
| `resource_config`      | Resource config (JSON)      |
| `deployment_name`      | Deployment name             |
| `deployment_variables` | Deployment variables (JSON) |

## Templating

You can use Go templates in your job agent config to dynamically configure
workflows:

```yaml theme={null}
jobAgentConfig:
  installationId: "{{.variables.github_installation_id}}"
  owner: "{{.variables.github_org}}"
  repo: "{{.deployment.slug}}"
  workflowId: "{{.variables.workflow_id}}"
  ref: "{{.version.tag}}"
```

## Status Reporting

### Automatic via Webhooks (Recommended)

When the GitHub App webhook is configured (see [GitHub App Setup](#creating-a-github-app)),
Ctrlplane automatically receives `workflow_run` events from GitHub and updates
job status without any extra steps in your workflow.

The API server listens at `/api/github/webhook` and maps GitHub workflow run
states to Ctrlplane job statuses:

| GitHub Conclusion | Ctrlplane Status  |
| ----------------- | ----------------- |
| *(in progress)*   | `in_progress`     |
| `success`         | `successful`      |
| `failure`         | `failure`         |
| `cancelled`       | `cancelled`       |
| `action_required` | `action_required` |
| `skipped`         | `skipped`         |
| `neutral`         | `skipped`         |

Ctrlplane matches the workflow run to a job by extracting the job ID from the
workflow run name. The `get-job-inputs` action includes the job ID in the run
name automatically.

<Note>
  For automatic status reporting to work, ensure:

  * The GitHub App webhook URL is set to `https://<your-ctrlplane-domain>/api/github/webhook`
  * The `GITHUB_WEBHOOK_SECRET` env var matches the secret configured in the GitHub App
  * The App is subscribed to **Workflow runs** events
</Note>

### Manual via API (Fallback)

If you are not using webhooks, you can manually report status from the workflow
using the Ctrlplane API:

```yaml theme={null}
- name: Mark job successful
  if: success()
  run: |
    curl -X PATCH "https://your-ctrlplane-instance.com/api/v1/jobs/${{ inputs.job_id }}" \
      -H "Authorization: Bearer ${{ secrets.CTRLPLANE_API_KEY }}" \
      -H "Content-Type: application/json" \
      -d '{"status": "successful"}'

- name: Mark job failed
  if: failure()
  run: |
    curl -X PATCH "https://your-ctrlplane-instance.com/api/v1/jobs/${{ inputs.job_id }}" \
      -H "Authorization: Bearer ${{ secrets.CTRLPLANE_API_KEY }}" \
      -H "Content-Type: application/json" \
      -d '{"status": "failure"}'
```

## Example: Kubernetes Deployment

```yaml theme={null}
name: Deploy to Kubernetes

on:
  workflow_dispatch:
    inputs:
      job_id:
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Get job context
        uses: ctrlplanedev/get-job-inputs@v1
        id: job
        with:
          base_url: ${{ secrets.CTRLPLANE_BASE_URL }}
          job_id: ${{ inputs.job_id }}
          api_key: ${{ secrets.CTRLPLANE_API_KEY }}

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Deploy
        run: |
          kubectl set image deployment/${{ steps.job.outputs.deployment_name }} \
            app=${{ steps.job.outputs.version_tag }} \
            -n ${{ fromJson(steps.job.outputs.resource_config).namespace }}
```

## Troubleshooting

### `GitHub bot not configured` error

The workspace engine cannot find the App credentials. Verify:

* `GITHUB_BOT_APP_ID` is set and is a valid numeric ID
* `GITHUB_BOT_PRIVATE_KEY` is set and contains the full PEM-encoded private key
* If using Helm, check that `global.integrations.github.bot.appId` and
  `global.integrations.github.bot.privateKey` are set in your values

### `failed to get installation token` error

The App credentials are set but the token exchange failed. Check:

* The **App ID** matches the GitHub App you created
* The **private key** has not been revoked or regenerated
* The **installation ID** in your deployment config is correct
* The GitHub App is still installed on the target organization/account

### Workflow not triggered

* Verify the workflow file has `workflow_dispatch` as a trigger
* Ensure the `ref` in your config points to a branch where the workflow file exists
* Check that the GitHub App has **Actions: Read and write** permission on the repository
* Confirm the **workflow ID** is correct (use the API to verify)

### Job status not updating automatically

* Verify the GitHub App webhook URL is set to `https://<your-ctrlplane-domain>/api/github/webhook`
* Ensure `GITHUB_WEBHOOK_SECRET` is set on the API server and matches the secret in GitHub App settings
* Check that the App is subscribed to **Workflow runs** events
* Confirm the API server is reachable from GitHub (not behind a firewall)

### Authentication errors in the workflow

* Ensure `CTRLPLANE_API_KEY` is set as a repository or organization secret in GitHub
* Verify the API key has permissions to read jobs in your Ctrlplane workspace


# Terraform Cloud
Source: https://docs.ctrlplane.dev/integrations/job-agents/terraform-cloud

Manage infrastructure deployments with Terraform Cloud/Enterprise

The Terraform Cloud job agent creates workspaces and triggers runs, enabling
infrastructure-as-code deployments with webhook-based status tracking.

## How It Works

```mermaid theme={null}
sequenceDiagram
    participant C as Ctrlplane
    participant T as Terraform Cloud
    participant I as Infrastructure

    C->>T: Create/Update Workspace
    C->>T: Sync Variables
    C->>T: Ensure Notification Config
    C->>T: Create Run
    T->>I: Plan & Apply
    T->>C: Webhook: run status updates
    C->>C: Update job status
```

1. Ctrlplane renders a workspace configuration from your template
2. The workspace is created or updated via Terraform Cloud API
3. Variables are synced to match your template
4. A webhook notification configuration (`ctrlplane-webhook`) is created on the workspace (idempotent)
5. A run is triggered with auto-apply
6. Terraform Cloud sends webhook notifications as the run progresses
7. The ctrlplane API receives webhooks and updates job status in the database

This is a **fire-and-forget** dispatch model — the workspace-engine does not poll
or maintain long-running goroutines. Status tracking is handled entirely by
TFC webhooks, making it resilient to engine restarts.

## Prerequisites

* Terraform Cloud or Terraform Enterprise account
* API token with workspace, run, and notification configuration permissions
* VCS connection (optional, for Git-based workflows)
* A reachable webhook endpoint (the ctrlplane API must be accessible from TFC)

## Configuration

### Job Agent Setup

Create a job agent with type `tfe`:

```yaml theme={null}
type: JobAgent
name: terraform-cloud
agentType: tfe
```

### Deployment Configuration

```yaml theme={null}
type: Deployment
name: infrastructure
jobAgent: terraform-cloud
jobAgentConfig:
  organization: your-org
  address: https://app.terraform.io
  token: "{{.variables.tfe_token}}"
  webhookUrl: https://ctrlplane.example.com/api/tfe/webhook
  triggerRunOnChange: true
  template: |
    name: {{.deployment.slug}}-{{.resource.identifier}}
    description: "Managed by Ctrlplane"
    execution_mode: remote
    auto_apply: true
    terraform_version: "1.6.0"
    working_directory: environments/{{.environment.name}}
    vcs_repo:
      identifier: your-org/infrastructure
      branch: main
      oauth_token_id: ot-xxxxxxxxxx
    variables:
      - key: environment
        value: {{.environment.name}}
        category: terraform
      - key: version
        value: {{.version.tag}}
        category: terraform
```

| Field                | Required | Default | Description                                                                                         |
| -------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------- |
| `organization`       | Yes      |         | Terraform Cloud organization                                                                        |
| `address`            | Yes      |         | Terraform Cloud/Enterprise URL                                                                      |
| `token`              | Yes      |         | API token                                                                                           |
| `template`           | Yes      |         | Go template for workspace configuration                                                             |
| `webhookUrl`         | Yes      |         | Ctrlplane API endpoint for TFC notifications (e.g. `https://ctrlplane.example.com/api/tfe/webhook`) |
| `triggerRunOnChange` | No       | `true`  | Whether to create a TFC run on dispatch. When `false`, only the workspace and variables are synced. |

### Environment Variables

| Variable             | Where            | Description                                                 |
| -------------------- | ---------------- | ----------------------------------------------------------- |
| `TFE_WEBHOOK_SECRET` | API              | HMAC secret for verifying incoming TFC webhooks             |
| `TFE_WEBHOOK_SECRET` | Workspace Engine | Same secret, used when creating notification configs on TFC |

Both the API and workspace-engine must share the same `TFE_WEBHOOK_SECRET`.

## Webhook Status Mapping

When TFC sends a notification, the webhook handler maps the trigger to a
ctrlplane job status:

| TFC Trigger           | Example Run Status                      | Ctrlplane Status |
| --------------------- | --------------------------------------- | ---------------- |
| `run:created`         | pending                                 | `pending`        |
| `run:planning`        | planning                                | `inProgress`     |
| `run:needs_attention` | planned (confirmable), policy\_override | `actionRequired` |
| `run:applying`        | applying                                | `inProgress`     |
| `run:completed`       | applied, planned\_and\_finished         | `successful`     |
| `run:errored`         | errored                                 | `failure`        |

## Workspace Template

The template defines the Terraform Cloud workspace:

| Field               | Type   | Description                      |
| ------------------- | ------ | -------------------------------- |
| `name`              | string | Workspace name (required)        |
| `description`       | string | Workspace description            |
| `execution_mode`    | string | `remote`, `local`, or `agent`    |
| `auto_apply`        | bool   | Auto-apply after plan            |
| `terraform_version` | string | Terraform version to use         |
| `working_directory` | string | Subdirectory for Terraform files |
| `vcs_repo`          | object | VCS repository settings          |
| `variables`         | array  | Workspace variables              |

### VCS Repository Settings

```yaml theme={null}
vcs_repo:
  identifier: org/repo
  branch: main
  oauth_token_id: ot-xxxxxxxxxx
  ingress_submodules: false
  tags_regex: ""
```

### Variable Configuration

```yaml theme={null}
variables:
  - key: aws_region
    value: us-east-1
    category: terraform # or "env"
    hcl: false
    sensitive: false
    description: "AWS region"
```

## Template Context

The template has access to the full dispatch context:

| Variable       | Description                        |
| -------------- | ---------------------------------- |
| `.deployment`  | Deployment details                 |
| `.environment` | Environment details                |
| `.resource`    | Target resource (config, metadata) |
| `.release`     | Release details                    |
| `.version`     | Deployment version (tag, name)     |
| `.variables`   | Merged deployment variables        |

## `triggerRunOnChange: false`

When `triggerRunOnChange` is set to `false`, the dispatcher will:

1. Upsert the workspace
2. Sync variables
3. Ensure the notification config exists
4. **Skip** creating a run

This is useful when you want VCS pushes to trigger runs instead of ctrlplane
creating them directly. The webhook notification config is still created so
that run status updates flow back to ctrlplane.

> **Note:** Correlating VCS-triggered runs back to ctrlplane jobs (by workspace
> name/ID instead of run ID) is a planned follow-up.

## Example: Multi-Environment Infrastructure

```yaml theme={null}
type: Deployment
name: vpc-infrastructure
jobAgent: terraform-cloud
jobAgentConfig:
  organization: "{{.variables.tfe_org}}"
  address: "{{.variables.tfe_address}}"
  token: "{{.variables.tfe_token}}"
  webhookUrl: "{{.variables.ctrlplane_webhook_url}}"
  template: |
    name: vpc-{{.environment.name}}-{{.resource.metadata.region}}
    description: "VPC for {{.environment.name}} in {{.resource.metadata.region}}"
    execution_mode: remote
    auto_apply: {{if eq .environment.name "production"}}false{{else}}true{{end}}
    terraform_version: "1.6.0"
    working_directory: modules/vpc

    vcs_repo:
      identifier: {{.variables.github_org}}/infrastructure
      branch: {{.version.tag}}
      oauth_token_id: {{.variables.vcs_oauth_token}}

    variables:
      - key: environment
        value: {{.environment.name}}
        category: terraform
      - key: region
        value: {{.resource.metadata.region}}
        category: terraform
      - key: vpc_cidr
        value: {{.resource.config.vpc_cidr}}
        category: terraform
      - key: AWS_ACCESS_KEY_ID
        value: {{.variables.aws_access_key}}
        category: env
        sensitive: true
      - key: AWS_SECRET_ACCESS_KEY
        value: {{.variables.aws_secret_key}}
        category: env
        sensitive: true
```

## Example: Agent-Based Execution

For private infrastructure, use agent execution mode:

```yaml theme={null}
template: |
  name: private-{{.resource.identifier}}
  execution_mode: agent
  agent_pool_id: {{.variables.agent_pool_id}}
  auto_apply: true

  variables:
    - key: target_host
      value: {{.resource.config.host}}
      category: terraform
```

## Terraform Provider Configuration

When using the ctrlplane Terraform provider:

```hcl theme={null}
resource "ctrlplane_job_agent" "tfc" {
  name = "terraform-cloud"

  terraform_cloud {
    address            = "https://app.terraform.io"
    organization       = "your-org"
    token              = var.tfc_token
    webhook_url        = "https://ctrlplane.example.com/api/tfe/webhook"
    trigger_run_on_change = true
    template           = file("workspace-template.yaml")
  }
}
```

## Troubleshooting

### Workspace creation fails

* Verify organization name is correct
* Check API token has workspace:write permission
* Ensure workspace name is valid (alphanumeric, hyphens, underscores)

### VCS connection errors

* Verify OAuth token ID is correct
* Check repository exists and is accessible
* Ensure branch or tag exists

### Run fails to start

* Check workspace has valid configuration
* Verify VCS connection is working
* Review workspace settings in Terraform Cloud UI

### Variables not updating

* Verify variable keys match expected format
* Check for duplicate variable definitions
* Sensitive variables won't show values in UI

### Webhook returns 401

* Check `TFE_WEBHOOK_SECRET` is set on the API
* Verify the same secret was used when creating the notification config on TFC
* Ensure the `x-tfe-notification-signature` header is present

### No webhook notifications received

* Verify the `webhookUrl` is reachable from Terraform Cloud
* Check the notification config exists on the TFC workspace (Settings > Notifications)
* Review TFC's notification delivery log for errors
* For local development, use smee.io or localtunnel to expose your API

### Job stays in `inProgress`

* Verify webhooks are reaching the API (check API logs for `POST /api/tfe/webhook`)
* Check the TFC run status directly in the TFC UI


# AWS Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/aws

Sync AWS resources into Ctrlplane

The AWS provider syncs resources from Amazon Web Services into Ctrlplane's
inventory—EKS clusters, EC2 instances, RDS databases, and VPC networks.

## Prerequisites

* `ctrlc` CLI installed
* AWS credentials configured (environment variables, `~/.aws/credentials`, or IAM role)
* Ctrlplane API key

## Supported Resources

| Command        | Resource Type  | Ctrlplane Kind          |
| -------------- | -------------- | ----------------------- |
| `aws eks`      | EKS Clusters   | `AWS/EKS`               |
| `aws ec2`      | EC2 Instances  | `AWS/EC2`               |
| `aws rds`      | RDS Instances  | `AWS/RDS`               |
| `aws networks` | VPCs & Subnets | `AWS/VPC`, `AWS/Subnet` |

## Authentication

Configure AWS credentials using any standard method:

```bash theme={null}
# Environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-east-1"

# Or use AWS CLI profile
export AWS_PROFILE="production"

# Or use IAM role (when running in AWS)
# Credentials are automatically retrieved
```

## EKS Clusters

Sync Amazon Elastic Kubernetes Service clusters:

```bash theme={null}
# Sync from a specific region
ctrlc sync aws eks --region us-east-1

# Sync from multiple regions
ctrlc sync aws eks --region us-east-1 --region us-west-2

# Sync from all regions
ctrlc sync aws eks

# Continuous sync
ctrlc sync aws eks --region us-east-1 --interval 5m
```

### Options

| Flag         | Description                      | Required                          |
| ------------ | -------------------------------- | --------------------------------- |
| `--region`   | AWS region(s) to sync from       | No (all regions if not specified) |
| `--provider` | Resource provider name           | No                                |
| `--interval` | Sync interval (e.g., `5m`, `1h`) | No                                |

### Resource Metadata

EKS clusters include metadata from AWS tags:

```yaml theme={null}
identifier: arn:aws:eks:us-east-1:123456789:cluster/prod-cluster
name: prod-cluster
kind: AWS/EKS
metadata:
  region: us-east-1
  account: "123456789"
  environment: production  # from AWS tag
  team: platform           # from AWS tag
config:
  endpoint: https://XXXXX.eks.us-east-1.amazonaws.com
  version: "1.28"
```

## EC2 Instances

Sync EC2 instances:

```bash theme={null}
# Sync from a specific region
ctrlc sync aws ec2 --region us-east-1

# Continuous sync
ctrlc sync aws ec2 --region us-east-1 --interval 5m
```

### Resource Metadata

```yaml theme={null}
identifier: i-0123456789abcdef0
name: web-server-1  # from Name tag
kind: AWS/EC2
metadata:
  region: us-east-1
  availability_zone: us-east-1a
  instance_type: t3.medium
  environment: production  # from AWS tag
config:
  private_ip: 10.0.1.100
  public_ip: 54.123.45.67
  vpc_id: vpc-12345
```

## RDS Instances

Sync RDS database instances:

```bash theme={null}
# Sync from a specific region
ctrlc sync aws rds --region us-east-1

# Continuous sync
ctrlc sync aws rds --region us-east-1 --interval 10m
```

### Resource Metadata

```yaml theme={null}
identifier: arn:aws:rds:us-east-1:123456789:db:prod-db
name: prod-db
kind: AWS/RDS
metadata:
  region: us-east-1
  engine: postgres
  engine_version: "15.4"
  instance_class: db.r5.large
  environment: production  # from AWS tag
config:
  endpoint: prod-db.xxxxx.us-east-1.rds.amazonaws.com
  port: 5432
```

## VPC Networks

Sync VPCs and subnets:

```bash theme={null}
# Sync from a specific region
ctrlc sync aws networks --region us-east-1
```

## Running in AWS

### ECS Task

```json theme={null}
{
  "family": "ctrlplane-sync",
  "containerDefinitions": [
    {
      "name": "sync",
      "image": "ghcr.io/ctrlplanedev/cli:latest",
      "command": [
        "ctrlc", "sync", "aws", "eks",
        "--region", "us-east-1",
        "--interval", "5m"
      ],
      "environment": [
        {
          "name": "CTRLPLANE_API_KEY",
          "value": "your-api-key"
        },
        {
          "name": "CTRLPLANE_WORKSPACE",
          "value": "your-workspace-id"
        }
      ]
    }
  ],
  "taskRoleArn": "arn:aws:iam::123456789:role/ctrlplane-sync-role"
}
```

### IAM Policy

The sync task needs read permissions:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "eks:ListClusters",
        "eks:DescribeCluster",
        "ec2:DescribeInstances",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "rds:DescribeDBInstances",
        "tag:GetResources"
      ],
      "Resource": "*"
    }
  ]
}
```

### Lambda Function

Run sync periodically with Lambda:

```python theme={null}
import subprocess

def handler(event, context):
    subprocess.run([
        "ctrlc", "sync", "aws", "eks",
        "--region", "us-east-1"
    ], check=True)
```

## Environment Targeting

Target AWS resources in environments:

```yaml theme={null}
# All production EKS clusters
type: Environment
name: Production EKS
resourceSelector: |
  resource.kind == "AWS/EKS" &&
  resource.metadata["environment"] == "production"
```

```yaml theme={null}
# US East resources only
type: Environment
name: US East
resourceSelector: |
  resource.metadata["region"] == "us-east-1"
```

```yaml theme={null}
# Production databases
type: Environment
name: Production Databases
resourceSelector: |
  resource.kind == "AWS/RDS" &&
  resource.metadata["environment"] == "production"
```

## Best Practices

### Tag Your Resources

Ensure AWS resources have meaningful tags:

```bash theme={null}
aws ec2 create-tags --resources i-12345 --tags \
  Key=environment,Value=production \
  Key=team,Value=platform \
  Key=tier,Value=critical
```

### Use Multiple Regions

Sync from all regions your infrastructure spans:

```bash theme={null}
ctrlc sync aws eks \
  --region us-east-1 \
  --region us-west-2 \
  --region eu-west-1 \
  --interval 5m
```

### Separate by Resource Type

Run separate sync processes for different resource types:

```bash theme={null}
# EKS sync
ctrlc sync aws eks --interval 5m &

# EC2 sync
ctrlc sync aws ec2 --interval 5m &

# RDS sync (less frequent)
ctrlc sync aws rds --interval 15m &
```

## Next Steps

<CardGroup>
  <Card title="Google Cloud" icon="google" href="./google-cloud">
    Sync GCP resources
  </Card>

  <Card title="Azure" icon="microsoft" href="./azure">
    Sync Azure resources
  </Card>

  <Card title="Selectors" icon="filter" href="../../concepts/selectors">
    Learn selector syntax
  </Card>

  <Card title="Environments" icon="layer-group" href="../../concepts/environments">
    Create dynamic environments
  </Card>
</CardGroup>


# Azure Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/azure

Sync Azure resources into Ctrlplane

The Azure provider syncs resources from Microsoft Azure into Ctrlplane's
inventory—AKS clusters and virtual networks.

## Prerequisites

* `ctrlc` CLI installed
* Azure credentials configured (Azure CLI, environment variables, or managed identity)
* Ctrlplane API key

## Supported Resources

| Command          | Resource Type    | Ctrlplane Kind |
| ---------------- | ---------------- | -------------- |
| `azure aks`      | AKS Clusters     | `Azure/AKS`    |
| `azure networks` | Virtual Networks | `Azure/VNet`   |

## Authentication

Configure Azure credentials:

```bash theme={null}
# Azure CLI (recommended for local development)
az login

# Service Principal (for CI/CD)
export AZURE_CLIENT_ID="your-client-id"
export AZURE_CLIENT_SECRET="your-client-secret"
export AZURE_TENANT_ID="your-tenant-id"

# Managed Identity (when running in Azure)
# Credentials are automatically retrieved
```

## AKS Clusters

Sync Azure Kubernetes Service clusters:

```bash theme={null}
# Sync from default subscription
ctrlc sync azure aks

# Sync from a specific subscription
ctrlc sync azure aks --subscription-id 00000000-0000-0000-0000-000000000000

# Continuous sync
ctrlc sync azure aks --interval 5m
```

### Options

| Flag                | Description                      | Required          |
| ------------------- | -------------------------------- | ----------------- |
| `--subscription-id` | Azure subscription ID            | No (uses default) |
| `--interval`        | Sync interval (e.g., `5m`, `1h`) | No                |

### Resource Metadata

```yaml theme={null}
identifier: /subscriptions/xxx/resourceGroups/prod-rg/providers/Microsoft.ContainerService/managedClusters/prod-cluster
name: prod-cluster
kind: Azure/AKS
metadata:
  subscription: 00000000-0000-0000-0000-000000000000
  resource_group: prod-rg
  location: eastus
  environment: production  # from Azure tag
  team: platform           # from Azure tag
config:
  fqdn: prod-cluster-xxxxx.hcp.eastus.azmk8s.io
  kubernetes_version: "1.28.3"
```

## Virtual Networks

Sync Azure Virtual Networks:

```bash theme={null}
# Sync from default subscription
ctrlc sync azure networks

# Sync from a specific subscription
ctrlc sync azure networks --subscription-id 00000000-0000-0000-0000-000000000000
```

### Resource Metadata

```yaml theme={null}
identifier: /subscriptions/xxx/resourceGroups/prod-rg/providers/Microsoft.Network/virtualNetworks/prod-vnet
name: prod-vnet
kind: Azure/VNet
metadata:
  subscription: 00000000-0000-0000-0000-000000000000
  resource_group: prod-rg
  location: eastus
config:
  address_space: ["10.0.0.0/16"]
```

## Running in Azure

### Azure Container Instances

```bash theme={null}
az container create \
  --resource-group ctrlplane-rg \
  --name ctrlplane-sync \
  --image ghcr.io/ctrlplanedev/cli:latest \
  --command-line "ctrlc sync azure aks --interval 5m" \
  --environment-variables \
    CTRLPLANE_API_KEY=your-api-key \
    CTRLPLANE_WORKSPACE=your-workspace-id \
  --assign-identity
```

### AKS Deployment with Workload Identity

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-azure-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-azure-sync
  template:
    metadata:
      labels:
        app: ctrlplane-azure-sync
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: ctrlplane-sync
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - azure
            - aks
            - --interval
            - "5m"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: your-workspace-id
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ctrlplane-sync
  annotations:
    azure.workload.identity/client-id: your-client-id
```

### Required Azure Permissions

The sync identity needs Reader permissions:

```bash theme={null}
# Assign Reader role at subscription level
az role assignment create \
  --assignee <identity-principal-id> \
  --role "Reader" \
  --scope /subscriptions/<subscription-id>

# Or at resource group level
az role assignment create \
  --assignee <identity-principal-id> \
  --role "Reader" \
  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>
```

## Environment Targeting

Target Azure resources in environments:

```yaml theme={null}
# All production AKS clusters
type: Environment
name: Production AKS
resourceSelector: |
  resource.kind == "Azure/AKS" &&
  resource.metadata["environment"] == "production"
```

```yaml theme={null}
# East US resources
type: Environment
name: East US
resourceSelector: |
  resource.metadata["location"] == "eastus"
```

```yaml theme={null}
# Specific resource group
type: Environment
name: Production Resource Group
resourceSelector: |
  resource.metadata["resource_group"] == "prod-rg"
```

## Best Practices

### Tag Your Resources

Ensure Azure resources have meaningful tags:

```bash theme={null}
az aks update \
  --resource-group prod-rg \
  --name prod-cluster \
  --tags environment=production team=platform tier=critical
```

### Sync Multiple Subscriptions

Run sync for each subscription:

```bash theme={null}
# Production subscription
ctrlc sync azure aks \
  --subscription-id prod-subscription-id \
  --interval 5m &

# Staging subscription
ctrlc sync azure aks \
  --subscription-id staging-subscription-id \
  --interval 5m &
```

## Next Steps

<CardGroup>
  <Card title="AWS" icon="aws" href="./aws">
    Sync AWS resources
  </Card>

  <Card title="Google Cloud" icon="google" href="./google-cloud">
    Sync GCP resources
  </Card>

  <Card title="Selectors" icon="filter" href="../../concepts/selectors">
    Learn selector syntax
  </Card>

  <Card title="Environments" icon="layer-group" href="../../concepts/environments">
    Create dynamic environments
  </Card>
</CardGroup>


# Custom Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/custom

Build your own resource provider via API

Create custom resource providers to sync any infrastructure into Ctrlplane's
inventory using the API or SDK.

## When to Use Custom Providers

Use custom providers when you need to sync:

* Internal infrastructure management systems
* Custom cloud platforms
* Database clusters
* Edge devices
* Any resource not covered by built-in providers

## Using the API

### Create or Update a Resource

```bash theme={null}
curl -X PUT "https://your-ctrlplane-instance.com/api/v1/workspaces/{workspaceId}/resources" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "identifier": "my-server-1",
    "name": "Production Server 1",
    "kind": "Server",
    "version": "1.0.0",
    "metadata": {
      "environment": "production",
      "region": "us-east-1",
      "team": "platform"
    },
    "config": {
      "host": "10.0.1.100",
      "port": 22
    }
  }'
```

### Delete a Resource

```bash theme={null}
curl -X DELETE \
  "https://your-ctrlplane-instance.com/api/v1/resources/{resourceId}" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY"
```

### List Resources

```bash theme={null}
curl "https://your-ctrlplane-instance.com/api/v1/workspaces/{workspaceId}/resources" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY"
```

## Using the Node.js SDK

```typescript theme={null}
import { Ctrlplane } from "@ctrlplane/node-sdk";

const client = new Ctrlplane({ apiKey: process.env.CTRLPLANE_API_KEY });

// Sync resources from your infrastructure
async function syncResources() {
  const servers = await discoverServers(); // Your discovery logic

  for (const server of servers) {
    await client.resources.upsert({
      workspaceId: "your-workspace-id",
      identifier: server.id,
      name: server.hostname,
      kind: "Server",
      version: "1.0.0",
      metadata: {
        environment: server.environment,
        region: server.region,
        team: server.team,
      },
      config: {
        host: server.ip,
        port: 22,
      },
    });
  }
}

// Run on interval
setInterval(syncResources, 5 * 60 * 1000); // Every 5 minutes
```

## Using a Shell Script

```bash theme={null}
#!/bin/bash
# sync-resources.sh

WORKSPACE_ID="your-workspace-id"
API_URL="https://your-ctrlplane-instance.com/api/v1"

# Discover resources (example: from cloud provider)
INSTANCES=$(aws ec2 describe-instances --query 'Reservations[].Instances[]')

# Sync each instance to Ctrlplane
echo "$INSTANCES" | jq -c '.[]' | while read instance; do
  INSTANCE_ID=$(echo "$instance" | jq -r '.InstanceId')
  NAME=$(echo "$instance" | jq -r '.Tags[] | select(.Key=="Name") | .Value')
  ENV=$(echo "$instance" | jq -r '.Tags[] | select(.Key=="Environment") | .Value')
  REGION=$(echo "$instance" | jq -r '.Placement.AvailabilityZone' | sed 's/.$//')

  curl -X PUT "${API_URL}/workspaces/${WORKSPACE_ID}/resources" \
    -H "Authorization: Bearer ${CTRLPLANE_API_KEY}" \
    -H "Content-Type: application/json" \
    -d "{
      \"identifier\": \"${INSTANCE_ID}\",
      \"name\": \"${NAME}\",
      \"kind\": \"AWS/EC2\",
      \"version\": \"1.0.0\",
      \"metadata\": {
        \"environment\": \"${ENV}\",
        \"region\": \"${REGION}\"
      }
    }"
done
```

## Using Python

```python theme={null}
import os
import requests
import time

API_KEY = os.environ["CTRLPLANE_API_KEY"]
WORKSPACE_ID = os.environ["CTRLPLANE_WORKSPACE"]
API_URL = "https://your-ctrlplane-instance.com/api/v1"

def sync_resource(resource):
    """Upsert a resource to Ctrlplane."""
    response = requests.put(
        f"{API_URL}/workspaces/{WORKSPACE_ID}/resources",
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        },
        json=resource
    )
    response.raise_for_status()
    return response.json()

def discover_resources():
    """Your custom discovery logic."""
    # Example: query your internal CMDB
    return [
        {
            "identifier": "server-001",
            "name": "Web Server 1",
            "kind": "Server",
            "version": "1.0.0",
            "metadata": {
                "environment": "production",
                "region": "us-east-1",
                "team": "platform"
            },
            "config": {
                "host": "10.0.1.100",
                "port": 22
            }
        }
    ]

def sync_all():
    """Sync all resources."""
    resources = discover_resources()
    for resource in resources:
        sync_resource(resource)
        print(f"Synced: {resource['name']}")

if __name__ == "__main__":
    while True:
        sync_all()
        time.sleep(300)  # Every 5 minutes
```

## Resource Schema

| Field        | Required | Description                                |
| ------------ | -------- | ------------------------------------------ |
| `identifier` | Yes      | Unique identifier for the resource         |
| `name`       | Yes      | Human-readable name                        |
| `kind`       | Yes      | Resource type (use `Category/Type` format) |
| `version`    | Yes      | Resource version/schema version            |
| `metadata`   | No       | Key-value pairs for filtering              |
| `config`     | No       | Configuration data for job agents          |

### Kind Naming Convention

Use a consistent naming convention:

```yaml theme={null}
# Good: Category/Type format
kind: Server/Linux
kind: Database/PostgreSQL
kind: Cache/Redis
kind: Queue/RabbitMQ

# Bad: inconsistent formats
kind: linux-server
kind: postgres
kind: redis
```

### Metadata vs Config

* **Metadata**: Used for environment selectors and filtering
* **Config**: Passed to job agents for deployment execution

```yaml theme={null}
# Metadata: for targeting
metadata:
  environment: production
  region: us-east-1
  team: platform
  tier: critical

# Config: for deployment
config:
  host: 10.0.1.100
  port: 22
  ssh_key_path: /path/to/key
```

## Running Continuously

### Kubernetes CronJob

```yaml theme={null}
apiVersion: batch/v1
kind: CronJob
metadata:
  name: sync-custom-resources
spec:
  schedule: "*/5 * * * *"  # Every 5 minutes
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: sync
              image: your-sync-image
              command: ["python", "sync.py"]
              env:
                - name: CTRLPLANE_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: ctrlplane-credentials
                      key: api-key
                - name: CTRLPLANE_WORKSPACE
                  value: your-workspace-id
          restartPolicy: OnFailure
```

### Kubernetes Deployment

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-resource-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-resource-sync
  template:
    metadata:
      labels:
        app: custom-resource-sync
    spec:
      containers:
        - name: sync
          image: your-sync-image
          command: ["python", "sync.py"]
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: your-workspace-id
```

## Best Practices

### Use Stable Identifiers

Use identifiers that won't change:

```yaml theme={null}
# Good: stable identifiers
identifier: server-001
identifier: db-prod-primary
identifier: cache-us-east-1

# Bad: may change
identifier: 10.0.1.100
identifier: ip-10-0-1-100
```

### Include Essential Metadata

Include metadata for effective targeting:

```yaml theme={null}
metadata:
  environment: production
  region: us-east-1
  team: platform
  tier: critical
```

### Handle Deletions

Remove resources that no longer exist:

```python theme={null}
def sync_all():
    # Get current resources from source
    current_resources = discover_resources()
    current_ids = {r["identifier"] for r in current_resources}
    
    # Get existing resources in Ctrlplane
    existing = get_ctrlplane_resources()
    existing_ids = {r["identifier"] for r in existing}
    
    # Sync current resources
    for resource in current_resources:
        sync_resource(resource)
    
    # Delete removed resources
    for resource in existing:
        if resource["identifier"] not in current_ids:
            delete_resource(resource["id"])
```

## Next Steps

<CardGroup>
  <Card title="Selectors" icon="filter" href="../../concepts/selectors">
    Learn selector syntax
  </Card>

  <Card title="Environments" icon="layer-group" href="../../concepts/environments">
    Create dynamic environments
  </Card>

  <Card title="API Reference" icon="code" href="../../api">
    Full API documentation
  </Card>

  <Card title="Node SDK" icon="npm" href="../../sdk/node">
    Node.js SDK reference
  </Card>
</CardGroup>


# GitHub Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/github

Sync GitHub resources into Ctrlplane

The GitHub provider syncs resources from GitHub into Ctrlplane's
inventory—currently supporting pull requests as deployment targets.

## Prerequisites

* `ctrlc` CLI installed
* GitHub personal access token or GitHub App
* Ctrlplane API key

## Supported Resources

| Command                | Resource Type | Ctrlplane Kind       |
| ---------------------- | ------------- | -------------------- |
| `github pull-requests` | Pull Requests | `GitHub/PullRequest` |

## Authentication

Set your GitHub token:

```bash theme={null}
export GITHUB_TOKEN="your-github-token"
```

## Pull Requests

Sync GitHub pull requests as resources for preview environments:

```bash theme={null}
# Sync pull requests from a repository
ctrlc sync github pull-requests \
  --owner my-org \
  --repo my-app

# Continuous sync
ctrlc sync github pull-requests \
  --owner my-org \
  --repo my-app \
  --interval 5m
```

## Resource Metadata

Each pull request is synced with metadata:

```yaml theme={null}
identifier: github.com/my-org/my-app/pull/123
name: "PR #123: Add new feature"
kind: GitHub/PullRequest
metadata:
  owner: my-org
  repo: my-app
  number: "123"
  state: open
  author: developer
  branch: feature/new-feature
  base_branch: main
config:
  url: https://github.com/my-org/my-app/pull/123
  head_sha: abc123def456
```

## Use Case: Preview Environments

Sync PRs to create dynamic preview environments:

```yaml theme={null}
# Environment for all open PRs
type: Environment
name: Preview Environments
resourceSelector: |
  resource.kind == "GitHub/PullRequest" &&
  resource.metadata["state"] == "open"
```

When a PR is opened:

1. GitHub provider syncs it as a resource
2. Environment selector matches the PR
3. Deployments can target the PR environment
4. When PR is closed/merged, resource is removed

## Running Continuously

### GitHub Actions

```yaml theme={null}
name: Sync PRs to Ctrlplane

on:
  schedule:
    - cron: '*/5 * * * *'  # Every 5 minutes
  pull_request:
    types: [opened, closed, reopened]

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - name: Sync PRs
        run: |
          ctrlc sync github pull-requests \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }}
        env:
          CTRLPLANE_API_KEY: ${{ secrets.CTRLPLANE_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

### Kubernetes Deployment

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-github-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-github-sync
  template:
    metadata:
      labels:
        app: ctrlplane-github-sync
    spec:
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - github
            - pull-requests
            - --owner
            - my-org
            - --repo
            - my-app
            - --interval
            - "5m"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: GITHUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-credentials
                  key: token
```

## Best Practices

### Sync on PR Events

Trigger sync immediately when PRs change:

```yaml theme={null}
# GitHub Actions workflow
on:
  pull_request:
    types: [opened, closed, synchronize, reopened]
```

### Label for Targeting

Use PR labels for more specific targeting:

```yaml theme={null}
# Only PRs with 'preview' label
type: Environment
name: Preview Environments
resourceSelector: |
  resource.kind == "GitHub/PullRequest" &&
  resource.metadata["labels"].contains("preview")
```

## Next Steps

<CardGroup>
  <Card title="GitHub Actions Job Agent" icon="github" href="../job-agents/github">
    Deploy to PRs with GitHub Actions
  </Card>

  <Card title="Dynamic Environments" icon="layer-group" href="../../use-cases/dynamic-environments">
    Learn about dynamic environments
  </Card>
</CardGroup>


# Google Cloud Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/google-cloud

Sync Google Cloud resources into Ctrlplane

The Google Cloud provider syncs resources from GCP into Ctrlplane's
inventory—GKE clusters, VMs, Cloud SQL, Cloud Run, and more.

## Prerequisites

* `ctrlc` CLI installed
* Google Cloud credentials (application default credentials or service account)
* Ctrlplane API key

## Supported Resources

| Command                 | Resource Type       | Ctrlplane Kind |
| ----------------------- | ------------------- | -------------- |
| `google-cloud gke`      | GKE Clusters        | `GCP/GKE`      |
| `google-cloud vms`      | Compute Engine VMs  | `GCP/VM`       |
| `google-cloud cloudsql` | Cloud SQL Instances | `GCP/CloudSQL` |
| `google-cloud cloudrun` | Cloud Run Services  | `GCP/CloudRun` |
| `google-cloud buckets`  | Storage Buckets     | `GCP/Bucket`   |
| `google-cloud bigtable` | Bigtable Instances  | `GCP/Bigtable` |
| `google-cloud redis`    | Memorystore Redis   | `GCP/Redis`    |
| `google-cloud secrets`  | Secret Manager      | `GCP/Secret`   |
| `google-cloud networks` | VPC Networks        | `GCP/VPC`      |
| `google-cloud projects` | GCP Projects        | `GCP/Project`  |

## Authentication

Configure GCP credentials:

```bash theme={null}
# Application Default Credentials (recommended for local development)
gcloud auth application-default login

# Service Account key file
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# Workload Identity (when running in GKE)
# Credentials are automatically retrieved
```

## GKE Clusters

Sync Google Kubernetes Engine clusters:

```bash theme={null}
# Sync from a specific project
ctrlc sync google-cloud gke --project my-project

# Continuous sync
ctrlc sync google-cloud gke --project my-project --interval 5m
```

### Resource Metadata

```yaml theme={null}
identifier: projects/my-project/locations/us-central1/clusters/prod-cluster
name: prod-cluster
kind: GCP/GKE
metadata:
  project: my-project
  region: us-central1
  environment: production  # from GCP label
  team: platform           # from GCP label
config:
  endpoint: https://XXX.XXX.XXX.XXX
  version: "1.28.3-gke.1286000"
```

## Compute Engine VMs

Sync virtual machine instances:

```bash theme={null}
# Sync from a project
ctrlc sync google-cloud vms --project my-project

# Continuous sync
ctrlc sync google-cloud vms --project my-project --interval 5m
```

### Resource Metadata

```yaml theme={null}
identifier: projects/my-project/zones/us-central1-a/instances/web-server-1
name: web-server-1
kind: GCP/VM
metadata:
  project: my-project
  zone: us-central1-a
  machine_type: e2-medium
  environment: production  # from GCP label
config:
  internal_ip: 10.128.0.2
  external_ip: 34.123.45.67
```

## Cloud SQL Instances

Sync Cloud SQL database instances:

```bash theme={null}
# Sync from a project
ctrlc sync google-cloud cloudsql --project my-project

# Continuous sync
ctrlc sync google-cloud cloudsql --project my-project --interval 10m
```

### Resource Metadata

```yaml theme={null}
identifier: projects/my-project/instances/prod-db
name: prod-db
kind: GCP/CloudSQL
metadata:
  project: my-project
  region: us-central1
  database_version: POSTGRES_15
  tier: db-custom-4-16384
  environment: production  # from GCP label
config:
  connection_name: my-project:us-central1:prod-db
  ip_address: 10.0.0.5
```

## Cloud Run Services

Sync Cloud Run services:

```bash theme={null}
# Sync from a project
ctrlc sync google-cloud cloudrun --project my-project

# Continuous sync
ctrlc sync google-cloud cloudrun --project my-project --interval 5m
```

### Resource Metadata

```yaml theme={null}
identifier: projects/my-project/locations/us-central1/services/api-service
name: api-service
kind: GCP/CloudRun
metadata:
  project: my-project
  region: us-central1
  environment: production  # from GCP label
config:
  url: https://api-service-xxxxx-uc.a.run.app
```

## Running in GCP

### Cloud Run Job

```yaml theme={null}
apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: ctrlplane-sync
spec:
  template:
    spec:
      containers:
        - image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - google-cloud
            - gke
            - --project
            - my-project
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: your-workspace-id
```

### GKE Deployment with Workload Identity

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-gcp-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-gcp-sync
  template:
    metadata:
      labels:
        app: ctrlplane-gcp-sync
    spec:
      serviceAccountName: ctrlplane-sync
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - google-cloud
            - gke
            - --project
            - my-project
            - --interval
            - "5m"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ctrlplane-sync
  annotations:
    iam.gke.io/gcp-service-account: ctrlplane-sync@my-project.iam.gserviceaccount.com
```

### IAM Permissions

The sync service account needs read permissions:

```bash theme={null}
# Create service account
gcloud iam service-accounts create ctrlplane-sync

# Grant permissions
gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:ctrlplane-sync@my-project.iam.gserviceaccount.com" \
  --role="roles/container.viewer"

gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:ctrlplane-sync@my-project.iam.gserviceaccount.com" \
  --role="roles/compute.viewer"

gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:ctrlplane-sync@my-project.iam.gserviceaccount.com" \
  --role="roles/cloudsql.viewer"
```

## Environment Targeting

Target GCP resources in environments:

```yaml theme={null}
# All production GKE clusters
type: Environment
name: Production GKE
resourceSelector: |
  resource.kind == "GCP/GKE" &&
  resource.metadata["environment"] == "production"
```

```yaml theme={null}
# US Central resources
type: Environment
name: US Central
resourceSelector: |
  resource.metadata["region"].startsWith("us-central")
```

```yaml theme={null}
# All Cloud Run services
type: Environment
name: Cloud Run Production
resourceSelector: |
  resource.kind == "GCP/CloudRun" &&
  resource.metadata["environment"] == "production"
```

## Best Practices

### Label Your Resources

Ensure GCP resources have meaningful labels:

```bash theme={null}
gcloud compute instances add-labels web-server-1 \
  --labels=environment=production,team=platform,tier=critical
```

### Sync Multiple Projects

Run sync for each project:

```bash theme={null}
# Production project
ctrlc sync google-cloud gke --project prod-project --interval 5m &

# Staging project
ctrlc sync google-cloud gke --project staging-project --interval 5m &
```

### Sync Multiple Resource Types

Run separate sync processes:

```bash theme={null}
# GKE clusters
ctrlc sync google-cloud gke --project my-project --interval 5m &

# Cloud SQL (less frequent)
ctrlc sync google-cloud cloudsql --project my-project --interval 15m &

# Cloud Run
ctrlc sync google-cloud cloudrun --project my-project --interval 5m &
```

## Next Steps

<CardGroup>
  <Card title="AWS" icon="aws" href="./aws">
    Sync AWS resources
  </Card>

  <Card title="Azure" icon="microsoft" href="./azure">
    Sync Azure resources
  </Card>

  <Card title="Selectors" icon="filter" href="../../concepts/selectors">
    Learn selector syntax
  </Card>

  <Card title="Environments" icon="layer-group" href="../../concepts/environments">
    Create dynamic environments
  </Card>
</CardGroup>


# Helm Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/helm

Sync Helm releases into Ctrlplane

The Helm provider syncs Helm releases from your Kubernetes clusters into
Ctrlplane's inventory.

## Prerequisites

* `ctrlc` CLI installed
* Kubernetes cluster access (kubeconfig or in-cluster)
* Ctrlplane API key

## Basic Usage

```bash theme={null}
# Sync all Helm releases from the cluster
ctrlc sync helm \
  --cluster-name my-cluster \
  --cluster-identifier k8s-prod-us-east-1

# Sync from a specific namespace
ctrlc sync helm \
  --cluster-name my-cluster \
  --cluster-identifier k8s-prod-us-east-1 \
  --namespace production

# Continuous sync
ctrlc sync helm \
  --cluster-name my-cluster \
  --cluster-identifier k8s-prod-us-east-1 \
  --interval 5m
```

## Options

| Flag                   | Description                                         | Required |
| ---------------------- | --------------------------------------------------- | -------- |
| `--cluster-name`       | Display name for the cluster                        | Yes      |
| `--cluster-identifier` | Unique identifier (or `CLUSTER_IDENTIFIER` env var) | Yes      |
| `--namespace`          | Kubernetes namespace (all namespaces if not set)    | No       |
| `--provider`           | Resource provider name                              | No       |
| `--interval`           | Sync interval (e.g., `5m`, `1h`)                    | No       |

## Resource Metadata

Each Helm release is synced with metadata:

```yaml theme={null}
identifier: k8s-prod-us-east-1/helm/production/api-gateway
name: api-gateway
kind: Helm/Release
metadata:
  cluster: k8s-prod-us-east-1
  namespace: production
  chart: api-gateway
  chart_version: "1.2.3"
  app_version: "2.0.0"
  status: deployed
config:
  cluster: k8s-prod-us-east-1
  namespace: production
  release_name: api-gateway
```

## Running in Kubernetes

Deploy as a Deployment:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-helm-sync
  namespace: ctrlplane
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-helm-sync
  template:
    metadata:
      labels:
        app: ctrlplane-helm-sync
    spec:
      serviceAccountName: ctrlplane-sync
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - helm
            - --cluster-name
            - "$(CLUSTER_NAME)"
            - --cluster-identifier
            - "$(CLUSTER_IDENTIFIER)"
            - --interval
            - "5m"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: your-workspace-id
            - name: CLUSTER_NAME
              value: "Production US-East"
            - name: CLUSTER_IDENTIFIER
              value: "prod-us-east-1"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ctrlplane-sync
  namespace: ctrlplane
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ctrlplane-helm-reader
rules:
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ctrlplane-helm-sync
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ctrlplane-helm-reader
subjects:
  - kind: ServiceAccount
    name: ctrlplane-sync
    namespace: ctrlplane
```

## Environment Targeting

Target Helm releases in environments:

```yaml theme={null}
# All production releases
type: Environment
name: Production Helm Releases
resourceSelector: |
  resource.kind == "Helm/Release" &&
  resource.metadata["namespace"] == "production"
```

```yaml theme={null}
# Specific chart releases
type: Environment
name: API Gateway Releases
resourceSelector: |
  resource.kind == "Helm/Release" &&
  resource.metadata["chart"] == "api-gateway"
```

## Best Practices

### Sync All Namespaces

Unless you have specific needs, sync from all namespaces:

```bash theme={null}
# Omit --namespace to sync all
ctrlc sync helm \
  --cluster-name my-cluster \
  --cluster-identifier prod-cluster
```

### Combine with Kubernetes Sync

Run both Kubernetes and Helm sync:

```bash theme={null}
# Kubernetes resources
ctrlc sync kubernetes \
  --cluster-name my-cluster \
  --cluster-identifier prod-cluster \
  --interval 5m &

# Helm releases
ctrlc sync helm \
  --cluster-name my-cluster \
  --cluster-identifier prod-cluster \
  --interval 5m &
```

## Next Steps

<CardGroup>
  <Card title="Kubernetes Provider" icon="dharmachakra" href="./kubernetes">
    Sync Kubernetes resources
  </Card>

  <Card title="vCluster Provider" icon="dharmachakra" href="./vcluster">
    Sync virtual clusters
  </Card>
</CardGroup>


# Kubernetes Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/kubernetes

Sync Kubernetes resources into Ctrlplane

The Kubernetes provider syncs resources from your Kubernetes clusters into
Ctrlplane's inventory—namespaces, deployments, services, and more.

## Prerequisites

* `ctrlc` CLI installed
* Kubernetes cluster access (kubeconfig or in-cluster)
* Ctrlplane API key

## Basic Usage

```bash theme={null}
# Sync all resources from the current cluster
ctrlc sync kubernetes \
  --cluster-name my-cluster \
  --cluster-identifier k8s-prod-us-east-1
```

## Options

| Flag                   | Description                                             | Required |
| ---------------------- | ------------------------------------------------------- | -------- |
| `--cluster-name`       | Display name for the cluster                            | Yes      |
| `--cluster-identifier` | Unique identifier (or set `CLUSTER_IDENTIFIER` env var) | Yes      |
| `--provider`           | Resource provider name                                  | No       |
| `--interval`           | Sync interval (e.g., `5m`, `1h`)                        | No       |

## Examples

### One-Time Sync

```bash theme={null}
# Sync from current kubeconfig context
ctrlc sync kubernetes \
  --cluster-name "Production US-East" \
  --cluster-identifier prod-us-east-1
```

### Continuous Sync

```bash theme={null}
# Sync every 5 minutes
ctrlc sync kubernetes \
  --cluster-name "Production US-East" \
  --cluster-identifier prod-us-east-1 \
  --interval 5m
```

### Using Environment Variables

```bash theme={null}
export CTRLPLANE_API_KEY="your-api-key"
export CTRLPLANE_WORKSPACE="your-workspace-id"
export CLUSTER_IDENTIFIER="prod-us-east-1"

ctrlc sync kubernetes --cluster-name "Production US-East"
```

## Synced Resources

The Kubernetes provider creates resources for:

| Kubernetes Resource | Ctrlplane Kind          | Description                   |
| ------------------- | ----------------------- | ----------------------------- |
| Namespaces          | `KubernetesNamespace`   | All namespaces in the cluster |
| Deployments         | `KubernetesDeployment`  | Deployment workloads          |
| StatefulSets        | `KubernetesStatefulSet` | Stateful workloads            |
| Services            | `KubernetesService`     | Service endpoints             |

### Resource Metadata

Each synced resource includes metadata from Kubernetes labels:

```yaml theme={null}
# Kubernetes Namespace with labels
apiVersion: v1
kind: Namespace
metadata:
  name: api-production
  labels:
    environment: production
    team: platform
    tier: critical
```

Becomes:

```yaml theme={null}
# Ctrlplane Resource
identifier: prod-us-east-1/namespace/api-production
name: api-production
kind: Kubernetes/Namespace
metadata:
  environment: production
  team: platform
  tier: critical
  cluster: prod-us-east-1
config:
  namespace: api-production
  cluster: prod-us-east-1
```

## Running in Kubernetes

Deploy as a Deployment with in-cluster authentication:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-k8s-sync
  namespace: ctrlplane
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-k8s-sync
  template:
    metadata:
      labels:
        app: ctrlplane-k8s-sync
    spec:
      serviceAccountName: ctrlplane-sync
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - kubernetes
            - --interval
            - "5m"
            - --cluster-name
            - "$(CLUSTER_NAME)"
            - --cluster-identifier
            - "$(CLUSTER_IDENTIFIER)"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: "your-workspace-id"
            - name: CLUSTER_NAME
              value: "Production US-East"
            - name: CLUSTER_IDENTIFIER
              value: "prod-us-east-1"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ctrlplane-sync
  namespace: ctrlplane
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ctrlplane-sync
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: ctrlplane-sync
    namespace: ctrlplane
```

## Best Practices

### Label Your Resources

Ensure Kubernetes resources have meaningful labels:

```yaml theme={null}
metadata:
  labels:
    environment: production
    team: platform
    app: api-gateway
    tier: critical
```

### Use Consistent Identifiers

Use stable cluster identifiers that won't change:

```bash theme={null}
# Good: descriptive and stable
--cluster-identifier prod-us-east-1

# Bad: may change
--cluster-identifier cluster-12345
```

### Sync Frequently

Keep resources up-to-date with short sync intervals:

```bash theme={null}
# Every 1 minutes is a good default
--interval 1m
```

## Next Steps

<CardGroup>
  <Card title="Helm Provider" icon="helm" href="./helm">
    Sync Helm releases
  </Card>

  <Card title="vCluster Provider" icon="dharmachakra" href="./vcluster">
    Sync virtual clusters
  </Card>

  <Card title="Selectors" icon="filter" href="../../concepts/selectors">
    Learn selector syntax
  </Card>

  <Card title="Environments" icon="layer-group" href="../../concepts/environments">
    Create dynamic environments
  </Card>
</CardGroup>


# Resource Providers Overview
Source: https://docs.ctrlplane.dev/integrations/resource-providers/overview

Sync infrastructure resources into Ctrlplane's inventory

Resource providers automatically discover and sync your infrastructure into
Ctrlplane's resource inventory. This enables dynamic environment targeting based
on resource metadata.

## Quick Start

All providers use the same pattern:

```bash theme={null}
# Set up authentication
export CTRLPLANE_API_KEY="your-api-key"

# Sync resources (one-time)
ctrlc sync <provider> [options]

# Sync on interval (continuous)
ctrlc sync <provider> --interval 5m [options]
```

## Common Patterns

### Continuous Sync

Run sync on an interval to keep resources up-to-date:

```bash theme={null}
# Sync every 5 minutes
ctrlc sync kubernetes --interval 5m --cluster-name prod-cluster

# Sync every hour
ctrlc sync aws eks --region us-east-1 --interval 1h
```

### Running in Kubernetes

Deploy a sync job as a Kubernetes CronJob or Deployment:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-sync
  template:
    metadata:
      labels:
        app: ctrlplane-sync
    spec:
      containers:
        - name: sync
          image: ctrlplane/cli:latest
          command:
            - ctrlc
            - sync
            - kubernetes
            - --interval
            - "5m"
            - --cluster-name
            - "$(CLUSTER_NAME)"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CLUSTER_NAME
              value: "my-cluster"
```

## Resource Schema

All resources follow the same schema:

| Field        | Required | Description                                     |
| ------------ | -------- | ----------------------------------------------- |
| `identifier` | Yes      | Unique identifier (auto-generated by providers) |
| `name`       | Yes      | Human-readable name                             |
| `kind`       | Yes      | Resource type (e.g., `Kubernetes/Namespace`)    |
| `version`    | Yes      | Resource version                                |
| `metadata`   | No       | Key-value pairs for filtering and selectors     |
| `config`     | No       | Configuration passed to job agents              |

### Metadata vs Config

* **Metadata** — Used for environment selectors and filtering
* **Config** — Passed to job agents for deployment execution

```yaml theme={null}
# Metadata: used for targeting
metadata:
  environment: production
  region: us-east-1
  team: platform

# Config: used by job agents
config:
  namespace: my-app
  cluster_url: https://k8s.example.com
```

## Environment Selectors

Resources are matched to environments using selectors:

```yaml theme={null}
type: Environment
name: Production US
resourceSelector: |
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"].startsWith("us-")
```

When resources are synced, environments automatically re-evaluate selectors.

## Next Steps

<CardGroup>
  <Card title="Kubernetes" icon="dharmachakra" href="./kubernetes">
    Sync Kubernetes resources
  </Card>

  <Card title="AWS" icon="aws" href="./aws">
    Sync AWS resources
  </Card>

  <Card title="Google Cloud" icon="google" href="./google-cloud">
    Sync GCP resources
  </Card>

  <Card title="Custom Provider" icon="code" href="./custom">
    Build your own
  </Card>
</CardGroup>


# Terraform Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/terraform

Sync Terraform Cloud/Enterprise workspaces into Ctrlplane

The Terraform provider syncs workspaces from Terraform Cloud or Terraform
Enterprise into Ctrlplane's inventory.

## Prerequisites

* `ctrlc` CLI installed
* Terraform Cloud/Enterprise API token
* Ctrlplane API key

## Authentication

Set your Terraform Cloud token:

```bash theme={null}
# Environment variable
export TFE_TOKEN="your-terraform-cloud-token"

# Optional: Custom Terraform Enterprise URL
export TFE_ADDRESS="https://tfe.example.com"
```

## Basic Usage

```bash theme={null}
# Sync all workspaces in an organization
ctrlc sync terraform \
  --organization my-org \
  --workspace <ctrlplane-workspace-id>

# Continuous sync
ctrlc sync terraform \
  --organization my-org \
  --workspace <ctrlplane-workspace-id> \
  --interval 5m
```

## Options

| Flag             | Description                      | Required |
| ---------------- | -------------------------------- | -------- |
| `--organization` | Terraform organization name      | Yes      |
| `--workspace`    | Ctrlplane workspace ID           | Yes      |
| `--interval`     | Sync interval (e.g., `5m`, `1h`) | No       |

## Resource Metadata

Each Terraform workspace is synced with metadata:

```yaml theme={null}
identifier: ws-abc123def456
name: production-infrastructure
kind: Terraform/Workspace
metadata:
  organization: my-org
  environment: production  # from workspace tags
  team: platform           # from workspace tags
config:
  workspace_id: ws-abc123def456
  vcs_repo: github.com/my-org/infrastructure
  working_directory: environments/production
```

## Running Continuously

### Kubernetes Deployment

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-terraform-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-terraform-sync
  template:
    metadata:
      labels:
        app: ctrlplane-terraform-sync
    spec:
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - terraform
            - --organization
            - my-org
            - --interval
            - "5m"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: your-workspace-id
            - name: TFE_TOKEN
              valueFrom:
                secretKeyRef:
                  name: terraform-credentials
                  key: token
```

## Environment Targeting

Target Terraform workspaces in environments:

```yaml theme={null}
# Production workspaces
type: Environment
name: Production Infrastructure
resourceSelector: |
  resource.kind == "Terraform/Workspace" &&
  resource.metadata["environment"] == "production"
```

```yaml theme={null}
# Platform team workspaces
type: Environment
name: Platform Infrastructure
resourceSelector: |
  resource.kind == "Terraform/Workspace" &&
  resource.metadata["team"] == "platform"
```

## Best Practices

### Tag Your Workspaces

Add tags to Terraform workspaces for better targeting:

```hcl theme={null}
# In Terraform Cloud UI or via API
# Add tags like: environment:production, team:platform
```

### Sync Frequently

Keep workspace state current:

```bash theme={null}
ctrlc sync terraform \
  --organization my-org \
  --interval 5m
```

## Next Steps

<CardGroup>
  <Card title="Terraform Cloud Job Agent" icon="cloud" href="../job-agents/terraform-cloud">
    Trigger Terraform runs from Ctrlplane
  </Card>

  <Card title="Selectors" icon="filter" href="../../concepts/selectors">
    Learn selector syntax
  </Card>
</CardGroup>


# vCluster Provider
Source: https://docs.ctrlplane.dev/integrations/resource-providers/vcluster

Sync vCluster virtual clusters into Ctrlplane

The vCluster provider syncs virtual Kubernetes clusters created with
[vCluster](https://www.vcluster.com/) into Ctrlplane's inventory.

## Prerequisites

* `ctrlc` CLI installed
* Access to the host Kubernetes cluster running vClusters
* Ctrlplane API key

## Basic Usage

```bash theme={null}
# Sync all vClusters from the host cluster
ctrlc sync vcluster \
  --cluster-identifier host-cluster-id

# Continuous sync
ctrlc sync vcluster \
  --cluster-identifier host-cluster-id \
  --interval 5m
```

## Options

| Flag                   | Description                                                      | Required |
| ---------------------- | ---------------------------------------------------------------- | -------- |
| `--cluster-identifier` | Identifier of the host cluster (or `CLUSTER_IDENTIFIER` env var) | Yes      |
| `--provider`           | Resource provider name                                           | No       |
| `--interval`           | Sync interval (e.g., `5m`, `1h`)                                 | No       |

## Resource Metadata

Each vCluster is synced with metadata:

```yaml theme={null}
identifier: host-cluster/vcluster/dev-team-cluster
name: dev-team-cluster
kind: vCluster/Cluster
metadata:
  host_cluster: host-cluster-id
  namespace: vcluster-dev-team
  release_name: dev-team-cluster
config:
  host_cluster: host-cluster-id
  namespace: vcluster-dev-team
```

## Running in Kubernetes

Deploy on the host cluster:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ctrlplane-vcluster-sync
  namespace: ctrlplane
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ctrlplane-vcluster-sync
  template:
    metadata:
      labels:
        app: ctrlplane-vcluster-sync
    spec:
      serviceAccountName: ctrlplane-sync
      containers:
        - name: sync
          image: ghcr.io/ctrlplanedev/cli:latest
          command:
            - ctrlc
            - sync
            - vcluster
            - --cluster-identifier
            - "$(CLUSTER_IDENTIFIER)"
            - --interval
            - "5m"
          env:
            - name: CTRLPLANE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ctrlplane-credentials
                  key: api-key
            - name: CTRLPLANE_WORKSPACE
              value: your-workspace-id
            - name: CLUSTER_IDENTIFIER
              value: host-cluster-id
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ctrlplane-sync
  namespace: ctrlplane
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ctrlplane-vcluster-sync
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: ctrlplane-sync
    namespace: ctrlplane
```

## Use Case: Development Environments

vClusters are ideal for dynamic development environments:

```yaml theme={null}
# All vClusters as dev environments
type: Environment
name: Development vClusters
resourceSelector: |
  resource.kind == "vCluster/Cluster"
```

```yaml theme={null}
# Team-specific vClusters
type: Environment
name: Backend Team Dev
resourceSelector: |
  resource.kind == "vCluster/Cluster" &&
  resource.metadata["namespace"].startsWith("vcluster-backend-")
```

## Best Practices

### Use Namespace Conventions

Name vCluster namespaces consistently for easy targeting:

```bash theme={null}
# Pattern: vcluster-{team}-{purpose}
vcluster-backend-dev
vcluster-frontend-preview
vcluster-data-staging
```

### Combine with Host Cluster Sync

Sync both the host cluster and vClusters:

```bash theme={null}
# Host cluster resources
ctrlc sync kubernetes \
  --cluster-identifier host-cluster \
  --interval 5m &

# vClusters
ctrlc sync vcluster \
  --cluster-identifier host-cluster \
  --interval 5m &
```

## Next Steps

<CardGroup>
  <Card title="Kubernetes Provider" icon="dharmachakra" href="./kubernetes">
    Sync Kubernetes resources
  </Card>

  <Card title="Dynamic Environments" icon="layer-group" href="../../use-cases/dynamic-environments">
    Create environments for vClusters
  </Card>
</CardGroup>


# Datadog Provider
Source: https://docs.ctrlplane.dev/integrations/verification-providers/datadog

Query metrics from Datadog's Metrics API for verification

The **Datadog provider** allows you to query metrics from Datadog's Metrics API
for verification checks. It uses the Datadog v2 Scalar Query API to fetch
aggregated metric values.

## Configuration

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  site: us5.datadoghq.com
  aggregator: last
  intervalSeconds: 300
  queries:
    a: "avg:kubernetes.cpu.usage{cluster:{{.resource.name}}}"
  formula: "a"
```

## Properties

<ParamField type="string">
  Must be `"datadog"`.
</ParamField>

<ParamField type="string">
  Datadog API key. Supports Go templates (e.g., `{{.variables.dd_api_key}}`).
</ParamField>

<ParamField type="string">
  Datadog Application key. Supports Go templates. Use the actual key value, NOT
  the Key ID.
</ParamField>

<ParamField type="map[string]string">
  Named queries map. Keys become accessible as `result.queries.<name>` in success conditions.
  Values support Go templates for dynamic filtering.
</ParamField>

<ParamField type="string">
  Formula to combine query results. Reference queries by their key names (e.g.,
  `"a / b * 100"`).
</ParamField>

<ParamField type="string">
  Aggregation method. Options: `last`, `avg`, `min`, `max`, `sum`, `mean`,
  `percentile`, `l2norm`, `area`.
</ParamField>

<ParamField type="integer">
  Time window in seconds for the query. Determines how far back to look for
  metric data.
</ParamField>

<ParamField type="string">
  Datadog site URL. Options: `datadoghq.com`, `datadoghq.eu`,
  `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`.
</ParamField>

## Aggregator Options

The `aggregator` property specifies how to aggregate metric values:

* `last` (default) - Most recent value
* `avg` - Average value
* `min` - Minimum value
* `max` - Maximum value
* `sum` - Sum of values
* `mean` - Mean value
* `percentile` - Percentile value
* `l2norm` - L2 norm
* `area` - Area under the curve

## Supported Sites

* `datadoghq.com` (US1 - default)
* `datadoghq.eu` (EU)
* `us3.datadoghq.com` (US3)
* `us5.datadoghq.com` (US5)
* `ap1.datadoghq.com` (AP1)

## Response Data Available in CEL

The Datadog provider makes the following data available in your CEL success
conditions:

| Field                   | Type              | Description                               |
| ----------------------- | ----------------- | ----------------------------------------- |
| `result.ok`             | boolean           | `true` if API call succeeded (2xx status) |
| `result.statusCode`     | integer           | HTTP status code from Datadog API         |
| `result.queries.<name>` | float64 (or null) | Value for each named query                |
| `result.json`           | object            | Full Datadog API response                 |
| `result.body`           | string            | Raw response body                         |
| `result.duration`       | integer           | Request duration in milliseconds          |

### Accessing Query Values

Since queries are named, you access them by name in your success condition:

```yaml theme={null}
queries:
  cpu: "avg:kubernetes.cpu.usage{cluster:prod}"
  memory: "avg:kubernetes.memory.usage{cluster:prod}"
successCondition: result.queries.cpu < 80 && result.queries.memory < 90
```

## Example Configurations

### Single Query

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    error_rate: "sum:requests.error.rate{service:api-service}"
successCondition: result.queries.error_rate < 0.01
```

### Multiple Queries with Formula

Use multiple queries with a formula to calculate ratios or complex metrics:

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:requests.errors{service:api}"
    total: "sum:requests.total{service:api}"
  formula: "errors / total * 100"
successCondition: result.queries.errors < 1
```

### With Environment and Resource Tags

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  site: us5.datadoghq.com
  intervalSeconds: 600
  queries:
    cpu: "avg:kubernetes.cpu.usage{cluster:{{.resource.name}},env:{{.environment.name}}}"
successCondition:
  result.ok && result.queries.cpu != null && result.queries.cpu < 80
```

### Latency Percentile

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  aggregator: percentile
  queries:
    p99: "trace.http.request.duration.by.service.99p{service:api-service}"
successCondition: result.queries.p99 < 200
```

## Example Success Conditions

```yaml theme={null}
# Simple threshold check
successCondition: result.queries.value < 0.01

# Check if value exists and is under threshold
successCondition: result.ok && result.queries.cpu != null && result.queries.cpu < 80

# Multiple query conditions
successCondition: result.queries.errors < 10 && result.queries.latency < 500

# Check API success first
successCondition: result.ok && result.statusCode == 200 && result.queries.rate < 0.1
```

## Template Variables

The Datadog provider supports Go templates in the `apiKey`, `appKey`, `site`,
`formula`, and `queries` fields:

```yaml theme={null}
# Resource information
{{.resource.name}}
{{.resource.identifier}}
{{.resource.kind}}

# Environment information
{{.environment.name}}
{{.environment.id}}

# Deployment information
{{.deployment.name}}
{{.deployment.slug}}

# Version information
{{.version.tag}}
{{.version.id}}

# Custom variables (from deployment variables)
{{.variables.dd_api_key}}
{{.variables.dd_app_key}}
```

## Storing Secrets in Variables

For sensitive values like API keys, use deployment variables:

1. **Create deployment variables** for your Datadog credentials
2. **Reference them in the provider configuration** using template syntax

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:errors{service:{{.resource.name}}}"
```

## Finding Your Datadog Keys

### API Key

1. Go to Datadog console → **Organization Settings** → **API Keys**
2. Create or copy an existing API key (40-character hex string)

### Application Key

1. Go to Datadog console → **Organization Settings** → **Application Keys**
2. Create or copy an existing Application key
3. **Important**: Use the actual key value, NOT the "Key ID" (which is a UUID)

## Best Practices

* **Use deployment variables** for API keys and application keys - never
  hardcode credentials
* **Name queries descriptively** to make success conditions readable
* **Use appropriate intervals** - shorter intervals provide faster feedback but
  may have less data
* **Tag your metrics** with service, environment, and version information for
  better filtering
* **Test queries manually** in Datadog before using them in verification
* **Handle missing data** by checking `result.ok` and
  `result.queries.<name> != null`
* **Use formulas** for calculated metrics like error rates or ratios


# HTTP Provider
Source: https://docs.ctrlplane.dev/integrations/verification-providers/http

Query any HTTP endpoint for verification metrics

The **HTTP provider** allows you to query any HTTP endpoint that returns JSON
data for verification metrics.

## Configuration

```yaml theme={null}
provider:
  type: http
  url: "http://{{.resource.name}}/health"
  method: GET
  headers:
    Authorization: "Bearer {{.variables.health_token}}"
  timeout: 30s
```

## Properties

<ParamField type="string">
  Must be `"http"`.
</ParamField>

<ParamField type="string">
  HTTP endpoint URL. Supports Go templates for dynamic URLs
  (e.g., `http://{{.resource.name}}/health`).
</ParamField>

<ParamField type="string">
  HTTP method. Options: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`,
  `OPTIONS`.
</ParamField>

<ParamField type="object">
  HTTP headers to include in the request. Values support Go templates
  (e.g., `Authorization: "Bearer {{.variables.token}}"`).
</ParamField>

<ParamField type="string">
  Request body for POST/PUT/PATCH requests. Supports Go templates for dynamic
  content.
</ParamField>

<ParamField type="string">
  Request timeout duration. Use format like `30s`, `1m`, `500ms`.
</ParamField>

### Supported HTTP Methods

* `GET` (default)
* `POST`
* `PUT`
* `PATCH`
* `DELETE`
* `HEAD`
* `OPTIONS`

## Response Data Available in CEL

The HTTP provider makes the following data available in your CEL success
conditions:

* `result.ok` - `true` if status code is 2xx
* `result.statusCode` - HTTP status code (e.g., `200`, `404`, `500`)
* `result.body` - Response body as string
* `result.json` - Parsed JSON response (if response is valid JSON)
* `result.headers` - Response headers as an object
* `result.duration` - Request duration in milliseconds

## Example Success Conditions

```yaml theme={null}
# Status code check
successCondition: result.ok

# JSON field check
successCondition: result.json.healthy == true

# Numeric threshold
successCondition: result.json.error_rate < 0.01

# Combined conditions
successCondition: result.ok && result.json.ready == true

# Check specific status code
successCondition: result.statusCode == 200

# Check response time
successCondition: result.duration < 500
```

## Examples

### Basic Health Check

```yaml theme={null}
provider:
  type: http
  url: "http://{{.resource.name}}/health"
  method: GET
successCondition: result.ok && result.json.status == "healthy"
```

### POST Request with Body

```yaml theme={null}
provider:
  type: http
  url: "http://smoke-test-runner/run"
  method: POST
  headers:
    Content-Type: "application/json"
  body: |
    {
      "service": "{{.resource.name}}",
      "version": "{{.version.tag}}",
      "environment": "{{.environment.name}}"
    }
successCondition: result.json.status == "passed"
```

### Authenticated Request

```yaml theme={null}
provider:
  type: http
  url: "http://api.example.com/health"
  method: GET
  headers:
    Authorization: "Bearer {{.variables.health_token}}"
    X-API-Key: "{{.variables.api_key}}"
successCondition: result.ok
```

### Custom Timeout

```yaml theme={null}
provider:
  type: http
  url: "http://slow-service/health"
  method: GET
  timeout: 60s
successCondition: result.ok
```

## Template Variables

The HTTP provider supports Go templates in the `url`, `headers`, and `body`
fields:

```yaml theme={null}
# Resource information
{{.resource.name}}
{{.resource.identifier}}
{{.resource.kind}}

# Environment information
{{.environment.name}}
{{.environment.id}}

# Deployment information
{{.deployment.name}}
{{.deployment.slug}}

# Version information
{{.version.tag}}
{{.version.id}}

# Custom variables (from deployment variables)
{{.variables.my_variable}}
{{.variables.health_token}}
```

## Best Practices

* **Use HTTPS** for production endpoints to ensure secure communication
* **Set appropriate timeouts** based on your service's expected response time
* **Handle authentication** using deployment variables for sensitive tokens
* **Validate JSON responses** by checking `result.json` exists before accessing
  fields
* **Use meaningful success conditions** that check both status codes and
  response content


# Sleep Provider
Source: https://docs.ctrlplane.dev/integrations/verification-providers/sleep

Wait for a specified duration before considering verification passed

The **Sleep provider** allows you to add a simple time-based delay to your
verification checks. This is useful for waiting for services to stabilize or for
scheduled tasks to complete.

## Configuration

```yaml theme={null}
provider:
  type: sleep
  durationSeconds: 30
```

## Properties

| Property          | Type    | Required | Description                          |
| ----------------- | ------- | -------- | ------------------------------------ |
| `type`            | string  | Yes      | Must be `"sleep"`                    |
| `durationSeconds` | integer | Yes      | Duration to wait in seconds (1-3600) |

## Response Data Available in CEL

The Sleep provider makes the following data available in your CEL success
conditions:

* `result.ok` - Always `true` after the sleep duration completes
* `result.value` - The duration that was waited (in seconds)
* `result.duration` - The actual duration waited in milliseconds

## Example Success Conditions

```yaml theme={null}
# Always passes after sleep completes
successCondition: result.ok

# Check that the expected duration was waited
successCondition: result.value == 30
```

## Examples

### Basic Sleep

```yaml theme={null}
provider:
  type: sleep
  durationSeconds: 30
successCondition: result.ok
```

### Wait for Service Stabilization

```yaml theme={null}
# Wait 2 minutes for service to stabilize after deployment
provider:
  type: sleep
  durationSeconds: 120
successCondition: result.ok
```

### Combined with Other Metrics

```yaml theme={null}
metrics:
  # First, wait for service to start
  - name: stabilization-wait
    intervalSeconds: 1
    count: 1
    provider:
      type: sleep
      durationSeconds: 60
    successCondition: result.ok

  # Then check health endpoint
  - name: health-check
    intervalSeconds: 10
    count: 5
    provider:
      type: http
      url: "http://{{.resource.name}}/health"
    successCondition: result.ok
```

## Use Cases

* **Service Stabilization**: Wait for services to fully start up before running
  health checks
* **Scheduled Tasks**: Wait for scheduled tasks or cron jobs to complete
* **Warm-up Period**: Allow caches or connections to warm up before verification
* **Sequential Verification**: Create a delay between different verification
  steps

## Best Practices

* **Use appropriate durations**: Keep sleep durations reasonable (typically
  30-300 seconds)
* **Combine with other metrics**: Use sleep as a first step, then follow with
  actual health checks
* **Consider deployment time**: Account for how long your service takes to start
* **Don't overuse**: Prefer actual health checks over sleep when possible


# Terraform Cloud Run Provider
Source: https://docs.ctrlplane.dev/integrations/verification-providers/terraform-cloud-run

Verify Terraform Cloud run status for infrastructure deployments

The **Terraform Cloud Run provider** allows you to verify that a Terraform Cloud
run has completed successfully. This is useful for infrastructure deployments
managed through Terraform Cloud.

## Configuration

```yaml theme={null}
provider:
  type: terraformCloudRun
  organization: "my-org"
  address: "https://app.terraform.io"
  token: "{{.variables.terraform_cloud_token}}"
  runId: "run-1234567890"
```

## Properties

| Property       | Type   | Required | Description                                                |
| -------------- | ------ | -------- | ---------------------------------------------------------- |
| `type`         | string | Yes      | Must be `"terraformCloudRun"`                              |
| `organization` | string | Yes      | Terraform Cloud organization name                          |
| `address`      | string | Yes      | Terraform Cloud address (e.g., `https://app.terraform.io`) |
| `token`        | string | Yes      | Terraform Cloud token (supports Go templates)              |
| `runId`        | string | Yes      | Terraform Cloud run ID                                     |

## Response Data Available in CEL

The Terraform Cloud Run provider makes the following data available in your CEL
success conditions:

* `result.ok` - `true` if the run completed successfully
* `result.statusCode` - HTTP status code from Terraform Cloud API
* `result.value` - Run status (e.g., `"applied"`, `"errored"`, `"planned"`)
* `result.json` - Full Terraform Cloud API response
* `result.duration` - Request duration in milliseconds

## Example Success Conditions

```yaml theme={null}
# Check that run was applied successfully
successCondition: result.ok && result.value == "applied"

# Check that run completed (applied or planned)
successCondition: result.ok && (result.value == "applied" || result.value == "planned")

# Ensure run didn't error
successCondition: result.ok && result.value != "errored"
```

## Examples

### Basic Run Verification

```yaml theme={null}
provider:
  type: terraformCloudRun
  organization: "my-org"
  address: "https://app.terraform.io"
  token: "{{.variables.terraform_cloud_token}}"
  runId: "{{.variables.terraform_run_id}}"
successCondition: result.ok && result.value == "applied"
```

### Using Template Variables

```yaml theme={null}
provider:
  type: terraformCloudRun
  organization: "{{.variables.tf_org}}"
  address: "https://app.terraform.io"
  token: "{{.variables.terraform_cloud_token}}"
  runId: "{{.variables.terraform_run_id}}"
successCondition: result.ok && result.value == "applied"
```

## Run Status Values

The provider returns the following status values:

* `"applied"` - Run was successfully applied
* `"planned"` - Run was planned but not yet applied
* `"errored"` - Run encountered an error
* `"canceled"` - Run was canceled
* `"pending"` - Run is pending
* `"planning"` - Run is currently planning
* `"applying"` - Run is currently applying

## Template Variables

The Terraform Cloud Run provider supports Go templates in the `organization`,
`address`, `token`, and `runId` fields:

```yaml theme={null}
# Resource information
{{.resource.name}}
{{.resource.identifier}}

# Environment information
{{.environment.name}}
{{.environment.id}}

# Deployment information
{{.deployment.name}}
{{.deployment.slug}}

# Version information
{{.version.tag}}
{{.version.id}}

# Custom variables (from deployment variables)
{{.variables.terraform_cloud_token}}
{{.variables.terraform_run_id}}
{{.variables.tf_org}}
```

## Storing Secrets in Variables

For sensitive values like tokens, use deployment variables:

```yaml theme={null}
provider:
  type: terraformCloudRun
  organization: "my-org"
  address: "https://app.terraform.io"
  token: "{{.variables.terraform_cloud_token}}"
  runId: "{{.variables.terraform_run_id}}"
```

## Best Practices

* **Use deployment variables** for tokens and run IDs - never hardcode sensitive
  values
* **Verify run completion** by checking that the status is `"applied"` or
  `"planned"`
* **Handle run failures** by checking for `"errored"` status
* **Use appropriate intervals** to poll for run status (typically 30-60 seconds)
* **Set failure limits** to avoid waiting indefinitely for stuck runs


# Inventory Overview
Source: https://docs.ctrlplane.dev/inventory/overview

Unified visibility into your infrastructure resources

Ctrlplane's inventory system provides a centralized view of all your
infrastructure resources with custom relationships and dynamic grouping.

## What is the Inventory?

The inventory is a real-time database of your deployment targets — Kubernetes
clusters, cloud functions, VMs, databases, or any custom infrastructure. It
enables:

* **Unified visibility** across providers (AWS, GCP, Kubernetes, custom)
* **Dynamic environments** that automatically include matching resources
* **Custom relationships** to model dependencies and ownership
* **Rich metadata** for filtering, grouping, and automation

```mermaid theme={null}
flowchart TB
    subgraph Providers["Resource Providers"]
        K8s["Kubernetes"]
        AWS["AWS"]
        Custom["Custom Scripts"]
    end

    subgraph Inventory["Ctrlplane Inventory"]
        Resources["Resources"]
        Metadata["Metadata & Config"]
        Relations["Relationships"]
    end

    subgraph Targeting["Environment Targeting"]
        Staging["Staging"]
        Prod["Production"]
        Regional["Regional"]
    end

    Providers -->|"sync"| Inventory
    Inventory -->|"selectors"| Targeting
```

## Core Concepts

<CardGroup>
  <Card title="Resources" icon="server" href="../concepts/resources">
    Deployment targets with metadata and configuration
  </Card>

  <Card title="Environments" icon="layer-group" href="../concepts/environments">
    Dynamic groups of resources using selectors
  </Card>

  <Card title="Relationships" icon="diagram-project" href="./relationships">
    Model dependencies between resources
  </Card>

  <Card title="Selectors" icon="filter" href="../concepts/selectors">
    Query language for matching resources
  </Card>

  <Card title="Resource Providers" icon="rotate" href="../integrations/resource-providers/overview">
    Sync infrastructure into the inventory
  </Card>
</CardGroup>

## How It Works

### 1. Resources are Synced

Resources are synced into Ctrlplane via resource providers or API:

```yaml theme={null}
type: Resource
name: production-cluster-us-east
kind: KubernetesCluster
identifier: k8s-prod-useast1
metadata:
  environment: production
  region: us-east-1
  team: platform
config:
  server: https://k8s.example.com
  namespace: default
```

### 2. Environments Match Resources

Environments use selectors to dynamically include resources:

```yaml theme={null}
type: Environment
name: Production US-East
resourceSelector: |
  resource.metadata["environment"] == "production" &&
  resource.metadata["region"] == "us-east-1"
```

When new resources are added that match the selector, they're automatically
included in the environment.

### 3. Deployments Target Environments

Deployments create release targets for each resource in each environment:

```
Deployment × Environment × Resource = Release Target
```

## Key Benefits

| Benefit                    | Description                                        |
| -------------------------- | -------------------------------------------------- |
| **Single source of truth** | All resources visible in one place                 |
| **Dynamic targeting**      | Environments auto-update as infrastructure changes |
| **Rich metadata**          | Filter and group by any attribute                  |
| **Cross-provider**         | Kubernetes, cloud, and custom in one view          |

## Next Steps

* [Resources](../concepts/resources) — Understand resource structure and
  lifecycle
* [Relationships](./relationships) — Model dependencies between resources
* [Environments](../concepts/environments) — Create dynamic resource groups
* [Selectors](../concepts/selectors) — Write powerful resource queries
* [Resource Providers](../integrations/resource-providers/overview) — Sync your
  infrastructure


# Relationships
Source: https://docs.ctrlplane.dev/inventory/relationships

Model dependencies and connections between resources

Relationships in Ctrlplane allow you to model how your infrastructure components
connect to each other—VPCs containing clusters, clusters running services,
databases backing applications.

## Why Relationships?

Understanding infrastructure dependencies helps you:

* **Visualize architecture** — See how resources connect across your stack
* **Impact analysis** — Understand what's affected when a resource changes
* **Deployment ordering** — Deploy dependencies before dependents
* **Troubleshooting** — Trace issues through connected resources

## Relationship Rules

Relationship rules automatically create connections between entities based on
matching criteria. When resources are synced, Ctrlplane evaluates rules and
creates relationships dynamically.

### Structure

```yaml theme={null}
type: RelationshipRule
id: vpc-to-cluster
name: VPC to Kubernetes Cluster
description: Links VPCs to K8s clusters in the same region
fromType: resource
toType: resource
relationshipType: contains
fromSelector:
  type: kind
  operator: equals
  value: vpc
toSelector:
  type: kind
  operator: equals
  value: kubernetes-cluster
propertyMatchers:
  - fromProperty: ["metadata", "region"]
    toProperty: ["metadata", "region"]
    operator: equals
```

### Components

| Field              | Description                                                  |
| ------------------ | ------------------------------------------------------------ |
| `fromType`         | Source entity type (`resource`, `deployment`, `environment`) |
| `toType`           | Target entity type (`resource`, `deployment`, `environment`) |
| `relationshipType` | Type of connection (see below)                               |
| `fromSelector`     | Selector matching source entities                            |
| `toSelector`       | Selector matching target entities                            |
| `propertyMatchers` | Property conditions for matching                             |

### Relationship Types

| Type          | Description                    | Example                       |
| ------------- | ------------------------------ | ----------------------------- |
| `contains`    | Parent contains child          | VPC contains clusters         |
| `runs-on`     | Service runs on infrastructure | App runs on cluster           |
| `depends-on`  | Requires another resource      | API depends on database       |
| `deployed-by` | Managed by a deployment        | Resource deployed by pipeline |
| `deploys-to`  | Deployment targets environment | Service deploys to production |

## Examples

### VPC to Kubernetes Clusters

Connect VPCs to the Kubernetes clusters running within them:

```yaml theme={null}
type: RelationshipRule
id: vpc-to-cluster
name: VPC Contains Clusters
fromType: resource
toType: resource
relationshipType: contains
fromSelector:
  type: kind
  operator: equals
  value: vpc
toSelector:
  type: kind
  operator: equals
  value: kubernetes-cluster
propertyMatchers:
  - fromProperty: ["metadata", "region"]
    toProperty: ["metadata", "region"]
    operator: equals
  - fromProperty: ["metadata", "account"]
    toProperty: ["metadata", "account"]
    operator: equals
```

This rule:

1. Finds all resources with `kind: vpc`
2. Finds all resources with `kind: kubernetes-cluster`
3. Creates a `contains` relationship when both have the same region AND account

### Database Dependencies

Model which services depend on which databases:

```yaml theme={null}
type: RelationshipRule
id: service-to-database
name: Service Database Dependencies
fromType: resource
toType: resource
relationshipType: depends-on
fromSelector:
  type: kind
  operator: equals
  value: kubernetes-deployment
toSelector:
  type: kind
  operator: equals
  value: rds-instance
propertyMatchers:
  - fromProperty: ["metadata", "database-cluster"]
    toProperty: ["metadata", "cluster-id"]
    operator: equals
```

### Deployment to Environment

Link deployments to the environments they target:

```yaml theme={null}
type: RelationshipRule
id: deployment-to-env
name: Deployment Targets Environment
fromType: deployment
toType: environment
relationshipType: deploys-to
fromSelector:
  type: metadata
  key: team
  operator: equals
  value: platform
toSelector:
  type: name
  operator: contains
  value: production
```

### Cross-Account Resources

Connect resources across AWS accounts:

```yaml theme={null}
type: RelationshipRule
id: transit-gateway-connections
name: Transit Gateway VPC Attachments
fromType: resource
toType: resource
relationshipType: contains
fromSelector:
  type: kind
  operator: equals
  value: transit-gateway
toSelector:
  type: kind
  operator: equals
  value: vpc
propertyMatchers:
  - fromProperty: ["config", "attachments"]
    toProperty: ["metadata", "vpc-id"]
    operator: contains
```

## Property Matchers

Property matchers define conditions for when relationships should be created
between matching entities.

### Operators

| Operator     | Description          | Example                  |
| ------------ | -------------------- | ------------------------ |
| `equals`     | Exact match          | Region equals region     |
| `contains`   | Array contains value | Tags contain team        |
| `startsWith` | String prefix match  | Name starts with "prod-" |
| `regex`      | Regular expression   | Name matches pattern     |

### Nested Properties

Access nested properties using array paths:

```yaml theme={null}
propertyMatchers:
  # Match metadata.region
  - fromProperty: ["metadata", "region"]
    toProperty: ["metadata", "region"]
    operator: equals
    
  # Match config.cluster.name
  - fromProperty: ["config", "cluster", "name"]
    toProperty: ["metadata", "cluster-name"]
    operator: equals
```

### Multiple Matchers

All property matchers must match for a relationship to be created:

```yaml theme={null}
propertyMatchers:
  # Must match region AND account AND environment
  - fromProperty: ["metadata", "region"]
    toProperty: ["metadata", "region"]
    operator: equals
  - fromProperty: ["metadata", "account"]
    toProperty: ["metadata", "account"]
    operator: equals
  - fromProperty: ["metadata", "environment"]
    toProperty: ["metadata", "environment"]
    operator: equals
```

## Selectors

Selectors filter which entities are considered for relationships.

### By Kind

```yaml theme={null}
fromSelector:
  type: kind
  operator: equals
  value: kubernetes-cluster
```

### By Metadata

```yaml theme={null}
fromSelector:
  type: metadata
  key: environment
  operator: equals
  value: production
```

### By Name

```yaml theme={null}
fromSelector:
  type: name
  operator: contains
  value: prod
```

## Managing Relationship Rules

### Create a Rule

```bash theme={null}
curl -X POST "https://your-ctrlplane-instance.com/api/v1/workspaces/{workspaceId}/relationship-rules" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "vpc-to-cluster",
    "name": "VPC to Kubernetes Cluster",
    "fromType": "resource",
    "toType": "resource",
    "relationshipType": "contains",
    "fromSelector": {
      "type": "kind",
      "operator": "equals",
      "value": "vpc"
    },
    "toSelector": {
      "type": "kind",
      "operator": "equals",
      "value": "kubernetes-cluster"
    },
    "propertyMatchers": [{
      "fromProperty": ["metadata", "region"],
      "toProperty": ["metadata", "region"],
      "operator": "equals"
    }]
  }'
```

### Update a Rule

```bash theme={null}
curl -X PUT "https://your-ctrlplane-instance.com/api/v1/relationship-rules/{ruleId}" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Updated Rule Name",
    "description": "New description"
  }'
```

### Delete a Rule

```bash theme={null}
curl -X DELETE "https://your-ctrlplane-instance.com/api/v1/relationship-rules/{ruleId}" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY"
```

Deleting a relationship rule removes the rule definition but does not delete the
underlying resources.

## Use Cases

### Infrastructure Topology

Model your complete infrastructure hierarchy:

```
Account
└── Region
    └── VPC
        └── Subnet
            └── Kubernetes Cluster
                └── Namespace
                    └── Deployment
```

### Service Dependencies

Track service-to-service and service-to-infrastructure dependencies:

```
API Gateway
├── depends-on → Auth Service
├── depends-on → User Database (RDS)
└── runs-on → EKS Cluster
```

### Multi-Cloud Relationships

Connect resources across cloud providers:

```
On-Prem Database
└── replicated-to → AWS RDS Read Replica
    └── accessed-by → EKS Application
```

## Best Practices

### Use Meaningful Relationship Types

Choose relationship types that reflect the actual connection:

```yaml theme={null}
# Good: specific relationship type
relationshipType: depends-on  # Service needs database
relationshipType: runs-on     # App runs on cluster
relationshipType: contains    # VPC contains subnets

# Avoid: generic types
relationshipType: related-to  # Too vague
```

### Keep Property Matchers Simple

Start with minimal matchers and add more only if needed:

```yaml theme={null}
# Good: simple and clear
propertyMatchers:
  - fromProperty: ["metadata", "region"]
    toProperty: ["metadata", "region"]
    operator: equals

# Avoid: overly complex
propertyMatchers:
  - fromProperty: ["metadata", "tags", "aws:region"]
    toProperty: ["config", "spec", "provider", "region"]
    operator: regex
    value: "^us-(east|west)-[0-9]$"
```

### Document Your Rules

Add descriptions to explain the purpose:

```yaml theme={null}
type: RelationshipRule
name: VPC to Cluster
description: |
  Links VPCs to Kubernetes clusters based on matching region and account.
  Used for infrastructure topology visualization and impact analysis.
```

## Next Steps

<CardGroup>
  <Card title="CEL Reference" icon="code" href="../reference/cel">
    Full expression language reference
  </Card>

  <Card title="Selectors" icon="filter" href="../concepts/selectors">
    Learn selector syntax
  </Card>

  <Card title="Dynamic Environments" icon="layer-group" href="../use-cases/dynamic-environments">
    Create environments from relationships
  </Card>

  <Card title="Resource Providers" icon="rotate" href="../integrations/resource-providers/overview">
    Sync infrastructure automatically
  </Card>
</CardGroup>


# 5-Minute Overview
Source: https://docs.ctrlplane.dev/overview

Understand Ctrlplane's mental model in 5 minutes

## The Core Flow

Every deployment in Ctrlplane follows this flow:

```mermaid theme={null}
flowchart LR
    CI --"1\. create version"--> Ctrlplane
    Ctrlplane --"2\. evaluate policies"--> Policies
    Policies --"3\. dispatch job"--> JobAgent
    JobAgent --"4\. deploy"--> Resource
    Resource --"5\. verify"--> Ctrlplane
    Ctrlplane --"6\. promote or rollback"--> Next
```

1. **CI creates a version** — After building, your CI tells Ctrlplane about the new version
2. **Policies are evaluated** — Ctrlplane checks if approvals, dependencies, and gates are satisfied
3. **Job is dispatched** — Ctrlplane tells your job agent (ArgoCD, GitHub Actions, etc.) to deploy
4. **Deployment executes** — The job agent performs the actual deployment
5. **Verification runs** — Ctrlplane checks metrics (Datadog, Prometheus, HTTP) to confirm health and can also validate resource presence/status (e.g., newly discovered clusters are visible and ready)
6. **Promote or rollback** — If verification passes, continue; if it fails, roll back

## The Key Entities

You only need to understand 5 things:

| Entity          | Question It Answers            | Example                                      |
| --------------- | ------------------------------ | -------------------------------------------- |
| **Resource**    | What infrastructure exists?    | `prod-us-east-1` K8s cluster                 |
| **Deployment**  | **What** to deploy and **how** | "API Gateway" deployed via ArgoCD            |
| **Environment** | **Where** to deploy            | "Production" = all clusters with `env: prod` |
| **Version**     | Which build to deploy?         | `v1.2.3` or `sha-abc123`                     |
| **Policy**      | **When** to deploy             | "After staging succeeds and SRE approves"    |

In short: **Deployments** define *what and how*, **Environments** define *where*, and **Policies** define *when*.

### How They Connect

```
Deployment × Environment × Resource = Release Target
```

When you deploy "API Gateway" to "Production" (which has 3 clusters), Ctrlplane creates 3 **release targets**:

* API Gateway → Production → us-east-1
* API Gateway → Production → us-west-2
* API Gateway → Production → eu-west-1

Each release target can have its own policies, verification, and rollout timing.

**Advanced example:** Version `v1.2.3` of the "API Gateway" deployment targets the "Production" environment,
which expands to multiple clusters including `us-west-2`. Ctrlplane creates one release target per cluster,
so each target can be governed independently. A deployment window policy can be scoped to the `us-west-2`
release target so it only deploys on weekdays before 9 a.m., while other production targets follow their
own timing. When a new version arrives, Ctrlplane waits for that window before dispatching the job for
`us-west-2`, even if other targets can deploy sooner.

## Dynamic Environments

Environments use **selectors** to automatically include resources:

```yaml theme={null}
type: Environment
name: Production
resourceSelector: resource.metadata["env"] == "production"
```

When you add a new cluster with `env: production` metadata, it automatically becomes part of the Production environment. No config changes needed.

```mermaid theme={null}
flowchart TB
    subgraph Resources["All Resources"]
        R1["us-east-1 cluster<br/>env: production"]
        R2["us-west-2 cluster<br/>env: production"]
        R3["dev cluster<br/>env: development"]
    end

    subgraph Envs["Environments"]
        Prod["Production"]
        Dev["Development"]
    end

    R1 -.->|"matches"| Prod
    R2 -.->|"matches"| Prod
    R3 -.->|"matches"| Dev
```

## Policies Define When to Deploy

Policies are the rules that govern *when* a version is allowed to deploy:

| Policy                      | What It Does                      |
| --------------------------- | --------------------------------- |
| **Approval**                | Require sign-off before deploying |
| **Environment Progression** | Wait for staging before prod      |
| **Gradual Rollout**         | Deploy to targets one at a time   |
| **Verification**            | Check Datadog/Prometheus metrics  |
| **Deployment Window**       | Only deploy during certain hours  |

Policies use selectors to target specific releases:

```yaml theme={null}
type: Policy
name: production-approval-policy
description: Production Approval Policy
selectors:
  - environments: environment.metadata['requires-approval'] == 'true'
rules:
  - approval:
      required: 1
```

## Job Agents Execute Deployments

Ctrlplane doesn't deploy directly—it tells **job agents** what to do:

| Agent               | What It Does                      |
| ------------------- | --------------------------------- |
| **GitHub Actions**  | Triggers workflow dispatch        |
| **ArgoCD**          | Creates/syncs ArgoCD Applications |
| **Terraform Cloud** | Triggers Terraform runs           |
| **Kubernetes**      | Applies manifests directly        |

This means Ctrlplane works with your existing deployment tooling.

## A Complete Example

```mermaid theme={null}
flowchart TB
    subgraph Build["1\. Build"]
        CI["GitHub Actions"]
    end

    subgraph Orchestrate["2\. Orchestrate (Ctrlplane)"]
        Version["Version v1.2.3"]
        Staging["Staging Release"]
        Verify1["Verify"]
        Prod["Production Release"]
        Approve["Approval Gate"]
        Verify2["Verify"]
    end

    subgraph Execute["3\. Execute"]
        ArgoStaging["ArgoCD Staging"]
        ArgoProd["ArgoCD Prod"]
    end

    CI -->|"create version"| Version
    Version --> Staging
    Staging -->|"dispatch"| ArgoStaging
    ArgoStaging -->|"complete"| Verify1
    Verify1 -->|"pass"| Prod
    Prod --> Approve
    Approve -->|"approved"| ArgoProd
    ArgoProd -->|"complete"| Verify2
```

1. GitHub Actions builds `v1.2.3` and creates a version in Ctrlplane
2. Ctrlplane creates a release for staging
3. ArgoCD deploys to staging
4. Verification checks Datadog metrics
5. Staging passes → production release is unblocked
6. Production requires approval → team lead approves
7. ArgoCD deploys to production
8. Verification confirms production is healthy

## What's Next?

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="./quickstart">
    Build this pipeline yourself in 15 minutes
  </Card>

  <Card title="Use Cases" icon="lightbulb" href="./use-cases/multi-region">
    See specific deployment scenarios
  </Card>
</CardGroup>


# Approval
Source: https://docs.ctrlplane.dev/policies/approval

Learn how to use approval rules to require manual sign-off before deployments proceed.

**Approval rules** require manual approval from authorized users before a
deployment can proceed. This adds a human checkpoint to your deployment pipeline
for high-risk changes.

## Overview

```mermaid theme={null}
flowchart TD
    A[Release Created] --> B{Approval Required?}
    B -->|No| C[Deployment Proceeds]
    B -->|Yes| D[Awaiting Approval]
    D --> E{Approvals Met?}
    E -->|Yes| C
    E -->|No| D
```

## Why Use Approval Rules?

Approval rules help you:

* **Add human oversight** - Require sign-off for production deployments
* **Enforce compliance** - Meet regulatory requirements for change management
* **Coordinate releases** - Ensure stakeholders are aware before deployment
* **Reduce risk** - Catch issues that automated checks might miss

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_approval" {
      name        = "Production Approval Policy"
      description = "Require approval for production deployments"

      selector = "environment.metadata['requires-approval'] == 'true'"

      any_approval {
        min_approvals = 1
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Production Approval Policy",
        "description": "Require approval for production deployments",
        "selector": "environment.metadata['\''requires-approval'\''] == '\''true'\''",
        "rules": [
          {
            "anyApproval": {
              "minApprovals": 1
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="integer">
  Minimum number of approvals required before the deployment can proceed.
</ParamField>

## Grandfathering

Versions that were created **before** an approval rule was added to a policy are
automatically allowed through (grandfathered in). This prevents newly added
approval rules from blocking already-in-flight releases.

## Common Patterns

### Single Approval for Production

Basic approval gate for production deployments:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_gate" {
      name     = "Production Gate Policy"
      selector = "environment.metadata['requires-approval'] == 'true'"

      any_approval {
        min_approvals = 1
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Production Gate Policy",
      "selector": "environment.metadata['requires-approval'] == 'true'",
      "rules": [
        {
          "anyApproval": { "minApprovals": 1 }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### Multiple Approvals for Critical Services

Require multiple sign-offs for high-risk deployments:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "critical_service_approval" {
      name     = "Critical Service Approval Policy"
      selector = "environment.metadata['requires-approval'] == 'true'"

      any_approval {
        min_approvals = 2
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Critical Service Approval Policy",
      "selector": "environment.metadata['requires-approval'] == 'true'",
      "rules": [
        {
          "anyApproval": { "minApprovals": 2 }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### Approval with Gradual Rollout

Approve once, then roll out gradually:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "controlled_production_release" {
      name     = "Controlled Production Release"
      selector = "environment.metadata['requires-approval'] == 'true'"

      any_approval {
        min_approvals = 1
      }

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 300
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Controlled Production Release",
      "selector": "environment.metadata['requires-approval'] == 'true'",
      "rules": [
        {
          "anyApproval": { "minApprovals": 1 }
        },
        {
          "gradualRollout": {
            "rolloutType": "linear",
            "timeScaleInterval": 300
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Approval Workflow

### 1. Release Created

When a new release is created that matches an approval policy, it enters an
"awaiting approval" state.

### 2. Approval Requested

Users with appropriate permissions can view pending approvals in the Ctrlplane
UI or via API.

### 3. Approval Granted

Authorized users approve (or reject) the release. Each approval is recorded with
the user and timestamp.

### 4. Deployment Proceeds

Once the required number of approvals is met, the deployment continues through
any remaining policy rules.

## Best Practices

### Environment-Based Approvals

| Environment | Approvals | Notes                        |
| ----------- | --------- | ---------------------------- |
| Development | 0         | No approval needed           |
| QA          | 0         | Automated testing sufficient |
| Staging     | 0-1       | Optional for visibility      |
| Production  | 1-2       | Always require approval      |

### Recommendations

* ✅ Require approvals for production environments
* ✅ Use multiple approvals for critical services
* ✅ Combine with verification for defense in depth
* ✅ Document approval requirements in runbooks
* ✅ Set up notifications for pending approvals

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace
* [Environment Progression](./environment-progression) - Enforce deployment order


# Deployment Dependency
Source: https://docs.ctrlplane.dev/policies/deployment-dependency

Learn how to use deployment dependency rules to create dependencies between different deployments.

**Deployment dependency rules** ensure that one deployment succeeds before
another can proceed. Use them to coordinate related services, enforce deployment
order across microservices, or manage infrastructure dependencies.

## Overview

```mermaid theme={null}
flowchart LR
    A[Database Migration] -->|Must succeed| B[API Service]
    B -->|Must succeed| C[Frontend]
```

## Why Use Deployment Dependencies?

Deployment dependency rules help you:

* **Coordinate services** - Deploy database before API, API before frontend
* **Manage infrastructure** - Infrastructure changes before application updates
* **Enforce order** - Shared libraries before dependent services
* **Reduce failures** - Prevent cascading failures from out-of-order deploys

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "api_requires_database" {
      name     = "API Requires Database"
      selector = "deployment.name == 'api-service'"

      deployment_dependency {
        depends_on_selector = "deployment.name == 'database-migration'"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "API Requires Database",
        "selector": "deployment.name == '\''api-service'\''",
        "rules": [
          {
            "deploymentDependency": {
              "dependsOn": "deployment.name == '\''database-migration'\''"
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="string">
  CEL expression to match upstream release targets that must exist before this
  deployment can proceed. The expression can reference both **deployment** and
  **version** properties of the currently deployed upstream release.
</ParamField>

### Available CEL Variables

The `dependsOn` expression is evaluated against each release target on the same
resource that has a successful release. Both `deployment.*` and `version.*`
fields are available:

| Variable              | Type      | Description                         |
| --------------------- | --------- | ----------------------------------- |
| `deployment.id`       | string    | Deployment ID                       |
| `deployment.name`     | string    | Deployment name                     |
| `deployment.slug`     | string    | Deployment slug                     |
| `deployment.metadata` | map       | Deployment metadata key-value pairs |
| `version.id`          | string    | Deployed version ID                 |
| `version.tag`         | string    | Version tag (e.g. `v2.1.0`)         |
| `version.name`        | string    | Version name                        |
| `version.status`      | string    | Version status                      |
| `version.metadata`    | map       | Version metadata key-value pairs    |
| `version.createdAt`   | timestamp | When the version was created        |

## How It Works

1. **Release created** - A new version is released for a deployment with
   dependency rules.
2. **Same-resource resolution** - Ctrlplane finds all release targets on the
   **same resource** as the current target.
3. **Version resolution** - For each release target, Ctrlplane resolves the
   deployment and its currently deployed version (from the latest successful
   job).
4. **CEL evaluation** - The `dependsOn` expression is evaluated against each
   `{deployment, version}` pair.
5. **Deployment allowed** - If at least one upstream release target matches the
   selector, the deployment can proceed.

## Common Patterns

### Database Before API

Ensure database migrations complete before API deploys:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "api_requires_db" {
      name     = "API Requires DB Migration"
      selector = "deployment.name == 'api-service'"

      deployment_dependency {
        depends_on_selector = "deployment.name == 'database-migration'"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "API Requires DB Migration",
      "selector": "deployment.name == 'api-service'",
      "rules": [
        {
          "deploymentDependency": {
            "dependsOn": "deployment.name == 'database-migration'"
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### Service Dependency Chain

Create a chain of dependencies:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "api_depends_on_db" {
      name     = "API Depends on DB"
      selector = "deployment.name == 'api-service'"

      deployment_dependency {
        depends_on_selector = "deployment.name == 'database-migration'"
      }
    }

    resource "ctrlplane_policy" "frontend_depends_on_api" {
      name     = "Frontend Depends on API"
      selector = "deployment.name == 'frontend'"

      deployment_dependency {
        depends_on_selector = "deployment.name == 'api-service'"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    [
      {
        "name": "API Depends on DB",
        "selector": "deployment.name == 'api-service'",
        "rules": [
          {
            "deploymentDependency": {
              "dependsOn": "deployment.name == 'database-migration'"
            }
          }
        ]
      },
      {
        "name": "Frontend Depends on API",
        "selector": "deployment.name == 'frontend'",
        "rules": [
          {
            "deploymentDependency": {
              "dependsOn": "deployment.name == 'api-service'"
            }
          }
        ]
      }
    ]
    ```
  </Tab>
</Tabs>

### Shared Library Dependencies

Ensure shared libraries are deployed before dependent services:

```hcl theme={null}
resource "ctrlplane_policy" "services_require_shared_lib" {
  name     = "Services Require Shared Lib"
  selector = "deployment.metadata['usesSharedLib'] == 'true'"

  deployment_dependency {
    depends_on_selector = "deployment.metadata['type'] == 'shared-library'"
  }
}
```

### Version-Scoped Dependencies

Require a specific version range of an upstream deployment:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "api_requires_db_v2" {
      name     = "API Requires DB Migration v2"
      selector = "deployment.name == 'api-service'"

      deployment_dependency {
        depends_on_selector = "deployment.name == 'database-migration' && version.tag.startsWith('v2.')"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "API Requires DB Migration v2",
      "selector": "deployment.name == 'api-service'",
      "rules": [
        {
          "deploymentDependency": {
            "dependsOn": "deployment.name == 'database-migration' && version.tag.startsWith('v2.')"
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### Version Metadata Filtering

Depend on an upstream deployment running a version with specific metadata:

```hcl theme={null}
resource "ctrlplane_policy" "frontend_requires_stable_api" {
  name     = "Frontend Requires Stable API"
  selector = "deployment.name == 'frontend'"

  deployment_dependency {
    depends_on_selector = "deployment.name == 'api-service' && version.metadata.channel == 'stable'"
  }
}
```

### Infrastructure First

Deploy infrastructure changes before application updates:

```hcl theme={null}
resource "ctrlplane_policy" "app_requires_infrastructure" {
  name     = "App Requires Infrastructure"
  selector = "deployment.metadata['type'] == 'application'"

  deployment_dependency {
    depends_on_selector = "deployment.metadata['type'] == 'infrastructure'"
  }
}
```

### Multi-Service Dependency

Depend on multiple services using CEL `in` operator:

```hcl theme={null}
resource "ctrlplane_policy" "gateway_requires_services" {
  name     = "Gateway Requires Backend Services"
  selector = "deployment.name == 'api-gateway'"

  deployment_dependency {
    depends_on_selector = "deployment.name in ['auth-service', 'user-service', 'billing-service']"
  }
}
```

## Combining with Other Rules

### With Environment Progression

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "api_full_gates" {
      name     = "API Full Gates"
      selector = "deployment.name == 'api-service' && environment.name == 'production'"

      deployment_dependency {
        depends_on_selector = "deployment.name == 'database-migration'"
      }

      environment_progression {
        depends_on_environment_selector = "environment.name == 'staging'"
      }

      any_approval {
        min_approvals = 1
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "API Full Gates",
      "selector": "deployment.name == 'api-service' && environment.name == 'production'",
      "rules": [
        {
          "deploymentDependency": {
            "dependsOn": "deployment.name == 'database-migration'"
          }
        },
        {
          "environmentProgression": {
            "dependsOnEnvironmentSelector": "environment.name == 'staging'"
          }
        },
        {
          "anyApproval": { "minApprovals": 1 }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### With Gradual Rollout

```hcl theme={null}
resource "ctrlplane_policy" "frontend_controlled_release" {
  name     = "Frontend Controlled Release"
  selector = "deployment.name == 'frontend'"

  deployment_dependency {
    depends_on_selector = "deployment.name == 'api-service'"
  }

  gradual_rollout {
    rollout_type        = "linear"
    time_scale_interval = 300
  }
}
```

## Best Practices

### Dependency Design

| Pattern               | Use Case                          |
| --------------------- | --------------------------------- |
| Database → API        | Schema changes before code        |
| API → Frontend        | API contracts before consumers    |
| Infrastructure → App  | Platform changes before workloads |
| Shared lib → Services | Common code before dependents     |
| Config → Application  | Configuration before apps         |

### Recommendations

* ✅ Keep dependency chains short (2-3 levels max)
* ✅ Use metadata to group related deployments
* ✅ Document why dependencies exist
* ✅ Test dependency resolution in staging
* ✅ Monitor for circular dependency issues

### Anti-Patterns

* ❌ Deep dependency chains (> 3 levels)
* ❌ Circular dependencies (A → B → A)
* ❌ Over-coupling unrelated services
* ❌ Using dependencies when environment progression would suffice

## Troubleshooting

### Deployment Blocked

If a deployment is blocked waiting for dependencies:

1. Check the dependency deployment's status
2. Verify the CEL expression matches the expected deployment
3. Remember that dependencies are resolved **per-resource** -- the upstream
   deployment must have a successful release on the same resource
4. Review the dependency deployment's success status

### Circular Dependencies

If you encounter circular dependency errors:

1. Review the dependency graph
2. Break the cycle by removing one dependency
3. Consider using environment progression instead

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Environment Progression](./environment-progression) - Cross-environment gates
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace


# Deployment Window
Source: https://docs.ctrlplane.dev/policies/deployment-window

Learn how to use deployment window rules to control when deployments can occur using time-based schedules.

**Deployment window rules** define specific time periods when deployments are
allowed or blocked. Using RFC 5545 recurrence rules (rrules), you can create
flexible schedules like "weekdays 9am-5pm" or "block deployments during
maintenance windows."

## Overview

```mermaid theme={null}
flowchart TD
    A[Release Created] --> B{Check Window}
    B --> C{Inside Window?}
    C -->|Allow Window + Inside| D[Deployment Proceeds]
    C -->|Allow Window + Outside| E[Wait for Window]
    C -->|Deny Window + Inside| E
    C -->|Deny Window + Outside| D
    E --> F[Re-evaluate when window changes]
    F --> B
```

## Why Use Deployment Windows?

Deployment windows help you:

* **Reduce risk** - Only deploy during business hours when teams are available
* **Coordinate operations** - Block deployments during maintenance windows
* **Meet compliance** - Enforce change control windows required by regulations
* **Protect stability** - Prevent deployments during high-traffic periods

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "business_hours_only" {
      name     = "Business Hours Only"
      selector = "environment.name == 'production'"

      deployment_window {
        rrule            = "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0"
        duration_minutes = 480
        timezone         = "America/New_York"
        allow_window     = true
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Business Hours Only",
        "selector": "environment.name == '\''production'\''",
        "rules": [
          {
            "deploymentWindow": {
              "rrule": "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0",
              "durationMinutes": 480,
              "timezone": "America/New_York",
              "allowWindow": true
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="string">
  RFC 5545 recurrence rule defining when windows start.
</ParamField>

<ParamField type="integer">
  Duration of each window in minutes.
</ParamField>

<ParamField type="string">
  IANA timezone for the rrule (e.g., `"America/New_York"`).
</ParamField>

<ParamField type="boolean">
  If `true`, deployments are only allowed during the window. If `false`,
  deployments are blocked during the window (deny window).
</ParamField>

## Understanding rrules

RFC 5545 recurrence rules define repeating patterns. Common components:

| Component  | Description             | Example                      |
| ---------- | ----------------------- | ---------------------------- |
| `FREQ`     | Frequency of recurrence | `DAILY`, `WEEKLY`, `MONTHLY` |
| `BYDAY`    | Days of the week        | `MO,TU,WE,TH,FR`             |
| `BYHOUR`   | Hours of the day (0-23) | `9` (9am)                    |
| `BYMINUTE` | Minutes of the hour     | `0`, `30`                    |

### Example rrules

| Pattern                  | rrule                                                  |
| ------------------------ | ------------------------------------------------------ |
| Weekdays at 9am          | `FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0` |
| Daily at 2am             | `FREQ=DAILY;BYHOUR=2;BYMINUTE=0`                       |
| Every Sunday at midnight | `FREQ=WEEKLY;BYDAY=SU;BYHOUR=0;BYMINUTE=0`             |
| First Monday of month    | `FREQ=MONTHLY;BYDAY=1MO;BYHOUR=9;BYMINUTE=0`           |

## Window Types

### Allow Windows

When `allowWindow: true` (default), deployments are **only allowed during** the
defined window:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    deployment_window {
      rrule            = "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0"
      duration_minutes = 480
      timezone         = "America/New_York"
      allow_window     = true
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "deploymentWindow": {
        "rrule": "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0",
        "durationMinutes": 480,
        "timezone": "America/New_York",
        "allowWindow": true
      }
    }
    ```
  </Tab>
</Tabs>

### Deny Windows (Blackouts)

When `allowWindow: false`, deployments are **blocked during** the defined
window:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    deployment_window {
      rrule            = "FREQ=WEEKLY;BYDAY=SU;BYHOUR=0;BYMINUTE=0"
      duration_minutes = 360
      timezone         = "America/New_York"
      allow_window     = false
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "deploymentWindow": {
        "rrule": "FREQ=WEEKLY;BYDAY=SU;BYHOUR=0;BYMINUTE=0",
        "durationMinutes": 360,
        "timezone": "America/New_York",
        "allowWindow": false
      }
    }
    ```
  </Tab>
</Tabs>

## Common Patterns

### Business Hours Only

Allow deployments only during business hours:

```hcl theme={null}
resource "ctrlplane_policy" "business_hours_deployments" {
  name     = "Business Hours Deployments"
  selector = "environment.name == 'production'"

  deployment_window {
    rrule            = "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0"
    duration_minutes = 480
    timezone         = "America/New_York"
  }
}
```

### Maintenance Window Blackout

Block deployments during scheduled maintenance:

```hcl theme={null}
resource "ctrlplane_policy" "maintenance_blackout" {
  name     = "Maintenance Blackout"
  selector = "environment.name == 'production'"

  deployment_window {
    rrule            = "FREQ=WEEKLY;BYDAY=SU;BYHOUR=2;BYMINUTE=0"
    duration_minutes = 240
    timezone         = "UTC"
    allow_window     = false
  }
}
```

### Weekend Freeze

Prevent deployments over weekends:

```hcl theme={null}
resource "ctrlplane_policy" "weekend_freeze" {
  name     = "Weekend Freeze"
  selector = "environment.name == 'production'"

  deployment_window {
    rrule            = "FREQ=WEEKLY;BYDAY=SA;BYHOUR=0;BYMINUTE=0"
    duration_minutes = 2880
    timezone         = "America/New_York"
    allow_window     = false
  }
}
```

### Late Night Deployments Only

For services that require off-peak deployments:

```hcl theme={null}
resource "ctrlplane_policy" "off_peak_deployments" {
  name     = "Off-Peak Deployments"
  selector = "deployment.metadata['requires_off_peak'] == 'true'"

  deployment_window {
    rrule            = "FREQ=DAILY;BYHOUR=2;BYMINUTE=0"
    duration_minutes = 180
    timezone         = "America/New_York"
  }
}
```

### Combined with Other Rules

Use deployment windows alongside other policy rules:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_controlled_release" {
      name     = "Production Controlled Release"
      selector = "environment.name == 'production'"

      any_approval {
        min_approvals = 1
      }

      deployment_window {
        rrule            = "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0"
        duration_minutes = 480
        timezone         = "America/New_York"
      }

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 300
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Production Controlled Release",
      "selector": "environment.name == 'production'",
      "rules": [
        { "anyApproval": { "minApprovals": 1 } },
        {
          "deploymentWindow": {
            "rrule": "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0",
            "durationMinutes": 480,
            "timezone": "America/New_York",
            "allowWindow": true
          }
        },
        {
          "gradualRollout": {
            "rolloutType": "linear",
            "timeScaleInterval": 300
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Behavior Details

### First Deployment Exemption

If a release target has never had a deployment before, deployment window rules
are bypassed. This ensures that initial deployments are not blocked by window
restrictions.

### Window Evaluation

* Deployments are evaluated against the current time
* If outside an allow window (or inside a deny window), the deployment waits
* Ctrlplane automatically re-evaluates when the window state changes, using the
  computed `nextEvaluationTime`

### Gradual Rollout Integration

When combined with gradual rollout rules:

* **Allow windows**: Rollout start time is adjusted to the next window opening
* **Deny windows**: Individual deployments within a rollout respect the deny
  period

### Timezone Handling

* Always specify a timezone for predictable behavior
* If omitted, UTC is used
* Use IANA timezone names (e.g., "America/New\_York", "Europe/London")

## Best Practices

### Environment-Based Windows

| Environment | Window Type             | Notes                        |
| ----------- | ----------------------- | ---------------------------- |
| Development | None                    | Deploy anytime               |
| QA          | None                    | Deploy anytime               |
| Staging     | Optional business hours | Mirror production if desired |
| Production  | Strict business hours   | When support is available    |

### Recommendations

* ✅ Use timezones matching your operational team's location
* ✅ Account for holidays with deny windows
* ✅ Provide adequate window duration for rollouts to complete
* ✅ Combine with approval rules for additional oversight
* ✅ Test rrule patterns in staging before production
* ❌ Don't create windows too narrow for deployments to complete
* ❌ Don't forget to account for gradual rollout duration

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace
* [Approval](./approval) - Add human approval gates


# Environment Progression
Source: https://docs.ctrlplane.dev/policies/environment-progression

Learn how to use environment progression rules to enforce deployment order across environments.

**Environment progression rules** ensure that releases are deployed to
prerequisite environments before they can proceed to downstream environments.
This enforces a deployment pipeline where changes must pass through QA before
staging, and staging before production.

## Overview

```mermaid theme={null}
flowchart LR
    A[QA] -->|Must succeed| B[Staging]
    B -->|Must succeed| C[Production]
```

## Why Use Environment Progression?

Environment progression rules help you:

* **Enforce deployment order** - Prevent skipping environments in your pipeline
* **Catch issues early** - Problems surface in lower environments first
* **Build confidence** - Each environment validates before the next
* **Meet compliance** - Satisfy audit requirements for change promotion

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "staging_requires_qa" {
      name     = "Staging Requires QA"
      selector = "environment.name == 'staging'"

      environment_progression {
        depends_on_environment_selector = "environment.name == 'qa'"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Staging Requires QA",
        "selector": "environment.name == '\''staging'\''",
        "rules": [
          {
            "environmentProgression": {
              "dependsOnEnvironmentSelector": "environment.name == '\''qa'\''"
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="string">
  CEL expression or selector matching the prerequisite environment(s). The
  release must be successfully deployed to environments matching this selector
  before proceeding.
</ParamField>

<ParamField type="float">
  Percentage of targets in the prerequisite environment that must succeed
  (0-100). Use a lower value if some targets are expected to be unavailable.
</ParamField>

<ParamField type="array">
  Job statuses considered successful. Defaults to `["successful"]`.
</ParamField>

<ParamField type="integer">
  Minutes to wait after the dependency succeeds before allowing progression. Use
  this to ensure the release has time to stabilize.
</ParamField>

<ParamField type="integer">
  Maximum age (in hours) of the dependency deployment. If the prerequisite
  deployment is older than this, progression is blocked until redeployed.
</ParamField>

<ParamField type="boolean">
  When enabled, jobs must also have passed verification to count toward the
  success percentage. Jobs with no verification metrics configured are still
  counted. Jobs with verification in a running, failed, or cancelled state are
  excluded from the success count.
</ParamField>

## How It Works

1. **Find dependency environments** - Ctrlplane finds all environments matching
   the selector that share at least one system with the current environment.
   This prevents cross-system dependency issues.
2. **Evaluate pass rate** - For each dependency environment, Ctrlplane checks
   if the success percentage of release targets meets the threshold.
3. **Check verification** - If `requireVerificationPassed` is enabled,
   only jobs that have passed verification count toward the success percentage.
   Jobs without verification metrics are still counted.
4. **Check soak time** - If `minimumSoakTimeMinutes` is configured, Ctrlplane
   verifies that enough time has elapsed since the most recent successful job.
5. **Check freshness** - If `maximumAgeHours` is configured, Ctrlplane ensures
   the dependency deployment is not too old.
6. **OR logic across environments** - If the selector matches multiple
   environments, the rule passes if **at least one** of them satisfies all
   criteria.

## Common Patterns

### Simple Linear Progression

Enforce QA → Staging → Production order:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "staging_requires_qa" {
      name     = "Staging Requires QA"
      selector = "environment.name == 'staging'"

      environment_progression {
        depends_on_environment_selector = "environment.name == 'qa'"
      }
    }

    resource "ctrlplane_policy" "production_requires_staging" {
      name     = "Production Requires Staging"
      selector = "environment.name == 'production'"

      environment_progression {
        depends_on_environment_selector = "environment.name == 'staging'"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    [
      {
        "name": "Staging Requires QA",
        "selector": "environment.name == 'staging'",
        "rules": [
          {
            "environmentProgression": {
              "dependsOnEnvironmentSelector": "environment.name == 'qa'"
            }
          }
        ]
      },
      {
        "name": "Production Requires Staging",
        "selector": "environment.name == 'production'",
        "rules": [
          {
            "environmentProgression": {
              "dependsOnEnvironmentSelector": "environment.name == 'staging'"
            }
          }
        ]
      }
    ]
    ```
  </Tab>
</Tabs>

### Soak Time Requirements

Require the release to "soak" in staging before production:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_soak_requirement" {
      name     = "Production Soak Requirement"
      selector = "environment.name == 'production'"

      environment_progression {
        depends_on_environment_selector = "environment.name == 'staging'"
        minimum_soak_time_minutes       = 60
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Production Soak Requirement",
      "selector": "environment.name == 'production'",
      "rules": [
        {
          "environmentProgression": {
            "dependsOnEnvironmentSelector": "environment.name == 'staging'",
            "minimumSoakTimeMinutes": 60
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### Freshness Requirements

Block promotion if the staging deployment is too old:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_freshness" {
      name     = "Production Freshness"
      selector = "environment.name == 'production'"

      environment_progression {
        depends_on_environment_selector = "environment.name == 'staging'"
        maximum_age_hours               = 24
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Production Freshness",
      "selector": "environment.name == 'production'",
      "rules": [
        {
          "environmentProgression": {
            "dependsOnEnvironmentSelector": "environment.name == 'staging'",
            "maximumAgeHours": 24
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

### Partial Success Threshold

Allow promotion when most (not all) targets succeed:

```hcl theme={null}
resource "ctrlplane_policy" "staging_partial_success" {
  name     = "Staging Partial Success"
  selector = "environment.name == 'staging'"

  environment_progression {
    depends_on_environment_selector = "environment.name == 'qa'"
    minimum_success_percentage      = 80
  }
}
```

### Complete Pipeline with All Options

Full-featured production gate:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_full_gate" {
      name     = "Production Full Gate"
      selector = "environment.name == 'production'"

      any_approval {
        min_approvals = 1
      }

      environment_progression {
        depends_on_environment_selector  = "environment.name == 'staging'"
        minimum_success_percentage       = 100
        minimum_soak_time_minutes        = 30
        maximum_age_hours                = 48
        require_verification_passed      = true
      }

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 300
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Production Full Gate",
      "selector": "environment.name == 'production'",
      "rules": [
        { "anyApproval": { "minApprovals": 1 } },
        {
          "environmentProgression": {
            "dependsOnEnvironmentSelector": "environment.name == 'staging'",
            "minimumSuccessPercentage": 100,
            "minimumSoakTimeMinutes": 30,
            "maximumAgeHours": 48,
            "requireVerificationPassed": true
          }
        },
        {
          "gradualRollout": {
            "rolloutType": "linear",
            "timeScaleInterval": 300
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Progression Lifecycle

### 1. Version Created

A new deployment version is created and ready for release.

### 2. Lower Environment Deployment

The version is deployed to the prerequisite environment (e.g., QA).

### 3. Success Evaluation

Ctrlplane evaluates if the deployment meets success criteria:

* Job status matches `successStatuses`
* Success percentage meets `minimumSuccessPercentage`
* If `requireVerificationPassed` is enabled, verification must have passed

### 4. Soak Time (if configured)

If `minimumSoakTimeMinutes` is set, the clock starts after success.

### 5. Progression Allowed

Once all criteria are met, the version can proceed to the next environment.

## Best Practices

### Timing Guidelines

| Transition           | Soak Time | Max Age | Notes                |
| -------------------- | --------- | ------- | -------------------- |
| Dev → QA             | 0         | -       | Fast iteration       |
| QA → Staging         | 0-15 min  | 24h     | Quick validation     |
| Staging → Production | 30-60 min | 48h     | Thorough soak        |
| Critical services    | 2-4 hours | 24h     | Extended observation |

### Recommendations

* ✅ Start with simple progression, add soak time later
* ✅ Use `maximumAgeHours` to prevent stale promotions
* ✅ Combine with verification for automated quality gates
* ✅ Use lower `minimumSuccessPercentage` for environments with flaky tests
* ✅ Document your progression requirements for the team

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace
* [Deployment Dependency](./deployment-dependency) - Cross-deployment gates


# Gradual Rollouts
Source: https://docs.ctrlplane.dev/policies/gradual-rollouts

Learn how to use gradual rollouts to control the pace of deployments across multiple release targets.

**Gradual rollouts** allow you to control the pace of deployments across
multiple release targets. Instead of deploying to all targets simultaneously,
you can stagger deployments over time to reduce risk and catch issues early.

## Overview

When deploying to multiple targets (e.g., multiple Kubernetes clusters in
production), gradual rollouts let you:

* **Reduce blast radius** - If something goes wrong, only a subset of targets
  are affected
* **Catch issues early** - Problems surface in early batches before full rollout
* **Control timing** - Space out deployments to manage load and monitoring

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_rollout" {
      name     = "Production Rollout"
      selector = "environment.name == 'production'"

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 300
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Production Rollout",
        "selector": "environment.name == '\''production'\''",
        "rules": [
          {
            "gradualRollout": {
              "rolloutType": "linear",
              "timeScaleInterval": 300
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="string">
  The rollout strategy to use:

  * `linear` — Deploy to each target at a fixed interval
  * `linear-normalized` — Space deployments evenly within the time window
</ParamField>

<ParamField type="integer">
  For `linear`: seconds between each target deployment. For
  `linear-normalized`: total seconds to complete the rollout.
</ParamField>

## Rollout Types

### Linear Rollout

Deploy to each target at a fixed interval:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    gradual_rollout {
      rollout_type        = "linear"
      time_scale_interval = 300
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "gradualRollout": {
        "rolloutType": "linear",
        "timeScaleInterval": 300
      }
    }
    ```
  </Tab>
</Tabs>

**Example with 5 targets**:

```
t+0m:   Target 1 deployed
t+5m:   Target 2 deployed
t+10m:  Target 3 deployed
t+15m:  Target 4 deployed
t+20m:  Target 5 deployed
```

### Linear Normalized Rollout

Deployments are spaced evenly so that the last target is scheduled at or before
the `timeScaleInterval`:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    gradual_rollout {
      rollout_type        = "linear-normalized"
      time_scale_interval = 600
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "gradualRollout": {
        "rolloutType": "linear-normalized",
        "timeScaleInterval": 600
      }
    }
    ```
  </Tab>
</Tabs>

**Example with 5 targets and 10 minute window**:

```
t+0m:   Target 1 deployed
t+2m:   Target 2 deployed
t+4m:   Target 3 deployed
t+6m:   Target 4 deployed
t+8m:   Target 5 deployed
```

## How Rollout Start Time Is Determined

The gradual rollout evaluator determines its start time by looking at other
policy rules that must be satisfied first:

1. **Approval rules** - Uses the `satisfiedAt` time when approvals were met
2. **Environment progression rules** - Uses the `satisfiedAt` time when
   progression criteria were met
3. **Policy skip overrides** - Uses the skip creation time
4. **Deployment windows** - Allow windows push the start to the window opening;
   deny windows push the start past the window closing

If no approval or progression rules exist, the rollout starts from the version
creation time. The rollout start is the **latest** time at which all
prerequisites were met.

Target ordering within a rollout is determined by a consistent hash of the
release target key and version ID, ensuring deterministic but varied ordering
across different versions.

## Common Patterns

### Conservative Production Rollout

Long intervals with approval:

```hcl theme={null}
resource "ctrlplane_policy" "conservative_rollout" {
  name     = "Conservative Rollout"
  selector = "environment.name == 'production'"

  any_approval {
    min_approvals = 1
  }

  gradual_rollout {
    rollout_type        = "linear"
    time_scale_interval = 900
  }
}
```

### Fast Staging Rollout

Quick rollout for non-production environments:

```hcl theme={null}
resource "ctrlplane_policy" "staging_rollout" {
  name     = "Staging Rollout"
  selector = "environment.name == 'staging'"

  gradual_rollout {
    rollout_type        = "linear-normalized"
    time_scale_interval = 120
  }
}
```

### Critical Service Rollout

Extra cautious rollout for critical services:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "critical_service_rollout" {
      name     = "Critical Service Rollout"
      selector = "deployment.metadata['tier'] == 'critical' && environment.name == 'production'"

      any_approval {
        min_approvals = 2
      }

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 1800
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Critical Service Rollout",
      "selector": "deployment.metadata['tier'] == 'critical' && environment.name == 'production'",
      "rules": [
        { "anyApproval": { "minApprovals": 2 } },
        {
          "gradualRollout": {
            "rolloutType": "linear",
            "timeScaleInterval": 1800
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Best Practices

### Timing Guidelines

| Environment | Rollout Type      | Interval  | Notes                         |
| ----------- | ----------------- | --------- | ----------------------------- |
| QA          | linear-normalized | 60-120s   | Fast feedback                 |
| Staging     | linear-normalized | 120-300s  | Reasonable pace               |
| Production  | linear            | 300-900s  | Conservative, time to monitor |
| Critical    | linear            | 900-1800s | Extra time for verification   |

### Recommendations

* ✅ Use longer intervals for production environments
* ✅ Combine with verification to catch issues between batches
* ✅ Use `linear-normalized` when you have a time constraint
* ✅ Use `linear` when you want consistent spacing regardless of target count
* ✅ Monitor each batch before the next one deploys

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Deployment Window](./deployment-window) - Time-based deployment control
* [Version Cooldown](./version-cooldown) - Batch frequent releases


# Overview
Source: https://docs.ctrlplane.dev/policies/overview



# Policies

**Policies** define **when** a version is allowed to deploy. They are the rules
that govern how releases progress through your environments — controlling
deployment timing through approvals, verification gates, progression
requirements, deployment windows, and more.

<Tip>
  In Ctrlplane's mental model: [**Deployments**](../concepts/deployments) = what & how, [**Environments**](../concepts/environments) = where, **Policies** = when.
</Tip>

## Building Confidence Through Policies

While [deployments](../concepts/deployments) define *what and how* and
[environments](../concepts/environments) define *where*, policies answer the
critical question of *when*: is this version ready to deploy? Has it passed
through the right gates? Policies help you deploy with confidence by ensuring
that each stage meets your quality standards before progressing to the next.

```
Development → QA → Staging → Production
     ↓          ↓       ↓          ↓
   (none)    smoke   integration  full
             tests     tests     gates
```

With policies, you can:

* **Start simple, grow complex** - Begin with basic health checks in QA, add
  integration tests in staging, require approvals and verification in production
* **Catch issues early** - Run smoke tests in QA to catch problems before they
  reach production
* **Automate quality gates** - Let verification results automatically determine
  if a release can proceed
* **Reduce deployment anxiety** - Know that every production deployment has
  passed through proven checks
* **Customize per environment** - Apply stricter rules where they matter most

## Policy Structure

A policy consists of:

1. **Name & Description** - Identify and document the policy's purpose
2. **Selector** - A CEL expression defining which release targets the policy
   applies to
3. **Rules** - One or more rules specifying behavior or requirements
4. **Priority** - Higher priority policies are evaluated first
5. **Metadata** - Arbitrary key-value pairs for organization

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_gates" {
      name        = "Production Gates"
      description = "Require approval and gradual rollout for production"
      priority    = 10
      enabled     = true

      selector = "environment.name == 'production'"

      any_approval {
        min_approvals = 1
      }

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 300
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Production Gates",
      "description": "Require approval and gradual rollout for production",
      "priority": 10,
      "enabled": true,
      "selector": "environment.name == 'production'",
      "rules": [
        {
          "anyApproval": { "minApprovals": 1 }
        },
        {
          "gradualRollout": {
            "rolloutType": "linear",
            "timeScaleInterval": 300
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Policy Selectors

The `selector` field is a CEL expression that determines which release targets
a policy applies to. Policies only affect releases that match the selector.

### Environment Selector

Target releases going to specific environments:

```
environment.name == "production"
environment.name in ["staging", "production"]
environment.name.startsWith("prod-")
environment.metadata['tier'] == "critical"
```

### Resource Selector

Target releases for specific resources:

```
resource.kind == "Kubernetes"
resource.metadata['region'] == "us-east-1"
resource.name.contains("critical")
```

### Deployment Selector

Target releases for specific deployments:

```
deployment.name == "api-service"
deployment.metadata['team'] == "platform"
```

### Combined Selectors

Combine conditions with `&&` (AND) and `||` (OR):

```
environment.name == "production" && deployment.metadata['tier'] == "critical"
```

## Policy Rules

Rules define what the policy enforces. A single policy can contain multiple
rules across different types. Each rule in the `rules` array contains exactly
one rule type.

### Approval Rule

Require manual approval before deployment:

| API Field                  | Terraform Attribute | Description                |
| -------------------------- | ------------------- | -------------------------- |
| `anyApproval.minApprovals` | `min_approvals`     | Number of approvals needed |

See [Approval](./approval) for details.

### Environment Progression Rule

Require successful deployment to a prerequisite environment:

| API Field                                             | Terraform Attribute               | Description                      |
| ----------------------------------------------------- | --------------------------------- | -------------------------------- |
| `environmentProgression.dependsOnEnvironmentSelector` | `depends_on_environment_selector` | CEL matching prerequisite env    |
| `environmentProgression.minimumSuccessPercentage`     | `minimum_success_percentage`      | Required success % (0-100)       |
| `environmentProgression.minimumSoakTimeMinutes`       | `minimum_soak_time_minutes`       | Soak time after success          |
| `environmentProgression.maximumAgeHours`              | `maximum_age_hours`               | Max age of dependency deployment |

See [Environment Progression](./environment-progression) for details.

### Gradual Rollout Rule

Control the pace of deployments across multiple targets:

| API Field                          | Terraform Attribute   | Description                          |
| ---------------------------------- | --------------------- | ------------------------------------ |
| `gradualRollout.rolloutType`       | `rollout_type`        | `"linear"` or `"linear-normalized"`  |
| `gradualRollout.timeScaleInterval` | `time_scale_interval` | Seconds between targets / total time |

See [Gradual Rollouts](./gradual-rollouts) for details.

### Deployment Dependency Rule

Require upstream deployments to succeed first:

| API Field                        | Terraform Attribute   | Description                      |
| -------------------------------- | --------------------- | -------------------------------- |
| `deploymentDependency.dependsOn` | `depends_on_selector` | CEL matching upstream deployment |

See [Deployment Dependency](./deployment-dependency) for details.

### Deployment Window Rule

Control when deployments are allowed using time-based schedules:

| API Field                          | Terraform Attribute | Description                         |
| ---------------------------------- | ------------------- | ----------------------------------- |
| `deploymentWindow.rrule`           | `rrule`             | RFC 5545 recurrence rule            |
| `deploymentWindow.durationMinutes` | `duration_minutes`  | Window duration in minutes          |
| `deploymentWindow.timezone`        | `timezone`          | IANA timezone                       |
| `deploymentWindow.allowWindow`     | `allow_window`      | Allow during window (deny if false) |

See [Deployment Window](./deployment-window) for details.

### Version Cooldown Rule

Batch frequent releases by enforcing a minimum time between deployments:

| API Field                         | Terraform Attribute | Description                  |
| --------------------------------- | ------------------- | ---------------------------- |
| `versionCooldown.intervalSeconds` | `duration`          | Minimum time between deploys |

See [Version Cooldown](./version-cooldown) for details.

### Retry Rule

Configure automatic retry behavior for failed jobs:

| API Field                 | Terraform Attribute | Description                   |
| ------------------------- | ------------------- | ----------------------------- |
| `retry.maxRetries`        | *Not available*     | Maximum retry attempts        |
| `retry.backoffSeconds`    | *Not available*     | Seconds between retries       |
| `retry.backoffStrategy`   | *Not available*     | `"linear"` or `"exponential"` |
| `retry.maxBackoffSeconds` | *Not available*     | Backoff cap                   |
| `retry.retryOnStatuses`   | *Not available*     | Which statuses trigger retry  |

See [Retry](./retry) for details.

### Version Selector Rule

Filter which versions can deploy to specific targets:

| API Field                     | Terraform Attribute | Description                     |
| ----------------------------- | ------------------- | ------------------------------- |
| `versionSelector.selector`    | *Not available*     | CEL expression or JSON selector |
| `versionSelector.description` | *Not available*     | Human-readable description      |

See [Version Selector](./version-selector) for details.

### Verification Rule

Run automated checks after deployment:

See [Verification](./verification) for detailed configuration options.

### Plan Validation Rule

Run OPA/Rego policies against a deployment plan to preview proposed changes.
This rule is **non-blocking** — it reports violations but does not gate
deployments.

| API Field                       | Terraform Attribute               | Description                   |
| ------------------------------- | --------------------------------- | ----------------------------- |
| `planValidationOpa.name`        | `plan_validation_opa.name`        | Rule name shown in violations |
| `planValidationOpa.description` | `plan_validation_opa.description` | Human-readable description    |
| `planValidationOpa.rego`        | `plan_validation_opa.rego`        | Rego v1 source code           |

See [Plan Validation](./plan-validation) for details.

## Terraform Provider Reference

The Terraform provider supports the following policy blocks:

| Block                     | Description                                     |
| ------------------------- | ----------------------------------------------- |
| `any_approval`            | Require manual approvals                        |
| `environment_progression` | Require success in prerequisite environments    |
| `gradual_rollout`         | Stagger deployments over time                   |
| `deployment_dependency`   | Require upstream deployments to succeed first   |
| `deployment_window`       | Time-based deployment scheduling                |
| `version_cooldown`        | Batch frequent releases                         |
| `verification`            | Automated health checks                         |
| `plan_validation_opa`     | OPA/Rego plan validation (non-blocking preview) |

<Note>
  `retry` and `version_selector` are not yet available in the Terraform
  provider. Use the REST API for these rule types.
</Note>

## REST API Reference

**Create a policy:**

```
POST /v1/workspaces/{workspaceId}/policies
```

**Get/Update/Delete a policy:**

```
GET    /v1/workspaces/{workspaceId}/policies/{policyId}
PUT    /v1/workspaces/{workspaceId}/policies/{policyId}
DELETE /v1/workspaces/{workspaceId}/policies/{policyId}
```

**List policies:**

```
GET /v1/workspaces/{workspaceId}/policies
```

Each rule in the `rules` array is an object with exactly one rule-type key
(e.g., `anyApproval`, `gradualRollout`, `deploymentWindow`).

## Policy Evaluation

When a release is created, Ctrlplane:

1. **Finds matching policies** - Evaluates the `selector` CEL expression
   against each release target
2. **Merges rules** - Combines rules from all matching policies
3. **Applies rules** - Enforces each rule type

### Rule Interactions

* **Approval + Gradual Rollout**: The rollout start time is determined by when
  the approval was satisfied
* **Environment Progression + Gradual Rollout**: The rollout waits until
  progression criteria are met
* **Deployment Window + Gradual Rollout**: Allow windows push the rollout start
  to the window opening; deny windows pause the rollout
* **Version Cooldown + Deployment Window**: Both must be satisfied -- cooldown
  determines which version, window determines when

## Common Patterns

### Environment Progression

Different requirements per environment:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "staging_policy" {
      name     = "Staging Policy"
      selector = "environment.name == 'staging'"

      environment_progression {
        depends_on_environment_selector = "environment.name == 'qa'"
      }
    }

    resource "ctrlplane_policy" "production_policy" {
      name     = "Production Policy"
      selector = "environment.name == 'production'"

      any_approval {
        min_approvals = 1
      }

      environment_progression {
        depends_on_environment_selector = "environment.name == 'staging'"
        minimum_soak_time_minutes       = 30
      }

      gradual_rollout {
        rollout_type        = "linear"
        time_scale_interval = 300
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    [
      {
        "name": "Staging Policy",
        "selector": "environment.name == 'staging'",
        "rules": [
          {
            "environmentProgression": {
              "dependsOnEnvironmentSelector": "environment.name == 'qa'"
            }
          }
        ]
      },
      {
        "name": "Production Policy",
        "selector": "environment.name == 'production'",
        "rules": [
          { "anyApproval": { "minApprovals": 1 } },
          {
            "environmentProgression": {
              "dependsOnEnvironmentSelector": "environment.name == 'staging'",
              "minimumSoakTimeMinutes": 30
            }
          },
          {
            "gradualRollout": {
              "rolloutType": "linear",
              "timeScaleInterval": 300
            }
          }
        ]
      }
    ]
    ```
  </Tab>
</Tabs>

### Critical Service Protection

Extra protection for critical services:

```hcl theme={null}
resource "ctrlplane_policy" "critical_service_gates" {
  name     = "Critical Service Gates"
  selector = "deployment.metadata['tier'] == 'critical' && environment.name == 'production'"
  priority = 20

  any_approval {
    min_approvals = 2
  }

  gradual_rollout {
    rollout_type        = "linear"
    time_scale_interval = 600
  }
}
```

## Best Practices

### Policy Organization

* ✅ Use descriptive policy names
* ✅ Document policy purpose in description
* ✅ Start with permissive policies and tighten over time
* ✅ Test policies in lower environments first
* ✅ Use `priority` to control evaluation order for overlapping policies

### Selector Design

* ✅ Be specific with selectors to avoid unexpected matches
* ✅ Use environment selectors for environment-specific rules
* ✅ Use metadata for cross-cutting concerns (team, tier, etc.)
* ✅ Test selector expressions before applying

### Rule Configuration

* ✅ Set reasonable timeouts and failure limits
* ✅ Use verification to catch issues before they impact users
* ✅ Require approvals for high-risk deployments
* ✅ Use gradual rollouts for large-scale deployments
* ✅ Combine complementary rules (e.g., approval + gradual rollout)

## Next Steps

* [Approval](./approval) - Require manual sign-off before deployment
* [Deployment Dependency](./deployment-dependency) - Create service dependencies
* [Deployment Window](./deployment-window) - Control when deployments can occur
* [Environment Progression](./environment-progression) - Enforce deployment
  order
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace
* [Retry](./retry) - Configure automatic retry behavior
* [Version Cooldown](./version-cooldown) - Batch frequent releases
* [Version Selector](./version-selector) - Filter deployable versions


# Plan Validation (OPA)
Source: https://docs.ctrlplane.dev/policies/plan-validation

Learn how to use OPA/Rego plan validation rules to inspect a deployment plan before you roll it out and surface policy violations as a preview.

**Plan validation rules** run [OPA](https://www.openpolicyagent.org/) (Open
Policy Agent) policies against a deployment **plan** — the diff between what is
currently deployed and what a new version proposes — and report any violations
back to you.

<Note>
  Plan validation is **non-blocking**. It does **not** gate or stop a
  deployment. It exists to give you a preview: when a new version is planned,
  you can see whether the proposed changes pass your policies before you decide
  to roll them out. Violations are surfaced on the deployment's GitHub check and
  in the ctrlplane UI, but the deployment is never automatically held back.
</Note>

## Overview

```mermaid theme={null}
flowchart TD
    A[Version Planned] --> B[Agent computes plan<br/>current vs. proposed]
    B --> C{Policy selector<br/>matches target?}
    C -->|No| D[No validation run]
    C -->|Yes| E[Evaluate Rego deny rules]
    E --> F{Any denials?}
    F -->|No| G[Reported as passed]
    F -->|Yes| H[Reported as violations]
```

A plan validation rule is a normal **policy rule**, so it shares the policy's
CEL `selector` that decides which release targets it applies to. The difference
is that instead of approvals or windows, the rule carries a snippet of **Rego**
that inspects the computed plan.

## When does it run?

Validation runs after an agent finishes computing a plan. Only agents that
produce a plan are evaluated:

* **ArgoCD** — the rendered manifest diff
* **Terraform Cloud** — the speculative plan output

If a deployment uses an agent that does not support plan operations, no
validation is run for that target.

## Why use plan validation?

* **Preview before you ship** — confirm a proposed change looks right before
  promoting it.
* **Catch risky diffs** — flag deletions, replacements, or scaling changes in a
  Terraform plan.
* **Guard against secrets or forbidden values** in rendered manifests.
* **Codify review checklists** that your team would otherwise eyeball by hand.

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "no_destroy_in_prod" {
      name     = "No destructive changes in production"
      selector = "environment.name == 'production'"

      plan_validation_opa {
        name        = "no-resource-deletions"
        description = "Flag plans that delete resources"
        rego        = <<-EOT
          package ctrlplane

          import rego.v1

          deny contains msg if {
            input.agentType == "terraform-cloud"
            plan := json.unmarshal(input.proposed)
            change := plan.resource_changes[_]
            change.change.actions[_] == "delete"
            msg := sprintf("resource %q would be deleted", [change.address])
          }
        EOT
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "No destructive changes in production",
        "selector": "environment.name == '\''production'\''",
        "rules": [
          {
            "planValidationOpa": {
              "name": "no-resource-deletions",
              "description": "Flag plans that delete resources",
              "rego": "package ctrlplane\n\nimport rego.v1\n\ndeny contains msg if {\n  input.agentType == \"terraform-cloud\"\n  plan := json.unmarshal(input.proposed)\n  change := plan.resource_changes[_]\n  change.change.actions[_] == \"delete\"\n  msg := sprintf(\"resource %q would be deleted\", [change.address])\n}"
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="string">
  Human-readable rule name. Shown in the check output to identify which rule
  produced a violation.
</ParamField>

<ParamField type="string">
  Optional human-readable explanation of what the rule does.
</ParamField>

<ParamField type="string">
  Rego v1 source code. Must define a `deny` rule set following the
  [Conftest](https://www.conftest.dev/) convention
  (`deny contains msg if { ... }`). Each member of the `deny` set becomes one
  violation message. An empty `deny` set means the plan passed.
</ParamField>

## Writing the Rego

Policies must be **Rego v1** (include `import rego.v1`) and define a `deny` rule
set. The package name can be anything — ctrlplane detects it automatically.

```rego theme={null}
package ctrlplane

import rego.v1

deny contains msg if {
    input.environment.name == "production"
    msg := "changes to production require manual review"
}
```

* The set is **empty by default** → the plan passes.
* Every string you add to `deny` becomes a separate violation line.
* Use the OPA standard library (`json.unmarshal`, `yaml.unmarshal`, `contains`,
  `sprintf`, regex, etc.) to parse and inspect the plan.

<Warning>
  If a policy contains invalid Rego (for example, a syntax error), it cannot be
  evaluated and no validation result is recorded for that plan. Test your
  policies with the [OPA playground](https://play.openpolicyagent.org/) or
  `opa eval` before saving them.
</Warning>

## The `input` document

Your Rego policy is evaluated against an `input` document with the following
fields:

| Field                   | Type    | Description                                                          |
| ----------------------- | ------- | -------------------------------------------------------------------- |
| `input.current`         | string  | The currently deployed state (raw, agent-specific format)            |
| `input.proposed`        | string  | The proposed state from the new version (raw, agent-specific format) |
| `input.hasChanges`      | boolean | Whether the plan contains any changes                                |
| `input.agentType`       | string  | The agent that produced the plan (e.g. `argo-cd`, `terraform-cloud`) |
| `input.environment`     | object  | The target environment                                               |
| `input.resource`        | object  | The target resource                                                  |
| `input.deployment`      | object  | The deployment                                                       |
| `input.proposedVersion` | object  | The deployment version being planned (the new version)               |
| `input.currentVersion`  | object  | The version currently deployed to this target (may be null)          |

<Note>
  `input.current` and `input.proposed` are **strings**, not parsed objects,
  because the format is agent-specific (Terraform JSON plan, rendered Kubernetes
  YAML, etc.). Parse them inside your policy with `json.unmarshal` or
  `yaml.unmarshal`.
</Note>

## Examples

### Flag resource deletions in a Terraform plan

```rego theme={null}
package ctrlplane

import rego.v1

deny contains msg if {
    input.agentType == "terraform-cloud"
    plan := json.unmarshal(input.proposed)
    change := plan.resource_changes[_]
    change.change.actions[_] == "delete"
    msg := sprintf("resource %q would be deleted", [change.address])
}
```

### Forbid hard-coded secrets in a rendered manifest

```rego theme={null}
package ctrlplane

import rego.v1

deny contains msg if {
    contains(input.proposed, "SECRET=")
    msg := "proposed manifest appears to contain a hard-coded secret"
}
```

### Require that a production change actually has a diff

```rego theme={null}
package ctrlplane

import rego.v1

deny contains "no changes detected for a production deploy" if {
    input.environment.name == "production"
    not input.hasChanges
}
```

### Flag a major version jump

```rego theme={null}
package ctrlplane

import rego.v1

deny contains msg if {
    current := input.currentVersion.tag
    proposed := input.proposedVersion.tag
    startswith(current, "v1.")
    startswith(proposed, "v2.")
    msg := sprintf("major version jump %s -> %s", [current, proposed])
}
```

## Viewing results

When a plan completes, ctrlplane records each rule's result against the plan.
Results surface in two places:

* **GitHub check run** — if the version carries GitHub metadata, the deployment
  check shows the plan diff and a **Policy violations** section listing each
  failing rule and its messages. A violation marks the check as failed so it is
  visible on the pull request, but it does not block the ctrlplane deployment.
* **ctrlplane UI** — the plan preview for a release target shows the diff
  alongside any validation results.

## Best practices

* ✅ Use a clear `name` — it is how violations are labeled in the output.
* ✅ Scope rules with the policy `selector` so they only run where they matter
  (e.g. production only).
* ✅ Branch on `input.agentType` when a policy is specific to Terraform or
  ArgoCD output.
* ✅ Test Rego in the OPA playground before saving.
* ❌ Don't rely on plan validation to *stop* a deployment — it is a preview, not
  a gate. Use [Approval](./approval) or
  [Environment Progression](./environment-progression) for gating.

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Approval](./approval) - Require sign-off before deploying
* [Verification](./verification/overview) - Check metrics after deploying


# Retry
Source: https://docs.ctrlplane.dev/policies/retry

Learn how to configure automatic retry behavior for failed deployments.

**Retry rules** configure how Ctrlplane handles failed jobs. You can control the
number of retry attempts, which failure types trigger retries, and the backoff
strategy between attempts.

## Overview

```mermaid theme={null}
flowchart TD
    A[Job Runs] --> B{Success?}
    B -->|Yes| C[Complete]
    B -->|No| D{Retries Left?}
    D -->|Yes| E[Wait for Backoff]
    E --> A
    D -->|No| F[Failed]
```

## Why Use Retry Rules?

Retry rules help you:

* **Handle transient failures** - Automatically recover from temporary issues
* **Reduce manual intervention** - Let the system retry before alerting
* **Configure per-environment** - More retries in dev, fewer in production
* **Control retry behavior** - Set backoff strategies to avoid thundering herd

## Configuration

<Tabs>
  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Retry on Failure",
        "selector": "environment.name == '\''production'\''",
        "rules": [
          {
            "retry": {
              "maxRetries": 3
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

<Note>
  Retry rules are not yet supported in the Terraform provider. Use the REST API
  to configure retry behavior.
</Note>

## Properties

<ParamField type="integer">
  Maximum retry attempts. `0` means no retries (1 attempt total), `3` means up
  to 4 attempts (1 initial + 3 retries).
</ParamField>

<ParamField type="array">
  Job statuses that trigger a retry. Defaults to `["failure",
      "invalidIntegration", "invalidJobAgent"]` when `maxRetries > 0`. When
  `maxRetries = 0`, also includes `"successful"` to enforce deploy-once
  semantics.
</ParamField>

<ParamField type="integer">
  Seconds to wait between retry attempts. If not set, retries are allowed
  immediately after job completion.
</ParamField>

<ParamField type="string">
  Backoff strategy: `linear` (constant delay) or `exponential` (doubling delay
  with each retry using `backoffSeconds * 2^(attempt-1)`).
</ParamField>

<ParamField type="integer">
  Maximum backoff cap in seconds (for exponential backoff). If not set, no
  maximum is enforced.
</ParamField>

## Job Statuses

The following job statuses can be used in `retryOnStatuses`:

| Status               | Description                     |
| -------------------- | ------------------------------- |
| `failure`            | Job failed during execution     |
| `successful`         | Job completed successfully      |
| `cancelled`          | Job was manually cancelled      |
| `skipped`            | Job was skipped                 |
| `invalidIntegration` | Integration configuration error |
| `invalidJobAgent`    | Job agent configuration error   |

<Note>
  Cancelled and skipped jobs never count toward the retry limit by default,
  allowing redeployment after manual cancellation.
</Note>

## Common Patterns

### Basic Retry

Retry failed jobs up to 3 times:

```json theme={null}
{
  "retry": {
    "maxRetries": 3
  }
}
```

### Retry with Backoff

Wait between retry attempts:

```json theme={null}
{
  "retry": {
    "maxRetries": 3,
    "backoffSeconds": 30
  }
}
```

### Exponential Backoff

Increase wait time with each retry:

```json theme={null}
{
  "retry": {
    "maxRetries": 5,
    "backoffSeconds": 10,
    "backoffStrategy": "exponential",
    "maxBackoffSeconds": 300
  }
}
```

With exponential backoff, wait times are: 10s → 20s → 40s → 80s → 160s (capped
at 300s)

### No Retries (Deploy-Once)

Disable retries for critical deployments. When `maxRetries` is `0`, the default
`retryOnStatuses` also includes `"successful"`, enforcing deploy-once semantics:

```json theme={null}
{
  "name": "No Retry Production",
  "selector": "environment.name == 'production'",
  "rules": [
    {
      "retry": { "maxRetries": 0 }
    }
  ]
}
```

### Retry Specific Statuses

Only retry on specific failure types:

```json theme={null}
{
  "retry": {
    "maxRetries": 3,
    "retryOnStatuses": ["failure", "invalidIntegration"],
    "backoffSeconds": 60
  }
}
```

### Environment-Specific Retry

Different retry behavior per environment:

```json theme={null}
[
  {
    "name": "Dev Retry",
    "selector": "environment.name == 'development'",
    "rules": [
      { "retry": { "maxRetries": 5, "backoffSeconds": 5 } }
    ]
  },
  {
    "name": "Staging Retry",
    "selector": "environment.name == 'staging'",
    "rules": [
      { "retry": { "maxRetries": 3, "backoffSeconds": 30 } }
    ]
  },
  {
    "name": "Production Retry",
    "selector": "environment.name == 'production'",
    "rules": [
      {
        "retry": {
          "maxRetries": 2,
          "backoffSeconds": 60,
          "backoffStrategy": "exponential"
        }
      }
    ]
  }
]
```

## Backoff Strategies

### Linear Backoff

Constant wait time between retries:

```
Attempt 1: immediate
Attempt 2: wait 30s
Attempt 3: wait 30s
Attempt 4: wait 30s
```

### Exponential Backoff

Doubling wait time with each retry:

```
Attempt 1: immediate
Attempt 2: wait 10s  (10 * 2^0)
Attempt 3: wait 20s  (10 * 2^1)
Attempt 4: wait 40s  (10 * 2^2)
Attempt 5: wait 80s  (10 * 2^3)
```

Use `maxBackoffSeconds` to cap the maximum wait time.

## Retry Lifecycle

### 1. Job Fails

A job completes with a status in `retryOnStatuses`.

### 2. Retry Check

Ctrlplane checks if retries remain (`attempt < maxRetries + 1`).

### 3. Backoff Wait

If `backoffSeconds` is configured, Ctrlplane waits before the next attempt. The
`nextEvaluationTime` is set to indicate when the retry will be allowed.

### 4. Retry Attempt

A new job is created for the retry attempt.

### 5. Success or Exhausted

The process continues until success or all retries are exhausted.

## Best Practices

### Retry Guidelines

| Scenario             | Max Retries | Backoff | Strategy    |
| -------------------- | ----------- | ------- | ----------- |
| Transient network    | 3-5         | 10-30s  | exponential |
| Rate limiting        | 3           | 60s     | exponential |
| Resource contention  | 2-3         | 30s     | linear      |
| Critical production  | 1-2         | 60s     | linear      |
| Flaky tests (dev/qa) | 5           | 5s      | linear      |

### Recommendations

* ✅ Use exponential backoff for external service failures
* ✅ Set `maxBackoffSeconds` to avoid excessive wait times
* ✅ Use fewer retries in production than in development
* ✅ Monitor retry rates to identify systemic issues
* ✅ Combine with alerting on final failure

### Anti-Patterns

* ❌ Infinite retries (always set `maxRetries`)
* ❌ No backoff for rate-limited APIs
* ❌ Same retry config across all environments
* ❌ Retrying on non-transient failures

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Environment Progression](./environment-progression) - Control promotion flow
* [Version Cooldown](./version-cooldown) - Batch frequent releases


# Overview
Source: https://docs.ctrlplane.dev/policies/verification/overview

Learn how to use verification to validate that a deployment is healthy after it completes.

**Verification** allows you to validate that a deployment is healthy after it
completes. Ctrlplane can automatically run verification checks by querying
metrics from external providers and evaluating success conditions.

## Overview

```mermaid theme={null}
flowchart TD
    A[Job Completed] --> B{Policy has<br/>verification?}
    B -->|No| C[Release Continues]
    B -->|Yes| D[Run Verification]
    D --> E[Query Metrics<br/>from Provider]
    E --> F[Evaluate CEL<br/>Success Condition]
    F --> G{Result}
    G -->|Pass| H[✓ Promote Release]
    G -->|Fail| I[✗ Trigger Rollback / Stop Promotion]
```

## Why Use Verification?

Verification helps you:

* **Catch Issues Early** - Detect problems before they impact users
* **Automate Rollbacks** - Trigger rollback policies when verification fails
* **Build Confidence** - Ensure deployments meet quality standards
* **Gate Promotions** - Block progression to production until QA verifies
* **Environment-Specific Checks** - Run different verifications per environment

## Basic Configuration

Add a verification rule to your policy:

```yaml theme={null}
policies:
  - name: qa-smoke-tests
    description: Run E2E smoke tests in QA before promotion
    selectors:
      - environment: environment.name == "qa"
    rules:
      - verification:
          metrics:
            - name: e2e-smoke-tests
              interval: 30s
              count: 5
              provider:
                type: http
                url: "http://e2e-runner.qa/run?service={{.resource.name}}"
              successCondition: result.ok && result.json.passed == true
              failureLimit: 1
```

## Environment-Specific Verifications

Different environments can have completely different verification requirements:

```yaml theme={null}
policies:
  # QA: Run E2E smoke tests
  - name: qa-verification
    selectors:
      - environment: environment.name == "qa"
    rules:
      - verification:
          metrics:
            - name: e2e-smoke-tests
              interval: 1m
              count: 3
              provider:
                type: http
                url: "http://e2e-runner/smoke?env=qa&service={{.resource.name}}"
              successCondition: result.json.all_passed == true

  # Staging: Check error rates and latency
  - name: staging-verification
    selectors:
      - environment: environment.name == "staging"
    rules:
      - verification:
          metrics:
            - name: error-rate
              interval: 30s
              count: 10
              provider:
                type: datadog
                apiKey: "{{.variables.dd_api_key}}"
                appKey: "{{.variables.dd_app_key}}"
                queries:
                  errors: "sum:errors{service:{{.resource.name}},env:staging}.as_rate()"
              successCondition: result.queries.errors < 0.01
              failureLimit: 2

  # Production: Comprehensive health checks
  - name: production-verification
    selectors:
      - environment: environment.name == "production"
    rules:
      - verification:
          metrics:
            - name: error-rate
              interval: 1m
              count: 10
              provider:
                type: datadog
                apiKey: "{{.variables.dd_api_key}}"
                appKey: "{{.variables.dd_app_key}}"
                queries:
                  errors: "sum:errors{service:{{.resource.name}},env:prod}.as_rate()"
              successCondition: result.queries.errors < 0.005
              failureLimit: 2
            - name: p99-latency
              interval: 1m
              count: 10
              provider:
                type: datadog
                apiKey: "{{.variables.dd_api_key}}"
                appKey: "{{.variables.dd_app_key}}"
                queries:
                  latency: "avg:latency.p99{service:{{.resource.name}},env:prod}"
              successCondition: result.queries.latency < 200
              failureLimit: 2
```

## Reusable Verification with Selectors

Use policy selectors to apply the same verification across multiple deployments
or environments:

```yaml theme={null}
policies:
  # Apply to all backend services
  - name: backend-health-verification
    selectors:
      - deployment: deployment.metadata.serviceType == "backend"
    rules:
      - verification:
          metrics:
            - name: health-check
              interval: 30s
              count: 5
              provider:
                type: http
                url: "http://{{.resource.name}}/health"
              successCondition: result.ok

  # Apply to all services with canary deployments
  - name: canary-verification
    selectors:
      - deployment: deployment.metadata.deploymentStrategy == "canary"
    rules:
      - verification:
          metrics:
            - name: canary-error-comparison
              interval: 2m
              count: 5
              provider:
                type: datadog
                apiKey: "{{.variables.dd_api_key}}"
                appKey: "{{.variables.dd_app_key}}"
                queries:
                  canary: "sum:errors{service:{{.resource.name}},version:canary}.as_rate()"
                  stable: "sum:errors{service:{{.resource.name}},version:stable}.as_rate()"
                formula: "canary / stable"
              successCondition:
                result.queries.canary < result.queries.stable * 1.1
```

## Progressive Delivery Gates

Use verification to gate promotion through environments:

```yaml theme={null}
policies:
  # QA must pass smoke tests before staging
  - name: qa-gate
    selectors:
      - environment: environment.name == "qa"
    rules:
      - verification:
          metrics:
            - name: smoke-tests
              interval: 30s
              count: 3
              provider:
                type: http
                url: "http://smoke-test-runner/run"
                method: POST
                body: |
                  {
                    "service": "{{.resource.name}}",
                    "version": "{{.version.tag}}",
                    "environment": "qa"
                  }
              successCondition: result.json.status == "passed"
              failureLimit: 0

  # Staging must pass before production
  - name: staging-gate
    selectors:
      - environment: environment.name == "staging"
    rules:
      - verification:
          metrics:
            - name: integration-tests
              interval: 1m
              count: 5
              provider:
                type: http
                url: "http://integration-runner/run?service={{.resource.name}}"
              successCondition:
                result.json.passed_count == result.json.total_count
```

## Metric Configuration

### Metric Properties

<ParamField type="string">
  Unique name for this verification metric. Used for identification in logs and
  UI.
</ParamField>

<ParamField type="integer">
  Seconds between each measurement. For example, `30` means check every 30
  seconds.
</ParamField>

<ParamField type="integer">
  Total number of measurements to take. Combined with `intervalSeconds`, this
  determines the verification duration.
</ParamField>

<ParamField type="object">
  Configuration for the metric provider (HTTP, Datadog, etc.). See
  provider-specific documentation for available options.
</ParamField>

<ParamField type="string">
  CEL expression evaluated against the provider response. Returns `true` for
  success. Example: `result.ok && result.statusCode == 200`
</ParamField>

<ParamField type="string">
  Optional CEL expression for explicit failure. If matched, verification fails
  immediately without waiting for more measurements.
</ParamField>

<ParamField type="integer">
  Number of consecutive failures allowed before the metric is considered failed.
  Set to `0` for no tolerance (fail on first failure).
</ParamField>

<ParamField type="integer">
  Number of consecutive successes required before the metric is considered
  passed.
</ParamField>

## Metric Providers

Ctrlplane supports multiple metric providers for collecting verification data.
Each provider has its own configuration and capabilities:

* **[HTTP Provider](../../integrations/verification-providers/http)** - Query any HTTP endpoint
  that returns JSON
* **[Datadog Provider](../../integrations/verification-providers/datadog)** - Query metrics from
  Datadog's Metrics API
* **[Sleep Provider](../../integrations/verification-providers/sleep)** - Wait for a specified
  duration before considering verification passed
* **[Terraform Cloud Run Provider](../../integrations/verification-providers/terraform-cloud-run)** -
  Verify Terraform Cloud run status for infrastructure deployments

See the individual provider documentation for detailed configuration options,
examples, and best practices.

## Template Variables

Provider configurations, success conditions, and failure conditions all support
Go templates with access to deployment context:

```yaml theme={null}
# Resource information
{{.resource.name}}
{{.resource.identifier}}
{{.resource.kind}}

# Environment information
{{.environment.name}}
{{.environment.id}}

# Deployment information
{{.deployment.name}}
{{.deployment.slug}}

# Version information
{{.version.tag}}
{{.version.id}}

# Custom variables (from deployment variables)
{{.variables.my_variable}}
{{.variables.dd_api_key}}
```

### Templated Conditions

Success and failure conditions can also use Go templates, which are rendered
before CEL evaluation:

```yaml theme={null}
metrics:
  - name: resource-check
    interval: 30s
    count: 5
    provider:
      type: http
      url: "http://api.internal/health/{{.resource.name}}"
    successCondition: result.json.name == "{{.resource.name}}"
    failureCondition: result.json.env != "{{.environment.name}}"
```

### Storing Secrets in Variables

For sensitive values like API keys, use deployment variables:

1. **Create a deployment variable**:

```bash theme={null}
curl -X POST https://your-ctrlplane-instance.com/api/v1/deployments/{deploymentId}/variables \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -d '{"key": "dd_api_key", "description": "Datadog API key"}'
```

2. **Set the value**:

```bash theme={null}
curl -X POST https://your-ctrlplane-instance.com/api/v1/deployments/{deploymentId}/variables/{variableId}/values \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -d '{"value": "your-actual-api-key"}'
```

3. **Reference in verification config**:

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.variables.dd_api_key}}"
  appKey: "{{.variables.dd_app_key}}"
  queries:
    errors: "sum:errors{service:api}"
```

## Success Conditions (CEL)

Success conditions are written in
[CEL (Common Expression Language)](https://github.com/google/cel-spec). The
measurement data is available as the `result` variable.

```yaml theme={null}
# Boolean check (HTTP provider)
successCondition: result.ok

# Numeric comparison (HTTP provider)
successCondition: result.json.value < 0.01

# Datadog provider - access queries by name
successCondition: result.queries.errors < 0.01

# String comparison
successCondition: result.json.status == "healthy"

# Logical operators
successCondition: result.ok && result.json.ready
successCondition: result.json.status == "healthy" || result.json.status == "degraded"

# Arithmetic
successCondition: result.json.success_count / result.json.total_count > 0.99

# Datadog - compare multiple queries
successCondition: result.queries.canary < result.queries.stable * 1.1
```

## Verification Lifecycle

### 1. Policy Evaluation

When a job completes, Ctrlplane evaluates policies to determine which
verifications apply based on the policy selectors.

### 2. Verification Starts

If a matching policy has verification rules, Ctrlplane creates a verification
record and starts the measurement process.

### 3. Measurements Taken

For each configured metric, measurements are taken at the specified interval:

```
Metric: error-rate (interval: 30s, count: 10)

t+0s:   Measurement 1 → Passed (value: 0.005)
t+30s:  Measurement 2 → Passed (value: 0.007)
t+60s:  Measurement 3 → Failed (value: 0.015)
t+90s:  Measurement 4 → Passed (value: 0.008)
...
t+270s: Measurement 10 → Passed (value: 0.006)
```

### 4. Verification Result

* **Passed**: All measurements passed, or failures stayed below `failureLimit`
* **Failed**: Failures exceeded `failureLimit`

### 5. Policy Action

Based on the verification result, the policy can:

* **Allow promotion** to the next environment
* **Trigger rollback** to a previous version
* **Block release** until manual intervention

### Verification Status

| Status      | Description                                   |
| ----------- | --------------------------------------------- |
| `running`   | Verification in progress, taking measurements |
| `passed`    | All checks passed within acceptable limits    |
| `failed`    | Too many measurements failed                  |
| `cancelled` | Verification was manually cancelled           |

## Best Practices

### Timing Recommendations

| Scenario              | Recommended Interval | Recommended Count |
| --------------------- | -------------------- | ----------------- |
| Quick smoke test      | 10-30s               | 3-5               |
| Standard verification | 30s-1m               | 5-10              |
| Extended soak test    | 5m                   | 12-24             |

### Failure Limits

| Risk Tolerance | Failure Limit | Notes                  |
| -------------- | ------------- | ---------------------- |
| Strict         | 1             | Fail on first failure  |
| Normal         | 2-3           | Allow transient issues |
| Lenient        | 5+            | For noisy metrics      |

### Environment-Specific Recommendations

| Environment | Verification Focus                  | Timing         |
| ----------- | ----------------------------------- | -------------- |
| QA          | Smoke tests, E2E tests              | Quick (1-3min) |
| Staging     | Integration tests, error rates      | Medium (5min)  |
| Production  | Error rates, latency, business KPIs | Extended (10m) |

## Troubleshooting

### Verification always fails

* Check if the provider can reach the target (network, DNS)
* Verify API credentials are correct
* Test the query manually
* Review measurement data for unexpected values
* Check if success condition is too strict

### Verification not running

* Verify the policy selector matches the release target
* Check that the policy is enabled
* Review policy evaluation logs
* Ensure verification is configured in the policy rules

### Wrong verification applied

* Review policy selectors
* Check policy priority/ordering
* Verify environment and metadata values
* Review which policies matched the release

## Provider Documentation

For detailed information about each metric provider, see:

* [HTTP Provider](../../integrations/verification-providers/http)
* [Datadog Provider](../../integrations/verification-providers/datadog)
* [Sleep Provider](../../integrations/verification-providers/sleep)
* [Terraform Cloud Run Provider](../../integrations/verification-providers/terraform-cloud-run)

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace
* [Selectors](../concepts/selectors) - Deep dive into selector syntax


# Version Cooldown
Source: https://docs.ctrlplane.dev/policies/version-cooldown

Learn how to use version cooldown rules to batch frequent releases and prevent rapid sequential deployments.

**Version cooldown rules** prevent rapid sequential deployments by requiring a
minimum time period to pass since the currently deployed (or in-progress)
version was created before allowing another deployment. This helps batch
frequent upstream releases into periodic deployments, reducing deployment churn
and infrastructure load.

## Why Use Version Cooldown?

Version cooldown helps you:

* **Reduce deployment frequency** - Batch multiple rapid releases into fewer
  deployments
* **Decrease infrastructure load** - Avoid constant rolling updates from CI/CD
  pipelines
* **Improve stability** - Give each deployment time to prove itself before the
  next
* **Save resources** - Reduce compute spent on deployment overhead

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "batch_deployments" {
      name     = "Batch Deployments"
      selector = "environment.name == 'production'"

      version_cooldown {
        duration = "1h"
      }
    }
    ```

    The Terraform provider accepts a Go duration string for `duration` (e.g.,
    `"1h"`, `"30m"`, `"90s"`). This is converted to seconds internally.
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Batch Deployments",
        "selector": "environment.name == '\''production'\''",
        "rules": [
          {
            "versionCooldown": {
              "intervalSeconds": 3600
            }
          }
        ]
      }'
    ```

    The API accepts `intervalSeconds` as an integer representing the minimum seconds
    that must pass.
  </Tab>
</Tabs>

## Properties

<ParamField type="integer">
  Minimum seconds that must pass since the currently deployed (or in-progress)
  version was created before allowing another deployment. Set to `0` to disable.
</ParamField>

## How It Works

Version cooldown checks if enough **time has elapsed** since the currently
deployed (or in-progress) version was created, not the time gap between version
creation times:

1. **Find reference version**: The currently deployed or in-progress version
   (in-progress takes precedence)
2. **Calculate elapsed time**: Time since the reference version was created
   (using current time)
3. **Apply cooldown**: If elapsed time >= interval, allow any version;
   otherwise, deny

### Example Timeline

```
Version Created:  v1.0 ────── v1.1 ── v1.2 ── v1.3 ────── v1.4
                  12:00       12:15   12:20   12:25       13:00

With intervalSeconds: 3600 (1 hour)
Currently deployed: v1.0 (created 12:00)

At 12:30 (30min elapsed since v1.0):
  Candidate v1.3 (created 12:25): DENIED - only 30min elapsed (need 60min)
  Candidate v1.2 (created 12:20): DENIED - only 30min elapsed
  Candidate v1.1 (created 12:15): DENIED - only 30min elapsed

At 13:05 (65min elapsed since v1.0):
  Candidate v1.3 (created 12:25): ALLOWED - 65min elapsed (>= 60min)
  Candidate v1.2 (created 12:20): ALLOWED - 65min elapsed
  Candidate v1.1 (created 12:15): ALLOWED - 65min elapsed
  Candidate v1.4 (created 13:00): ALLOWED - 65min elapsed
```

Once the cooldown period has elapsed, **any** version can be deployed,
regardless of when it was created. This enables batching rapid releases.

## Common Patterns

### Hourly Batching

Deploy at most once per hour:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "hourly_deployments" {
      name     = "Hourly Deployments"
      selector = "environment.name == 'production'"

      version_cooldown {
        duration = "1h"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    {
      "name": "Hourly Deployments",
      "selector": "environment.name == 'production'",
      "rules": [
        { "versionCooldown": { "intervalSeconds": 3600 } }
      ]
    }
    ```
  </Tab>
</Tabs>

### Per-Environment Cooldown

Different intervals for different environments:

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "staging_cooldown" {
      name     = "Staging Cooldown"
      selector = "environment.name == 'staging'"

      version_cooldown {
        duration = "15m"
      }
    }

    resource "ctrlplane_policy" "production_cooldown" {
      name     = "Production Cooldown"
      selector = "environment.name == 'production'"

      version_cooldown {
        duration = "1h"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```json theme={null}
    [
      {
        "name": "Staging Cooldown",
        "selector": "environment.name == 'staging'",
        "rules": [
          { "versionCooldown": { "intervalSeconds": 900 } }
        ]
      },
      {
        "name": "Production Cooldown",
        "selector": "environment.name == 'production'",
        "rules": [
          { "versionCooldown": { "intervalSeconds": 3600 } }
        ]
      }
    ]
    ```
  </Tab>
</Tabs>

### Combined with Other Rules

Use cooldown alongside other policy rules:

```hcl theme={null}
resource "ctrlplane_policy" "production_controlled_release" {
  name     = "Production Controlled Release"
  selector = "environment.name == 'production'"

  version_cooldown {
    duration = "30m"
  }

  any_approval {
    min_approvals = 1
  }

  gradual_rollout {
    rollout_type        = "linear"
    time_scale_interval = 300
  }
}
```

### With Deployment Windows

Combine cooldown with deployment windows for comprehensive control:

```hcl theme={null}
resource "ctrlplane_policy" "controlled_production" {
  name     = "Controlled Production"
  selector = "environment.name == 'production'"

  deployment_window {
    rrule            = "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=9;BYMINUTE=0"
    duration_minutes = 480
    timezone         = "America/New_York"
  }

  version_cooldown {
    duration = "2h"
  }
}
```

### Weekly Scheduled Deployments

Deploy updates on a specific day (e.g., every Monday):

```hcl theme={null}
resource "ctrlplane_policy" "monday_deployments" {
  name     = "Monday Deployments"
  selector = "deployment.name == 'datadog-agent'"

  deployment_window {
    rrule            = "FREQ=WEEKLY;BYDAY=MO;BYHOUR=0;BYMINUTE=0"
    duration_minutes = 1440
    timezone         = "America/New_York"
    allow_window     = true
  }

  version_cooldown {
    duration = "120h"
  }

  gradual_rollout {
    rollout_type        = "linear-normalized"
    time_scale_interval = 82800
  }
}
```

## Behavior Details

### Version Selection

When a candidate version fails cooldown:

1. Ctrlplane tries the next older version
2. This continues until a qualifying version is found
3. If no versions qualify, the release target waits

### Same Version Redeploys

Redeploying the currently deployed version is always allowed, regardless of
cooldown settings. This enables:

* Manual re-runs of failed deployments
* Rollback-and-redeploy workflows
* **Configuration-only changes** - Deploy the same version with updated
  configuration immediately, bypassing cooldown

### Urgent Deployments

For urgent deployments that need to bypass cooldown (e.g., security patches,
critical fixes):

* **Same-version redeploy**: If the urgent change uses the same version ID, it
  automatically bypasses cooldown
* **Policy Skip**: Create a PolicySkip to bypass cooldown for a specific
  version. This allows urgent deployments while maintaining cooldown for regular
  releases

### In-Progress Deployments

If a deployment is in progress, the cooldown uses that version as the reference:

* Prevents deploying a newer version while one is still rolling out
* Ensures sequential deployments respect the interval

## Best Practices

### Interval Guidelines

| Use Case             | Suggested Interval | Notes                      |
| -------------------- | ------------------ | -------------------------- |
| High-frequency CI/CD | 15-30 minutes      | Balance freshness vs churn |
| Standard services    | 1-2 hours          | Reasonable batching        |
| Stable/low-priority  | 4-24 hours         | Significant batching       |
| Development/staging  | 5-15 minutes       | Faster feedback loops      |

### Recommendations

* ✅ Start with shorter intervals and increase as needed
* ✅ Use longer intervals for production vs staging
* ✅ Combine with gradual rollouts for safer deployments
* ✅ Monitor deployment frequency to tune intervals
* ✅ Use same-version redeploys for urgent configuration changes
* ✅ Use PolicySkip for urgent deployments that need to bypass cooldown
* ❌ Don't set intervals so long that critical fixes are delayed
* ❌ Don't use cooldown on environments that need immediate updates
* ⚠️ Ensure deployment windows are long enough for gradual rollouts to complete

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Gradual Rollouts](./gradual-rollouts) - Control deployment pace
* [Deployment Window](./deployment-window) - Time-based deployment control


# Version Selector
Source: https://docs.ctrlplane.dev/policies/version-selector

Learn how to use version selector rules to control which versions can be deployed to specific environments.

**Version selector rules** filter which deployment versions are allowed to
deploy to matching environments. Use them to restrict production to stable
releases, enforce naming conventions, or block specific versions.

## Overview

```mermaid theme={null}
flowchart TD
    A[Version Created] --> B{Matches Selector?}
    B -->|Yes| C[Allowed to Deploy]
    B -->|No| D[Blocked]
```

## Why Use Version Selectors?

Version selector rules help you:

* **Enforce release channels** - Only stable versions in production
* **Block bad versions** - Prevent known-bad releases from deploying
* **Naming conventions** - Require specific version formats
* **Feature flags** - Control rollout of experimental features
* **Context-aware filtering** - Use environment, resource, or deployment data
  to make dynamic version decisions

## Configuration

<Tabs>
  <Tab title="Terraform">
    ```hcl theme={null}
    resource "ctrlplane_policy" "production_stable" {
      name     = "Production Stable Only"
      selector = "environment.name == 'production'"

      version_selector {
        selector    = "!version.tag.contains('-rc')"
        description = "Only stable versions (no release candidates)"
      }
    }
    ```
  </Tab>

  <Tab title="API">
    ```bash theme={null}
    curl -X POST https://api.ctrlplane.com/v1/workspaces/{workspaceId}/policies \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "name": "Production Stable Only",
        "selector": "environment.name == '\''production'\''",
        "rules": [
          {
            "versionSelector": {
              "selector": "!version.tag.contains('\''-rc'\'')",
              "description": "Only stable versions (no release candidates)"
            }
          }
        ]
      }'
    ```
  </Tab>
</Tabs>

## Properties

<ParamField type="string | object">
  A CEL expression (string) or JSON selector (object) to match allowed versions.
  CEL expressions have access to `version`, `environment`, `resource`, and
  `deployment` variables, enabling context-aware version filtering.
</ParamField>

<ParamField type="string">
  Human-readable explanation of the rule. Shown to users when a version is
  blocked.
</ParamField>

## CEL Expression Variables

When using CEL expressions, you have access to the following variables:

| Variable      | Type | Description                            |
| ------------- | ---- | -------------------------------------- |
| `version`     | map  | The deployment version being evaluated |
| `environment` | map  | The target environment                 |
| `resource`    | map  | The target resource                    |
| `deployment`  | map  | The deployment                         |

### Version Fields

| Field               | Type   | Description                    |
| ------------------- | ------ | ------------------------------ |
| `version.tag`       | string | Version tag (e.g., "v1.2.3")   |
| `version.metadata`  | map    | Custom metadata on the version |
| `version.createdAt` | string | When the version was created   |
| `version.status`    | string | Version status (e.g., "ready") |

### Environment Fields

| Field                  | Type   | Description      |
| ---------------------- | ------ | ---------------- |
| `environment.name`     | string | Environment name |
| `environment.metadata` | map    | Custom metadata  |

### Resource Fields

| Field               | Type   | Description     |
| ------------------- | ------ | --------------- |
| `resource.name`     | string | Resource name   |
| `resource.metadata` | map    | Custom metadata |

### Deployment Fields

| Field                 | Type   | Description     |
| --------------------- | ------ | --------------- |
| `deployment.name`     | string | Deployment name |
| `deployment.metadata` | map    | Custom metadata |

The CEL environment includes standard extensions (string functions, math, etc.)
and all entity fields are accessible as map keys.

## Common Patterns

### Stable Versions Only

Block pre-release versions from production:

```json theme={null}
{
  "name": "Production Stable",
  "selector": "environment.name == 'production'",
  "rules": [
    {
      "versionSelector": {
        "selector": "!version.tag.contains('-')",
        "description": "No pre-release versions (no hyphens in tag)"
      }
    }
  ]
}
```

### Semantic Version Pattern

Require semantic versioning format:

```json theme={null}
{
  "versionSelector": {
    "selector": "version.tag.matches('^v[0-9]+\\\\.[0-9]+\\\\.[0-9]+$')",
    "description": "Must be semantic version (vX.Y.Z)"
  }
}
```

### Release Channel by Metadata

Use version metadata for release channels:

```json theme={null}
[
  {
    "name": "Production Channel",
    "selector": "environment.name == 'production'",
    "rules": [
      {
        "versionSelector": {
          "selector": "version.metadata['channel'] == 'stable'",
          "description": "Only stable channel versions"
        }
      }
    ]
  },
  {
    "name": "Staging Channels",
    "selector": "environment.name == 'staging'",
    "rules": [
      {
        "versionSelector": {
          "selector": "version.metadata['channel'] in ['stable', 'beta']",
          "description": "Stable and beta channels allowed"
        }
      }
    ]
  }
]
```

### Major Version Restriction

Restrict major version changes:

```json theme={null}
{
  "versionSelector": {
    "selector": "version.tag.startsWith('v2.')",
    "description": "Only v2.x versions allowed"
  }
}
```

### Context-Aware Version Filtering

Use environment or resource data to dynamically filter versions. This is
uniquely powerful because the selector has access to all four entity types:

```json theme={null}
{
  "name": "Region-Specific Versions",
  "selector": "environment.name == 'production'",
  "rules": [
    {
      "versionSelector": {
        "selector": "resource.metadata['region'] != 'us-east-1' || version.metadata['us_east_certified'] == 'true'",
        "description": "US-East-1 requires certified versions"
      }
    }
  ]
}
```

### Branch-Based Filtering

Only deploy versions from the main branch:

```json theme={null}
{
  "versionSelector": {
    "selector": "version.metadata['branch'] == 'main'",
    "description": "Only deploy versions from main branch"
  }
}
```

### Feature Flag Versions

Control feature rollout by version metadata and resource:

```json theme={null}
{
  "name": "New UI Rollout",
  "selector": "environment.name == 'production'",
  "rules": [
    {
      "versionSelector": {
        "selector": "resource.metadata['region'] == 'us-east-1' || !has(version.metadata['feature_new_ui']) || version.metadata['feature_new_ui'] != 'true'",
        "description": "New UI only enabled for us-east-1"
      }
    }
  ]
}
```

## JSON Selector Format

As an alternative to CEL expressions, you can use JSON selectors that match
against the version object:

```json theme={null}
{
  "versionSelector": {
    "selector": {
      "matchExpression": [
        {
          "key": "tag",
          "operator": "DoesNotContain",
          "value": "-rc"
        }
      ]
    },
    "description": "Only stable versions (no release candidates)"
  }
}
```

### JSON Selector Operators

| Operator         | Description             | Example                         |
| ---------------- | ----------------------- | ------------------------------- |
| `Equals`         | Exact match             | `tag Equals "v1.0.0"`           |
| `NotEquals`      | Not equal               | `tag NotEquals "v1.0.0"`        |
| `In`             | Value in list           | `tag In ["v1.0.0", "v1.0.1"]`   |
| `NotIn`          | Value not in list       | `tag NotIn ["v1.0.0"]`          |
| `Contains`       | String contains         | `tag Contains "beta"`           |
| `DoesNotContain` | String does not contain | `tag DoesNotContain "rc"`       |
| `StartsWith`     | String starts with      | `tag StartsWith "v2."`          |
| `EndsWith`       | String ends with        | `tag EndsWith "-stable"`        |
| `Matches`        | Regex match             | `tag Matches "^v[0-9]+"`        |
| `Exists`         | Field exists            | `metadata.approved Exists`      |
| `DoesNotExist`   | Field does not exist    | `metadata.blocked DoesNotExist` |

<Note>
  JSON selectors only match against the `version` object. For cross-entity
  filtering (e.g., using environment or resource data), use CEL expressions
  instead.
</Note>

## Best Practices

### Environment Guidelines

| Environment | Version Policy     |
| ----------- | ------------------ |
| Development | Allow all versions |
| QA          | Allow all or beta+ |
| Staging     | Stable and beta    |
| Production  | Stable only        |

### Recommendations

* ✅ Use `description` to explain why versions are restricted
* ✅ Prefer CEL expressions for context-aware filtering (access to environment,
  resource, deployment)
* ✅ Use metadata for release channels instead of parsing tags
* ✅ Document blocked versions with links to issues
* ✅ Test selectors in lower environments first
* ✅ Start permissive and tighten over time

### Anti-Patterns

* ❌ Overly complex regex patterns
* ❌ Blocking without documentation
* ❌ Inconsistent version tagging conventions
* ❌ Forgetting to update blocked version lists

## Next Steps

* [Policies Overview](./overview) - Learn about policy structure
* [Environment Progression](./environment-progression) - Control promotion flow
* [Version Cooldown](./version-cooldown) - Batch frequent releases


# Quickstart
Source: https://docs.ctrlplane.dev/quickstart

Set up deployment orchestration with environment promotion and verification in 15 minutes.

This guide walks you through setting up a complete deployment pipeline with
staging → production promotion and automated verification. By the end, you'll
have a working example of Ctrlplane's core capabilities.

## What You'll Build

```mermaid theme={null}
flowchart LR
    CI["CI (build)"] --> Staging

    subgraph Staging["Staging"]
        direction TB
        S1["Deploy"] --> S2["Verify"]
    end

    Staging --> Production

    subgraph Production["Production"]
        direction TB
        P0["Approval"] --> P1["Deploy"] --> P2["Verify"]
    end
```

* **Deployment** with automatic version creation from CI
* **Two environments** (staging, production) with resource selectors
* **Verification** that checks deployment health before promotion
* **Approval policy** requiring sign-off for production

## Prerequisites

* Ctrlplane account ([self-hosted](./installation))
* API key (Settings → API Keys)
* GitHub repository with CI workflow

## Step 1: Create a System

A system groups related deployments. This is typically a product, platform, or
bounded context.

```hcl Terraform theme={null}
resource "ctrlplane_system" "quickstart" {
  name        = "Tutorial Quickstart"
  description = "A tutorial on how to use Ctrlplane found in the Quickstart guide."
}
```

## Step 2: Register Resources

Resources are your deployment targets. In production, you'd sync these from
Kubernetes or cloud providers. For this quickstart, we'll create them manually.

<CodeGroup>
  ```yaml YAML theme={null}
  # ctrlc apply -f resource.yaml
  ---
  type: Resource
  identifier: tutorial-quickstart-resource-staging
  kind: KubernetesCluster
  name: quickstart-staging-cluster
  version: tutorial/quickstart/v1
  metadata:
    tutorial: quickstart
    environment: staging
    region: us-east-1
  config:
    number: 1
    string: "one"
    boolean: true
    array:
      - one
      - two
      - three
  ---
  type: Resource
  identifier: tutorial-quickstart-resource-production
  kind: KubernetesCluster
  name: quickstart-production-cluster
  version: tutorial/quickstart/v1
  metadata:
    tutorial: quickstart
    environment: production
    region: us-east-1
  config:
    number: 1
    string: "one"
    boolean: true
    array:
      - one
      - two
      - three
  ```

  ```hcl Terraform theme={null}
  resource "ctrlplane_resource" "quickstart_staging" {
    identifier = "tutorial-quickstart-resource-staging"
    kind       = "KubernetesCluster"
    name       = "quickstart-staging-cluster"
    version    = "tutorial/quickstart/v1"

    metadata = {
      tutorial    = "quickstart"
      environment = "staging"
      region      = "us-east-1"
    }

    config = {
      number  = 1
      string  = "one"
      boolean = true
      array   = ["one", "two", "three"]
    }
  }

  resource "ctrlplane_resource" "quickstart_production" {
    identifier = "tutorial-quickstart-resource-production"
    kind       = "KubernetesCluster"
    name       = "quickstart-production-cluster"
    version    = "tutorial/quickstart/v1"

    metadata = {
      tutorial    = "quickstart"
      environment = "production"
      region      = "us-east-1"
    }

    config = {
      number  = 1
      string  = "one"
      boolean = true
      array   = ["one", "two", "three"]
    }
  }
  ```
</CodeGroup>

```bash theme={null}
ctrlc apply -f https://raw.githubusercontent.com/ctrlplanedev/ctrlplane/main/examples/quickstart/2-resources.yaml
```

## Step 3: Create Environments

Environments use selectors to dynamically match resources. When you add new
clusters with matching metadata, they're automatically included.

```hcl Terraform theme={null}
resource "ctrlplane_environment" "staging" {
  name              = "Staging"
  description       = "Pre-production validation"
  resource_selector = "resource.metadata['environment'] == 'staging'"
}

resource "ctrlplane_environment" "production" {
  name              = "Production"
  description       = "Live production environment"
  resource_selector = "resource.metadata['environment'] == 'production'"
}
```

## Step 4: Create a Job Agent

Job agents execute your deployments. We'll use a simple test runner, but
Ctrlplane supports GitHub Actions, Kubernetes jobs, ArgoCD, or custom agents.

```hcl Terraform theme={null}
resource "ctrlplane_job_agent" "quickstart" {
  name        = "Tutorial Quickstart Job Agent"
  description = "A tutorial on how to use Ctrlplane found in the Quickstart guide."

  test_runner {
    sleep = "5s"
  }
}
```

```bash theme={null}
ctrlc apply -f https://raw.githubusercontent.com/ctrlplanedev/ctrlplane/main/examples/quickstart/4-job-agent.yaml
```

## Step 5: Create a Deployment

A deployment represents your application. The job agent config tells Ctrlplane
how to trigger deployments.

```hcl Terraform theme={null}
# Production environment
resource "ctrlplane_deployment" "quickstart" {
  name        = "Tutorial Quickstart Deployment"
  description = "A tutorial on how to use Ctrlplane found in the Quickstart guide."

  job_agent {
    id = ctrlplane_job_agent.quickstart.id
  }
}
```

## Step 6: Add Deployment Workflow

Create `.github/workflows/deploy.yml` in your repository:

```yaml theme={null}
name: Deploy

on:
  workflow_dispatch:
    inputs:
      job_id:
        description: "Ctrlplane Job ID"
        required: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Get deployment context
        uses: ctrlplanedev/get-job-inputs@v1
        id: job
        with:
          job_id: ${{ inputs.job_id }}
          api_key: ${{ secrets.CTRLPLANE_API_KEY }}

      - name: Deploy to Kubernetes
        run: |
          echo "Deploying ${{ steps.job.outputs.version_tag }}"
          echo "Environment: ${{ steps.job.outputs.environment_name }}"
          echo "Cluster: ${{ steps.job.outputs.resource_identifier }}"

          # Your deployment logic here
          # kubectl set image deployment/api-gateway \
          #   api-gateway=${{ steps.job.outputs.version_tag }}
```

Add `CTRLPLANE_API_KEY` to your repository secrets.

## Step 7: Integrate CI Build

Add version creation to your build workflow (`.github/workflows/build.yml`):

```yaml theme={null}
- name: Install Ctrlplane CLI
  uses: ctrlplanedev/cli@main
  with:
    api_key: ${{ secrets.CTRLPLANE_API_KEY }}

- name: Create deployment version
  if: github.ref == 'refs/heads/main'
  run: |
    ctrlc api upsert version \
      --workspace <your-workspace> \
      --deployment ${{ secrets.CTRLPLANE_DEPLOYMENT_ID }}  \
      --tag ${{ github.sha }} \
      --name "${{ github.sha::7 }}" \
      --metadata github/owner=${{ github.repository_owner }} \
      --metadata github/repo=${{ github.event.repository.name }} \
      --metadata git/sha=${{ github.sha }} \
      --metadata git/branch=${{ github.ref_name }} \
      --metadata github/run-number=${{ github.run_number }} \
      --metadata github/run-id=${{ github.run_id }} \
      --metadata github/run-attempt=${{ github.run_attempt }}
```

## Step 8: Add Production Approval

Create a policy requiring approval before production deployments

```hcl Terraform theme={null}
resource "ctrlplane_policy" "production_approval" {
  name        = "Production Approval Policy"
  description = "Production Approval Policy"

  selector {
    environments = "environment.metadata['requires-approval'] == 'true'"
  }

  approval {
    required = 1
  }
}
```

## Step 9: Test the Pipeline

1. Push a commit to `main`
2. CI builds and creates a version in Ctrlplane
3. Ctrlplane creates releases for staging and production
4. Staging deployment executes immediately
5. Verification runs health checks
6. Production waits for approval
7. After approval, production deploys and verifies

View the pipeline in the Ctrlplane UI:

* **Deployments** → See version progression across environments
* **Releases** → Track release progression across environments
* **Jobs** → View execution details and logs

## What You've Built

✅ **Deployment orchestration** with automatic environment progression\
✅ **Resource inventory** with metadata-based environment selectors\
✅ **Verification** ensuring deployment health before promotion\
✅ **Policy gates** requiring approval for production

## Next Steps

<CardGroup>
  <Card title="Policies" icon="shield" href="./policies/overview">
    Configure gradual rollouts, concurrency limits, and custom gates
  </Card>

  <Card title="Verification" icon="check" href="./policies/verification/overview">
    Add Datadog metrics, custom HTTP checks, and more
  </Card>

  <Card title="Resources" icon="server" href="./concepts/resources">
    Sync resources from Kubernetes, AWS, or custom providers
  </Card>

  <Card title="Job Agents" icon="robot" href="./integrations/job-agents/github">
    Deploy with GitHub Actions, ArgoCD, or custom agents
  </Card>
</CardGroup>

## Troubleshooting

**Jobs not being created:**

* Verify resource metadata matches environment selectors
* Check deployment has a job agent configured
* Review policy denials in the Releases view

**Verification failing:**

* Test the health endpoint manually
* Check the success condition syntax
* Review measurement data in the verification details

**GitHub workflow not triggering:**

* Ensure job agent type is `github`
* Verify workflow filename matches `jobAgentConfig`
* Check GitHub App permissions

Need help?
[GitHub Discussions](https://github.com/ctrlplanedev/ctrlplane/discussions)


# CEL Expression Language
Source: https://docs.ctrlplane.dev/reference/cel

Reference guide for Common Expression Language (CEL) in Ctrlplane

Ctrlplane uses [CEL (Common Expression Language)](https://github.com/google/cel-spec)
for writing powerful selectors and matching expressions. CEL is a simple, fast,
and safe expression language developed by Google.

## Overview

CEL expressions are used in:

* **Resource selectors** — Match resources to environments
* **Policy selectors** — Target policies to specific releases
* **Relationship rules** — Define how entities connect

```yaml theme={null}
# Example: Environment resource selector
resourceSelector: resource.metadata["environment"] == "production"

# Example: Policy selector
selector: environment.name == "Production" && deployment.metadata["critical"] == "true"
```

## Basic Syntax

### Accessing Fields

```cel theme={null}
# Direct field access
resource.name
resource.kind
resource.identifier

# Metadata access (map)
resource.metadata["environment"]
resource.metadata["region"]

# Config access
resource.config["namespace"]
```

### Comparison Operators

| Operator | Description      | Example                                |
| -------- | ---------------- | -------------------------------------- |
| `==`     | Equal            | `resource.kind == "KubernetesCluster"` |
| `!=`     | Not equal        | `resource.metadata["env"] != "dev"`    |
| `<`      | Less than        | `resource.metadata["priority"] < "5"`  |
| `>`      | Greater than     | `resource.metadata["replicas"] > "1"`  |
| `<=`     | Less or equal    | `version.metadata["build"] <= "100"`   |
| `>=`     | Greater or equal | `resource.metadata["tier"] >= "2"`     |

### Logical Operators

| Operator | Description | Example                            |
| -------- | ----------- | ---------------------------------- |
| `&&`     | Logical AND | `a == "x" && b == "y"`             |
| `\|\|`   | Logical OR  | `a == "x" \|\| a == "y"`           |
| `!`      | Logical NOT | `!resource.metadata["deprecated"]` |

### Arithmetic Operators

| Operator | Description              | Example                     |
| -------- | ------------------------ | --------------------------- |
| `+`      | Addition / Concatenation | `"prefix-" + resource.name` |
| `-`      | Subtraction              | `int(a) - int(b)`           |
| `*`      | Multiplication           | `int(a) * 2`                |
| `/`      | Division                 | `int(a) / 2`                |
| `%`      | Modulo                   | `int(a) % 2 == 0`           |

## Available Variables

### Resource Context

When writing resource selectors:

| Variable              | Type   | Description                               |
| --------------------- | ------ | ----------------------------------------- |
| `resource.id`         | string | Unique resource ID                        |
| `resource.name`       | string | Resource display name                     |
| `resource.kind`       | string | Resource type (e.g., `KubernetesCluster`) |
| `resource.identifier` | string | External identifier                       |
| `resource.version`    | string | Resource version                          |
| `resource.metadata`   | map    | Key-value metadata                        |
| `resource.config`     | map    | Resource configuration                    |

### Environment Context

When writing environment selectors:

| Variable               | Type   | Description          |
| ---------------------- | ------ | -------------------- |
| `environment.id`       | string | Environment ID       |
| `environment.name`     | string | Environment name     |
| `environment.metadata` | map    | Environment metadata |

### Deployment Context

When writing deployment selectors:

| Variable              | Type   | Description         |
| --------------------- | ------ | ------------------- |
| `deployment.id`       | string | Deployment ID       |
| `deployment.name`     | string | Deployment name     |
| `deployment.metadata` | map    | Deployment metadata |

### Version Context

When writing version selectors:

| Variable           | Type   | Description      |
| ------------------ | ------ | ---------------- |
| `version.id`       | string | Version ID       |
| `version.tag`      | string | Version tag      |
| `version.name`     | string | Version name     |
| `version.metadata` | map    | Version metadata |

## String Functions

### contains

Check if a string contains a substring:

```cel theme={null}
resource.name.contains("prod")
resource.metadata["tags"].contains("critical")
```

### startsWith

Check if a string starts with a prefix:

```cel theme={null}
resource.identifier.startsWith("k8s-")
resource.metadata["region"].startsWith("us-")
```

### endsWith

Check if a string ends with a suffix:

```cel theme={null}
resource.name.endsWith("-cluster")
resource.metadata["zone"].endsWith("a")
```

### matches

Regular expression matching:

```cel theme={null}
# Match identifiers like prod-cluster-1, prod-cluster-2
resource.identifier.matches("^prod-cluster-[0-9]+$")

# Match any US region
resource.metadata["region"].matches("^us-(east|west)-[0-9]$")
```

### size

Get string length:

```cel theme={null}
resource.name.size() > 0
resource.metadata["description"].size() < 100
```

### toLowerCase / toUpperCase

Case conversion:

```cel theme={null}
resource.metadata["env"].toLowerCase() == "production"
```

## List Operations

### in

Check if a value is in a list:

```cel theme={null}
resource.metadata["region"] in ["us-east-1", "us-west-2", "eu-west-1"]
resource.kind in ["KubernetesCluster", "KubernetesNamespace"]
```

### size

Get list length:

```cel theme={null}
resource.metadata["tags"].size() > 0
```

### exists

Check if any element matches:

```cel theme={null}
# Check if any tag starts with "team-"
resource.metadata["tags"].exists(t, t.startsWith("team-"))
```

### all

Check if all elements match:

```cel theme={null}
# Check if all regions are in US
resource.metadata["regions"].all(r, r.startsWith("us-"))
```

## Map Operations

### has

Check if a key exists in a map:

```cel theme={null}
has(resource.metadata["team"])
has(resource.config["namespace"])
```

This is safer than direct access when the key might not exist.

### Key Access

Access map values:

```cel theme={null}
resource.metadata["environment"]
resource.config["server"]
```

## Conditional Expressions

### Ternary Operator

```cel theme={null}
resource.metadata["tier"] == "critical" ? "high-priority" : "normal"
```

### Null-Safe Access

Use `has()` for optional fields:

```cel theme={null}
has(resource.metadata["deprecated"]) && resource.metadata["deprecated"] == "true"
```

Or use the default pattern:

```cel theme={null}
# Default to empty string if not present
(has(resource.metadata["team"]) ? resource.metadata["team"] : "") == "platform"
```

## Common Patterns

### Production Resources

```cel theme={null}
resource.metadata["environment"] == "production"
```

### Multi-Region Targeting

```cel theme={null}
resource.metadata["region"] in ["us-east-1", "us-west-2"]
```

### Kind Filtering

```cel theme={null}
resource.kind == "KubernetesCluster" && 
resource.metadata["environment"] == "production"
```

### Team-Based Selection

```cel theme={null}
resource.metadata["team"] == "platform" || 
resource.metadata["team"] == "infrastructure"
```

### Exclude Deprecated

```cel theme={null}
!has(resource.metadata["deprecated"]) || 
resource.metadata["deprecated"] != "true"
```

### Critical Tier in Production

```cel theme={null}
resource.metadata["environment"] == "production" && 
resource.metadata["tier"] == "critical"
```

### US Regions Only

```cel theme={null}
resource.metadata["region"].startsWith("us-")
```

### Canary Resources

```cel theme={null}
resource.metadata["environment"] == "production" && 
has(resource.metadata["canary"]) && 
resource.metadata["canary"] == "true"
```

### Complex Multi-Condition

```cel theme={null}
(resource.metadata["environment"] == "production" && 
 resource.metadata["region"] in ["us-east-1", "us-west-2"]) ||
(resource.metadata["environment"] == "staging" && 
 resource.metadata["region"] == "us-east-1")
```

## Policy Selector Examples

### Target Production Environment

```cel theme={null}
environment.name == "Production"
```

### Target Critical Deployments

```cel theme={null}
deployment.metadata["tier"] == "critical"
```

### Target Specific Environment + Deployment

```cel theme={null}
environment.name == "Production" && 
deployment.metadata["requires-approval"] == "true"
```

### Version Filtering

```cel theme={null}
version.tag.startsWith("v2.") && 
!version.tag.contains("beta")
```

## Relationship Rule Examples

### Match by Region

```cel theme={null}
# fromSelector for VPCs
resource.kind == "vpc"

# toSelector for clusters  
resource.kind == "KubernetesCluster"

# Property matcher expression
from.metadata["region"] == to.metadata["region"]
```

### Match by Account

```cel theme={null}
from.metadata["account"] == to.metadata["account"] &&
from.metadata["region"] == to.metadata["region"]
```

## Type Coercion

CEL is strongly typed. Use these functions to convert types:

### String to Integer

```cel theme={null}
int(resource.metadata["replicas"]) > 3
```

### Integer to String

```cel theme={null}
string(42) == resource.metadata["count"]
```

### Type Checking

```cel theme={null}
type(resource.metadata["count"]) == string
```

## Error Handling

### Safe Field Access

Always use `has()` for optional fields to avoid runtime errors:

```cel theme={null}
# Bad: may error if "team" doesn't exist
resource.metadata["team"] == "platform"

# Good: safe access
has(resource.metadata["team"]) && resource.metadata["team"] == "platform"
```

### Default Values

Provide defaults for optional fields:

```cel theme={null}
# Use empty string as default
(has(resource.metadata["tier"]) ? resource.metadata["tier"] : "standard") == "critical"
```

## Performance Tips

### Prefer Simple Expressions

```cel theme={null}
# Fast: simple equality
resource.metadata["env"] == "production"

# Slower: regex matching
resource.identifier.matches("^prod-.*")
```

### Short-Circuit Evaluation

CEL uses short-circuit evaluation. Put cheap checks first:

```cel theme={null}
# Good: check env first (fast), then regex (slow)
resource.metadata["env"] == "production" && 
resource.identifier.matches("^prod-cluster-[0-9]+$")
```

### Avoid Redundant Checks

```cel theme={null}
# Bad: redundant
resource.kind == "KubernetesCluster" && 
resource.kind != "VM"

# Good: single check
resource.kind == "KubernetesCluster"
```

## Debugging

### Test Expressions

Use the API to test your CEL expressions:

```bash theme={null}
curl -X POST "https://your-ctrlplane-instance.com/api/v1/workspaces/{workspaceId}/resources/query" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": "resource.metadata[\"environment\"] == \"production\""
  }'
```

### Common Errors

| Error                  | Cause                     | Fix                    |
| ---------------------- | ------------------------- | ---------------------- |
| `no such key`          | Accessing missing map key | Use `has()` check      |
| `type mismatch`        | Comparing different types | Use type coercion      |
| `syntax error`         | Invalid CEL syntax        | Check quotes, brackets |
| `undeclared reference` | Unknown variable          | Check variable names   |

### Escaping Quotes

In YAML, escape quotes properly:

```yaml theme={null}
# Single quotes around expression, double quotes inside
resourceSelector: 'resource.metadata["environment"] == "production"'

# Or use YAML literal block
resourceSelector: |
  resource.metadata["environment"] == "production"
```

In JSON:

```json theme={null}
{
  "filter": "resource.metadata[\"environment\"] == \"production\""
}
```

## Reference

### Reserved Keywords

These words are reserved in CEL:

* `true`, `false`, `null`
* `in`
* `as`
* `break`, `const`, `continue`, `else`, `for`, `function`, `if`, `import`, `let`, `loop`, `package`, `namespace`, `return`, `var`, `void`, `while`

### Operator Precedence

From highest to lowest:

1. `()` - Grouping
2. `.` `[]` - Member access
3. `-` `!` - Unary
4. `*` `/` `%` - Multiplicative
5. `+` `-` - Additive
6. `<` `<=` `>` `>=` `in` - Relational
7. `==` `!=` - Equality
8. `&&` - Logical AND
9. `||` - Logical OR
10. `?:` - Conditional

## Next Steps

<CardGroup>
  <Card title="Selectors" icon="filter" href="../concepts/selectors">
    Use CEL in selectors
  </Card>

  <Card title="Relationships" icon="diagram-project" href="../inventory/relationships">
    CEL in relationship rules
  </Card>

  <Card title="Policies" icon="shield" href="../policies/overview">
    Target policies with CEL
  </Card>

  <Card title="Environments" icon="layer-group" href="../concepts/environments">
    Dynamic environment membership
  </Card>
</CardGroup>


# Glossary & Reference
Source: https://docs.ctrlplane.dev/reference/glossary

Complete reference for all Ctrlplane entities and concepts

This page provides definitions and details for every entity in Ctrlplane. For a
gentler introduction, see [5-Minute Overview](/overview).

## Entity Overview

```
System
  ├── Environments (group resources via selectors)
  ├── Deployments (what to deploy)
  │     └── Versions (specific builds)
  └── Resources (deployment targets)

Release Target = Deployment × Environment × Resource

Release = Version deployed to a Release Target
  └── Job (executed by Job Agent)

Policy → Rules that govern releases
```

## System

A **System** is a logical grouping of related deployments, environments, and
resources. Think of it as a workspace for a product or team.

| Property      | Description                  |
| ------------- | ---------------------------- |
| `name`        | Display name                 |
| `slug`        | URL-friendly identifier      |
| `description` | What this system encompasses |

**Example**: "E-commerce Platform" system containing API, Frontend, and Payment
deployments.

**When to create a System**: One per product, platform, or team boundary.

## Resource

A **Resource** is a deployment target—the actual infrastructure where your code
runs.

| Property     | Description                                    |
| ------------ | ---------------------------------------------- |
| `name`       | Human-readable name                            |
| `kind`       | Type (e.g., `KubernetesCluster`, `AWS/Lambda`) |
| `identifier` | Unique identifier                              |
| `metadata`   | Key-value pairs for classification             |
| `config`     | Resource-specific configuration                |
| `version`    | Current version/state                          |

**Examples**: Kubernetes cluster, EC2 instance, Lambda function, VM.

**How created**: Via Resource Providers (auto-sync from K8s, AWS, GCP) or API.

```yaml theme={null}
name: prod-us-east-1
kind: KubernetesCluster
identifier: k8s-prod-use1
metadata:
  region: us-east-1
  environment: production
  tier: critical
config:
  server: https://k8s.example.com
```

## Environment

An **Environment** represents a logical deployment stage (dev, staging, prod)
that groups resources using selectors.

| Property           | Description                                 |
| ------------------ | ------------------------------------------- |
| `name`             | Environment name                            |
| `systemId`         | Parent system                               |
| `resourceSelector` | Selector determining which resources belong |
| `directory`        | Optional path for hierarchical organization |

**Key concept**: Environments are *dynamic*. When you add a new resource
matching the selector, it automatically joins the environment.

```yaml theme={null}
name: Production
resourceSelector: resource.metadata["environment"] == "production"
```

## Deployment

A **Deployment** represents a service or application you want to deploy.

| Property           | Description                                      |
| ------------------ | ------------------------------------------------ |
| `name`             | Deployment name                                  |
| `slug`             | URL-friendly identifier                          |
| `description`      | What this deployment does                        |
| `systemId`         | Parent system                                    |
| `resourceSelector` | Optional filter for which resources can run this |
| `jobAgentId`       | Which job agent executes deployments             |
| `jobAgentConfig`   | Configuration passed to the job agent            |

**Examples**: "API Service", "Frontend App", "Payment Processor".

## Version

A **Version** is a specific build or release of a deployment, typically created
by your CI pipeline.

| Property         | Description                                       |
| ---------------- | ------------------------------------------------- |
| `deploymentId`   | Parent deployment                                 |
| `tag`            | Version identifier (e.g., `v1.2.3`, `sha-abc123`) |
| `name`           | Optional human-readable name                      |
| `status`         | `building`, `ready`, or `failed`                  |
| `metadata`       | Arbitrary metadata (git commit, build number)     |
| `config`         | Version-specific configuration                    |
| `jobAgentConfig` | Overrides deployment's job agent config           |

**Status meanings**:

* `building` — Still being built, won't be deployed
* `ready` — Ready for deployment (default for policies)
* `failed` — Build failed, won't be deployed

```bash theme={null}
# CI creates a version after building
curl -X POST ".../deployments/{id}/versions" \
  -d '{
    "tag": "v1.2.3",
    "status": "ready",
    "metadata": {"commit": "abc123"}
  }'
```

## Release Target

A **Release Target** is the combination of a Deployment, Environment, and
Resource. It represents a specific place where a deployment can be released.

```
Release Target = Deployment × Environment × Resource
```

**Example**:

* Deployment: "API Service"
* Environment: "Production"
* Resource: "us-east-1 cluster"
* **Release Target**: "API Service on Production/us-east-1"

**Automatic creation**: Release targets are computed from the intersection of:

1. Environment's resource selector → which resources
2. Deployment's resource selector (if any) → further filtering

## Release

A **Release** is an instance of deploying a specific Version to a Release
Target.

| Property        | Description                  |
| --------------- | ---------------------------- |
| `versionId`     | Which version to deploy      |
| `deploymentId`  | Which deployment             |
| `environmentId` | Which environment            |
| `resourceId`    | Which resource               |
| `createdAt`     | When the release was created |

Releases do not carry their own status. The state of a release is inferred from
the **Release Target State** — which tracks the desired release, current release,
and latest job for each target.

## Job

A **Job** is the actual deployment task executed by a Job Agent.

| Property     | Description                                     |
| ------------ | ----------------------------------------------- |
| `releaseId`  | The release this job deploys                    |
| `jobAgentId` | Which agent executes this job                   |
| `status`     | `pending`, `in_progress`, `completed`, `failed` |
| `externalId` | External identifier (e.g., GitHub run ID)       |
| `message`    | Status message or error details                 |

**Job lifecycle**:

1. Ctrlplane creates job for approved release
2. Job agent polls and receives the job
3. Agent acknowledges and executes
4. Agent updates status as it progresses
5. Agent marks completed or failed

## Job Agent

A **Job Agent** is the executor that performs deployments. It bridges Ctrlplane
to your infrastructure.

| Agent Type      | What It Does                           |
| --------------- | -------------------------------------- |
| GitHub Actions  | Triggers workflow dispatch             |
| ArgoCD          | Creates/syncs ArgoCD Applications      |
| Terraform Cloud | Creates workspaces, triggers runs      |
| Kubernetes      | Applies manifests directly             |
| ArgoWorkflows   | Applies an inline workflow or template |

## Policy

A **Policy** defines rules governing when and how deployments happen.

| Component   | Description                                  |
| ----------- | -------------------------------------------- |
| `name`      | Policy name                                  |
| `selectors` | Which release targets this policy applies to |
| `rules`     | The specific rules to enforce                |

**Policy types**:

| Type                    | Description                                   |
| ----------------------- | --------------------------------------------- |
| Approval                | Requires manual sign-off                      |
| Environment Progression | Wait for another environment to succeed first |
| Gradual Rollout         | Deploy to targets sequentially with delays    |
| Deployment Window       | Only deploy during certain hours              |
| Version Selector        | Filter which versions can deploy              |
| Version Cooldown        | Minimum time between deployments              |
| Deployment Dependency   | Wait for another deployment to complete       |
| Verification            | Check metrics after deployment                |

**Policy evaluation**: When a release target needs deployment:

1. Find all policies matching the target
2. Evaluate each rule
3. All pass → create job
4. Any requires action → release is pending
5. Any denies → release is blocked

## Selector

**Selectors** are query expressions used to match resources, environments, or
deployments.

```yaml theme={null}
# Match resources in production
resource.metadata["environment"] == "production"

# Match critical deployments in any region
deployment.metadata["tier"] == "critical" &&
  resource.metadata["region"] in ["us-east-1", "eu-west-1"]
```

Used in:

* Environment resource selectors
* Deployment resource selectors
* Policy target selectors

See [Selectors](/concepts/selectors) for full syntax.

## Variables

**Variables** provide dynamic configuration for deployments.

| Type                 | Description                                 |
| -------------------- | ------------------------------------------- |
| Deployment Variables | Defined per deployment, vary by environment |
| Resource Variables   | Defined on resources                        |

```json theme={null}
{
  "deploymentVariable": "replicas",
  "values": [
    { "environmentId": "dev", "value": "1" },
    { "environmentId": "prod", "value": "3" }
  ]
}
```

## Quick Reference Table

| Entity         | What It Is                                      |
| -------------- | ----------------------------------------------- |
| System         | Workspace grouping deployments and environments |
| Resource       | Deployment target (cluster, VM, function)       |
| Environment    | Logical stage grouping resources via selectors  |
| Deployment     | Service/app to deploy                           |
| Version        | Specific build of a deployment                  |
| Release Target | Deployment × Environment × Resource             |
| Release        | Version being deployed to a release target      |
| Job            | Execution task sent to a job agent              |
| Job Agent      | Executor (ArgoCD, GitHub Actions, etc.)         |
| Policy         | Rules controlling deployments                   |
| Selector       | Query expression matching entities              |

## Next Steps

* [5-Minute Overview](/overview) — Understand the flow
* [Quickstart](/quickstart) — Hands-on tutorial
* [Selectors](/concepts/selectors) — Deep dive into selector syntax
* [Policies](/policies/overview) — Configure deployment rules


# RFC 0001: Scoped Versions
Source: https://docs.ctrlplane.dev/rfc/0001-scoped-versions



| Category | Status               | Created    | Author        |
| -------- | -------------------- | ---------- | ------------- |
| Policies | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add an optional `targetSelector` field to deployment versions that limits which
release targets a version flows through the promotion lifecycle for. This allows
deployers to express "this version only affects these targets" at creation time,
so unaffected targets skip the full policy pipeline entirely.

## Motivation

Ctrlplane already handles two kinds of changes differently:

* **Variable changes** roll out instantly. When a deployment variable or
  resource variable is updated, the affected release targets are re-reconciled
  with the new variable values. The version hasn't changed, so policies that
  already passed (approval, environment progression, verification) remain
  satisfied. The new release is created with updated variables and a job is
  dispatched immediately.

* **Version changes** go through the full promotion lifecycle. When a new
  deployment version is created with `status: ready`, ctrlplane creates releases
  for **every** release target in the deployment's matrix (Deployment x
  Environment x Resource). Each release goes through environment progression,
  approval gates, verification, gradual rollout, and cooldown before a job is
  created.

The problem is that many version changes only affect a subset of release
targets. A hotfix for a single region, a config embedded in the version for one
service variant, a change to a Helm chart that only impacts certain clusters —
all of these trigger the full promotion lifecycle across **all** targets.

This creates unnecessary latency. A deployer who knows their change only impacts
3 out of 50 clusters must still wait for staging verification, production
approval, and gradual rollout to complete across all 50. Unlike variable
changes, there is no way to express "this version change is narrow" — every
version is treated as a full rollout.

### Why ctrlplane cannot derive version impact automatically

For variable changes, ctrlplane can detect impact mechanically: it resolves the
new variable values, compares them to the current release's variables, and only
creates new releases where the resolved values actually differ. This is why
variable changes can roll out instantly — ctrlplane knows exactly what changed.

Version changes are fundamentally different. A release is defined as
`Version + Environment + Resource + Resolved Variables`. When a new version is
created, the version component is always new — that is the entire reason the
release exists. Even if every resolved variable is identical across targets, the
version ID differs, so every release is "different" from ctrlplane's
perspective. You cannot diff away the version itself.

The knowledge of which targets are truly impacted by a version change comes from
the deployer's understanding of what the change *means* — which config files
changed in the Helm chart, which services are affected by the new image, which
regions need the update. This is semantic knowledge about the change that exists
outside ctrlplane's data model. Ctrlplane sees a new version and treats it as a
new version for all targets; it cannot know that "this Helm chart change only
affects the payment service" or "this image bump doesn't change behavior for
clusters running the old schema."

Scoped versions acknowledge this reality by giving the deployer a structured way
to express their knowledge, rather than trying to derive it mechanically. The
same way ctrlplane already trusts that variable selectors correctly express
which targets a variable value applies to, scoped versions let the deployer
express which targets a version applies to.

### Comparison with existing mechanisms

**Version Selectors** are policy rules that answer "is this version *allowed* to
deploy to this target?" They are eligibility gates — a version that fails a
selector shows as **blocked/denied** in the UI and in rule evaluations. This is
semantically wrong for the scoped version use case: the version is not *bad* for
unaffected targets, it is simply *irrelevant*. Version selectors also don't
exempt matching targets from other policy rules — a version that passes the
selector still goes through the full promotion chain.

**Policy Skips** allow bypassing individual policy rules for a version +
environment. They work today, but require the deployer to know specific rule
IDs, create skips per-rule per-environment, and the version still appears in the
evaluation pipeline for every target. They are an escape hatch, not a
first-class workflow.

**Scoped Versions** operate *before* the policy pipeline. The reconciler skips
the version entirely for non-matching targets — no releases created, no policy
evaluations run, no "denied" entries in the UI. The intent ("this version is for
these targets") lives on the version itself, making it auditable and
declarative.

## Proposal

### Schema

Add an optional `target_selector` column to the `deployment_version` table:

```sql theme={null}
ALTER TABLE deployment_version
  ADD COLUMN target_selector TEXT;
```

When `NULL`, the version targets all release targets (current behavior). When
set, it contains a CEL expression evaluated against the release target's
resource, environment, and deployment.

### API

Extend the version creation endpoints to accept the new field.

**REST API:**

```
POST /v1/deployments/{deploymentId}/versions
```

```json theme={null}
{
  "tag": "v1.2.3-hotfix",
  "status": "ready",
  "targetSelector": "resource.metadata['region'] == 'us-east-1'",
  "metadata": {
    "commit": "abc123",
    "scope": "us-east-1 payment hotfix"
  }
}
```

**Terraform:**

```hcl theme={null}
resource "ctrlplane_deployment_version" "hotfix" {
  deployment_id   = ctrlplane_deployment.api.id
  tag             = "v1.2.3-hotfix"
  status          = "ready"
  target_selector = "resource.metadata['region'] == 'us-east-1'"
}
```

The CEL expression has access to the same variables as version selectors:
`resource`, `environment`, and `deployment`.

### Reconciler changes

In the desired release reconciler, the `findDeployableVersion` function iterates
candidate versions newest-first and evaluates policy rules. The target selector
check should be inserted **before** policy evaluation, as a pre-filter on the
candidate version list:

```
loadInput
  → getCandidateVersions
  → filterByTargetSelector   ← NEW: remove versions whose targetSelector
  → findDeployableVersion       does not match this release target
  → resolveVariables
  → persistRelease
```

Concretely, in `reconcile.go`, after `GetCandidateVersions` returns, filter the
list:

```go theme={null}
func (r *reconciler) filterByTargetSelector(ctx context.Context) error {
    if len(r.versions) == 0 {
        return nil
    }

    filtered := make([]*oapi.DeploymentVersion, 0, len(r.versions))
    for _, v := range r.versions {
        if v.TargetSelector == "" {
            filtered = append(filtered, v)
            continue
        }
        matches, err := selector.MatchCEL(ctx, v.TargetSelector, r.scope)
        if err != nil {
            log.Warn("target selector eval failed, including version",
                "version", v.Id, "error", err)
            filtered = append(filtered, v)
            continue
        }
        if matches {
            filtered = append(filtered, v)
        }
    }
    r.versions = filtered
    return nil
}
```

Versions with a `targetSelector` that does not match the current release target
are silently removed from the candidate list. The reconciler then proceeds as
normal with the remaining candidates. If no candidates remain, the release
target keeps its current state.

### UI

The web UI should surface scoped versions in a few places:

1. **Version list** — Show a badge or indicator when a version has a
   `targetSelector`, with the expression visible on hover.
2. **Release target view** — When a version is scoped and doesn't match a
   target, it should not appear in that target's version evaluation list at all
   (as opposed to appearing as "denied").
3. **Version creation** — Optionally expose the `targetSelector` field in the UI
   when creating versions manually.

### Behavior with other policy rules

Scoped versions interact cleanly with existing policy rules:

* **Environment progression:** Only evaluated for targets that match the scope.
  If a scoped version targets production directly and no staging targets match,
  the environment progression rule is only evaluated for production targets. The
  deployer is responsible for ensuring this makes sense — the scope is an
  explicit declaration of intent.
* **Approval:** Approvals are per-environment. Only environments with matching
  targets will require approval.
* **Gradual rollout:** Rollout only applies across matching targets, naturally
  reducing the rollout surface.
* **Version cooldown:** Evaluated per-target as before, but only for targets in
  scope.

### Fallback behavior

If `targetSelector` evaluation fails (malformed CEL, missing fields), the
version should be **included** in the candidate list (fail-open). This prevents
a typo in a selector from silently dropping a version for all targets. The
failure should be logged as a warning.

## Examples

### Hotfix for a single region

```bash theme={null}
curl -X POST ".../deployments/{id}/versions" \
  -d '{
    "tag": "v1.2.3-hotfix-use1",
    "status": "ready",
    "targetSelector": "resource.metadata[\"region\"] == \"us-east-1\""
  }'
```

Only us-east-1 release targets enter the promotion pipeline. All other targets
remain on their current version undisturbed.

### Config change for a specific environment

```bash theme={null}
curl -X POST ".../deployments/{id}/versions" \
  -d '{
    "tag": "v2.0.1-staging-config",
    "status": "ready",
    "targetSelector": "environment.name == \"staging\""
  }'
```

Only staging targets are considered. This version never reaches production
targets, so no environment progression or production approval is triggered.

### Broad rollout (default behavior)

```bash theme={null}
curl -X POST ".../deployments/{id}/versions" \
  -d '{
    "tag": "v2.1.0",
    "status": "ready"
  }'
```

No `targetSelector` — all release targets are considered. Identical to current
behavior.

## Migration

* The schema change is additive (`ADD COLUMN ... NULL`), requiring no data
  migration.
* Existing versions have `target_selector = NULL`, preserving current behavior.
* No changes to existing policies or release targets are needed.
* The reconciler change is backwards-compatible: versions without a selector
  pass through the filter unchanged.

## Open Questions

1. **Should scoped versions interact with environment progression?** If a
   version scopes to production only, should environment progression rules block
   it (staging hasn't seen it) or should the scope be treated as an explicit
   override of progression? The current proposal lets the deployer handle this —
   they can combine the scope with policy skips if needed.

2. **Should there be a permission or policy guard on scoping?** Scoped versions
   let deployers bypass the normal promotion surface area. Organizations may
   want to restrict who can create scoped versions, or require that scoped
   versions still pass through certain gates.

3. **Should the selector support resource-only, or also environment and
   deployment fields?** The proposal includes all three for flexibility, but
   simpler scoping (resource-only) might be sufficient and easier to reason
   about.

4. **Naming:** `targetSelector` vs `scope` vs `affectedTargets` — what conveys
   the intent most clearly?


# RFC 0002: Plan-Based Diff Detection
Source: https://docs.ctrlplane.dev/rfc/0002-plan-based-diff-detection



| Category | Status               | Created    | Author        |
| -------- | -------------------- | ---------- | ------------- |
| Policies | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add a `Plannable` interface to job agents that lets ctrlplane compute the
rendered deployment output for a release target *without* dispatching a job. By
comparing the rendered output hash of a proposed version against the hash of the
currently deployed release, ctrlplane can mechanically determine which release
targets are actually affected by a version change. Unaffected targets can then
be fast-tracked through the promotion lifecycle.

## Motivation

RFC 0001 (Scoped Versions) introduces a way for deployers to *declare* which
targets a version affects. This works well when the deployer knows the impact
upfront — a regional hotfix, a single-service config change. But it relies on
the deployer providing accurate scope. If the scope is wrong, targets are either
unnecessarily delayed (too broad) or silently skipped (too narrow).

The external systems ctrlplane dispatches to — ArgoCD, Terraform Cloud, Helm,
Kubernetes — already know how to compute what a deployment *would* produce
without actually applying it. ArgoCD renders Application manifests from
templates. Terraform produces execution plans. Helm has `helm template`. These
systems can answer the question "would this version change anything for this
target?" with mechanical precision.

Today, ctrlplane cannot leverage this knowledge. The rendering happens inside
the job agent at dispatch time, and the result is never captured or compared.
ctrlplane treats every new version as a change for every target because it
operates on version identity (version ID differs → release differs), not on
rendered output identity (rendered manifest is the same → nothing changed).

### Why version identity is insufficient

A release's content hash (`Release.ContentHash()`) includes the version ID and
tag:

```go theme={null}
func (r *Release) ContentHash() string {
    var sb strings.Builder
    sb.WriteString(r.Version.Id)
    sb.WriteString(r.Version.Tag)
    // ... variables and release target key
}
```

This means two releases with different versions *always* have different content
hashes, even if the rendered deployment output is byte-for-byte identical. The
content hash answers "is this the same release?" but not "would this produce the
same deployed state?"

### Why rendering is the right level to compare

The rendered output is what actually gets applied to the target system.
Crucially, this is not the intermediate representation ctrlplane produces (like
an ArgoCD Application CRD or a Terraform variable file) — it is the *final*
output that the external system produces after it processes that intermediate
input. For ArgoCD, this means the Kubernetes manifests after fetching the git
repo and rendering the Helm chart. For Terraform, this means the execution plan
after evaluating all modules and state.

ctrlplane's in-process template rendering (e.g., `TemplateApplication`) produces
the *input* to the external system, not the deployed output. A version change
almost always changes this input (the `targetRevision`, the image tag in Helm
values, etc.). But the external system may still produce identical output — for
example, when a git commit only modifies files for a different service than the
one this Helm chart deploys.

The only way to know whether the deployed state would actually change is to ask
the external system to render the final output. This is what `terraform plan`,
`argocd app diff`, and `helm template` do. Plan-based diff detection brings this
capability into ctrlplane's promotion lifecycle by delegating the rendering to
the system that owns it.

### Relationship to RFC 0001

Scoped versions (RFC 0001) and plan-based diff detection are complementary:

* **Scoped versions** are fast and explicit — the deployer states intent, the
  reconciler filters instantly, no external calls needed.
* **Plan-based diffs** are accurate and automatic — the external system computes
  impact, no deployer knowledge required, but adds latency from the plan call.

A typical workflow might use scoped versions as the primary mechanism and
plan-based diffs as a validation step or as the basis for auto-generating the
scope.

## Proposal

### New interface: `Plannable`

Add an optional interface to the job agent type system alongside the existing
`Dispatchable` and `Verifiable`:

```go theme={null}
// Plannable is optionally implemented by a Dispatchable to compute the
// rendered deployment output without dispatching a job. The reconciler
// uses this to detect whether a version change would produce a different
// deployed state for a given release target.
type Plannable interface {
    Plan(ctx context.Context, dispatchCtx *oapi.DispatchContext) (*PlanResult, error)
}

type PlanResult struct {
    // ContentHash is a deterministic hash of the rendered deployment output.
    // Two plans with the same ContentHash produce identical deployed state.
    ContentHash string

    // HasChanges indicates whether the rendered output differs from the
    // currently deployed state. When false, the target is unaffected.
    HasChanges bool

    // Diff is an optional human-readable summary of what changed. Stored
    // for audit and displayed in the UI. May be empty for no-diff results.
    Diff string
}
```

This follows the same pattern as `Verifiable`:

```go theme={null}
// Existing pattern in types/types.go:
type Dispatchable interface {
    Type() string
    Dispatch(ctx context.Context, job *oapi.Job) error
}

type Verifiable interface {
    Verifications(config oapi.JobAgentConfig) ([]oapi.VerificationMetricSpec, error)
}

// New:
type Plannable interface {
    Plan(ctx context.Context, dispatchCtx *oapi.DispatchContext) (*PlanResult, error)
}
```

### Registry extension

The job agent registry already checks for optional interfaces. Add a `Plan`
method following the same pattern as `AgentVerifications`:

```go theme={null}
func (r *Registry) Plan(
    ctx context.Context,
    agentType string,
    dispatchCtx *oapi.DispatchContext,
) (*types.PlanResult, error) {
    dispatcher, ok := r.dispatchers[agentType]
    if !ok {
        return nil, nil
    }

    p, ok := dispatcher.(types.Plannable)
    if !ok {
        return nil, nil
    }

    return p.Plan(ctx, dispatchCtx)
}
```

When an agent does not implement `Plannable`, the registry returns nil and the
reconciler falls back to treating the version as a change for all targets
(current behavior).

### Schema

Store the rendered content hash on the release target state so it can be
compared against future plan results:

```sql theme={null}
ALTER TABLE release_target
  ADD COLUMN rendered_content_hash TEXT;
```

This column is updated when a job completes successfully. It represents the hash
of the output that was actually deployed.

Additionally, store plan results for audit and UI display:

```sql theme={null}
CREATE TABLE release_target_plan (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    deployment_id UUID NOT NULL REFERENCES deployment(id) ON DELETE CASCADE,
    environment_id UUID NOT NULL REFERENCES environment(id) ON DELETE CASCADE,
    resource_id UUID NOT NULL REFERENCES resource(id) ON DELETE CASCADE,
    version_id UUID NOT NULL REFERENCES deployment_version(id) ON DELETE CASCADE,
    content_hash TEXT NOT NULL,
    has_changes BOOLEAN NOT NULL,
    diff TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```

### Reconciler integration

The plan step fits into the desired release reconciler as an optional phase
between candidate selection and policy evaluation:

```text theme={null}
loadInput
  → getCandidateVersions
  → filterByTargetSelector         (RFC 0001)
  → computePlanForTopCandidate     ← NEW
  → findDeployableVersion
  → resolveVariables
  → persistRelease
```

The plan is computed only for the top candidate version (newest first) to
minimize external calls. If the plan shows no changes, the reconciler can
either:

1. Skip the version for this target (move to the next candidate)
2. Fast-track the version through policy evaluation (auto-satisfy gates)

Which behavior applies depends on the policy configuration (see "Policy
integration" below).

```go theme={null}
func (r *reconciler) computePlan(ctx context.Context) error {
    if len(r.versions) == 0 {
        return nil
    }

    version := r.versions[0]

    // Build a dispatch context for the candidate version
    dispatchCtx, err := r.buildDispatchContext(ctx, version)
    if err != nil {
        log.Warn("failed to build dispatch context for plan", "error", err)
        return nil // fail-open: proceed without plan
    }

    // Ask the job agent to compute the rendered output
    result, err := r.planner.Plan(ctx, r.agentType, dispatchCtx)
    if err != nil {
        log.Warn("plan failed", "error", err)
        return nil // fail-open
    }
    if result == nil {
        return nil // agent does not support planning
    }

    // Store the plan result
    r.planResult = result

    // Compare against the currently deployed hash
    if r.currentRenderedHash != "" && result.ContentHash == r.currentRenderedHash {
        result.HasChanges = false
    }

    return nil
}
```

### Policy integration

Plan results feed into the policy pipeline through a new optional policy rule
type: `diffCheck`. This rule evaluates the plan result and can auto-satisfy
other gates when no diff is detected:

```hcl theme={null}
resource "ctrlplane_policy" "fast_track_no_diff" {
  name     = "Fast-track unchanged targets"
  selector = "environment.name == 'production'"

  diff_check {
    skip_when_no_diff = [
      "environment_progression",
      "approval",
      "verification",
    ]
  }
}
```

When the plan result for a release target shows `HasChanges = false`, the rules
listed in `skip_when_no_diff` are automatically satisfied. The version still
advances through the pipeline (the release is created, the release target state
updates), but blocking gates are bypassed.

If no `diffCheck` policy is configured, plan results are informational only —
stored for audit and displayed in the UI but not used to alter promotion flow.

The `diffCheck` evaluator would be added to the evaluator set in `policyeval.go`
alongside the existing evaluators:

```go theme={null}
func RuleTypes() []string {
    return []string{
        (&versionselector.Evaluator{}).RuleType(),
        (&approval.AnyApprovalEvaluator{}).RuleType(),
        (&environmentprogression.EnvironmentProgressionEvaluator{}).RuleType(),
        (&gradualrollout.GradualRolloutEvaluator{}).RuleType(),
        (&deploymentdependency.DeploymentDependencyEvaluator{}).RuleType(),
        (&deploymentwindow.DeploymentWindowEvaluator{}).RuleType(),
        (&versioncooldown.VersionCooldownEvaluator{}).RuleType(),
        // NEW:
        (&diffcheck.DiffCheckEvaluator{}).RuleType(),
    }
}
```

### Agent implementations

#### ArgoCD

The ArgoCD agent's in-process `TemplateApplication` function renders a Go
template into an ArgoCD Application CRD. This is **not** the right level to
diff. The Application spec contains fields like `targetRevision` and
`helm.values` that reference `version.tag` — so the rendered Application CRD
will always differ between versions, even when the final deployed manifests are
identical.

The actual deployed state is what ArgoCD produces *from* the Application spec:
it fetches the git repo at the specified revision, renders the Helm chart (or
kustomize overlay, or plain manifests), and produces the final Kubernetes
manifests that get applied to the cluster. Two different git revisions can
produce identical rendered manifests if the files that changed in the commit are
irrelevant to the chart or overlay being used.

To compute a real diff, the `Plan` implementation must call the ArgoCD API to
get the fully rendered manifests. The ArgoCD Go client already used by the agent
(`ApplicationServiceClient`) exposes `GetManifests` for exactly this:

```go theme={null}
func (a *ArgoApplication) Plan(
    ctx context.Context,
    dispatchCtx *oapi.DispatchContext,
) (*types.PlanResult, error) {
    serverAddr, apiKey, template, err := ParseJobAgentConfig(
        dispatchCtx.JobAgentConfig,
    )
    if err != nil {
        return nil, err
    }

    // First, render the Application CRD from the dispatch context
    // (same as dispatch time)
    app, err := TemplateApplication(dispatchCtx, template)
    if err != nil {
        return nil, err
    }
    MakeApplicationK8sCompatible(app)

    // Connect to ArgoCD
    client, err := argocdclient.NewClient(&argocdclient.ClientOptions{
        ServerAddr: serverAddr,
        AuthToken:  apiKey,
    })
    if err != nil {
        return nil, fmt.Errorf("create ArgoCD client: %w", err)
    }
    ioCloser, appClient, err := client.NewApplicationClient()
    if err != nil {
        return nil, fmt.Errorf("create application client: %w", err)
    }
    defer ioCloser.Close()

    // Get the fully rendered manifests that ArgoCD would produce
    // for this Application spec. ArgoCD fetches the git repo,
    // renders Helm/kustomize/plain manifests, and returns the
    // final Kubernetes resources.
    manifests, err := appClient.GetManifests(ctx,
        &argocdapplication.ApplicationManifestQuery{
            Name:     &app.Name,
            Revision: &app.Spec.Source.TargetRevision,
        },
    )
    if err != nil {
        return nil, fmt.Errorf("get manifests from ArgoCD: %w", err)
    }

    // Hash the rendered manifests (sorted for determinism)
    rendered := sortAndJoinManifests(manifests.Manifests)
    hash := sha256.Sum256([]byte(rendered))

    return &types.PlanResult{
        ContentHash: hex.EncodeToString(hash[:]),
        HasChanges:  true, // caller compares against stored hash
    }, nil
}

func sortAndJoinManifests(manifests []string) string {
    sorted := make([]string, len(manifests))
    copy(sorted, manifests)
    sort.Strings(sorted)
    return strings.Join(sorted, "\n---\n")
}
```

This requires a network call to the ArgoCD server, which in turn fetches the git
repo and renders the chart. The latency depends on the size of the repo and
chart complexity, but is typically 1-10 seconds. ArgoCD caches rendered
manifests aggressively, so repeated plans for the same revision are fast.

For applications that don't yet exist in ArgoCD (first deploy), the plan step
can use ArgoCD's dry-run create or fall back to treating the version as changed.

**Why in-process rendering is insufficient:** The Go template in the job agent
config produces the Application CRD — it defines *where* to look for manifests
(which repo, which revision, which Helm values). It does not produce the actual
Kubernetes resources. A template like:

```yaml theme={null}
spec:
  source:
    repoURL: https://github.com/org/charts
    targetRevisicon: "{{ .release.version.tag }}"
    helm:
      values: |
        replicas: {{ .release.variables.REPLICA_COUNT }}
```

will always produce a different Application CRD when `version.tag` changes. But
if the Helm chart at the new tag only changed a values file for a different
service, the rendered Kubernetes manifests for *this* resource may be identical.
Only ArgoCD — which actually fetches and renders the chart — can tell you that.

#### Terraform Cloud

Terraform Cloud has native plan support. The `Plan` implementation would trigger
a speculative plan run via the API and return the plan's resource change
summary:

```go theme={null}
func (t *TerraformCloud) Plan(
    ctx context.Context,
    dispatchCtx *oapi.DispatchContext,
) (*types.PlanResult, error) {
    // Trigger a speculative plan (does not apply)
    run, err := t.client.CreateRun(ctx, RunConfig{
        IsDestroy:   false,
        PlanOnly:    true,
        Variables:   dispatchCtx.Variables,
    })
    if err != nil {
        return nil, err
    }

    // Wait for plan to complete
    plan, err := t.client.WaitForPlan(ctx, run.ID)
    if err != nil {
        return nil, err
    }

    hasChanges := plan.ResourceAdditions > 0 ||
        plan.ResourceChanges > 0 ||
        plan.ResourceDestructions > 0

    return &types.PlanResult{
        ContentHash: plan.StateHash,
        HasChanges:  hasChanges,
        Diff:        plan.Summary,
    }, nil
}
```

This involves a network call and takes longer (seconds to minutes). The
reconciler should handle this asynchronously.

#### GitHub Actions

GitHub Actions does not have a native plan/dry-run concept. The agent would not
implement `Plannable`, and the registry returns nil. Targets using GitHub
Actions fall back to current behavior — every version is treated as a change.

### Storing the deployed hash

When a job completes successfully, the reconciler updates the release target's
`rendered_content_hash`:

```go theme={null}
func (s *Setter) UpdateRenderedHash(
    ctx context.Context,
    rt *ReleaseTarget,
    hash string,
) error {
    q := db.GetQueries(ctx)
    return q.UpdateReleaseTargetRenderedHash(ctx, db.UpdateReleaseTargetRenderedHashParams{
        ResourceID:          rt.ResourceID,
        EnvironmentID:       rt.EnvironmentID,
        DeploymentID:        rt.DeploymentID,
        RenderedContentHash: hash,
    })
}
```

For the initial deployment (no previous hash), `HasChanges` defaults to true.

### UI

1. **Release target view** — When a plan result exists, show a "No changes
   detected" or "Changes detected" indicator alongside the version evaluation.
   For targets with no changes, display a muted state to signal the version is
   advancing without operational impact.
2. **Diff viewer** — When `Diff` is populated, provide an expandable panel
   showing the human-readable diff (YAML diff for ArgoCD, resource summary for
   Terraform).
3. **Version detail** — Aggregate plan results across all release targets to
   show "X of Y targets affected" on the version page.

### Async plan execution

All `Plannable` agents involve network calls — ArgoCD must fetch the git repo
and render charts, Terraform Cloud must run a speculative plan. Plans should run
asynchronously:

1. The reconciler enqueues a plan request when it encounters a new candidate
   version for a release target.
2. A plan worker processes the request, calls the agent's `Plan` method, and
   stores the result in `release_target_plan`.
3. On the next reconciliation pass, the stored plan result is available and the
   reconciler uses it to determine diff status.

This follows the same async pattern used by verification metrics, which also
enqueue work items and store results that the reconciler picks up on subsequent
passes.

## Examples

### ArgoCD: Helm chart change affecting one service

A deployment manages 20 clusters across 4 environments. A new version points to
a new git commit that updates the Helm chart's `values.yaml` for the payment
service. The ArgoCD Application template sets `targetRevision` to the version
tag.

1. Version `v3.1.0` is created. The git commit behind this tag only modifies
   `charts/payment/values.yaml`.
2. The reconciler enqueues a plan for each release target.
3. For each target, the plan worker renders the ArgoCD Application CRD (which
   differs for every target because `targetRevision` changed), then calls
   ArgoCD's `GetManifests` API to get the fully rendered Kubernetes manifests at
   that revision.
4. For the 4 clusters that deploy the payment chart, ArgoCD's rendered manifests
   differ from the stored hash — `HasChanges = true`.
5. For the 16 clusters that deploy other charts from the same repo, ArgoCD
   renders the same manifests as the previous version (the files that changed
   are irrelevant to their charts) — `HasChanges = false`.
6. The `diffCheck` policy auto-satisfies environment progression and approval
   for the 16 unaffected clusters.
7. The 4 affected clusters go through the full promotion lifecycle.

### Terraform Cloud: Infrastructure change scoped to one region

A Terraform deployment manages infrastructure in 3 regions. A version changes an
IAM policy that only applies to us-east-1.

1. Version `v1.5.0` is created.
2. The reconciler triggers speculative plans for each region's release target.
3. The us-east-1 plan shows 1 resource change. The other two plans show 0
   changes.
4. Only the us-east-1 target enters the full promotion pipeline.

### GitHub Actions: No plan support (fallback)

A deployment uses GitHub Actions as its job agent. GitHub Actions does not
implement `Plannable`.

1. Version `v2.0.0` is created.
2. The reconciler calls `registry.Plan()` — returns nil.
3. All release targets enter the promotion pipeline as usual.
4. No change from current behavior.

## Migration

* The `rendered_content_hash` column is additive and nullable. Existing release
  targets start with `NULL`, meaning the first plan comparison always treats the
  target as changed (fail-open).
* The `release_target_plan` table is new and requires no data migration.
* Agents that do not implement `Plannable` continue to work without changes.
* The `diffCheck` policy rule is optional. Without it, plan results are
  informational only.

## Open Questions

1. **Should plan results block or only fast-track?** The current proposal only
   uses plan results to *skip* policy gates (fast-track). An alternative is to
   *block* versions that show no changes from creating releases at all, similar
   to how scoped versions filter candidates. The risk is that a plan bug could
   prevent legitimate deployments.

2. **Cost of Plans.** Each plan consumes resources. For deployments with many
   release targets, the plan step could generate significant API load. Should
   planning be opt-in per deployment, or rate-limited? For deployments with many
   release targets, the plan step could generate significant API load. Should
   planning be opt-in per deployment, or rate-limited?

3. **Interaction with RFC 0001.** If a version has a `targetSelector` (RFC 0001)
   that excludes a target, should the plan still run for that target? The
   proposed order (filter by target selector, then plan) means excluded targets
   are never planned, which is efficient but means you cannot use plan results
   to validate a target selector's correctness.


# RFC 0003: Lifecycle Brackets
Source: https://docs.ctrlplane.dev/rfc/0003-lifecycle-brackets



| Category | Status               | Created    | Author        |
| -------- | -------------------- | ---------- | ------------- |
| Policies | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add two new policy rule types — **deployment bracket** and **resource
concurrency** — that together enable coordinated multi-deployment rollouts with
cluster-wide capacity limits. Deployment brackets group related deployments so
they execute as a unit against each resource (drain a node once, apply all
upgrades, uncordon). Resource concurrency limits how many resources in a group
can be simultaneously undergoing deployment (only 20% of nodes offline at a
time).

Both are policy rules, not new entity types. They slot into the existing
evaluator pipeline alongside gradual rollout, deployment dependency, and the
other rule types.

## Motivation

When deploying software to Kubernetes nodes, two operational constraints exist
that ctrlplane's current policy engine cannot express:

### Drain cost amortization

Deploying to a node requires draining pods first — evicting workloads,
respecting PodDisruptionBudgets, waiting for graceful termination. This is slow
(minutes to tens of minutes) and disruptive. If multiple deployments target the
same node (kubelet upgrade, containerd upgrade, OS patch), each deployment today
triggers its own independent drain/uncordon cycle because each release target is
evaluated independently.

For 3 deployments across 10 nodes, this means 30 drain cycles instead of 10. The
overhead scales linearly with the number of bracketed deployments.

### Cluster-wide concurrency limits

Only a percentage of nodes in a cluster can be offline simultaneously. If 20% of
a 10-node cluster goes down, the remaining 8 nodes must absorb all workloads.
Exceeding this threshold risks cascading failures.

The existing concurrency control (`ReleaseTargetConcurrencyEvaluator`) enforces
a hard limit of 1 active job per release target:

```go theme={null}
// releasetargetconcurrency.go
func (e *ReleaseTargetConcurrencyEvaluator) Evaluate(
    ctx context.Context, release *oapi.Release,
) *oapi.RuleEvaluation {
    processingJobs := e.store.Jobs.GetJobsInProcessingStateForReleaseTarget(
        &release.ReleaseTarget,
    )
    if len(processingJobs) != 0 {
        return results.NewDeniedResult("Release target has an active job")
    }
    return results.NewAllowedResult("Release target has no active jobs")
}
```

This prevents overlapping jobs on the *same* release target but provides no
cross-resource limit. There is no way to say "at most 2 of these 10 nodes can be
draining at once."

### Why existing rules are insufficient

**Gradual rollout** staggers deployments over time using hash-based position
ordering. It controls *when* each target's turn arrives but not *how many* can
be active simultaneously. If multiple targets' rollout times pass while earlier
targets are still executing (e.g., a slow drain), all of them proceed at once.

**Deployment dependency** checks whether an upstream deployment has succeeded
for the same resource. It controls *ordering* (drain before upgrade, upgrade
before uncordon) but not *grouping*. Each deployment's policy pipeline runs
independently — there is no mechanism to hold all deployments for a resource
until a collection of new versions is ready.

**Version cooldown** batches frequent releases for a *single* deployment. It
does not coordinate across deployments.

These three rules address different dimensions (time, order, frequency) but none
addresses the two missing dimensions: cross-deployment grouping and
cross-resource capacity limits.

## Proposal

### New policy rule: deployment bracket

A deployment bracket groups deployments that should execute as a coordinated
unit per resource. The bracket rule gates deployment until all member
deployments have versions ready, using a configurable collection window.

#### Schema

```sql theme={null}
CREATE TABLE policy_rule_deployment_bracket (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    policy_id UUID NOT NULL REFERENCES policy(id) ON DELETE CASCADE,

    -- CEL selector defining which deployments are co-bracketed.
    -- Evaluated against deployment context.
    deployment_selector TEXT NOT NULL,

    -- How to batch versions across the bracket group.
    readiness_mode TEXT NOT NULL DEFAULT 'collection_window',

    -- Duration of collection/wait window in seconds (e.g., 86400 = 24h).
    readiness_window_seconds INTEGER,

    -- What to do with members that have no new version during window.
    unchanged_member_strategy TEXT NOT NULL DEFAULT 'skip_unchanged',

    -- What to do when a new version group is ready while a previous one
    -- is still executing on a resource.
    overlap_strategy TEXT NOT NULL DEFAULT 'queue',

    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```

**`readiness_mode`** controls when a bracket starts executing:

| Mode                | Behavior                                                                                                                                 |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `collection_window` | Wait `readiness_window_seconds` after the first member version is published. Take the latest version of each member at window close.     |
| `wait_for_all`      | Wait until every member deployment has a new version. Fall back to `unchanged_member_strategy` after `readiness_window_seconds` timeout. |
| `immediate`         | Proceed as soon as any member has a new version. Use latest available version for all members.                                           |

**`unchanged_member_strategy`** controls members with no new version when the
window closes:

| Strategy           | Behavior                                                                                                  |
| ------------------ | --------------------------------------------------------------------------------------------------------- |
| `skip_unchanged`   | Only deploy members that have new versions. Others are no-ops.                                            |
| `redeploy_current` | Re-deploy the currently running version for all members. Useful when hooks have side effects.             |
| `require_all`      | Don't close the window until all members have a new version. Fall back to `skip_unchanged` after timeout. |

**`overlap_strategy`** controls what happens when a new version group becomes
ready while a previous group is still executing on a resource:

| Strategy | Behavior                                                                                                                                               |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `queue`  | Wait for the active group to fully complete (including post-hooks), then start a fresh cycle. Safe default.                                            |
| `merge`  | If the resource is already in a pre-hook state (e.g., drained), deploy the new group's versions before running post-hooks. Avoids double drain cycles. |

#### Version group state

The bracket evaluator needs to track which versions belong to an active
collection window or executing group. A lightweight state table supports this:

```sql theme={null}
CREATE TABLE bracket_version_group (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    bracket_rule_id UUID NOT NULL
        REFERENCES policy_rule_deployment_bracket(id) ON DELETE CASCADE,
    resource_id UUID NOT NULL,
    environment_id UUID NOT NULL,

    -- When collection started (first triggering version)
    collection_started_at TIMESTAMPTZ NOT NULL,
    -- When collection should end
    collection_ends_at TIMESTAMPTZ NOT NULL,

    -- collecting | ready | executing | completed | failed
    status TEXT NOT NULL DEFAULT 'collecting',

    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE bracket_version_group_member (
    group_id UUID NOT NULL
        REFERENCES bracket_version_group(id) ON DELETE CASCADE,
    deployment_id UUID NOT NULL,
    -- NULL while still collecting, set when window closes
    version_id UUID,
    -- When this member's version was locked
    locked_at TIMESTAMPTZ,
    PRIMARY KEY (group_id, deployment_id)
);
```

#### Evaluator

The bracket evaluator implements the standard `Evaluator` interface:

```go theme={null}
type DeploymentBracketEvaluator struct {
    getters   Getters
    ruleId    string
    rule      *oapi.DeploymentBracketRule
    timeGetter func() time.Time
}

func (e *DeploymentBracketEvaluator) ScopeFields() evaluator.ScopeFields {
    return evaluator.ScopeDeployment | evaluator.ScopeVersion | evaluator.ScopeReleaseTarget
}

func (e *DeploymentBracketEvaluator) RuleType() string {
    return evaluator.RuleTypeDeploymentBracket
}
```

The `Evaluate` method performs three checks:

1. **Overlap check** — if a version group is executing for this bracket +
   resource, and the candidate version is not part of that group, return
   `Pending` (for `queue` strategy) or `Allowed` (for `merge`, if pre-hooks are
   complete).

2. **Collection window check** — if no active group exists, find or create a
   `bracket_version_group`. If the window hasn't closed, return `Pending` with
   `NextEvaluationTime` set to `collection_ends_at`.

3. **Readiness check** — if the window has closed, verify the candidate version
   matches the locked version for this deployment in the version group. If so,
   return `Allowed`.

```go theme={null}
func (e *DeploymentBracketEvaluator) Evaluate(
    ctx context.Context,
    scope evaluator.EvaluatorScope,
) *oapi.RuleEvaluation {
    resourceId := scope.ReleaseTarget().ResourceId
    now := e.timeGetter()

    // 1. Check for active executing group
    activeGroup := e.getters.GetActiveVersionGroup(e.ruleId, resourceId)
    if activeGroup != nil && activeGroup.Status == "executing" {
        lockedVersion := activeGroup.GetLockedVersion(scope.Deployment.Id)
        if lockedVersion != nil && lockedVersion.Id == scope.Version.Id {
            return results.NewAllowedResult(
                "Version is part of active bracket group")
        }

        if e.rule.OverlapStrategy == "merge" {
            if e.getters.IsPreHookComplete(activeGroup.Id, resourceId) {
                return results.NewAllowedResult(
                    "Merging into active bracket — resource already prepared")
            }
        }

        return results.NewPendingResult(results.ActionTypeWait,
            fmt.Sprintf("Bracket group %s executing on this resource",
                activeGroup.Id))
    }

    // 2. Find or create collection window
    group := e.getters.GetOrCreateVersionGroup(
        e.ruleId, resourceId, scope.ReleaseTarget().EnvironmentId,
        e.rule.ReadinessWindowSeconds, now,
    )

    if now.Before(group.CollectionEndsAt) {
        return results.NewPendingResult(results.ActionTypeWait,
            fmt.Sprintf("Collection window closes at %s",
                group.CollectionEndsAt.Format(time.RFC3339)),
        ).WithNextEvaluationTime(group.CollectionEndsAt)
    }

    // 3. Window closed — check version lock
    lockedVersion := group.GetLockedVersion(scope.Deployment.Id)
    if lockedVersion == nil {
        switch e.rule.UnchangedMemberStrategy {
        case "skip_unchanged":
            return results.NewAllowedResult(
                "No new version for this member — skipping")
        case "redeploy_current":
            return results.NewAllowedResult(
                "Redeploying current version as part of bracket")
        default:
            return results.NewDeniedResult(
                "No version available for this bracket member")
        }
    }

    if lockedVersion.Id != scope.Version.Id {
        return results.NewDeniedResult(
            "Version does not match locked bracket version")
    }

    return results.NewAllowedResult(
        fmt.Sprintf("Bracket ready — version locked at window close"))
}
```

### New policy rule: resource concurrency

A resource concurrency rule limits how many resources in a group can
simultaneously be undergoing deployment. The group is defined by a CEL selector,
and the limit can be a percentage or absolute count.

#### Schema

```sql theme={null}
CREATE TABLE policy_rule_resource_concurrency (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    policy_id UUID NOT NULL REFERENCES policy(id) ON DELETE CASCADE,

    -- CEL selector defining the concurrency group.
    -- e.g., "resource.metadata['cluster'] == 'prod-east'"
    group_selector TEXT NOT NULL,

    -- 'percentage' or 'count'
    limit_type TEXT NOT NULL,

    -- The limit value (e.g., 20 for 20% or 5 for 5 nodes)
    limit_value INTEGER NOT NULL,

    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```

#### Evaluator

```go theme={null}
type ResourceConcurrencyEvaluator struct {
    getters       Getters
    ruleId        string
    rule          *oapi.ResourceConcurrencyRule
}

func (e *ResourceConcurrencyEvaluator) ScopeFields() evaluator.ScopeFields {
    return evaluator.ScopeResource | evaluator.ScopeReleaseTarget
}

func (e *ResourceConcurrencyEvaluator) Evaluate(
    ctx context.Context,
    scope evaluator.EvaluatorScope,
) *oapi.RuleEvaluation {
    groupResources := e.getters.GetResourcesBySelector(ctx, e.rule.GroupSelector)
    totalInGroup := len(groupResources)

    activeCount := e.getters.CountResourcesWithActiveJobs(ctx, groupResources)

    var maxConcurrent int
    if e.rule.LimitType == "percentage" {
        maxConcurrent = int(math.Ceil(
            float64(totalInGroup) * float64(e.rule.LimitValue) / 100.0,
        ))
    } else {
        maxConcurrent = int(e.rule.LimitValue)
    }

    if activeCount >= maxConcurrent {
        return results.NewPendingResult(results.ActionTypeWait,
            fmt.Sprintf("Concurrency limit reached: %d/%d resources active (max %d)",
                activeCount, maxConcurrent, maxConcurrent))
    }

    return results.NewAllowedResult(
        fmt.Sprintf("Within concurrency limit: %d/%d active (max %d)",
            activeCount, maxConcurrent, maxConcurrent))
}
```

The "active" count includes any resource that has at least one job in a
processing state across any deployment matched by the parent policy's selector.
For brackets, this means a resource in any phase (pre-hook, deploying,
post-hook) counts against the limit.

### Enhancement: scoped deployment dependencies

Today, a deployment dependency rule on a policy applies to every release target
matched by the policy's `selector`. This forces separate policies for each
ordering constraint in a bracket. Adding an optional `applies_to` field lets
multiple dependency rules coexist on one policy:

```sql theme={null}
ALTER TABLE policy_rule_deployment_dependency
  ADD COLUMN applies_to TEXT;
```

When `NULL`, the rule applies to all matched release targets (current behavior).
When set, the rule is only evaluated for release targets whose deployment
matches the CEL expression:

```go theme={null}
func (e *DeploymentDependencyEvaluator) Evaluate(
    ctx context.Context, scope evaluator.EvaluatorScope,
) *oapi.RuleEvaluation {
    if e.rule.AppliesTo != nil {
        matched, err := selector.Match(ctx,
            celToSelector(*e.rule.AppliesTo), scope.Deployment)
        if err != nil || !matched {
            return results.NewAllowedResult(
                "Dependency rule does not apply to this deployment")
        }
    }

    // ...existing evaluation logic unchanged...
}
```

### Bracket-aware gradual rollout

When a bracket rule exists on the same policy as a gradual rollout rule, the
rollout evaluator should hash on `resourceId + bracketRuleId` instead of the
individual release target key. This ensures all deployments in a bracket for a
given resource receive the same rollout position.

The gradual rollout evaluator already receives the full policy context through
the store. The change is in the hash input for rollout position calculation:

```go theme={null}
func (e *GradualRolloutEvaluator) getHashKey(
    releaseTarget *oapi.ReleaseTarget,
    versionId string,
) string {
    if e.bracketRuleId != "" {
        // All bracket members for this resource get the same position
        return releaseTarget.ResourceId + e.bracketRuleId
    }
    return releaseTarget.Key() + versionId
}
```

Without this, kubelet-upgrade and containerd-upgrade on the same node would get
different rollout positions and arrive at different times, defeating the purpose
of bracketing.

### Evaluator chain integration

Both new evaluators slot into the existing factory:

```go theme={null}
func EvaluatorsForPolicy(store *store.Store, rule *oapi.PolicyRule) []evaluator.Evaluator {
    return evaluator.CollectEvaluators(
        deployableversions.NewEvaluatorFromStore(store),
        approval.NewEvaluatorFromStore(store, rule),
        environmentprogression.NewEvaluatorFromStore(store, rule),
        deploymentbracket.NewEvaluatorFromStore(store, rule),
        resourceconcurrency.NewEvaluatorFromStore(store, rule),
        gradualrollout.NewEvaluatorFromStore(store, rule),
        versionselector.NewEvaluator(rule),
        deploymentdependency.NewEvaluator(store, rule),
        deploymentwindow.NewEvaluatorFromStore(store, rule),
        versioncooldown.NewEvaluatorFromStore(store, rule),
    )
}
```

For any release target, all evaluators must return `Allowed` for a job to be
created. The evaluators gate at different levels:

| Evaluator             | Gates at           | Question it answers                  |
| --------------------- | ------------------ | ------------------------------------ |
| Deployment bracket    | Version group      | Are all sibling deployments ready?   |
| Gradual rollout       | Resource ordering  | Is it this resource's turn?          |
| Resource concurrency  | Cluster capacity   | Is there a concurrency slot?         |
| Deployment dependency | Execution ordering | Have upstream deployments succeeded? |

### Pre/post hooks as deployments

Rather than introducing a separate hook mechanism, lifecycle hooks (drain,
uncordon) are modeled as regular deployments that are bracket members with
dependency ordering. A "drain" deployment's job agent runs `kubectl drain`. An
"uncordon" deployment's job agent runs `kubectl uncordon`. Deployment dependency
rules control execution order within the bracket.

This reuses the existing job agent, job dispatch, retry, rollback, and
verification infrastructure. Hooks get observability (traces, job status) and
policy controls (retry on failure, rollback) for free.

## Examples

### Node upgrade with ordered bracket

A cluster has 10 nodes. Three workload deployments (kubelet, containerd,
os-patch) plus two lifecycle deployments (drain, uncordon) are tagged with
`metadata.layer = "node"`.

os-patch must run before kubelet and containerd. All five share a 24-hour
collection window and a 20% concurrency limit.

**Policy configuration:**

```json theme={null}
{
  "name": "Node Lifecycle",
  "selector": "deployment.metadata['layer'] == 'node'",
  "rules": [
    {
      "deploymentBracket": {
        "deploymentSelector": "deployment.metadata['layer'] == 'node'",
        "readinessMode": "collection_window",
        "readinessWindowSeconds": 86400,
        "unchangedMemberStrategy": "skip_unchanged",
        "overlapStrategy": "queue"
      }
    },
    {
      "resourceConcurrency": {
        "groupSelector": "resource.metadata['cluster'] == 'prod-east'",
        "limitType": "percentage",
        "limitValue": 20
      }
    },
    {
      "gradualRollout": {
        "rolloutType": "linear",
        "timeScaleInterval": 300
      }
    },
    {
      "deploymentDependency": {
        "dependsOn": "deployment.name == 'node-drain'",
        "appliesTo": "deployment.name in ['os-patch', 'kubelet-upgrade', 'containerd-upgrade']"
      }
    },
    {
      "deploymentDependency": {
        "dependsOn": "deployment.name == 'os-patch'",
        "appliesTo": "deployment.name in ['kubelet-upgrade', 'containerd-upgrade']"
      }
    },
    {
      "deploymentDependency": {
        "dependsOn": "deployment.name in ['kubelet-upgrade', 'containerd-upgrade']",
        "appliesTo": "deployment.name == 'node-uncordon'"
      }
    }
  ]
}
```

This expresses the execution DAG:

```
node-drain
    │
    ▼
os-patch
    │
    ├──────────┐
    ▼          ▼
kubelet    containerd     (parallel)
    │          │
    └────┬─────┘
         ▼
   node-uncordon
```

**Execution trace for node-3 (position 1 in rollout):**

```
Day 1, 09:00   kubelet v1.29.2 published
               Collection window opens, closes Day 2 09:00

Day 1, 14:00   containerd v1.7.3 published
Day 1, 22:00   os-patch 2026-03 published

Day 2, 09:00   Collection window closes. Versions locked.
               Gradual rollout: node-3 is position 1, offset 300s.
               Resource concurrency: 0/2 active.

Day 2, 09:05   node-3's rollout position reached. Concurrency slot available.
               5 release targets evaluated:

               node-drain:       bracket ✓  rollout ✓  concurrency ✓  deps: none ✓
                 → JOB CREATED: kubectl drain node-3

               os-patch:         bracket ✓  rollout ✓  concurrency ✓  deps: drain? ✗
                 → Pending

               kubelet-upgrade:  deps: os-patch? ✗ → Pending
               containerd:       deps: os-patch? ✗ → Pending
               node-uncordon:    deps: kubelet+containerd? ✗ → Pending

Day 2, 09:13   node-drain succeeds. Reconciliation triggers.
               os-patch: deps: drain ✓ → JOB CREATED

Day 2, 09:16   os-patch succeeds.
               kubelet:    deps: drain ✓, os-patch ✓ → JOB CREATED
               containerd: deps: drain ✓, os-patch ✓ → JOB CREATED
               (parallel execution)

Day 2, 09:19   kubelet + containerd succeed.
               node-uncordon: deps: kubelet ✓, containerd ✓ → JOB CREATED

Day 2, 09:20   node-uncordon succeeds. node-3 complete. Concurrency slot freed.
```

### Different clusters, different policies

The same deployments can have different bracket configurations per cluster by
using the policy selector to scope rules:

```json theme={null}
[
  {
    "name": "Staging Node Lifecycle",
    "selector": "deployment.metadata['layer'] == 'node' AND resource.metadata['cluster'] == 'staging'",
    "rules": [
      {
        "deploymentBracket": {
          "readinessMode": "immediate",
          "deploymentSelector": "deployment.metadata['layer'] == 'node'"
        }
      },
      {
        "resourceConcurrency": {
          "groupSelector": "resource.metadata['cluster'] == 'staging'",
          "limitType": "percentage",
          "limitValue": 50
        }
      }
    ]
  },
  {
    "name": "Prod Node Lifecycle",
    "selector": "deployment.metadata['layer'] == 'node' AND resource.metadata['cluster'] == 'prod-east'",
    "rules": [
      {
        "deploymentBracket": {
          "readinessMode": "collection_window",
          "readinessWindowSeconds": 86400,
          "deploymentSelector": "deployment.metadata['layer'] == 'node'"
        }
      },
      {
        "resourceConcurrency": {
          "groupSelector": "resource.metadata['cluster'] == 'prod-east'",
          "limitType": "percentage",
          "limitValue": 20
        }
      }
    ]
  }
]
```

Staging deploys immediately with 50% concurrency. Production waits 24 hours and
limits to 20%. Same deployments, different operational posture.

### Partial readiness

Only kubelet gets a new version within the 24-hour window. containerd and
os-patch have no updates.

```
Day 1, 09:00   kubelet v1.29.2 published. Window opens.
Day 2, 09:00   Window closes. Only kubelet has a new version.

               unchangedMemberStrategy = "skip_unchanged":
                 kubelet-upgrade → deploys v1.29.2
                 containerd, os-patch → desired version = current version → no-op
                 node-drain → runs (prepares the node)
                 node-uncordon → runs (restores the node)

               Net effect: one drain cycle for one deployment upgrade.
```

### Overlapping version groups

A new version group becomes ready while the previous group is still executing.

```
Day 2, 09:00   Group A starts executing on node-3.
               [drain → os-patch → kubelet v1.29.2 + containerd v1.7.3 → uncordon]

Day 2, 09:10   Group B collection window closes (new versions ready).
               kubelet v1.29.3, containerd v1.7.4, os-patch 2026-04.

               overlapStrategy = "queue":
                 Group B status for node-3: "queued"
                 Bracket evaluator: "Group A executing → Pending"
                 Group B CAN proceed on nodes that finished Group A.

Day 2, 09:20   Group A completes on node-3.
               Group B promoted to "ready".
               node-3 starts a fresh bracket cycle with Group B versions:
               [drain → os-patch 2026-04 → kubelet v1.29.3 + containerd v1.7.4 → uncordon]

               overlapStrategy = "merge":
                 Node is already drained from Group A.
                 Group B skips drain, deploys new versions, then uncordons.
                 One drain cycle covers both groups.
```

### Failure mid-bracket

kubelet fails on node-3 while containerd succeeds.

```
Day 2, 09:16   kubelet-upgrade FAILS on node-3.
               containerd-upgrade succeeds.

               node-uncordon: deps = kubelet + containerd
                 kubelet failed → dependency not met → Pending

               With retry rule on the same policy:
                 kubelet retries with backoff → succeeds on retry 2
                 node-uncordon: all deps met → uncordons

               Without retry rule:
                 node-3 stays drained. Concurrency slot held.
                 Remaining nodes blocked behind concurrency limit.
                 Operator must intervene (policy skip, manual fix).
```

This is why retry and/or rollback rules should always accompany bracket
policies. A stuck resource holds a concurrency slot indefinitely.

## Migration

* The `policy_rule_deployment_bracket` and `policy_rule_resource_concurrency`
  tables are new. No data migration required.
* The `bracket_version_group` and `bracket_version_group_member` tables are new.
* The `applies_to` column on `policy_rule_deployment_dependency` is additive and
  nullable. Existing rules have `applies_to = NULL`, preserving current behavior
  (rule applies to all matched release targets).
* The new evaluators return `nil` from their factory functions when the policy
  rule does not contain the relevant configuration, following the same pattern
  as all existing evaluators.
* Agents that execute lifecycle hooks (drain, uncordon) are standard job agents.
  No new agent interfaces are needed.

## Open Questions

1. **Collection window trigger semantics.** Should the collection window start
   when the first member's version is *published* or when the first member's
   version *passes other policy rules* (approval, version selector)? Starting at
   publication is simpler but means the window runs concurrently with approval —
   a 24h window with a 20h approval process only leaves 4h of actual collection
   time.

2. **Gradual rollout interaction.** The proposal changes the hash input when a
   bracket rule is present. This means adding or removing a bracket rule changes
   the rollout order for all targets. Should the bracket-aware hashing be opt-in
   to avoid surprising rollout order changes?

3. **Bracket membership dynamism.** The `deploymentSelector` on the bracket rule
   is evaluated dynamically. If a new deployment is added mid-rollout that
   matches the selector, should in-progress version groups absorb it? The
   simplest behavior is to only affect future version groups.

4. **Merge strategy completeness.** The `merge` overlap strategy avoids double
   drain cycles but requires the dependency evaluator to distinguish between
   "upstream succeeded with Group A's version" and "upstream succeeded with
   Group B's version." The current evaluator checks success status but not which
   version succeeded. This may need a version-aware dependency check for merge
   to work correctly.

5. **Failure blast radius.** A bracket failure on one resource holds a
   concurrency slot. With 20% concurrency on a 10-node cluster, 2 stuck nodes
   block the entire rollout. Should there be a configurable timeout that
   auto-releases concurrency slots after a bracket has been stuck for too long,
   even if the resource is in an unknown state?

6. **Hook idempotency.** The `queue` overlap strategy runs drain → uncordon →
   drain → uncordon for consecutive groups. This assumes drain and uncordon are
   idempotent. The `merge` strategy assumes the resource remains in a drained
   state between groups. Should the bracket rule have an explicit field
   declaring whether hooks are idempotent and/or whether the prepared state
   persists?

7. **Auto rollback interaction.** If a rollback policy triggers for one member
   deployment mid-bracket (e.g., kubelet fails health checks and auto-rolls
   back to v1.28.x), the resource is in a partially-upgraded state — some
   bracket members succeeded with new versions, others rolled back. Several
   sub-questions arise:
   * Should a rollback of any bracket member trigger a rollback of *all*
     bracket members on that resource to restore a consistent version set?
     This is the safe default for tightly coupled components (kubelet +
     containerd), but overly aggressive for loosely coupled ones.
   * If only the failed member rolls back, do post-hooks (uncordon) still
     run? The dependency graph may be satisfied (kubelet "completed" via
     rollback, containerd succeeded), but the resource is in a mixed state
     that the operator may not have intended to uncordon.
   * Should the bracket rule have a `rollback_strategy` field (e.g.,
     `individual`, `all_members`, `halt_and_notify`) that controls whether
     rollback is scoped to the failing member, cascaded to the full bracket,
     or paused for manual intervention?
   * How does a bracket-wide rollback interact with the concurrency limit?
     Rolling back N members on a resource means N additional jobs — does
     each count against the concurrency slot, or does the resource's
     existing slot cover the rollback work?


# RFC 0004: Dry-Run Deployment Plans
Source: https://docs.ctrlplane.dev/rfc/0004-dry-run-plans



| Category | Status               | Created    | Author        |
| -------- | -------------------- | ---------- | ------------- |
| Policies | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add an ephemeral plan API that CI pipelines call on pull requests to compute
**full rendered diffs** for each release target — showing exactly what
Kubernetes manifests, Terraform resources, or other deployed artifacts would
change, like `terraform plan` output. Results can optionally be posted back to
GitHub as PR comments or check runs. No version is created; the plan is computed
on the fly and returned to the caller.

## Motivation

When a developer opens a pull request that will eventually become a new
deployment version, two questions arise before merging:

1. **Which release targets will this version affect?**
2. **What exactly will change on each affected target?**

Today, neither question can be answered without merging the PR, creating the
version, and letting the full promotion lifecycle run. The deployer may have
intuition about the impact, but there is no way to get a concrete, rendered diff
— the kind of output `terraform plan` provides — before committing to a
deployment.

RFC 0002 introduces the `Plannable` interface on job agents, which can compute
rendered output without dispatching a job. But RFC 0002 focuses on the
reconciler: plans are computed during the promotion lifecycle to detect no-diff
targets and fast-track them. There is no way to trigger a plan *before* a
version exists.

### The PR workflow gap

The typical CI workflow for ctrlplane today:

```text theme={null}
Developer opens PR
  → CI builds artifact
  → PR is reviewed and merged
  → CI creates version (POST /v1/.../versions, status: ready)
  → ctrlplane creates releases for ALL release targets
  → full promotion lifecycle runs (staging → verification → approval → production)
  → deployer discovers which targets were actually affected
```

The deployer only learns what changed *after* committing to deployment. For
large deployments with tens or hundreds of release targets, this is a
significant blind spot. A PR that changes a single Helm values file for one
service triggers releases across every cluster, and the deployer won't know
which clusters are truly impacted until the pipeline is running.

`terraform plan` solved this for infrastructure: before applying, you see the
full execution plan with resource-level diffs. The same pattern should exist for
ctrlplane deployments.

### What "plan" means in this context

A dry-run plan computes, for each release target, the **full rendered output**
that the external system (ArgoCD, Terraform Cloud, etc.) would produce for the
proposed version — then diffs it against the current deployed state. This is not
a hash comparison (RFC 0002) or an affected/unaffected classification. It is the
actual diff content:

* For **ArgoCD**: the per-resource Kubernetes manifest diff (like
  `argocd app diff`)
* For **Terraform Cloud**: the resource-level before/after diff (like
  `terraform plan`)
* For **Helm**: the rendered template diff (like `helm diff upgrade`)

The diff is what the deployer reviews on the PR, the same way they review
`terraform plan` output today.

### Relationship to prior RFCs

* **RFC 0001 (Scoped Versions)** — The deployer declares which targets a version
  affects. Dry-run plans can inform that decision: review the plan on the PR,
  then create the version with a `targetSelector` that matches only the affected
  targets.
* **RFC 0002 (Plan-Based Diff Detection)** — Provides the `Plannable` interface
  and agent implementations that this RFC consumes. RFC 0002 runs plans inside
  the reconciler; this RFC exposes plans via an API endpoint before any version
  exists.

## Proposal

### API

Add a new endpoint that accepts proposed version data and returns rendered diffs
per release target. Nothing is persisted — the plan is ephemeral.

**Endpoint:**

```text theme={null}
POST /v1/workspaces/{workspaceId}/deployments/{deploymentId}/plan
```

**Request body:**

```json theme={null}
{
  "tag": "pr-123-abc123",
  "config": {},
  "jobAgentConfig": {},
  "metadata": {
    "pr": "123",
    "commit": "abc123"
  }
}
```

The fields mirror the version creation endpoint but no version row is inserted.
The API constructs a transient version object in memory and uses it to build
dispatch contexts.

**Synchronous response** (when all agents complete quickly):

```json theme={null}
{
  "id": "plan_abc123",
  "status": "completed",
  "summary": {
    "total": 50,
    "changed": 3,
    "unchanged": 47,
    "errored": 0,
    "resourceChanges": {
      "add": 1,
      "modify": 4,
      "delete": 0
    }
  },
  "targets": [
    {
      "environmentId": "env_prod",
      "environmentName": "production",
      "resourceId": "res_use1",
      "resourceName": "us-east-1-cluster",
      "hasChanges": true,
      "diff": {
        "raw": "--- current\n+++ proposed\n@@ -12,3 +12,3 @@\n-  image: payments:v1.2.3\n+  image: payments:v1.2.4\n",
        "resources": [
          {
            "kind": "Deployment",
            "name": "payment-api",
            "namespace": "payments",
            "action": "modify",
            "diff": "--- current\n+++ proposed\n@@ -12,3 +12,3 @@\n-  image: payments:v1.2.3\n+  image: payments:v1.2.4\n"
          }
        ]
      }
    },
    {
      "environmentId": "env_prod",
      "environmentName": "production",
      "resourceId": "res_euw1",
      "resourceName": "eu-west-1-cluster",
      "hasChanges": false,
      "diff": null
    }
  ]
}
```

Each target in the response includes:

* `hasChanges` — whether the rendered output differs from the current state
* `diff.raw` — human-readable unified diff of the full rendered output
* `diff.resources` — structured breakdown of per-resource changes with `kind`,
  `name`, `namespace`, `action` (add/modify/delete), and a per-resource `diff`

**Async response** (when agents require slow external calls):

```json theme={null}
{
  "id": "plan_abc123",
  "status": "computing",
  "summary": null,
  "targets": []
}
```

The CI polls
`GET /v1/workspaces/{workspaceId}/deployments/{deploymentId}/plan/{planId}`
until `status` transitions to `completed` or `failed`.

### Extended `PlanResult` type

RFC 0002 defines `PlanResult` with `ContentHash`, `HasChanges`, and a simple
`Diff` string. The dry-run plan requires richer diff data. The type is extended:

```go theme={null}
type PlanResult struct {
    ContentHash    string
    HasChanges     bool
    RenderedOutput string
    Diff           *PlanDiff
}

type PlanDiff struct {
    Raw       string
    Resources []ResourceChange
}

type ResourceChange struct {
    Kind      string // "Deployment", "Service", "aws_iam_policy"
    Name      string // "payment-api", "module.vpc.aws_subnet"
    Namespace string // Kubernetes namespace, empty for non-k8s
    Action    string // "add", "modify", "delete", "no-op"
    Before    string // Rendered YAML/JSON before (empty for adds)
    After     string // Rendered YAML/JSON after (empty for deletes)
    Diff      string // Unified diff for this resource
}
```

RFC 0002's reconciler integration only uses `ContentHash` and `HasChanges`. The
additional fields (`RenderedOutput`, `Diff`) are populated by agents when called
through the dry-run plan API and ignored by the reconciler path.

### How agents produce diffs

The `Plannable` interface from RFC 0002 is unchanged — agents return a
`PlanResult`. The difference is what the caller does with it:

* **Reconciler (RFC 0002):** Only inspects `ContentHash` and `HasChanges`.
* **Dry-run plan API (this RFC):** Inspects the full `PlanDiff` and returns it
  to the caller.

Agents that want to participate in dry-run plans must populate the `Diff` field.
Agents that only implement hash-based comparison (no diff capability) can still
participate — the API response will show `hasChanges: true/false` but `diff`
will be null.

#### ArgoCD

The ArgoCD agent calls the ArgoCD API to produce a real diff. The in-process
`TemplateApplication` function only renders the Application CRD (which always
differs because `targetRevision` changes). The actual diff lives in the
Kubernetes manifests that ArgoCD produces after fetching the git repo and
rendering the Helm chart or kustomize overlay.

The `Plan` implementation uses a **temporary Application** strategy. Calling
`GetManifests` on the existing Application only overrides the revision — it does
not pick up changes to Helm values, parameters, kustomize patches, or any other
spec field derived from deployment variables. To get a fully accurate manifest
diff for *any* kind of change (revision, variables, config), the agent creates a
short-lived Application with auto-sync disabled, waits for ArgoCD to render
manifests for it, fetches those manifests, then cleans it up.

The flow:

1. Renders the proposed Application CRD from the dispatch context (same as
   dispatch time). This CRD reflects all variable and config changes.
2. Strips any auto-sync policy and sets the sync policy to manual, so the
   temporary Application will never deploy to the cluster.
3. Creates the temporary Application in ArgoCD with a deterministic plan-scoped
   name (e.g., `<original-name>-plan-<short-hash>`).
4. Waits for ArgoCD to compute the desired manifests for the temporary
   Application. ArgoCD fetches the git repo, renders Helm/kustomize with the
   full proposed spec (including new values, parameters, revisions), and
   populates the manifest cache.
5. Calls `GetManifests` on the temporary Application to retrieve the fully
   rendered proposed manifests.
6. Calls `GetManifests` on the original Application to retrieve the current
   manifests.
7. Deletes the temporary Application.
8. Computes a per-resource unified diff between the two manifest sets.

**Multi-source Applications.** ArgoCD v2.6+ supports `spec.sources` (plural) for
Applications that pull from multiple Git repos or Helm charts (e.g., a chart
from one repo and values from another). Because the temporary Application is
created from the full rendered spec, multi-source applications are handled
naturally — the proposed spec's `sources` list (with all target revisions) is
preserved as-is.

```go theme={null}
const (
    planLabelKey      = "ctrlplane.dev/plan"
    planCreatedAtKey  = "ctrlplane.dev/plan-created-at"
    planTTL           = 10 * time.Minute
)

func planAppName(originalName string) string {
    h := sha256.Sum256([]byte(originalName + time.Now().String()))
    return fmt.Sprintf("%s-plan-%s", originalName, hex.EncodeToString(h[:4]))
}

func prepareTmpApp(app *v1alpha1.Application, tmpName string) *v1alpha1.Application {
    tmp := app.DeepCopy()
    tmp.Name = tmpName
    tmp.ResourceVersion = ""

    if tmp.Labels == nil {
        tmp.Labels = map[string]string{}
    }
    tmp.Labels[planLabelKey] = "true"
    if tmp.Annotations == nil {
        tmp.Annotations = map[string]string{}
    }
    tmp.Annotations[planCreatedAtKey] = time.Now().UTC().Format(time.RFC3339)

    tmp.Spec.SyncPolicy = &v1alpha1.SyncPolicy{Automated: nil}
    tmp.Operation = nil
    return tmp
}

func (a *ArgoApplication) Plan(
    ctx context.Context,
    dispatchCtx *oapi.DispatchContext,
) (*types.PlanResult, error) {
    serverAddr, apiKey, template, err := ParseJobAgentConfig(
        dispatchCtx.JobAgentConfig,
    )
    if err != nil {
        return nil, err
    }

    proposedApp, err := TemplateApplication(dispatchCtx, template)
    if err != nil {
        return nil, err
    }
    MakeApplicationK8sCompatible(proposedApp)

    client, err := argocdclient.NewClient(&argocdclient.ClientOptions{
        ServerAddr: serverAddr,
        AuthToken:  apiKey,
    })
    if err != nil {
        return nil, fmt.Errorf("create ArgoCD client: %w", err)
    }
    ioCloser, appClient, err := client.NewApplicationClient()
    if err != nil {
        return nil, fmt.Errorf("create application client: %w", err)
    }
    defer ioCloser.Close()

    originalName := proposedApp.Name
    tmpName := planAppName(originalName)
    tmpApp := prepareTmpApp(proposedApp, tmpName)

    upsert := true
    _, err = appClient.Create(ctx, &argocdapplication.ApplicationCreateRequest{
        Application: tmpApp,
        Upsert:      &upsert,
    })
    if err != nil {
        return nil, fmt.Errorf("create temporary plan application: %w", err)
    }
    defer func() {
        cascade := false
        _, _ = appClient.Delete(ctx, &argocdapplication.ApplicationDeleteRequest{
            Name:    &tmpName,
            Cascade: &cascade,
        })
    }()

    if err := waitForManifests(ctx, appClient, tmpName); err != nil {
        return nil, fmt.Errorf("wait for temporary app manifests: %w", err)
    }

    proposedManifests, err := appClient.GetManifests(ctx,
        &argocdapplication.ApplicationManifestQuery{Name: &tmpName},
    )
    if err != nil {
        return nil, fmt.Errorf("get proposed manifests: %w", err)
    }

    currentManifests, err := appClient.GetManifests(ctx,
        &argocdapplication.ApplicationManifestQuery{Name: &originalName},
    )
    if err != nil {
        return buildAddAllResult(proposedManifests)
    }

    return diffManifestSets(currentManifests.Manifests, proposedManifests.Manifests)
}
```

The `waitForManifests` helper polls the temporary Application until ArgoCD
reports a non-empty manifest set or the context deadline expires:

```go theme={null}
func waitForManifests(
    ctx context.Context,
    appClient argocdapplication.ApplicationServiceClient,
    name string,
) error {
    ticker := time.NewTicker(2 * time.Second)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-ticker.C:
            resp, err := appClient.GetManifests(ctx,
                &argocdapplication.ApplicationManifestQuery{Name: &name},
            )
            if err != nil {
                continue
            }
            if len(resp.Manifests) > 0 {
                return nil
            }
        }
    }
}
```

**Cleanup guarantees.** The temporary Application is deleted in a `defer` with
`cascade: false` (the Application never synced, so there are no cluster
resources to remove). If the agent crashes before cleanup, orphaned Applications
remain in ArgoCD. ArgoCD has no native TTL mechanism for Applications — cleanup
of orphans is ctrlplane's responsibility.

Every temporary Application is labelled `ctrlplane.dev/plan: "true"` and
annotated with `ctrlplane.dev/plan-created-at: <RFC3339 timestamp>`. A
background goroutine in the workspace engine periodically lists Applications
matching the plan label, parses the created-at annotation, and deletes any older
than `planTTL` (default 10 minutes):

```go theme={null}
func (gc *PlanAppGC) Run(ctx context.Context) {
    ticker := time.NewTicker(1 * time.Minute)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            gc.cleanup(ctx)
        }
    }
}

func (gc *PlanAppGC) cleanup(ctx context.Context) {
    selector := fmt.Sprintf("%s=true", planLabelKey)
    apps, err := gc.appClient.List(ctx, &argocdapplication.ApplicationQuery{
        Selector: &selector,
    })
    if err != nil {
        log.Warn("plan GC: list failed", "error", err)
        return
    }
    for _, app := range apps.Items {
        createdAt, err := time.Parse(time.RFC3339, app.Annotations[planCreatedAtKey])
        if err != nil || time.Since(createdAt) < planTTL {
            continue
        }
        cascade := false
        name := app.Name
        _, _ = gc.appClient.Delete(ctx, &argocdapplication.ApplicationDeleteRequest{
            Name:    &name,
            Cascade: &cascade,
        })
        log.Info("plan GC: deleted orphaned plan app", "name", name,
            "age", time.Since(createdAt))
    }
}
```

The GC runs per ArgoCD server. When the workspace engine starts, it registers a
`PlanAppGC` instance for each configured ArgoCD connection.

The `diffManifestSets` function parses each manifest as a Kubernetes resource,
matches resources by `apiVersion/kind/namespace/name`, and produces a
`ResourceChange` for each:

* Resources in proposed but not current → `action: "add"`
* Resources in current but not proposed → `action: "delete"`
* Resources in both with different content → `action: "modify"` with unified
  diff
* Resources in both with identical content → omitted (no-op)

#### Terraform Cloud

Terraform Cloud speculative plans already produce structured diff output. The
`Plan` implementation triggers a speculative plan run and maps the result:

```go theme={null}
func (t *TerraformCloud) Plan(
    ctx context.Context,
    dispatchCtx *oapi.DispatchContext,
) (*types.PlanResult, error) {
    run, err := t.client.CreateRun(ctx, RunConfig{
        IsDestroy: false,
        PlanOnly:  true,
        Variables: dispatchCtx.Variables,
    })
    if err != nil {
        return nil, err
    }

    plan, err := t.client.WaitForPlan(ctx, run.ID)
    if err != nil {
        return nil, err
    }

    resources := make([]types.ResourceChange, 0, len(plan.ResourceChanges))
    for _, rc := range plan.ResourceChanges {
        resources = append(resources, types.ResourceChange{
            Kind:   rc.Type,
            Name:   rc.Address,
            Action: mapTerraformAction(rc.Change.Actions),
            Before: rc.Change.Before,
            After:  rc.Change.After,
            Diff:   rc.Change.Diff,
        })
    }

    hasChanges := plan.ResourceAdditions > 0 ||
        plan.ResourceChanges > 0 ||
        plan.ResourceDestructions > 0

    return &types.PlanResult{
        ContentHash: plan.StateHash,
        HasChanges:  hasChanges,
        Diff: &types.PlanDiff{
            Raw:       plan.HumanReadableOutput,
            Resources: resources,
        },
    }, nil
}
```

#### GitHub Actions / unsupported agents

Agents that do not implement `Plannable` return nil from the registry's `Plan`
method. The dry-run plan endpoint reports these targets as:

```json theme={null}
{
  "resourceName": "some-target",
  "hasChanges": null,
  "diff": null,
  "status": "unsupported"
}
```

The CI can still post a PR comment noting that some targets could not be
planned.

### Plan execution flow

The plan endpoint does not create a version or trigger the reconciler. It
constructs the necessary context in-memory and calls agents directly:

```text theme={null}
1. Parse request body into transient version object
2. Look up deployment and its job agents
3. Find all release targets for this deployment
    (same query as enqueueReleaseTargetsForDeployment)
4. For each release target:
    a. Resolve variables (reuse variableresolver.Resolve)
    b. Build DispatchContext (reuse jobs.Factory.BuildDispatchContext)
    c. Call registry.Plan(agentType, dispatchCtx)
    d. Collect PlanResult
5. Aggregate results into response
6. If github field present, post results to PR
```

For agents with fast plan steps (ArgoCD with cached manifests), the endpoint can
complete synchronously. For slow agents (Terraform Cloud speculative plans
taking minutes), the endpoint:

1. Creates a plan record in a lightweight `deployment_plan` table with
   `status: "computing"`.
2. Enqueues plan computation as background work.
3. Returns the plan ID immediately.
4. The CI polls `GET .../plan/{planId}` until status is `completed`.

```sql theme={null}
CREATE TABLE deployment_plan (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
    deployment_id UUID NOT NULL REFERENCES deployment(id) ON DELETE CASCADE,
    status TEXT NOT NULL DEFAULT 'computing',
    request JSONB NOT NULL,
    result JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    completed_at TIMESTAMPTZ,
    expires_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + INTERVAL '1 hour'
);
```

Plans are ephemeral — the `expires_at` column enables periodic cleanup. No
long-term storage is needed.

### GitHub integration

When the plan request includes a `github` field, ctrlplane posts results back to
the PR using the GitHub App that is already configured for workflow dispatch:

**Request with GitHub integration:**

```json theme={null}
{
  "tag": "pr-123-abc123",
  "config": {},
  "metadata": { "pr": "123", "commit": "abc123" },
  "github": {
    "owner": "org",
    "repo": "myapp",
    "sha": "abc123def456",
    "prNumber": 123
  }
}
```

**PR comment format:**

The comment follows the pattern established by Atlantis and Terraform Cloud,
adapted for ctrlplane's multi-target model:

````markdown theme={null}
### Ctrlplane Deployment Plan

**Deployment:** API Service **Version:** pr-123-abc123

| Environment | Resource           | Changes    | Details                  |
| ----------- | ------------------ | ---------- | ------------------------ |
| production  | us-east-1-cluster  | 1 modified | `Deployment/payment-api` |
| production  | eu-west-1-cluster  | No changes | —                        |
| production  | ap-south-1-cluster | No changes | —                        |
| staging     | staging-cluster    | 1 modified | `Deployment/payment-api` |

**Summary:** 2 of 4 targets affected (1 resource modified)

<details>
<summary>us-east-1-cluster diff</summary>

\```diff --- Deployment/payments/payment-api (current) +++
Deployment/payments/payment-api (proposed) @@ -15,3 +15,3 @@ containers: - name:
payment-api

-        image: payments:v1.2.3

*        image: payments:v1.2.4
  \```

</details>
````

**GitHub Check Run** (alternative or complement to PR comment):

The plan can also be reported as a GitHub Check Run with status
`success`/`neutral`/`failure` and structured annotations per changed resource.
Check runs integrate with branch protection rules, allowing teams to require a
passing plan before merge.

**Implementation:** The existing GitHub App integration in the workspace engine
uses the ArgoCD Go client pattern for API calls. The PR comment/check run
posting uses the GitHub App's installation token (the same token acquisition
flow used by `GoGitHubWorkflowDispatcher` in
`apps/workspace-engine/svc/controllers/jobdispatch/jobagents/github/`).

### Optional: `pull_request` webhook handler

As a convenience layer, ctrlplane can optionally react to GitHub `pull_request`
webhook events to auto-trigger plans without CI changes.

The GitHub webhook handler in `apps/api/src/routes/github/index.ts` currently
only handles `workflow_run` events:

```typescript theme={null}
if (eventType === "workflow_run")
  await handleWorkflowRunEvent(req.body as WorkflowRunEvent);
```

Adding a `pull_request` handler:

```typescript theme={null}
if (eventType === "workflow_run")
  await handleWorkflowRunEvent(req.body as WorkflowRunEvent);
else if (eventType === "pull_request")
  await handlePullRequestEvent(req.body as PullRequestEvent);
```

The PR metadata types already exist in `packages/validators/src/github/index.ts`
(`GithubPullRequestVersion`, `PullRequestMetadataKey`, `PullRequestConfigKey`)
but are not wired up to any handler.

The `handlePullRequestEvent` function would:

1. Extract the repo owner/name and head SHA from the event payload.
2. Find deployments whose job agent config references this repo (by matching
   `owner` and `repo` fields in the GitHub job agent config).
3. For each matching deployment, trigger a plan using the head SHA as the
   proposed version tag.
4. Post results back as a PR comment or check run.

This is optional — the CI-triggered API is the primary integration path. The
webhook handler is a convenience for teams that want automatic plans without
modifying their CI pipelines.

## Examples

### ArgoCD: Helm chart change on a PR

A deployment manages 20 clusters across 4 environments using ArgoCD with a
monorepo Helm chart. A developer opens a PR that modifies
`charts/payment/values.yaml`.

```bash theme={null}
# In the CI pipeline triggered by the PR:
curl -X POST \
  "https://api.ctrlplane.dev/v1/workspaces/$WS/deployments/$DEPLOY/plan" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "tag": "pr-456-'$(git rev-parse --short HEAD)'",
    "config": {},
    "metadata": {
      "commit": "'$(git rev-parse HEAD)'",
      "pr": "456",
      "branch": "'$(git branch --show-current)'"
    },
    "github": {
      "owner": "myorg",
      "repo": "platform",
      "sha": "'$(git rev-parse HEAD)'",
      "prNumber": 456
    }
  }'
```

ctrlplane:

1. Builds a transient version with the PR's head commit.
2. For each of the 20 release targets, calls ArgoCD's `GetManifests` API with
   the PR commit as the target revision.
3. Diffs the proposed manifests against the currently deployed manifests.
4. Returns: 4 targets show changes (the clusters running the payment service),
   16 show no changes.
5. Posts a PR comment showing the diff table with expandable per-target diffs.

The developer sees exactly which clusters are affected and what Kubernetes
resources change — before merging.

### Terraform Cloud: Infrastructure PR

A deployment manages Terraform infrastructure across 3 regions. A PR changes an
IAM policy module.

```bash theme={null}
curl -X POST \
  "https://api.ctrlplane.dev/v1/workspaces/$WS/deployments/$DEPLOY/plan" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "tag": "pr-789-abc123",
    "config": {},
    "metadata": { "pr": "789" }
  }'
```

Response (after async completion):

```json theme={null}
{
  "id": "plan_xyz",
  "status": "completed",
  "summary": {
    "total": 3,
    "changed": 1,
    "unchanged": 2,
    "errored": 0,
    "resourceChanges": { "add": 0, "modify": 2, "delete": 0 }
  },
  "targets": [
    {
      "environmentName": "production",
      "resourceName": "us-east-1",
      "hasChanges": true,
      "diff": {
        "raw": "Terraform will perform the following actions:\n\n  # aws_iam_policy.service_policy will be updated in-place\n  ~ resource \"aws_iam_policy\" \"service_policy\" {\n      ~ policy = jsonencode(\n          ~ {\n              ~ Statement = [\n                  ~ {\n                      ~ Action = [\n                          + \"s3:GetObject\",\n                        ]\n                    },\n                ]\n            }\n        )\n    }\n\nPlan: 0 to add, 2 to change, 0 to destroy.",
        "resources": [
          {
            "kind": "aws_iam_policy",
            "name": "module.auth.aws_iam_policy.service_policy",
            "action": "modify",
            "diff": "..."
          },
          {
            "kind": "aws_iam_role_policy_attachment",
            "name": "module.auth.aws_iam_role_policy_attachment.service",
            "action": "modify",
            "diff": "..."
          }
        ]
      }
    },
    {
      "environmentName": "production",
      "resourceName": "eu-west-1",
      "hasChanges": false,
      "diff": null
    },
    {
      "environmentName": "production",
      "resourceName": "ap-south-1",
      "hasChanges": false,
      "diff": null
    }
  ]
}
```

The PR shows that only us-east-1 is affected, with exactly 2 IAM resources
changing.

### GitHub Actions: Unsupported agent

A deployment uses GitHub Actions (no `Plannable` implementation). The plan
endpoint still runs but cannot produce diffs:

```json theme={null}
{
  "id": "plan_def",
  "status": "completed",
  "summary": {
    "total": 5,
    "changed": 0,
    "unchanged": 0,
    "errored": 0,
    "unsupported": 5
  },
  "targets": [
    {
      "environmentName": "production",
      "resourceName": "cluster-1",
      "hasChanges": null,
      "diff": null,
      "status": "unsupported"
    }
  ]
}
```

The CI can still post a PR comment noting that plan output is not available for
this deployment type.

## Migration

* The `deployment_plan` table is new and requires no data migration.
* Plans are ephemeral with a 1-hour TTL by default. No long-term storage
  concerns.
* The `Plannable` interface (RFC 0002) is unchanged. Agents that already
  implement it gain dry-run plan support automatically; they only need to
  populate the `Diff` field for rich output.
* The `pull_request` webhook handler is additive. The existing `workflow_run`
  handler is unchanged.
* No changes to the version creation flow, reconciler, or promotion lifecycle.

## Open Questions

1. **Rate limiting.** Plans involve external API calls (ArgoCD manifest
   rendering, Terraform speculative plans). For deployments with many release
   targets, a single PR could trigger hundreds of external calls. Should there
   be a per-deployment or per-workspace rate limit on plan requests? Should
   callers be able to scope the plan to specific environments or resources?

2. **Plan scope.** The proposal plans against all release targets. For large
   deployments, the caller may want to plan only for specific environments or
   resources. Should the request body accept an optional filter
   (`environmentSelector`, `resourceSelector`) to narrow the plan scope?

3. **Diff format standardization.** ArgoCD produces YAML diffs, Terraform
   produces HCL-style diffs. Should the `raw` field in `PlanDiff` be
   agent-specific (each agent returns its native format), or should ctrlplane
   normalize to a common diff format?

4. **Cost of plans.** Each plan consumes Terraform Cloud compute resources. For
   deployments with many targets across many PRs, this could become expensive.
   Should Terraform plans require explicit opt-in per deployment?

5. **Temporary Application permissions.** Creating and deleting Applications
   requires write access to the ArgoCD API. Some teams restrict Application
   creation to specific ArgoCD projects or RBAC roles. Should the plan
   Application be created in a dedicated ArgoCD project (e.g.,
   `ctrlplane-plans`) with limited permissions, or inherit the project from the
   original Application?

6. **ArgoCD rendering latency.** After creating the temporary Application, the
   agent polls until ArgoCD renders manifests. For large Helm charts or slow git
   repos this could take significant time. Should there be a configurable
   timeout per agent, and how should the plan endpoint report rendering timeouts
   vs. real errors?


# RFC 0005: Argo Workflows Job Agent
Source: https://docs.ctrlplane.dev/rfc/0005-argo-workflows-job-agent



| Category   | Status               | Created    | Author        |
| ---------- | -------------------- | ---------- | ------------- |
| Job Agents | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add a new `argo-workflows` job agent type that submits Argo Workflow CRDs to
Kubernetes, templates workflow specs from the dispatch context, and monitors
workflow execution to completion. This enables teams to use Argo Workflows as a
native deployment execution engine within ctrlplane's promotion lifecycle.

## Motivation

Ctrlplane's job agent model today covers three execution patterns:

* **ArgoCD** — declarative GitOps sync of Kubernetes Applications
* **GitHub Actions** — CI-triggered workflow dispatch via the GitHub API
* **Terraform Cloud** — speculative and apply runs via the TFC API

These patterns share a common shape: ctrlplane constructs a dispatch context,
the agent translates it into an external system call, and the external system
reports back when done. But a significant class of deployment operations does
not fit neatly into any of them.

### The gap: orchestrated multi-step deployments

Many deployment procedures involve multiple steps that must execute in sequence
or in a DAG structure on a Kubernetes cluster:

* **Database migrations** before application rollout
* **Canary analysis** with traffic splitting, metric collection, and rollback
* **Blue/green cutover** with health checks between steps
* **Infrastructure provisioning** (create namespace, install CRDs, deploy app)
* **Integration test suites** that run against a freshly deployed environment
* **Custom scripts** (data backfixes, cache warming, feature flag toggling)

Today, teams using these patterns have two options:

1. **GitHub Actions** — The workflow runs outside the cluster, requiring
   kubeconfig secrets, network access to the cluster API, and manual status
   reporting back to ctrlplane. Multi-cluster deployments need per-cluster
   credentials. The workflow has no native access to in-cluster resources.

2. **ArgoCD sync hooks** — ArgoCD supports PreSync/Sync/PostSync hooks, but
   these are limited to single Jobs or Pods with linear ordering. Complex DAGs,
   conditional branching, retries with backoff, artifact passing between steps,
   and parameterized templates are not expressible.

Argo Workflows fills this gap. It is a Kubernetes-native workflow engine that
runs inside the cluster, supports DAG and step-based orchestration, has
first-class retry/backoff semantics, handles artifact passing between steps, and
is already widely deployed alongside ArgoCD in GitOps environments.

### Why not use GitHub Actions for everything?

GitHub Actions can technically orchestrate any deployment, but it operates
outside the cluster boundary:

```text theme={null}
GitHub Actions (external)          Kubernetes cluster
┌──────────────────────┐           ┌──────────────────────┐
│ deploy.yml           │           │                      │
│  step 1: migrate db ─┼──kubectl──┼─→ run migration pod  │
│  step 2: deploy app ─┼──kubectl──┼─→ update deployment  │
│  step 3: run tests  ─┼──kubectl──┼─→ create test pod    │
│  step 4: report     ─┼──api─────┼─→ ctrlplane callback │
└──────────────────────┘           └──────────────────────┘
```

Every step crosses the network boundary. This requires:

* Kubeconfig or service account token stored as GitHub secrets
* Network connectivity from GitHub's runners to the cluster API
* Per-cluster credential management for multi-cluster deployments
* Manual status reporting back to ctrlplane's API

With Argo Workflows, the entire execution stays in-cluster:

```text theme={null}
Kubernetes cluster
┌────────────────────────────────────────┐
│ Argo Workflow (submitted by ctrlplane) │
│  step 1: migrate db → migration pod   │
│  step 2: deploy app → kubectl apply   │
│  step 3: run tests  → test pod        │
│  status: reported via workflow CRD     │
└────────────────────────────────────────┘
```

No external credentials. No network boundary crossings. Native access to
in-cluster resources. Status is read from the Workflow CRD status field.

### Why not extend the ArgoCD agent?

ArgoCD and Argo Workflows are separate projects with different APIs, CRDs, and
operational models:

* **ArgoCD** is declarative: you describe a desired state (Application CRD) and
  ArgoCD continuously reconciles toward it. The agent upserts an Application and
  verifies it reaches Healthy+Synced.
* **Argo Workflows** is imperative: you submit a workflow (Workflow CRD) and it
  runs to completion. Each submission is a discrete execution with a start and
  end.

The dispatch lifecycle is fundamentally different. ArgoCD's `UpsertApplication`
→ poll health model does not map to Argo Workflows' submit → watch completion
model. Combining them in one agent would conflate two distinct execution
semantics behind a single `argo-cd` type, making configuration confusing and
error handling ambiguous.

## Proposal

### Agent type and config

Register a new agent type `argo-workflows` in the workspace engine's job agent
registry. The job agent config provides cluster access and a workflow template:

```json theme={null}
{
  "type": "argo-workflows",
  "serverUrl": "https://argo-workflows.example.com",
  "token": "argo-token-or-service-account",
  "namespace": "argo",
  "template": "apiVersion: argoproj.io/v1alpha1\nkind: Workflow\n..."
}
```

| Field       | Required | Description                                                 |
| ----------- | -------- | ----------------------------------------------------------- |
| `serverUrl` | Yes      | Argo Workflows server URL (API endpoint)                    |
| `token`     | Yes      | Bearer token or service account token for authentication    |
| `namespace` | No       | Default namespace for workflow submission (default: `argo`) |
| `template`  | Yes      | Go template rendering an Argo Workflow YAML                 |

The `template` field follows the same pattern as the ArgoCD agent's template: a
Go template string that receives the dispatch context and produces a valid Argo
Workflow CRD. This is rendered at dispatch time using the `templatefuncs`
pipeline with custom delimiters `{[` / `]}` instead of Go's default `{{` / `}}`
(see "Template delimiters" below).

### Workflow template

The template renders a complete Argo Workflow spec from the dispatch context.
The dispatch context provides deployment, environment, resource, version, and
variable data — the same data available to all job agent templates.

**Example template for a database migration + deploy workflow:**

```yaml theme={null}
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: deploy-{[.deployment.slug]}-
  namespace: { [.resource.config.namespace | default "argo"] }
  labels:
    ctrlplane.dev/job-id: "{[.job.id]}"
    ctrlplane.dev/deployment: "{[.deployment.slug]}"
    ctrlplane.dev/environment: "{[.environment.name]}"
spec:
  entrypoint: deploy
  serviceAccountName: argo-deployer
  arguments:
    parameters:
      - name: image-tag
        value: "{[.release.version.tag]}"
      - name: target-namespace
        value: "{[.resource.config.namespace]}"
      - name: replica-count
        value: "{[.release.variables.REPLICA_COUNT]}"
  templates:
    - name: deploy
      dag:
        tasks:
          - name: migrate-db
            template: run-migration
          - name: deploy-app
            template: apply-manifests
            dependencies: [migrate-db]
          - name: smoke-test
            template: run-tests
            dependencies: [deploy-app]

    - name: run-migration
      container:
        image: "{[.release.variables.MIGRATION_IMAGE]}:{[.release.version.tag]}"
        command: ["/migrate", "--target", "latest"]

    - name: apply-manifests
      container:
        image: bitnami/kubectl:latest
        command:
          - kubectl
          - set
          - image
          - "deployment/{[.deployment.slug]}"
          - "app={[.release.variables.APP_IMAGE]}:{[.release.version.tag]}"
          - "-n"
          - "{{workflow.parameters.target-namespace}}"

    - name: run-tests
      container:
        image: "{[.release.variables.TEST_IMAGE]}:latest"
        command:
          [
            "/run-tests",
            "--endpoint",
            "http://{[.deployment.slug]}.{{workflow.parameters.target-namespace}}.svc",
          ]
      retryStrategy:
        limit: 3
        backoff:
          duration: "10s"
          factor: 2
```

### Template delimiters

Argo Workflows uses `{{` / `}}` for its own parameter substitution at runtime
(e.g., `{{workflow.parameters.target-namespace}}`). Go's `text/template` uses
the same delimiters by default. With standard Go templates, users must escape
every Argo expression — an error-prone and unreadable approach.

Instead, ctrlplane templates use **`{[` / `]}`** as delimiters. This is a clean
separation: `{[ ]}` is ctrlplane's template language, `{{ }}` is Argo's. Both
coexist in the same YAML file without escaping:

```yaml theme={null}
# {[ ]} — resolved by ctrlplane at dispatch time
image: "{[.release.version.tag]}"

# {{ }} — resolved by Argo Workflows at workflow runtime
namespace: "{{workflow.parameters.target-namespace}}"
```

The `templatefuncs` package already provides a `New` function that configures
template options. The custom delimiters are set via Go's `Delims` method:

```go theme={null}
func NewWithDelims(name string) *template.Template {
    return template.New(name).
        Delims("{[", "]}").
        Funcs(funcs).
        Option("missingkey=zero")
}
```

This delimiter change applies to **all** ctrlplane job agent templates, not just
Argo Workflows. The ArgoCD agent benefits too — ArgoCD Application specs that
embed Helm value overrides containing `{{ }}` expressions currently require
escaping, and custom delimiters eliminate that. Existing templates using `{{` /
`}}` would be migrated to `{[` / `]}` as part of this change.

### Implementation

#### Go types

```go theme={null}
package argoworkflows

type ArgoWorkflows struct {
    setter   Setter
    submitter WorkflowSubmitter
}

// WorkflowSubmitter submits and monitors Argo Workflows.
type WorkflowSubmitter interface {
    SubmitWorkflow(
        ctx context.Context,
        serverAddr, token, namespace string,
        workflow *unstructured.Unstructured,
    ) (workflowName string, err error)

    GetWorkflowStatus(
        ctx context.Context,
        serverAddr, token, namespace, name string,
    ) (*WorkflowStatus, error)
}

type WorkflowStatus struct {
    Phase      string // Pending, Running, Succeeded, Failed, Error
    Message    string
    StartedAt  *time.Time
    FinishedAt *time.Time
    Nodes      map[string]NodeStatus
}

type NodeStatus struct {
    Name      string
    Phase     string
    Message   string
    StartedAt *time.Time
    FinishedAt *time.Time
}
```

#### Dispatchable implementation

```go theme={null}
var (
    _ types.Dispatchable = &ArgoWorkflows{}
    _ types.Verifiable   = &ArgoWorkflows{}
)

func (a *ArgoWorkflows) Type() string {
    return "argo-workflows"
}

func (a *ArgoWorkflows) Dispatch(ctx context.Context, job *oapi.Job) error {
    dispatchCtx := job.DispatchContext
    if dispatchCtx == nil {
        return fmt.Errorf("job %s has no dispatch context", job.Id)
    }

    serverAddr, token, namespace, template, err := ParseJobAgentConfig(
        dispatchCtx.JobAgentConfig,
    )
    if err != nil {
        return fmt.Errorf("parse job agent config: %w", err)
    }

    wf, err := TemplateWorkflow(dispatchCtx, job, template)
    if err != nil {
        return fmt.Errorf("template workflow: %w", err)
    }

    EnsureLabels(wf, job)

    go func() {
        parentSpanCtx := trace.SpanContextFromContext(ctx)
        asyncCtx, span := tracer.Start(context.Background(),
            "ArgoWorkflows.AsyncDispatch",
            trace.WithLinks(trace.Link{SpanContext: parentSpanCtx}),
        )
        defer span.End()

        name, err := a.submitter.SubmitWorkflow(
            asyncCtx, serverAddr, token, namespace, wf,
        )
        if err != nil {
            _ = a.setter.UpdateJob(asyncCtx, job.Id,
                oapi.JobStatusFailure,
                fmt.Sprintf("failed to submit workflow: %s", err.Error()),
                nil,
            )
            return
        }

        metadata := map[string]string{
            "ctrlplane/links": fmt.Sprintf(
                `{"Argo Workflow":"%s/workflows/%s/%s"}`,
                serverAddr, namespace, name,
            ),
            "argo-workflows/name":      name,
            "argo-workflows/namespace": namespace,
        }
        _ = a.setter.UpdateJob(asyncCtx, job.Id,
            oapi.JobStatusInProgress, "", metadata,
        )

        a.pollUntilComplete(asyncCtx, job.Id, serverAddr, token, namespace, name)
    }()

    return nil
}
```

#### Workflow templating

The template rendering follows the same pattern as ArgoCD but uses `{[` / `]}`
delimiters and produces an unstructured Kubernetes object instead of a typed
Application CRD. This avoids importing Argo Workflows' full type system as a
dependency:

```go theme={null}
func TemplateWorkflow(
    dispatchCtx *oapi.DispatchContext,
    job *oapi.Job,
    tmpl string,
) (*unstructured.Unstructured, error) {
    t, err := templatefuncs.NewWithDelims("argoWorkflowsAgentConfig").Parse(tmpl)
    if err != nil {
        return nil, fmt.Errorf("parse template: %w", err)
    }

    data := dispatchCtx.Map()
    data["job"] = structToMap(job)

    var buf bytes.Buffer
    if err := t.Execute(&buf, data); err != nil {
        return nil, fmt.Errorf("execute template: %w", err)
    }

    obj := &unstructured.Unstructured{}
    if err := yaml.Unmarshal(buf.Bytes(), &obj.Object); err != nil {
        return nil, fmt.Errorf("unmarshal workflow: %w", err)
    }

    if obj.GetAPIVersion() != "argoproj.io/v1alpha1" {
        return nil, fmt.Errorf(
            "expected apiVersion argoproj.io/v1alpha1, got %s",
            obj.GetAPIVersion(),
        )
    }
    if obj.GetKind() != "Workflow" {
        return nil, fmt.Errorf("expected kind Workflow, got %s", obj.GetKind())
    }

    return obj, nil
}

func EnsureLabels(wf *unstructured.Unstructured, job *oapi.Job) {
    labels := wf.GetLabels()
    if labels == nil {
        labels = make(map[string]string)
    }
    labels["ctrlplane.dev/job-id"] = job.Id
    labels["ctrlplane.dev/managed-by"] = "ctrlplane"
    wf.SetLabels(labels)
}
```

#### Completion polling

After submission, the agent polls the Argo Workflows API for workflow status.
The polling follows an exponential backoff pattern capped at 30 seconds:

```go theme={null}
func (a *ArgoWorkflows) pollUntilComplete(
    ctx context.Context,
    jobID, serverAddr, token, namespace, name string,
) {
    backoff := 2 * time.Second
    maxBackoff := 30 * time.Second
    timeout := 2 * time.Hour

    deadline := time.Now().Add(timeout)
    for time.Now().Before(deadline) {
        select {
        case <-ctx.Done():
            return
        case <-time.After(backoff):
        }

        status, err := a.submitter.GetWorkflowStatus(
            ctx, serverAddr, token, namespace, name,
        )
        if err != nil {
            backoff = min(backoff*2, maxBackoff)
            continue
        }

        switch status.Phase {
        case "Succeeded":
            _ = a.setter.UpdateJob(ctx, jobID,
                oapi.JobStatusSuccessful, "", nil)
            return
        case "Failed", "Error":
            _ = a.setter.UpdateJob(ctx, jobID,
                oapi.JobStatusFailure, status.Message, nil)
            return
        }

        backoff = min(backoff*2, maxBackoff)
    }

    _ = a.setter.UpdateJob(ctx, jobID,
        oapi.JobStatusFailure, "workflow timed out", nil)
}
```

#### Verification

The agent implements `Verifiable` to provide a health check that monitors the
Workflow CRD's status. Unlike ArgoCD's continuous health check (which polls
indefinitely because ArgoCD applications are long-lived), the Argo Workflows
verification checks that the workflow reaches a terminal state:

```go theme={null}
func (a *ArgoWorkflows) Verifications(
    config oapi.JobAgentConfig,
) ([]oapi.VerificationMetricSpec, error) {
    serverAddr, ok := config["serverUrl"].(string)
    if !ok || serverAddr == "" {
        return nil, nil
    }
    token, ok := config["token"].(string)
    if !ok || token == "" {
        return nil, nil
    }

    baseURL := serverAddr
    if !strings.HasPrefix(baseURL, "https://") {
        baseURL = "https://" + baseURL
    }
    workflowURL := fmt.Sprintf("%s/api/v1/workflows", baseURL)

    method := oapi.GET
    timeout := "10s"
    headers := map[string]string{
        "Authorization": fmt.Sprintf("Bearer %s", token),
    }
    var provider oapi.MetricProvider
    if err := provider.FromHTTPMetricProvider(oapi.HTTPMetricProvider{
        Url:     workflowURL,
        Method:  &method,
        Timeout: &timeout,
        Headers: &headers,
        Type:    oapi.Http,
    }); err != nil {
        return nil, fmt.Errorf("build argo workflows health check provider: %w", err)
    }

    successThreshold := 1
    failureCondition := "result.statusCode != 200 || result.json.status.phase == 'Failed' || result.json.status.phase == 'Error'"
    spec := oapi.VerificationMetricSpec{
        Name:             "argo-workflow-status",
        IntervalSeconds:  30,
        Count:            120,
        SuccessThreshold: &successThreshold,
        SuccessCondition: "result.statusCode == 200 && result.json.status.phase == 'Succeeded'",
        FailureCondition: &failureCondition,
        Provider:         provider,
    }
    return []oapi.VerificationMetricSpec{spec}, nil
}
```

#### Cancellation

When ctrlplane cancels a job, the agent should stop the Argo Workflow. The
`WorkflowSubmitter` interface includes a stop method:

```go theme={null}
type WorkflowSubmitter interface {
    SubmitWorkflow(...) (string, error)
    GetWorkflowStatus(...) (*WorkflowStatus, error)
    StopWorkflow(
        ctx context.Context,
        serverAddr, token, namespace, name string,
    ) error
}
```

The stop call uses Argo Workflows'
`PUT /api/v1/workflows/{namespace}/{name}/stop` endpoint, which terminates
running nodes and marks the workflow as failed. This integrates with the
existing job cancellation flow — when the reconciler transitions a job to
`cancelled`, the agent's poll loop detects this and calls `StopWorkflow`.

### Registry registration

The agent is registered alongside the existing agents in the job dispatch
controller:

```go theme={null}
func New(workerID string, pgxPool *pgxpool.Pool) *reconcile.Worker {
    // ...existing setup...

    dispatcher := jobagents.NewRegistry(&PostgresGetter{})
    dispatcher.Register(
        argo.New(&argo.GoApplicationUpserter{}, &PostgresSetter{Queue: enqueueQueue}),
    )
    dispatcher.Register(testrunner.New(&PostgresSetter{Queue: enqueueQueue}))
    dispatcher.Register(
        github.New(&github.GoGitHubWorkflowDispatcher{}, &PostgresSetter{Queue: enqueueQueue}),
    )
    dispatcher.Register(
        argoworkflows.New(
            &argoworkflows.HTTPWorkflowSubmitter{},
            &PostgresSetter{Queue: enqueueQueue},
        ),
    )

    // ...rest unchanged...
}
```

### API communication

The agent communicates with Argo Workflows via its REST API rather than
Kubernetes client-go. This matches the pattern established by the ArgoCD agent
(which uses the ArgoCD API, not the Kubernetes API) and avoids requiring
in-cluster access from the workspace engine:

| Operation       | Method | Endpoint                                    |
| --------------- | ------ | ------------------------------------------- |
| Submit workflow | POST   | `/api/v1/workflows/{namespace}`             |
| Get status      | GET    | `/api/v1/workflows/{namespace}/{name}`      |
| Stop workflow   | PUT    | `/api/v1/workflows/{namespace}/{name}/stop` |
| Get logs        | GET    | `/api/v1/workflows/{namespace}/{name}/log`  |

The `HTTPWorkflowSubmitter` implementation makes standard HTTP calls with the
bearer token:

```go theme={null}
type HTTPWorkflowSubmitter struct{}

func (s *HTTPWorkflowSubmitter) SubmitWorkflow(
    ctx context.Context,
    serverAddr, token, namespace string,
    workflow *unstructured.Unstructured,
) (string, error) {
    body, err := json.Marshal(map[string]any{
        "workflow": workflow.Object,
    })
    if err != nil {
        return "", fmt.Errorf("marshal workflow: %w", err)
    }

    url := fmt.Sprintf("%s/api/v1/workflows/%s", serverAddr, namespace)
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
    if err != nil {
        return "", err
    }
    req.Header.Set("Authorization", "Bearer "+token)
    req.Header.Set("Content-Type", "application/json")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return "", fmt.Errorf("submit workflow: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        respBody, _ := io.ReadAll(resp.Body)
        return "", fmt.Errorf("submit workflow: status %d: %s",
            resp.StatusCode, string(respBody))
    }

    var result struct {
        Metadata struct {
            Name string `json:"name"`
        } `json:"metadata"`
    }
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return "", fmt.Errorf("decode response: %w", err)
    }

    return result.Metadata.Name, nil
}
```

### TRPC and UI integration

#### Job agent config type

Add the `argo-workflows` type to the job agent config discriminated union in the
TRPC router:

```typescript theme={null}
const jobAgentConfig = z.discriminatedUnion("type", [
  // ...existing types...
  z.object({
    type: z.literal("argo-workflows"),
    serverUrl: z.string().url(),
    token: z.string(),
    namespace: z.string().optional(),
    template: z.string(),
  }),
]);
```

#### Deployment configuration

In the Terraform provider and CLI:

```hcl theme={null}
resource "ctrlplane_deployment" "api" {
  name = "API Service"
  slug = "api-service"

  job_agent {
    id = ctrlplane_job_agent.argo_wf.id

    argo_workflows {
      server_url = "https://argo-workflows.prod.example.com"
      token      = var.argo_workflows_token
      namespace  = "deployments"
      template   = file("${path.module}/deploy-workflow.yaml")
    }
  }
}
```

```yaml theme={null}
# CLI
type: Deployment
name: API Service
slug: api-service
jobAgent:
  ref: argo-workflows-agent
jobAgentConfig:
  serverUrl: https://argo-workflows.prod.example.com
  token: "${ARGO_WORKFLOWS_TOKEN}"
  namespace: deployments
  template: |
    apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    ...
```

### Plannable implementation (RFC 0002 integration)

The Argo Workflows agent can optionally implement `Plannable` by performing a
dry-run submission. Argo Workflows supports `--dry-run` and `--server-dry-run`
flags that validate and render the workflow without executing it. The rendered
output includes the fully resolved template with all parameter substitutions
applied:

```go theme={null}
func (a *ArgoWorkflows) Plan(
    ctx context.Context,
    dispatchCtx *oapi.DispatchContext,
) (*types.PlanResult, error) {
    serverAddr, token, namespace, template, err := ParseJobAgentConfig(
        dispatchCtx.JobAgentConfig,
    )
    if err != nil {
        return nil, err
    }

    wf, err := TemplateWorkflow(dispatchCtx, nil, template)
    if err != nil {
        return nil, err
    }

    rendered, err := json.Marshal(wf.Object)
    if err != nil {
        return nil, err
    }

    hash := sha256.Sum256(rendered)
    return &types.PlanResult{
        ContentHash: hex.EncodeToString(hash[:]),
        HasChanges:  true,
    }, nil
}
```

This enables plan-based diff detection (RFC 0002) and dry-run deployment plans
(RFC 0004) for Argo Workflows deployments. The hash comparison detects when a
version change produces an identical workflow spec — for example, when a
monorepo version bump only affects files unrelated to this deployment's workflow
template.

## Examples

### Database migration + application deploy

A deployment uses Argo Workflows to run a database migration before updating the
application. The workflow has a DAG structure: migrate → deploy → smoke test.

```yaml theme={null}
# Deployment config
type: Deployment
name: Payment Service
slug: payment-service
jobAgent:
  ref: argo-workflows
jobAgentConfig:
  serverUrl: https://argo.internal.example.com
  token: "${ARGO_TOKEN}"
  namespace: deploy-workflows
  template: |
    apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    metadata:
      generateName: payment-deploy-
      labels:
        ctrlplane.dev/job-id: "{[.job.id]}"
    spec:
      entrypoint: deploy-pipeline
      serviceAccountName: deploy-sa
      templates:
        - name: deploy-pipeline
          dag:
            tasks:
              - name: migrate
                template: db-migrate
              - name: deploy
                template: rolling-update
                dependencies: [migrate]
              - name: verify
                template: smoke-test
                dependencies: [deploy]

        - name: db-migrate
          container:
            image: "payment-service/migrator:{[.release.version.tag]}"
            command: ["/migrate", "up"]
            env:
              - name: DATABASE_URL
                value: "{[.release.variables.DATABASE_URL]}"

        - name: rolling-update
          container:
            image: bitnami/kubectl:latest
            command:
              - sh
              - -c
              - |
                kubectl set image deployment/payment-service \
                  app=payment-service:{[.release.version.tag]} \
                  -n {[.resource.config.namespace]} \
                  --record
                kubectl rollout status deployment/payment-service \
                  -n {[.resource.config.namespace]} \
                  --timeout=300s

        - name: smoke-test
          container:
            image: payment-service/tests:latest
            command: ["/run-smoke-tests"]
          retryStrategy:
            limit: 3
            backoff:
              duration: "5s"
              factor: 2
```

When version `v2.3.1` is deployed to the `us-east-1-cluster` resource in the
`production` environment:

1. Ctrlplane creates a job and dispatches to the `argo-workflows` agent.
2. The agent renders the template with the dispatch context (version tag
   `v2.3.1`, resource config, environment, variables).
3. The rendered Workflow CRD is submitted to Argo Workflows at
   `https://argo.internal.example.com`.
4. Argo Workflows executes: migrate → deploy → smoke test.
5. The agent polls workflow status. On `Succeeded`, it marks the ctrlplane job
   as successful.
6. Ctrlplane's promotion lifecycle advances to the next resource.

### Canary deployment with traffic splitting

A more complex workflow that performs a canary rollout with metric-based
validation:

```yaml theme={null}
# template (abbreviated)
spec:
  entrypoint: canary-rollout
  templates:
    - name: canary-rollout
      steps:
        - - name: deploy-canary
            template: deploy
            arguments:
              parameters:
                - name: variant
                  value: canary
                - name: replicas
                  value: "1"
        - - name: shift-traffic
            template: traffic-split
            arguments:
              parameters:
                - name: canary-weight
                  value: "10"
        - - name: validate-metrics
            template: check-metrics
        - - name: promote
            template: deploy
            arguments:
              parameters:
                - name: variant
                  value: stable
                - name: replicas
                  value: "{[.release.variables.REPLICA_COUNT]}"
        - - name: full-traffic
            template: traffic-split
            arguments:
              parameters:
                - name: canary-weight
                  value: "0"
```

This pattern is not expressible with ArgoCD sync hooks or GitHub Actions without
significant external tooling. Argo Workflows handles it natively.

### Lifecycle bracket hooks (RFC 0003 integration)

Argo Workflows is well-suited for the lifecycle hook deployments described in
RFC 0003. A "drain" deployment can use an Argo Workflow that runs
`kubectl drain` with proper PDB handling and timeout logic:

```yaml theme={null}
# drain deployment's job agent config
type: Deployment
name: node-drain
jobAgent:
  ref: argo-workflows
jobAgentConfig:
  template: |
    apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    metadata:
      generateName: drain-{[.resource.name]}-
    spec:
      entrypoint: drain
      templates:
        - name: drain
          container:
            image: bitnami/kubectl:latest
            command:
              - kubectl
              - drain
              - "{[.resource.name]}"
              - --ignore-daemonsets
              - --delete-emptydir-data
              - --timeout=600s
          activeDeadlineSeconds: 900
```

This gives the drain operation full workflow semantics: timeout handling, retry
on transient failure, observable status via the Argo Workflows UI, and job
metadata linking back to ctrlplane.

## Migration

* No schema changes required. The `job_agent` table already supports arbitrary
  agent types via the `type` column.
* The new agent type is registered in the workspace engine's controller. No
  changes to the API, reconciler, or promotion lifecycle.
* The `argo-workflows` type is added to the TRPC job agent config union. This is
  an additive change — existing types are unaffected.
* No dependency on the Argo Workflows Go SDK. The agent uses the REST API via
  standard `net/http` and `k8s.io/apimachinery/pkg/apis/meta/v1/unstructured`
  for CRD manipulation.
* Existing deployments using ArgoCD are unaffected. The `argo-cd` and
  `argo-workflows` types are fully independent.
* **Template delimiter migration.** Changing from `{{ }}` to `{[ ]}` delimiters
  affects all existing job agent templates. Existing ArgoCD and GitHub Actions
  deployment configs that use `{{ }}` must be updated to `{[ ]}`. This can be
  done in two phases: (1) add `{[ ]}` support alongside `{{ }}` with
  auto-detection of which delimiter style is present, (2) deprecate `{{ }}`
  after a migration window. The auto-detection checks whether the template
  contains `{[` — if so, use `{[ ]}` delimiters; otherwise fall back to `{{ }}`.

## Open Questions

1. **Authentication model.** The proposal uses a bearer token for Argo Workflows
   API access. In production, teams often use Kubernetes service account tokens
   with RBAC, OIDC tokens via Dex, or SSO. Should the agent support multiple
   auth methods (bearer token, kubeconfig, OIDC client credentials), or is
   bearer token sufficient as the initial implementation with others added
   later?

2. **Workflow cleanup.** Argo Workflows supports TTL-based cleanup
   (`ttlStrategy`) on the Workflow CRD. Should ctrlplane set a default TTL on
   submitted workflows to prevent accumulation, or leave this to the user's
   template? A sensible default (e.g., 24 hours) prevents resource leaks but may
   surprise users who want to inspect completed workflows.

3. **Log streaming.** The Argo Workflows API supports streaming logs from
   individual workflow nodes. Should the agent capture and surface step-level
   logs in ctrlplane's job detail view, or is linking to the Argo Workflows UI
   sufficient? Log streaming adds complexity but improves observability for
   teams that don't have direct access to the Argo Workflows dashboard.

4. **Restorable semantics.** The workspace engine supports `Restorable` agents
   that can re-establish in-flight jobs after a process restart. The Argo
   Workflows agent should implement this by querying workflow status on restore
   and resuming the poll loop. Should the initial implementation include restore
   support, or is it acceptable to mark orphaned jobs as failed on restart?

5. **Template validation.** The ArgoCD agent validates templates at dispatch
   time. Should the Argo Workflows agent provide pre-submission validation
   (e.g., via Argo's `--dry-run` API) to catch template errors before
   submission, or is post-submission error handling sufficient?

6. **Cluster-scoped vs namespaced Workflows.** Argo Workflows supports both
   `Workflow` (namespaced) and `ClusterWorkflowTemplate` (cluster-scoped)
   resources. The proposal only supports `Workflow`. Should `WorkflowTemplate`
   references be supported, where the template field specifies a
   `workflowTemplateRef` instead of inline specs? This would enable reuse of
   pre-defined cluster workflow templates.

7. **Template delimiter migration scope.** The `{[` / `]}` delimiter change
   benefits all agents but requires migrating every existing template. Should
   this be scoped to the Argo Workflows agent only (using a per-agent delimiter
   config), or applied globally across all agents? A global change is cleaner
   long-term but has a larger migration surface. The auto-detection fallback
   mitigates breakage, but dual-delimiter support adds complexity to the
   template engine.


# RFC 0006: Secret Provider Integration
Source: https://docs.ctrlplane.dev/rfc/0006-secret-provider-integration



| Category       | Status               | Created    | Author        |
| -------------- | -------------------- | ---------- | ------------- |
| Infrastructure | <Badge>Draft</Badge> | 2026-03-06 | Justin Brooks |

## Summary

Add a pluggable secret provider system that lets workspaces connect to external
secret managers (Doppler, HashiCorp Vault, AWS Secrets Manager, etc.) and
reference secrets by path rather than storing sensitive values in ctrlplane. A
new workspace-scoped `secret_provider` entity holds encrypted provider
credentials. A `SecretReference` type describes a secret's location. The
workspace-engine resolves references at variable resolution or job dispatch time
by calling the external provider.

## Motivation

Ctrlplane currently stores sensitive credentials at two levels:

**Job agent configs** — When an operator creates an ArgoCD runner, the API token
is stored in the `job_agent.config` JSON column. The same applies to Terraform
Cloud tokens.

**Deployment variables** — The `SensitiveValue` type exists in the OpenAPI
schema and the variable resolver recognizes it as a distinct value type, but
resolution is explicitly rejected:

```go theme={null}
case "sensitive":
    return nil, fmt.Errorf("sensitive values are not resolved by the variable resolver")
```

The `release_variable.encrypted` column, `job_variable.sensitive` flag, and
`@ctrlplane/secrets` AES-256 service all exist but are not wired end-to-end. The
infrastructure for handling secrets was scaffolded but never completed.

### Why this matters

1. **Compliance** — SOC 2, ISO 27001, and similar frameworks require that
   secrets are encrypted at rest and access-controlled.

2. **Secret rotation** — When credentials are stored directly in ctrlplane,
   rotation requires updating every agent config and variable value that
   references the credential. With an external provider, rotation happens in
   Doppler/Vault/AWS and ctrlplane picks up the new value on the next
   resolution.

3. **Separation of concerns** — Platform teams manage secrets in their existing
   secret management infrastructure. Application teams reference secrets by name
   in ctrlplane without needing access to the actual values.

4. **Multi-workspace** — Each workspace may use a different secret provider or
   account. The provider credentials themselves need per-workspace encrypted
   storage.

### Existing mechanisms and their limitations

**`@ctrlplane/secrets` AES-256** — A TypeScript encryption service exists in
`packages/secrets/` but is not imported anywhere in the application code. It
provides encrypt/decrypt with a 256-bit key from `VARIABLES_AES_256_KEY`. This
could handle encryption at rest but does not solve external provider
integration.

**`SensitiveValue` type** — Defined in the OpenAPI schema with a `valueHash`
field. The variable resolver detects it but refuses to resolve it, returning an
error. The intent was that a separate decryption path would handle these values,
but that path was never built.

**Go template interpolation** — The ArgoCD agent config already supports Go
templates, and the docs show `apiKey: "{{.variables.argocd_token}}"`. This means
credentials can technically flow through the variable system into agent configs
at dispatch time. The missing piece is a way to resolve variables whose values
come from an external source.

## Proposal

### New entity: `secret_provider`

A workspace-scoped entity that holds the credentials for connecting to an
external secret management service:

```sql theme={null}
CREATE TABLE secret_provider (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    config BYTEA NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(workspace_id, name)
);
```

| Column   | Description                                                                  |
| -------- | ---------------------------------------------------------------------------- |
| `type`   | Provider type: `doppler`, `vault`, `aws-secretsmanager`, `kubernetes`, `env` |
| `config` | AES-256 encrypted JSON blob with provider-specific credentials               |
| `name`   | Human-readable name, unique per workspace                                    |

The `config` column uses `BYTEA` for the encrypted payload rather than `JSON`,
since the ciphertext is opaque binary. The workspace-engine decrypts it in
memory using the instance-level `VARIABLES_AES_256_KEY` when a secret resolution
is needed.

Example configs (before encryption):

```json theme={null}
// Doppler
{ "serviceToken": "dp.st.xxxxxxxxxxxx" }

// Vault
{ "address": "https://vault.internal.company.com", "authMethod": "kubernetes", "role": "ctrlplane" }

// AWS Secrets Manager
{ "region": "us-east-1", "accessKeyId": "AKIA...", "secretAccessKey": "..." }

// Kubernetes
{ "namespace": "default" }

// Environment variables (instance-level, no config needed)
{}
```

### `SecretReference` type

A value object that describes where a secret lives:

```go theme={null}
type SecretReference struct {
    Provider string `json:"provider"`
    Path     string `json:"path,omitempty"`
    Key      string `json:"key"`
}
```

| Field      | Description                                                                                        |
| ---------- | -------------------------------------------------------------------------------------------------- |
| `Provider` | Matches `secret_provider.name` within the workspace                                                |
| `Path`     | Provider-specific path (e.g., `my-project/production` for Doppler, `secret/data/argocd` for Vault) |
| `Key`      | The specific secret key within the path                                                            |

The `Provider` field matches by `name`, not by `type`. This allows a workspace
to have multiple providers of the same type (e.g., two Doppler connections for
different teams).

### Go interfaces

A new package `pkg/secrets/` in the workspace-engine:

```go theme={null}
// Provider resolves secret references against an external secret store.
type Provider interface {
    Name() string
    Resolve(ctx context.Context, workspaceID string, ref SecretReference) (string, error)
}

// ProviderConfig holds the decrypted configuration for a workspace's
// secret provider integration.
type ProviderConfig struct {
    ID          string         `json:"id"`
    WorkspaceID string         `json:"workspaceId"`
    Type        string         `json:"type"`
    Name        string         `json:"name"`
    Config      map[string]any `json:"config"`
}

// ProviderConfigStore retrieves decrypted provider configuration for a
// workspace. The implementation handles AES-256 decryption transparently.
type ProviderConfigStore interface {
    GetProviderConfig(ctx context.Context, workspaceID string, providerName string) (*ProviderConfig, error)
    ListProviderConfigs(ctx context.Context, workspaceID string) ([]*ProviderConfig, error)
}

// Resolver holds a registry of provider implementations and resolves
// references by dispatching to the appropriate one.
type Resolver struct {
    store         ProviderConfigStore
    implementations map[string]ProviderFactory
}

// ProviderFactory creates a Provider instance from decrypted config.
type ProviderFactory func(config map[string]any) (Provider, error)
```

The `Resolver` uses the `ProviderConfigStore` to look up the workspace's
provider credentials, then passes them to the appropriate `ProviderFactory` to
construct a `Provider`, then calls `Resolve`. Provider instances can be cached
per workspace with a TTL to avoid repeated decryption and construction.

### Resolution points

The secret resolver integrates at two points in the workspace-engine:

#### 1. Variable resolution (deployment variables)

The `SensitiveValue` type is extended to carry a `SecretReference`:

```jsonnet theme={null}
SensitiveValue: {
  type: 'object',
  required: ['valueHash'],
  properties: {
    valueHash: { type: 'string' },
    secretRef: {
      type: 'object',
      required: ['provider', 'key'],
      properties: {
        provider: { type: 'string' },
        path: { type: 'string' },
        key: { type: 'string' },
      },
    },
  },
},
```

In `variableresolver/value.go`, the `"sensitive"` case changes from an error to
a resolution call:

```go theme={null}
case "sensitive":
    return resolveSensitive(ctx, secretResolver, workspaceID, value)
```

The resolved value becomes a `LiteralValue` and flows into the release like any
other variable. The variable's key is added to `release.EncryptedVariables` so
downstream consumers know it originated from a sensitive source.

#### 2. Job agent config resolution

For agent configs that use Go template interpolation (the existing mechanism),
no changes are needed. The credential flows through the variable system:

1. Deployment variable `argocd_token` is a `SensitiveValue` with a
   `SecretReference` pointing to Doppler
2. Variable resolver calls the secret provider, gets the plaintext token
3. Token lands in `release.Variables` and then `DispatchContext.Variables`
4. The agent config template `{{.variables.argocd_token}}` renders the value

This reuses the existing template interpolation rather than adding a second
resolution path in the agent config.

### Provider implementations

Each provider is a Go package under `pkg/secrets/<provider>/`:

**Doppler** (`pkg/secrets/doppler/`)

```
Path format: <project>/<config>
Key: secret name
API: GET /v3/configs/config/secret?project=X&config=Y&name=Z
Auth: Bearer <serviceToken> from ProviderConfig
```

**HashiCorp Vault** (`pkg/secrets/vault/`)

```
Path format: <mount>/<path> (e.g., secret/data/argocd)
Key: field within the secret data
API: Vault KV v2 read
Auth: Kubernetes auth, AppRole, or token from ProviderConfig
```

**AWS Secrets Manager** (`pkg/secrets/awssm/`)

```
Path format: secret ARN or name
Key: JSON field within the secret string
API: GetSecretValue
Auth: accessKeyId/secretAccessKey from ProviderConfig, or IAM role
```

**Kubernetes** (`pkg/secrets/kubernetes/`)

```
Path format: <namespace>/<secret-name>
Key: data key within the Secret
API: K8s client-go, in-cluster or kubeconfig
Auth: service account or kubeconfig from ProviderConfig
```

**Environment** (`pkg/secrets/env/`)

```
Path: unused
Key: environment variable name
Auth: none (reads from the workspace-engine process env)
```

The `env` provider is special — it does not use `ProviderConfigStore` and does
not require a `secret_provider` row. It reads directly from the process
environment and is available in every workspace by default. This is useful for
development and for instance-level secrets that are the same across all
workspaces.

### Bootstrap chain

The bootstrap dependency chain for secrets is:

```
Operator deploys workspace-engine
  → sets VARIABLES_AES_256_KEY as env var (one key per instance)

Workspace admin connects a secret provider via the UI
  → enters Doppler service token / Vault address / AWS credentials
  → API encrypts with AES-256, stores in secret_provider.config

User creates a deployment variable as SensitiveValue
  → references provider by name + path + key
  → no secret value touches the API or database

Workspace-engine resolves at release time
  → decrypts secret_provider.config in memory
  → calls external provider to fetch the actual secret
  → resolved value exists only in memory during resolution
```

One symmetric key on the process (`VARIABLES_AES_256_KEY`) gates access to all
provider credentials. This key is the same one that `@ctrlplane/secrets` in the
TypeScript layer uses, so both sides can encrypt/decrypt interchangeably.

### API

**REST endpoints:**

```
PUT  /v1/workspaces/{workspaceId}/secret-providers/{providerId}
GET  /v1/workspaces/{workspaceId}/secret-providers
GET  /v1/workspaces/{workspaceId}/secret-providers/{providerId}
DELETE /v1/workspaces/{workspaceId}/secret-providers/{providerId}
```

The `PUT` body:

```json theme={null}
{
  "name": "doppler-production",
  "type": "doppler",
  "config": {
    "serviceToken": "dp.st.xxxxxxxxxxxx"
  }
}
```

The API layer encrypts `config` before storing. The `GET` response never
includes the decrypted config — it returns the provider metadata only:

```json theme={null}
{
  "id": "...",
  "name": "doppler-production",
  "type": "doppler",
  "createdAt": "...",
  "updatedAt": "..."
}
```

### Web UI

**Settings > Secret Providers** — A workspace-level settings page to manage
provider connections. CRUD for `secret_provider` entities. The config form is
type-specific (Doppler shows a service token field, Vault shows address + auth
method, etc.).

**Variable editor** — When creating a deployment variable value, a "Secret"
option is available alongside "Literal" and "Reference". Selecting it shows a
provider dropdown (populated from the workspace's `secret_provider` entries) and
path/key inputs. The submitted value is a `SensitiveValue` with a `secretRef`.

**Runner creation** — The ArgoCD dialog can suggest using a variable reference
for the API key instead of a direct value, linking to the variable editor
workflow.

### Caching

External provider calls add latency to variable resolution. The resolver should
cache resolved values with a configurable TTL (default: 5 minutes). The cache is
keyed by `(workspaceID, providerName, path, key)` and held in memory on the
workspace-engine instance. Cache entries are invalidated when:

* The `secret_provider` config is updated (provider credentials changed)
* The TTL expires
* The workspace-engine restarts

This means secret rotation in the external provider takes effect within the TTL
window. For immediate rotation, the operator can update the `secret_provider`
entity to flush the cache.

### Audit

Secret resolution events should be recorded in the existing `event` table:

```json theme={null}
{
  "action": "secret.resolved",
  "workspaceId": "...",
  "payload": {
    "provider": "doppler-production",
    "path": "backend/production",
    "key": "ARGOCD_TOKEN",
    "releaseId": "...",
    "releaseTargetId": "..."
  }
}
```

The resolved value is never included in the audit event — only the reference
metadata.

## Examples

### ArgoCD API token from Doppler

1. Workspace admin creates a secret provider:

```
PUT /v1/workspaces/{wsId}/secret-providers/{id}
{
  "name": "doppler-platform",
  "type": "doppler",
  "config": { "serviceToken": "dp.st.xxxx" }
}
```

2. Deployment author creates a variable:

```
PUT /v1/workspaces/{wsId}/deployment-variable-values/{id}
{
  "deploymentVariableId": "...",
  "value": {
    "valueHash": "sha256:abcdef...",
    "secretRef": {
      "provider": "doppler-platform",
      "path": "backend/production",
      "key": "ARGOCD_TOKEN"
    }
  }
}
```

3. The ArgoCD agent config template uses the variable:

```yaml theme={null}
serverUrl: argocd.example.com:443
apiKey: "{{.variables.argocd_token}}"
template: |
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  ...
```

4. At release time, the variable resolver hits the `SensitiveValue`, calls the
   Doppler provider, and the resolved token flows through `release.Variables`
   into the dispatch context.

### Vault for database credentials

```json theme={null}
{
  "valueHash": "sha256:...",
  "secretRef": {
    "provider": "vault-prod",
    "path": "database/creds/api-service",
    "key": "password"
  }
}
```

The Vault provider reads the dynamic credential, and it resolves as a regular
`LiteralValue` in the release variables.

### Environment variable fallback (development)

For local development where no external provider is configured, the `env`
provider reads from the workspace-engine process:

```json theme={null}
{
  "valueHash": "sha256:...",
  "secretRef": {
    "provider": "env",
    "path": "",
    "key": "ARGOCD_TOKEN"
  }
}
```

## Migration

* The `secret_provider` table is additive — no existing tables are modified.
* Existing job agent configs with plaintext credentials continue to work. The
  migration path is to create a secret provider, create a deployment variable
  with a `SensitiveValue` reference, update the agent config template to use
  `{{.variables.xxx}}`, and remove the plaintext credential from the agent
  config.
* The `SensitiveValue` schema extension (`secretRef` field) is additive.
  Existing `SensitiveValue` entries without a `secretRef` will fail resolution
  with a clear error message.
* The `EncryptedVariables` field on releases, currently always `[]string{}`,
  will begin to be populated for variables resolved from secret providers.

## Open questions

1. **Should the `env` provider require a `secret_provider` row?** The current
   proposal makes it available by default without configuration. This is
   convenient for development but means any variable can read arbitrary
   environment variables from the workspace-engine process.

2. **Secret versioning** — Should the `SecretReference` support pinning to a
   specific secret version (e.g., Vault lease ID, AWS version stage)? This would
   improve reproducibility but complicates the reference format.

3. **Cross-workspace providers** — Should an instance admin be able to define
   providers at the instance level that are available to all workspaces? This
   avoids duplicating provider configs across workspaces but adds a
   multi-tenancy concern.

4. **Release snapshotting** — Should the resolved secret value be stored
   (encrypted) in `release_variable` for reproducibility, or should it always be
   re-resolved from the external provider? Snapshotting means a release is fully
   reproducible; re-resolving means secrets are never in the database but a
   provider outage blocks dispatch.

5. **RBAC** — Should creating `SensitiveValue` variables that reference a
   provider require a specific permission? This would prevent unprivileged users
   from reading arbitrary secrets from the workspace's providers.


# RFC 0009: Manual Action Job Agent
Source: https://docs.ctrlplane.dev/rfc/0007-manual-action-job-agent



| Category   | Status               | Created    | Author        |
| ---------- | -------------------- | ---------- | ------------- |
| Job Agents | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add a `manual-action` job agent type that represents a human task within a
deployment pipeline. When dispatched, the agent transitions the job to an
"action required" state, notifies assignees through configured channels (Slack,
email, webhook), and waits indefinitely until a human explicitly marks the task
as completed. This enables teams to embed manual operational steps — hardware
swaps, vendor coordination, compliance sign-offs, manual DNS changes — directly
into ctrlplane's promotion lifecycle, ensuring downstream deployments do not
proceed until the manual work is confirmed done.

## Motivation

### Automated agents assume automated execution

Ctrlplane's job agent model is built around dispatching work to external
systems that execute autonomously: ArgoCD syncs an Application, GitHub Actions
runs a workflow, Terraform Cloud applies a plan, Argo Workflows orchestrates a
DAG. In each case, ctrlplane sends a dispatch, the external system does the
work, and the agent reports back when it finishes.

But not every step in a deployment pipeline can be automated. Real-world
deployment procedures frequently include steps that require a human to
physically do something:

* **Hardware provisioning** — rack and cable a new server before software
  deployment can target it.
* **Manual DNS changes** — update DNS records in a provider that lacks API
  access or is managed by a different team.
* **Vendor coordination** — contact a third-party provider to enable a feature
  flag, update a firewall rule, or rotate a certificate.
* **Compliance checkpoints** — obtain a sign-off from a security or compliance
  officer that a change has been reviewed and meets regulatory requirements.
* **Customer communication** — notify a customer before a maintenance window
  begins, and confirm they have acknowledged.
* **Manual database operations** — run a migration in a restricted production
  environment where automated access is prohibited by policy.
* **Physical verification** — inspect that a deployment to an edge device or
  kiosk is functioning correctly before proceeding to the next location.

Today, teams handle these steps outside of ctrlplane — a Slack message, a Jira
ticket, a verbal confirmation — and then manually advance the pipeline by
updating the job status via the API or UI. This works but has three problems:

1. **No orchestration signal.** Ctrlplane does not know that a manual step
   exists. The pipeline appears stalled with no indication of what is being
   waited on or who is responsible.
2. **No notification routing.** There is no mechanism to automatically notify
   the right person when a manual step is ready. The deployer must remember to
   ping someone.
3. **No audit trail.** There is no record of who completed the manual step,
   when, or what evidence they provided. The job status update only records that
   the job transitioned to success.

### Distinct from the approval policy

The existing approval policy (see `policies/approval`) gates whether a release
*should proceed* — it is a governance checkpoint. A user reviews the proposed
change and approves or rejects it. The release itself has not started executing;
the approval decides whether it will.

A manual action is different. It represents *work that must be performed* as
part of the deployment execution. The deployment has already been approved and
is in progress. The manual action is a step within that execution that happens
to require a human instead of a machine:

```text theme={null}
Approval policy                Manual action agent
─────────────────              ────────────────────
"Should we deploy v2.3.1       "Swap the failed disk in
 to production?"                rack-7-slot-3 before we
                                deploy to this node."

Gate before execution.          Step during execution.
Evaluator in policy pipeline.   Job agent in dispatch pipeline.
Blocks job creation.            Blocks job completion.
```

Conflating the two creates semantic confusion. An approval is a policy decision.
A manual action is an execution step. They have different lifecycles, different
actors, different notification requirements, and different audit semantics.

### Why not use an external ticketing system?

Teams could model manual steps as GitHub Actions workflows that create a Jira
ticket and poll for its resolution. But this requires:

* A CI runner continuously polling an external system.
* Credential management for the ticketing system API.
* Custom logic to map ticket state transitions to ctrlplane job status updates.
* No native integration with ctrlplane's notification system, audit log, or UI.

The manual action agent keeps the orchestration within ctrlplane. The external
integration is limited to notification delivery (Slack, email, webhook) rather
than execution tracking.

## Proposal

### Agent type and config

Register a new agent type `manual-action` in the workspace engine's job agent
registry. The job agent config describes what the human needs to do and who
should be notified:

```json theme={null}
{
  "type": "manual-action",
  "name": "Swap failed disk",
  "description": "Replace the failed disk in {[.resource.name]} before deployment proceeds.",
  "assignees": ["ops-team"],
  "channels": [
    {
      "type": "slack",
      "channelId": "C04XXXXXX"
    }
  ],
  "timeout": "PT24H",
  "requireEvidence": true
}
```

| Field             | Required | Description                                                                                                   |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------- |
| `name`            | Yes      | Short name for the manual task, displayed in the UI and notifications.                                        |
| `description`     | Yes      | Go template string describing what the human needs to do. Receives the dispatch context.                      |
| `assignees`       | No       | List of team slugs or user emails to notify. If omitted, the notification goes to the configured channel.     |
| `channels`        | No       | Notification channels for this task. Falls back to workspace notification defaults if omitted.                |
| `timeout`         | No       | ISO 8601 duration after which the job is marked as failed if not completed. Default: no timeout.              |
| `requireEvidence` | No       | If true, the completion request must include an `evidence` field (URL, description, or attachment reference). |

The `description` field is a Go template rendered with `{[ ]}` delimiters
(matching the convention from RFC 0005). This allows the task description to
include deployment-specific context:

```text theme={null}
Replace the failed disk in {[.resource.name]} (rack {[.resource.metadata.rack]},
slot {[.resource.metadata.slot]}). After replacement, verify the disk is online
with `lsblk` and confirm the RAID array is rebuilding.

Deployment: {[.deployment.slug]}
Environment: {[.environment.name]}
Version: {[.release.version.tag]}
```

### Dispatch lifecycle

When the workspace engine dispatches a job to the `manual-action` agent, the
following sequence occurs:

```text theme={null}
┌─────────────────┐     ┌──────────────┐     ┌─────────────────┐
│ workspace-engine│     │ manual-action│     │  notification   │
│   (dispatch)    │     │    agent     │     │    system       │
└────────┬────────┘     └──────┬───────┘     └────────┬────────┘
         │  Dispatch(job)      │                      │
         │────────────────────►│                      │
         │                     │  render description  │
         │                     │  from template       │
         │                     │                      │
         │                     │  UpdateJob(           │
         │                     │    action_required)   │
         │                     │                      │
         │                     │  Send notifications  │
         │                     │─────────────────────►│
         │                     │                      │  Slack message
         │                     │                      │  with action button
         │                     │                      │
         │                     │        (waiting for human)
         │                     │                      │
         │                     │  ◄── human clicks    │
         │                     │      "Complete" in   │
         │                     │      Slack / UI / API│
         │                     │                      │
         │                     │  UpdateJob(successful)│
         │                     │                      │
         ▼                     ▼                      ▼
```

The key difference from other agents: after dispatch, there is no polling loop.
The agent transitions the job to `action_required` and returns. The job remains
in this state until an external signal (API call, Slack interaction, UI button)
advances it. There is no background goroutine watching an external system.

### Job status: `action_required`

A new job status `action_required` is added to the `JobStatus` enum. This
status indicates that the job has been dispatched and is waiting for a human to
complete a task. It is semantically distinct from:

* `pending` — job has not been dispatched yet.
* `in_progress` — job has been dispatched and an external system is actively
  executing it.
* `action_required` — job has been dispatched but requires a human to do
  something before it can complete.

The workspace engine treats `action_required` the same as `in_progress` for
promotion lifecycle purposes: downstream deployments wait for the job to reach
a terminal state (`successful` or `failure`).

```sql theme={null}
ALTER TYPE job_status ADD VALUE 'action_required' AFTER 'in_progress';
```

The UI renders `action_required` jobs with a distinct visual treatment — an
amber indicator with a call-to-action button — to differentiate them from
automated jobs that are still running.

### Implementation

#### Go types

```go theme={null}
package manualaction

type ManualAction struct {
    setter   Setter
    notifier Notifier
}

type Setter interface {
    UpdateJob(
        ctx context.Context,
        jobID string,
        status oapi.JobStatus,
        message string,
        metadata map[string]string,
    ) error
}

type Notifier interface {
    SendManualActionNotification(
        ctx context.Context,
        notification ManualActionNotification,
    ) error
}

type ManualActionNotification struct {
    JobID         string
    WorkspaceID   string
    Name          string
    Description   string
    Assignees     []string
    Channels      []NotificationChannel
    DeploymentCtx *oapi.DispatchContext
    CallbackURL   string
}

type NotificationChannel struct {
    Type      string // "slack", "email", "webhook"
    ChannelID string
}
```

#### Dispatchable implementation

```go theme={null}
var _ types.Dispatchable = &ManualAction{}

func New(setter Setter, notifier Notifier) *ManualAction {
    return &ManualAction{setter: setter, notifier: notifier}
}

func (a *ManualAction) Type() string {
    return "manual-action"
}

func (a *ManualAction) Dispatch(ctx context.Context, job *oapi.Job) error {
    dispatchCtx := job.DispatchContext
    if dispatchCtx == nil {
        return fmt.Errorf("job %s has no dispatch context", job.Id)
    }

    cfg, err := ParseConfig(dispatchCtx.JobAgentConfig)
    if err != nil {
        return fmt.Errorf("parse manual-action config: %w", err)
    }

    description, err := RenderDescription(cfg.Description, dispatchCtx, job)
    if err != nil {
        return fmt.Errorf("render description: %w", err)
    }

    metadata := map[string]string{
        "manual-action/name":        cfg.Name,
        "manual-action/description": description,
    }
    if cfg.RequireEvidence {
        metadata["manual-action/require-evidence"] = "true"
    }

    if err := a.setter.UpdateJob(
        ctx, job.Id, oapi.JobStatusActionRequired, "", metadata,
    ); err != nil {
        return fmt.Errorf("update job to action_required: %w", err)
    }

    callbackURL := fmt.Sprintf(
        "/api/v1/jobs/%s/complete",
        job.Id,
    )

    notification := ManualActionNotification{
        JobID:         job.Id,
        WorkspaceID:   dispatchCtx.WorkspaceId,
        Name:          cfg.Name,
        Description:   description,
        Assignees:     cfg.Assignees,
        Channels:      cfg.Channels,
        DeploymentCtx: dispatchCtx,
        CallbackURL:   callbackURL,
    }

    go func() {
        asyncCtx := context.WithoutCancel(ctx)
        if err := a.notifier.SendManualActionNotification(
            asyncCtx, notification,
        ); err != nil {
            _ = a.setter.UpdateJob(
                asyncCtx, job.Id, oapi.JobStatusActionRequired,
                fmt.Sprintf("notification delivery failed: %s", err.Error()),
                nil,
            )
        }
    }()

    if cfg.Timeout != "" {
        go a.enforceTimeout(context.WithoutCancel(ctx), job.Id, cfg.Timeout)
    }

    return nil
}
```

#### Timeout enforcement

If a timeout is configured, a background goroutine waits for the duration and
then checks whether the job is still in `action_required` state. If so, it
transitions the job to `failure`:

```go theme={null}
func (a *ManualAction) enforceTimeout(
    ctx context.Context,
    jobID string,
    timeoutStr string,
) {
    duration, err := iso8601.ParseDuration(timeoutStr)
    if err != nil {
        return
    }

    select {
    case <-ctx.Done():
        return
    case <-time.After(duration):
    }

    job, err := a.getter.GetJob(ctx, uuid.MustParse(jobID))
    if err != nil {
        return
    }

    if job.Status == oapi.JobStatusActionRequired {
        _ = a.setter.UpdateJob(
            ctx, jobID, oapi.JobStatusFailure,
            fmt.Sprintf("manual action timed out after %s", timeoutStr),
            nil,
        )
    }
}
```

#### Description rendering

The description template is rendered using the same `templatefuncs` pipeline
as other job agents, with `{[` / `]}` delimiters:

```go theme={null}
func RenderDescription(
    tmpl string,
    dispatchCtx *oapi.DispatchContext,
    job *oapi.Job,
) (string, error) {
    t, err := templatefuncs.NewWithDelims("manualActionDescription").Parse(tmpl)
    if err != nil {
        return "", fmt.Errorf("parse template: %w", err)
    }

    data := dispatchCtx.Map()
    data["job"] = structToMap(job)

    var buf bytes.Buffer
    if err := t.Execute(&buf, data); err != nil {
        return "", fmt.Errorf("execute template: %w", err)
    }

    return buf.String(), nil
}
```

### Completion API

A new endpoint allows humans (or integrations) to mark a manual action job as
completed:

```
POST /v1/jobs/{jobId}/complete
```

**Request body:**

```json theme={null}
{
  "status": "successful",
  "message": "Disk replaced and RAID rebuild verified.",
  "evidence": "https://runbook.internal/disk-swap/RUN-4521"
}
```

| Field      | Required    | Description                                                                           |
| ---------- | ----------- | ------------------------------------------------------------------------------------- |
| `status`   | No          | `successful` (default) or `failure`. Allows the human to report that the task failed. |
| `message`  | No          | Free-text message describing what was done or why it failed.                          |
| `evidence` | Conditional | Required if `requireEvidence = true` in the agent config. URL or description.         |

The endpoint validates:

1. The job exists and is in `action_required` status.
2. The caller has permission to complete jobs in this workspace.
3. If `requireEvidence` is configured, the `evidence` field is present and
   non-empty.

On success, the job transitions to the requested terminal status and the
promotion lifecycle advances.

```go theme={null}
func (h *Handler) CompleteJob(w http.ResponseWriter, r *http.Request) {
    jobID := chi.URLParam(r, "jobId")

    var req CompleteJobRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "invalid request body", http.StatusBadRequest)
        return
    }

    job, err := h.getter.GetJob(r.Context(), uuid.MustParse(jobID))
    if err != nil {
        http.Error(w, "job not found", http.StatusNotFound)
        return
    }

    if job.Status != oapi.JobStatusActionRequired {
        http.Error(w,
            fmt.Sprintf("job is in %s status, expected action_required", job.Status),
            http.StatusConflict,
        )
        return
    }

    requireEvidence := job.Metadata["manual-action/require-evidence"] == "true"
    if requireEvidence && req.Evidence == "" {
        http.Error(w, "evidence is required for this manual action", http.StatusBadRequest)
        return
    }

    status := oapi.JobStatusSuccessful
    if req.Status == "failure" {
        status = oapi.JobStatusFailure
    }

    metadata := map[string]string{
        "manual-action/completed-by": r.Context().Value(ctxUserID).(string),
        "manual-action/completed-at": time.Now().UTC().Format(time.RFC3339),
    }
    if req.Evidence != "" {
        metadata["manual-action/evidence"] = req.Evidence
    }

    if err := h.setter.UpdateJob(
        r.Context(), jobID, status, req.Message, metadata,
    ); err != nil {
        http.Error(w, "failed to update job", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{
        "jobId":  jobID,
        "status": string(status),
    })
}
```

### Slack integration

The Slack integration is the primary notification channel for manual actions.
When a manual action job is dispatched, a Slack message is sent to the
configured channel with an interactive Block Kit layout:

#### Message format

```json theme={null}
{
  "channel": "C04XXXXXX",
  "blocks": [
    {
      "type": "header",
      "text": {
        "type": "plain_text",
        "text": "🔧 Manual Action Required"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Swap failed disk*\n\nReplace the failed disk in us-east-1-node-7 (rack 7, slot 3). After replacement, verify the disk is online with `lsblk` and confirm the RAID array is rebuilding."
      }
    },
    {
      "type": "section",
      "fields": [
        {
          "type": "mrkdwn",
          "text": "*Deployment:*\ninfra-rollout"
        },
        {
          "type": "mrkdwn",
          "text": "*Environment:*\nproduction"
        },
        {
          "type": "mrkdwn",
          "text": "*Resource:*\nus-east-1-node-7"
        },
        {
          "type": "mrkdwn",
          "text": "*Version:*\nv1.4.2"
        }
      ]
    },
    {
      "type": "actions",
      "elements": [
        {
          "type": "button",
          "text": {
            "type": "plain_text",
            "text": "✅ Mark as Completed"
          },
          "style": "primary",
          "action_id": "manual_action_complete",
          "value": "<job-id>"
        },
        {
          "type": "button",
          "text": {
            "type": "plain_text",
            "text": "❌ Report Failure"
          },
          "style": "danger",
          "action_id": "manual_action_fail",
          "value": "<job-id>"
        },
        {
          "type": "button",
          "text": {
            "type": "plain_text",
            "text": "View in Ctrlplane"
          },
          "url": "https://your-ctrlplane-instance.com/workspaces/.../jobs/<job-id>"
        }
      ]
    }
  ]
}
```

#### Interaction handler

When a user clicks a button in Slack, the Slack API sends an interaction
payload to ctrlplane's Slack integration endpoint. The handler:

1. Verifies the Slack request signature.
2. Extracts the `action_id` and `value` (job ID).
3. Resolves the Slack user to a ctrlplane user via the workspace's Slack
   integration mapping.
4. Calls the completion API internally.
5. Updates the original Slack message to reflect the new status.

```go theme={null}
func (h *SlackInteractionHandler) HandleInteraction(
    ctx context.Context,
    payload slack.InteractionCallback,
) error {
    action := payload.ActionCallback.BlockActions[0]
    jobID := action.Value
    slackUserID := payload.User.ID

    ctrlplaneUser, err := h.userMapper.ResolveSlackUser(ctx, slackUserID)
    if err != nil {
        return fmt.Errorf("resolve slack user %s: %w", slackUserID, err)
    }

    var status oapi.JobStatus
    var message string

    switch action.ActionID {
    case "manual_action_complete":
        status = oapi.JobStatusSuccessful
        message = fmt.Sprintf(
            "Completed by %s via Slack", ctrlplaneUser.Name,
        )
    case "manual_action_fail":
        status = oapi.JobStatusFailure
        message = fmt.Sprintf(
            "Reported as failed by %s via Slack", ctrlplaneUser.Name,
        )
    default:
        return fmt.Errorf("unknown action: %s", action.ActionID)
    }

    metadata := map[string]string{
        "manual-action/completed-by":     ctrlplaneUser.Id,
        "manual-action/completed-at":     time.Now().UTC().Format(time.RFC3339),
        "manual-action/completed-via":    "slack",
        "manual-action/slack-user-id":    slackUserID,
        "manual-action/slack-channel-id": payload.Channel.ID,
    }

    if err := h.setter.UpdateJob(
        ctx, jobID, status, message, metadata,
    ); err != nil {
        return fmt.Errorf("update job: %w", err)
    }

    return h.updateSlackMessage(ctx, payload, status, ctrlplaneUser.Name)
}
```

#### Message update on completion

After the job is completed (via Slack or any other method), the original Slack
message is updated to show the resolved state. The action buttons are removed
and replaced with a status block:

```json theme={null}
{
  "type": "context",
  "elements": [
    {
      "type": "mrkdwn",
      "text": "✅ Completed by @jane.doe at 2026-03-13 14:32 UTC"
    }
  ]
}
```

This prevents double-completion and provides an at-a-glance record in the Slack
channel.

#### Evidence collection via Slack modal

When `requireEvidence = true`, clicking "Mark as Completed" opens a Slack modal
instead of immediately completing the job. The modal prompts for:

* A text description of what was done.
* An optional URL to supporting evidence (runbook, screenshot, monitoring
  dashboard).

```json theme={null}
{
  "type": "modal",
  "title": {
    "type": "plain_text",
    "text": "Complete Manual Action"
  },
  "submit": {
    "type": "plain_text",
    "text": "Complete"
  },
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Swap failed disk*\nReplace the failed disk in us-east-1-node-7..."
      }
    },
    {
      "type": "input",
      "block_id": "evidence_message",
      "label": {
        "type": "plain_text",
        "text": "What was done?"
      },
      "element": {
        "type": "plain_text_input",
        "action_id": "evidence_message_input",
        "multiline": true,
        "placeholder": {
          "type": "plain_text",
          "text": "Describe the action taken..."
        }
      }
    },
    {
      "type": "input",
      "block_id": "evidence_url",
      "optional": true,
      "label": {
        "type": "plain_text",
        "text": "Evidence URL"
      },
      "element": {
        "type": "url_text_input",
        "action_id": "evidence_url_input",
        "placeholder": {
          "type": "plain_text",
          "text": "https://..."
        }
      }
    }
  ],
  "private_metadata": "{\"job_id\": \"<job-id>\"}"
}
```

The modal submission handler extracts the evidence, calls the completion API
with the evidence payload, and updates the Slack message.

### Notification channels

The manual action agent supports multiple notification channels through the
existing notification system. Each channel type has a specific renderer:

| Channel   | Behavior                                                                           |
| --------- | ---------------------------------------------------------------------------------- |
| `slack`   | Sends a Block Kit message with interactive buttons. Supports completion via Slack. |
| `email`   | Sends an email with task description and a deep link to the ctrlplane UI.          |
| `webhook` | POSTs a JSON payload to a configured URL. Used for custom integrations.            |

The notification is sent once at dispatch time. Reminders can be configured to
re-send the notification at intervals while the job remains in
`action_required`:

```json theme={null}
{
  "type": "manual-action",
  "name": "Update DNS records",
  "description": "...",
  "channels": [
    {
      "type": "slack",
      "channelId": "C04XXXXXX"
    }
  ],
  "reminder": {
    "interval": "PT1H",
    "maxReminders": 3
  }
}
```

| Field                   | Description                                                                    |
| ----------------------- | ------------------------------------------------------------------------------ |
| `reminder.interval`     | ISO 8601 duration between reminders.                                           |
| `reminder.maxReminders` | Maximum number of reminders to send before stopping. Default 0 (no reminders). |

### Registry registration

```go theme={null}
func New(workerID string, pgxPool *pgxpool.Pool) *reconcile.Worker {
    // ...existing setup...

    dispatcher := jobagents.NewRegistry(&PostgresGetter{})
    dispatcher.Register(
        argo.New(&argo.GoApplicationUpserter{}, &PostgresSetter{Queue: enqueueQueue}),
    )
    dispatcher.Register(testrunner.New(&PostgresSetter{Queue: enqueueQueue}))
    dispatcher.Register(
        github.New(
            &github.GoGitHubWorkflowDispatcher{},
            &PostgresSetter{Queue: enqueueQueue},
        ),
    )
    dispatcher.Register(
        manualaction.New(
            &PostgresSetter{Queue: enqueueQueue},
            &NotificationSender{},
        ),
    )

    // ...rest unchanged...
}
```

### TRPC and UI integration

#### Job agent config type

```typescript theme={null}
const jobAgentConfig = z.discriminatedUnion("type", [
  // ...existing types...
  z.object({
    type: z.literal("manual-action"),
    name: z.string(),
    description: z.string(),
    assignees: z.array(z.string()).optional(),
    channels: z
      .array(
        z.object({
          type: z.enum(["slack", "email", "webhook"]),
          channelId: z.string(),
        }),
      )
      .optional(),
    timeout: z.string().optional(),
    requireEvidence: z.boolean().optional(),
    reminder: z
      .object({
        interval: z.string(),
        maxReminders: z.number().int().min(0).optional(),
      })
      .optional(),
  }),
]);
```

#### Job completion tRPC route

```typescript theme={null}
job.complete: protectedProcedure
  .input(
    z.object({
      jobId: z.string().uuid(),
      status: z.enum(["successful", "failure"]).default("successful"),
      message: z.string().optional(),
      evidence: z.string().optional(),
    }),
  )
  .mutation(async ({ ctx, input }) => {
    const job = await ctx.db
      .select()
      .from(schema.job)
      .where(eq(schema.job.id, input.jobId))
      .then(takeFirstOrNull);

    if (job == null)
      throw new TRPCError({ code: "NOT_FOUND" });

    if (job.status !== "action_required")
      throw new TRPCError({
        code: "PRECONDITION_FAILED",
        message: `Job is ${job.status}, expected action_required`,
      });

    const requireEvidence =
      job.metadata?.["manual-action/require-evidence"] === "true";
    if (requireEvidence && !input.evidence)
      throw new TRPCError({
        code: "BAD_REQUEST",
        message: "Evidence is required for this manual action",
      });

    await ctx.db
      .update(schema.job)
      .set({
        status: input.status,
        message: input.message ?? "",
        metadata: {
          ...job.metadata,
          "manual-action/completed-by": ctx.session.user.id,
          "manual-action/completed-at": new Date().toISOString(),
          "manual-action/completed-via": "ui",
          ...(input.evidence
            ? { "manual-action/evidence": input.evidence }
            : {}),
        },
      })
      .where(eq(schema.job.id, input.jobId));

    await enqueuePolicyEval(ctx.db, job.releaseTargetId);
  })
```

#### UI: job detail view

When a job has status `action_required`, the job detail view displays:

1. **Task description** — the rendered description from the agent config,
   formatted as markdown.
2. **Assignees** — who is responsible for completing the task.
3. **Status timeline** — when the job was dispatched, when notifications were
   sent, when reminders were sent.
4. **Action buttons** — "Mark as Completed" and "Report Failure" buttons.
5. **Evidence field** — if `requireEvidence` is true, a text input and URL
   field that must be filled before completion.
6. **Timeout indicator** — if a timeout is configured, a countdown showing
   remaining time.

The release target overview shows `action_required` jobs with an amber badge
and the task name, making it immediately visible which deployments are waiting
on human action.

### Deployment configuration

#### Terraform

```hcl theme={null}
resource "ctrlplane_deployment" "infra_rollout" {
  name = "Infrastructure Rollout"
  slug = "infra-rollout"

  job_agent {
    id = ctrlplane_job_agent.manual.id

    manual_action {
      name        = "Hardware verification"
      description = <<-EOT
        Verify that node {[.resource.name]} has been physically
        provisioned and is network-reachable.

        1. Confirm the node is racked and cabled.
        2. Verify IPMI connectivity: ping {[.resource.metadata.ipmi_ip]}
        3. Confirm the node appears in the inventory system.
      EOT

      assignees = ["platform-ops"]

      channel {
        type       = "slack"
        channel_id = "C04XXXXXX"
      }

      timeout          = "PT8H"
      require_evidence = true
    }
  }
}
```

#### CLI YAML

```yaml theme={null}
type: Deployment
name: Infrastructure Rollout
slug: infra-rollout
jobAgent:
  ref: manual-action-agent
jobAgentConfig:
  name: Hardware verification
  description: |
    Verify that node {[.resource.name]} has been physically
    provisioned and is network-reachable.
  assignees:
    - platform-ops
  channels:
    - type: slack
      channelId: C04XXXXXX
  timeout: PT8H
  requireEvidence: true
```

## Examples

### Multi-step deployment with manual checkpoint

A system has three deployments in sequence: database migration (automated),
hardware verification (manual), and application deploy (automated). The manual
step ensures a human confirms the target node is ready before the application
is deployed to it:

```yaml theme={null}
# System: edge-rollout
# Environment: production
# Deployments (ordered by dependency):

# 1. Automated — runs database migration via Argo Workflows
type: Deployment
name: Database Migration
slug: db-migration
jobAgent:
  ref: argo-workflows
jobAgentConfig:
  template: |
    apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    metadata:
      generateName: migrate-
    spec:
      entrypoint: migrate
      templates:
        - name: migrate
          container:
            image: "db-migrator:{[.release.version.tag]}"

---

# 2. Manual — human verifies the edge device
type: Deployment
name: Device Verification
slug: device-verification
jobAgent:
  ref: manual-action
jobAgentConfig:
  name: "Verify edge device {[.resource.name]}"
  description: |
    The edge device {[.resource.name]} at location
    {[.resource.metadata.location]} needs physical verification
    before the v{[.release.version.tag]} firmware is deployed.

    Checklist:
    - Device is powered on and network-reachable
    - Current firmware version matches expected baseline
    - No hardware alerts in the device management console
    - Device storage has >20% free space
  assignees:
    - field-ops
  channels:
    - type: slack
      channelId: C04FIELD_OPS
  timeout: PT48H
  requireEvidence: true
  reminder:
    interval: PT4H
    maxReminders: 3

---

# 3. Automated — deploys firmware via ArgoCD
type: Deployment
name: Firmware Deploy
slug: firmware-deploy
jobAgent:
  ref: argo-cd
jobAgentConfig:
  # ...standard ArgoCD config...
```

The deployment dependency policy ensures these run in order. When the database
migration completes, ctrlplane dispatches the device verification job. A Slack
message appears in `#field-ops`:

```
🔧 Manual Action Required

Verify edge device us-west-2-kiosk-14

The edge device us-west-2-kiosk-14 at location "Portland Store #42"
needs physical verification before the v3.1.0 firmware is deployed.

Checklist:
- Device is powered on and network-reachable
- Current firmware version matches expected baseline
- No hardware alerts in the device management console
- Device storage has >20% free space

Deployment: device-verification
Environment: production
Resource: us-west-2-kiosk-14
Version: v3.1.0

[✅ Mark as Completed]  [❌ Report Failure]  [View in Ctrlplane]
```

A field technician visits the kiosk, verifies the checklist, clicks "Mark as
Completed" in Slack, enters "All checks passed. Device firmware at v3.0.2
baseline. 45% storage free." as evidence, and the firmware deploy proceeds
automatically.

### Customer notification gate

Before deploying a breaking API change to a customer's dedicated environment,
the account team must confirm the customer has been notified and has
acknowledged the maintenance window:

```yaml theme={null}
type: Deployment
name: Customer Notification
slug: customer-notification
jobAgent:
  ref: manual-action
jobAgentConfig:
  name: "Notify customer for {[.environment.name]}"
  description: |
    Contact the customer for environment {[.environment.name]}
    regarding the upcoming v{[.release.version.tag]} deployment.

    This version includes breaking API changes documented at:
    https://docs.example.com/changelog/{[.release.version.tag]}

    Steps:
    1. Send the maintenance notification email using the template
       in the runbook.
    2. Wait for customer acknowledgment (email reply or portal
       confirmation).
    3. Mark as completed only after receiving acknowledgment.
  assignees:
    - account-management
  channels:
    - type: slack
      channelId: C04ACCOUNTS
    - type: email
      channelId: account-team@example.com
  timeout: PT72H
  requireEvidence: true
```

### Compliance sign-off

A regulated deployment requires a compliance officer to review and sign off
before proceeding:

```yaml theme={null}
type: Deployment
name: Compliance Review
slug: compliance-review
jobAgent:
  ref: manual-action
jobAgentConfig:
  name: "Compliance review for {[.deployment.slug]} v{[.release.version.tag]}"
  description: |
    Review the deployment of {[.deployment.slug]} version
    {[.release.version.tag]} to {[.environment.name]} for
    compliance with SOC 2 change management requirements.

    Review items:
    - Change request ticket has been approved
    - Rollback plan is documented
    - Monitoring alerts are configured
    - Change window is within approved schedule

    Provide the change request ticket URL as evidence.
  assignees:
    - compliance-team
  timeout: PT24H
  requireEvidence: true
```

## Migration

* The `action_required` value is added to the `job_status` enum. This is an
  additive change — existing jobs are unaffected.
* No schema changes to existing tables. The manual action metadata is stored
  in the job's existing `metadata` JSONB column.
* The completion API endpoint is new. No changes to existing endpoints.
* The Slack interaction handler is new. It is registered alongside the existing
  Slack integration webhook handlers.
* The `manual-action` agent type is registered in the workspace engine's
  controller. No changes to the reconciler or promotion lifecycle beyond
  recognizing the new `action_required` status.
* The notification system must support the `SendManualActionNotification`
  method. This extends the existing `Notifier` interface. If the notification
  system is not configured, the agent still transitions the job to
  `action_required` — the task is visible in the UI but no external
  notification is sent.

## Open Questions

1. **Reassignment.** The initial proposal assigns the task at dispatch time via
   the `assignees` field in the agent config. Should the UI and API support
   reassigning a manual action to a different user or team after dispatch? This
   is useful when the original assignee is unavailable, but adds complexity to
   the notification flow (the new assignee needs to be notified, the original
   assignee's notification should be updated).

2. **Escalation.** If a manual action is not completed within a configurable
   period (shorter than the timeout), should the system escalate to a different
   set of assignees? For example, after 2 hours notify the team lead, after 4
   hours notify the on-call manager. This is a common pattern in incident
   management tools but adds significant complexity.

3. **Partial completion.** Some manual tasks have multiple steps (a checklist).
   Should the agent support partial completion where each checklist item is
   tracked independently, or is a single "completed/failed" status sufficient?
   Partial completion provides better visibility but the checklist structure
   must be defined in the agent config and rendered in both the UI and Slack.

4. **Restorable semantics.** After a workspace-engine restart, `action_required`
   jobs with configured timeouts need their timeout goroutines restarted. The
   agent should implement `Restorable` to query for `action_required` jobs on
   startup and re-establish timeout enforcement. Should the initial
   implementation include restore support, or is it acceptable to lose timeout
   enforcement on restart (the job remains in `action_required` indefinitely
   until manually completed or failed)?

5. **Slack app permissions.** The interactive Slack integration requires the
   ctrlplane Slack app to have `chat:write`, `commands`, and
   `interactions` scopes. If the workspace does not have a Slack integration
   configured, should the agent fall back to a non-interactive notification
   (plain message without buttons), or should it fail at dispatch time with a
   configuration error?

6. **Idempotent completion.** If multiple people click "Complete" in Slack
   simultaneously, the second request should be a no-op (the job is already in
   a terminal state). The current proposal handles this via the status check in
   the completion endpoint. Should the UI also show who else attempted to
   complete the task, or is the first completion sufficient?

7. **Webhook completion.** The `webhook` notification channel sends a JSON
   payload with the task details. Should the webhook payload include a callback
   URL and a signed token that allows the external system to call the
   completion API without separate authentication? This enables "complete via
   webhook callback" for systems that can process and respond programmatically
   (e.g., a ServiceNow integration that auto-completes the ctrlplane job when
   a change request is approved).

8. **Interaction with deployment freeze.** If a deployment freeze (RFC 0008) is
   activated while a manual action job is in `action_required` state, should
   the freeze prevent the job from being completed? The freeze blocks new job
   creation, but an `action_required` job has already been dispatched. The
   safe default is to allow completion (the freeze prevents downstream jobs,
   not in-flight ones), but some organizations may want the freeze to also
   prevent manual action completion.


# RFC 0008: Deployment Freeze / Emergency Lock
Source: https://docs.ctrlplane.dev/rfc/0008-deployment-freeze



| Category   | Status               | Created    | Author        |
| ---------- | -------------------- | ---------- | ------------- |
| Operations | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add a first-class deployment freeze primitive that instantly halts all
deployments within a configurable scope (workspace, system, environment, or
deployment). Freezes are imperative operations — created and lifted via API or
UI — with an audit trail recording who activated the freeze, why, and when it
was lifted. An optional TTL auto-thaws the freeze after a configurable duration
to prevent forgotten freezes from blocking deployments indefinitely.

## Motivation

### No emergency halt exists

Ctrlplane's deployment window evaluator (`policy_rule_deployment_window`)
provides scheduled allow/deny windows using rrule patterns. This covers planned
maintenance windows and business-hours-only deployment policies. But there is no
mechanism for an operator to say "stop everything now" during an incident.

When a production incident occurs, the response today requires one of:

1. **Disabling policies.** Setting `enabled = false` on every relevant policy.
   This stops deployments but also disables approval requirements, version
   selectors, and every other policy rule. Re-enabling them requires remembering
   which policies were active. There is no audit trail of the freeze itself.

2. **Creating deny windows.** Adding a `policy_rule_deployment_window` with
   `allow_window = false` that covers the incident duration. This requires
   knowing the duration in advance, does not surface clearly as an emergency
   action in the UI, and leaves orphan policy rules that must be cleaned up.

3. **Manual intervention.** Telling the team on Slack to stop pushing versions
   and hoping no automation triggers. This provides no system-level enforcement.

None of these are satisfactory for incident response. The operator needs a
single action that:

* Takes effect immediately across the target scope.
* Does not disable other policy rules (approval, verification, etc. remain
  configured for when the freeze lifts).
* Records who activated it, why, and links to an incident.
* Automatically lifts after a TTL if not manually thawed.
* Notifies relevant stakeholders when activated and when lifted.

### Deployment windows are the wrong abstraction

Deployment windows are **scheduled, recurring patterns** — "deploy only on
weekdays 9am–5pm." They are defined ahead of time and repeat on a cadence. An
emergency freeze is an **imperative, one-shot action** — "stop deploying right
now because production is on fire." Overloading the window concept for emergency
freezes creates several problems:

* **Discoverability.** An emergency freeze buried in policy rules is hard to
  find. Operators need a top-level indicator — a banner in the UI, a status
  endpoint — that shows whether a freeze is active.
* **Audit semantics.** A deployment window rule has no concept of "who activated
  this" or "why." It's a configuration, not an action.
* **Scope mismatch.** Deployment windows are scoped to policies, which are
  scoped by selector. An emergency freeze often needs to cover an entire
  workspace or environment regardless of which policies are configured.
* **Lifecycle mismatch.** Windows are permanent configuration. Freezes are
  transient — they are created and destroyed. TTL-based auto-expiry makes no
  sense for a recurring window rule.

### RFC 0003 does not address this

RFC 0003 introduces resource concurrency limits that cap how many resources can
be simultaneously undergoing deployment. This addresses capacity concerns (don't
overwhelm the cluster) but not the "stop everything" incident response case.
Concurrency limits still allow deployments — just fewer at a time. A freeze
allows zero.

## Proposal

### Deployment freeze as a standalone entity

A deployment freeze is not a policy rule. It is a workspace-level entity with
its own lifecycle (create, extend, thaw), its own API surface, and its own
audit trail. The workspace-engine checks for active freezes early in the
evaluator pipeline and denies all matching deployments while a freeze is active.

This separation means:

* Freezes can be managed by anyone with the appropriate permission without
  touching policy configuration.
* The policy configuration remains unchanged during a freeze — approval rules,
  gradual rollout settings, deployment windows, etc. are all preserved.
* When the freeze lifts, deployments resume exactly where they left off in the
  policy pipeline.

### Schema

```sql theme={null}
CREATE TYPE deployment_freeze_scope AS ENUM (
    'workspace',
    'system',
    'environment',
    'deployment'
);

CREATE TABLE deployment_freeze (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,

    -- What scope this freeze covers.
    scope deployment_freeze_scope NOT NULL,

    -- The specific entity ID for non-workspace scopes.
    -- NULL when scope = 'workspace' (the workspace_id is the scope).
    scope_entity_id UUID,

    -- Human-readable reason for the freeze.
    reason TEXT NOT NULL,

    -- Optional link to an incident tracker (PagerDuty, Jira, etc.)
    incident_url TEXT,

    -- Who activated the freeze.
    created_by UUID NOT NULL REFERENCES "user"(id),

    -- When the freeze was activated.
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),

    -- When the freeze should auto-thaw (NULL = no auto-thaw).
    expires_at TIMESTAMPTZ,

    -- When the freeze was manually thawed (NULL = still active or expired).
    thawed_at TIMESTAMPTZ,

    -- Who manually thawed the freeze (NULL if auto-expired or still active).
    thawed_by UUID REFERENCES "user"(id),

    -- Optional note when thawing (e.g., "Incident resolved, RCA pending").
    thaw_reason TEXT,

    -- CEL selector to further narrow the freeze within the scope.
    -- e.g., within an environment scope, only freeze deployments matching
    -- "deployment.metadata['tier'] == 'critical'"
    selector TEXT,

    CONSTRAINT valid_scope_entity CHECK (
        (scope = 'workspace' AND scope_entity_id IS NULL) OR
        (scope != 'workspace' AND scope_entity_id IS NOT NULL)
    )
);

CREATE INDEX idx_deployment_freeze_workspace
    ON deployment_freeze (workspace_id);
CREATE INDEX idx_deployment_freeze_active
    ON deployment_freeze (workspace_id)
    WHERE thawed_at IS NULL;
```

A freeze is **active** when:

* `thawed_at IS NULL` (not manually thawed), AND
* `expires_at IS NULL OR expires_at > now()` (no TTL, or TTL not yet reached).

The `selector` field is optional and allows narrowing within a scope. A
workspace-wide freeze with `selector = "deployment.metadata['tier'] == 'critical'"`
freezes only critical-tier deployments across the workspace, leaving non-critical
deployments unaffected.

### Freeze audit log

Every state change is recorded in a dedicated audit table:

```sql theme={null}
CREATE TYPE deployment_freeze_action AS ENUM (
    'activated',
    'extended',
    'thawed',
    'expired'
);

CREATE TABLE deployment_freeze_event (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    freeze_id UUID NOT NULL
        REFERENCES deployment_freeze(id) ON DELETE CASCADE,
    action deployment_freeze_action NOT NULL,
    actor_id UUID REFERENCES "user"(id),
    note TEXT,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_deployment_freeze_event_freeze
    ON deployment_freeze_event (freeze_id, created_at);
```

The `expired` action is written by the workspace-engine when it detects a freeze
past its `expires_at`. The `actor_id` is NULL for system-initiated actions
(auto-expiry).

### API

#### REST

```
POST   /v1/workspaces/{workspaceId}/freezes          Create a freeze
GET    /v1/workspaces/{workspaceId}/freezes          List freezes (active + recent)
GET    /v1/workspaces/{workspaceId}/freezes/active    List only active freezes
GET    /v1/workspaces/{workspaceId}/freezes/{id}      Get freeze details + events
PATCH  /v1/workspaces/{workspaceId}/freezes/{id}      Extend or modify a freeze
POST   /v1/workspaces/{workspaceId}/freezes/{id}/thaw Thaw a freeze
```

**Create:**

```json theme={null}
POST /v1/workspaces/{workspaceId}/freezes

{
  "scope": "environment",
  "scopeEntityId": "<environment-uuid>",
  "reason": "Production incident INC-4521 — elevated error rates on payment service",
  "incidentUrl": "https://pagerduty.com/incidents/INC-4521",
  "expiresIn": "PT4H",
  "selector": null
}
```

`expiresIn` is an ISO 8601 duration. The server computes `expires_at = now() +
duration`. If omitted, the freeze has no auto-thaw and must be manually lifted.

**Thaw:**

```json theme={null}
POST /v1/workspaces/{workspaceId}/freezes/{id}/thaw

{
  "reason": "Incident resolved. RCA scheduled for Monday."
}
```

**Extend:**

```json theme={null}
PATCH /v1/workspaces/{workspaceId}/freezes/{id}

{
  "expiresIn": "PT8H",
  "reason": "Extending freeze — incident still under investigation"
}
```

Extending resets the TTL from `now()`, not from the original `created_at`.

#### tRPC

```typescript theme={null}
deploymentFreeze.create
deploymentFreeze.list
deploymentFreeze.listActive
deploymentFreeze.get
deploymentFreeze.extend
deploymentFreeze.thaw
```

### Evaluator integration

A `DeploymentFreezeEvaluator` is added to the evaluator pipeline with
`Complexity() = 0` (cheapest possible) so it runs before all other evaluators.
If a matching freeze is active, it short-circuits with a `Denied` result — the
remaining evaluators are never called.

```go theme={null}
type DeploymentFreezeEvaluator struct {
    getters Getters
    ruleId  string
}

func (e *DeploymentFreezeEvaluator) ScopeFields() evaluator.ScopeFields {
    return evaluator.ScopeReleaseTarget
}

func (e *DeploymentFreezeEvaluator) RuleType() string {
    return evaluator.RuleTypeDeploymentFreeze
}

func (e *DeploymentFreezeEvaluator) Complexity() int {
    return 0
}
```

The `Evaluate` method checks for any active freeze matching the release
target's workspace, system, environment, and deployment:

```go theme={null}
func (e *DeploymentFreezeEvaluator) Evaluate(
    ctx context.Context,
    scope evaluator.EvaluatorScope,
) *oapi.RuleEvaluation {
    freezes := e.getters.GetActiveFreezes(ctx, scope)
    if len(freezes) == 0 {
        return results.NewAllowedResult("No active deployment freeze")
    }

    freeze := freezes[0] // Most specific or most recent
    result := results.NewDeniedResult(
        fmt.Sprintf("Deployment frozen: %s", freeze.Reason),
    ).
        WithDetail("freeze_id", freeze.Id).
        WithDetail("frozen_by", freeze.CreatedBy).
        WithDetail("frozen_at", freeze.CreatedAt.Format(time.RFC3339)).
        WithDetail("scope", string(freeze.Scope)).
        WithDetail("reason", freeze.Reason)

    if freeze.IncidentUrl != "" {
        result = result.WithDetail("incident_url", freeze.IncidentUrl)
    }

    if freeze.ExpiresAt != nil {
        result = result.
            WithDetail("expires_at", freeze.ExpiresAt.Format(time.RFC3339)).
            WithNextEvaluationTime(*freeze.ExpiresAt)
    }

    return result
}
```

The `GetActiveFreezes` query checks all scopes that could apply to the release
target:

```sql theme={null}
SELECT * FROM deployment_freeze
WHERE workspace_id = $1
  AND thawed_at IS NULL
  AND (expires_at IS NULL OR expires_at > now())
  AND (
    scope = 'workspace'
    OR (scope = 'system' AND scope_entity_id = $2)
    OR (scope = 'environment' AND scope_entity_id = $3)
    OR (scope = 'deployment' AND scope_entity_id = $4)
  )
ORDER BY created_at DESC;
```

After fetching candidate freezes, those with a non-null `selector` are
post-filtered by evaluating the CEL expression against the release target
context. This keeps the SQL query simple while supporting fine-grained
filtering.

### Evaluator pipeline placement

The freeze evaluator does not originate from a policy rule — it is injected
unconditionally for every evaluation:

```go theme={null}
func CollectEvaluators(
    ctx context.Context,
    getter Getter,
    workspaceId string,
    policies []*oapi.PolicyWithRules,
) []evaluator.Evaluator {
    evals := []evaluator.Evaluator{
        deploymentfreeze.NewEvaluator(getter, workspaceId),
    }

    for _, policy := range policies {
        for _, rule := range policy.Rules {
            evals = append(evals, ruleEvaluators(ctx, getter, rule)...)
        }
    }

    slices.SortFunc(evals, func(a, b evaluator.Evaluator) int {
        return cmp.Compare(a.Complexity(), b.Complexity())
    })

    return evals
}
```

Because it has `Complexity() = 0` and the evaluators are sorted cheapest-first,
the freeze check always runs first. If a freeze is active, the `Denied` result
prevents job creation without evaluating any policy rules.

### Override mechanism

Some deployments must proceed even during a freeze — a hotfix for the very
incident that caused the freeze, or a rollback to a known-good version. The
freeze supports an explicit override via a `bypass_freeze` field on the version
or a policy skip:

```sql theme={null}
ALTER TABLE deployment_version
    ADD COLUMN bypass_freeze BOOLEAN NOT NULL DEFAULT FALSE;
```

When `bypass_freeze = true`, the freeze evaluator returns `Allowed` with a
detail noting the bypass:

```go theme={null}
if scope.Version != nil && scope.Version.BypassFreeze {
    return results.NewAllowedResult(
        "Deployment freeze bypassed for this version",
    ).
        WithDetail("freeze_id", freeze.Id).
        WithDetail("bypass_reason", "version marked bypass_freeze=true")
}
```

Creating a version with `bypass_freeze = true` requires a specific permission
(`deployment_freeze.bypass`) and is recorded in the freeze event log.

### Auto-thaw

The workspace-engine runs a periodic check (every 60 seconds) for freezes past
their `expires_at`:

```go theme={null}
func (w *Worker) expireFreezes(ctx context.Context) error {
    expired, err := w.store.ExpireActiveFreezes(ctx)
    if err != nil {
        return err
    }

    for _, freeze := range expired {
        w.store.CreateFreezeEvent(ctx, FreezeEvent{
            FreezeId: freeze.Id,
            Action:   "expired",
            Note:     fmt.Sprintf("Auto-expired after TTL (%s)",
                freeze.ExpiresAt.Sub(freeze.CreatedAt).String()),
        })

        w.notifications.Send(ctx, FreezeExpiredNotification{
            Freeze:    freeze,
            Workspace: w.workspace,
        })
    }

    return nil
}
```

The `ExpireActiveFreezes` query atomically sets `thawed_at = now()` on all
freezes where `expires_at <= now() AND thawed_at IS NULL`:

```sql theme={null}
UPDATE deployment_freeze
SET thawed_at = now()
WHERE expires_at IS NOT NULL
  AND expires_at <= now()
  AND thawed_at IS NULL
RETURNING *;
```

After auto-thaw, the workspace-engine triggers a reconciliation cycle for all
release targets in the affected scope so that pending deployments resume
immediately.

### UI

#### Freeze banner

When any freeze is active in the current workspace, a persistent banner appears
at the top of the workspace layout:

```
⚠ Deployment freeze active — "Production incident INC-4521" — Expires in 3h 22m
  [View details]  [Thaw now]
```

The banner is dismissible per-session but reappears on page refresh if the
freeze is still active. Multiple active freezes show as a count with a dropdown.

#### Freeze management page

A new page at `/workspaces/{id}/freezes` shows:

* **Active freezes** — scope, reason, who activated, when, TTL remaining, thaw
  button.
* **Recent freezes** — last 30 days, with full event history (activated,
  extended, thawed/expired).
* **Create freeze** button — opens a form with scope picker, reason, incident
  URL, TTL selector, and optional CEL selector.

#### Environment and deployment views

The environment detail page and deployment detail page show a freeze indicator
when a freeze is active that covers them. Frozen release targets display the
freeze reason in their status column instead of the normal policy evaluation
status.

### Notifications

Freeze lifecycle events trigger notifications through the existing notification
system:

| Event     | Recipients                               | Content                                         |
| --------- | ---------------------------------------- | ----------------------------------------------- |
| Activated | Workspace members with deploy permission | Scope, reason, incident URL, TTL, who activated |
| Extended  | Same as activated                        | New TTL, extension reason                       |
| Thawed    | Same as activated                        | Who thawed, thaw reason, duration               |
| Expired   | Same as activated                        | Original TTL, total duration                    |
| Bypassed  | Workspace admins                         | Which version bypassed, who created the version |

Notifications are sent via configured channels (Slack, webhook, email) based on
workspace notification settings.

## Examples

### Workspace-wide emergency freeze

An SRE detects elevated error rates across all services:

```bash theme={null}
curl -X POST ".../workspaces/{id}/freezes" \
  -d '{
    "scope": "workspace",
    "reason": "Elevated 5xx rates across all services — investigating root cause",
    "incidentUrl": "https://pagerduty.com/incidents/INC-4521",
    "expiresIn": "PT2H"
  }'
```

All deployments in the workspace are frozen. The evaluator denies every release
target evaluation with the freeze reason. After 2 hours, the freeze auto-thaws
and pending deployments resume.

### Environment-scoped freeze during rollback

A bad deployment reaches production. The operator freezes production while
rolling back:

```bash theme={null}
# Freeze production
curl -X POST ".../workspaces/{id}/freezes" \
  -d '{
    "scope": "environment",
    "scopeEntityId": "<prod-env-id>",
    "reason": "Rolling back payment-service v2.3.1 — customer-facing errors",
    "expiresIn": "PT1H"
  }'

# Push rollback version with bypass
curl -X POST ".../deployments/{id}/versions" \
  -d '{
    "tag": "v2.3.0-rollback",
    "status": "ready",
    "bypassFreeze": true
  }'
```

The rollback version proceeds through the policy pipeline normally. All other
deployments to production are blocked until the freeze is lifted.

### Freeze with selector

A database migration is running and only deployments that write to the database
should be frozen:

```bash theme={null}
curl -X POST ".../workspaces/{id}/freezes" \
  -d '{
    "scope": "workspace",
    "reason": "Database migration in progress — freezing DB-dependent services",
    "selector": "deployment.metadata[\"depends_on_db\"] == \"true\"",
    "expiresIn": "PT30M"
  }'
```

Deployments without `depends_on_db: true` in their metadata continue
unaffected.

### Extending a freeze

The incident is taking longer than expected:

```bash theme={null}
curl -X PATCH ".../workspaces/{id}/freezes/{freezeId}" \
  -d '{
    "expiresIn": "PT6H",
    "reason": "Root cause identified but fix requires database migration — extending freeze"
  }'
```

The TTL resets to 6 hours from now. An `extended` event is recorded.

### Manual thaw

The incident is resolved:

```bash theme={null}
curl -X POST ".../workspaces/{id}/freezes/{freezeId}/thaw" \
  -d '{
    "reason": "Incident resolved. Payment service stable at v2.3.2. RCA: https://wiki/RCA-4521"
  }'
```

The freeze is lifted immediately. The workspace-engine triggers reconciliation
and pending deployments resume.

### Multiple overlapping freezes

A workspace freeze is active when an environment-specific freeze is also
created:

```
Active freezes:
  1. Workspace — "Company-wide change freeze for Q1 close" (expires in 47h)
  2. Environment (prod) — "Hotfix in progress" (expires in 1h)

Release target evaluation:
  - Staging deployment → Denied by freeze #1 (workspace scope)
  - Prod deployment → Denied by freeze #1 AND #2
  - Prod deployment with bypass_freeze version → Denied by freeze #1
    (workspace freeze still applies; bypass only exempts the version
     from freeze evaluation, not from workspace-level freezes)
```

The `bypass_freeze` field on a version exempts it from **all** matching
freezes. If the operator intends for the bypass to respect workspace-level
freezes, they should scope the bypass to a specific freeze via metadata
convention (see Open Questions).

## Migration

* The `deployment_freeze` and `deployment_freeze_event` tables are new. No data
  migration required.
* The `bypass_freeze` column on `deployment_version` is additive with a default
  of `false`. Existing versions are unaffected.
* The `deployment_freeze_scope` and `deployment_freeze_action` enum types are
  new.
* The freeze evaluator is injected unconditionally and returns `Allowed` when
  no freezes are active. Existing behavior is preserved.
* No changes to existing policy rules or evaluators.

## Open Questions

1. **Bypass granularity.** The current proposal has a boolean `bypass_freeze` on
   the version that bypasses all matching freezes. Should bypass be scoped to a
   specific freeze ID instead? This would allow a version to bypass an
   environment freeze but still respect a workspace freeze. The trade-off is
   complexity — the deployer must know the freeze ID at version creation time.

2. **Freeze inheritance.** If a workspace freeze is active, should an
   environment-level thaw override it for that environment? The current proposal
   says no — a workspace freeze blocks everything regardless of
   environment-level freeze state. This is the safe default but may be too
   rigid for organizations that want hierarchical freeze management.

3. **In-flight jobs.** A freeze prevents new jobs from being created. Should it
   also cancel or pause jobs that are already running? Cancellation is
   destructive and may leave resources in an inconsistent state. The safe
   default is to let in-flight jobs complete but prevent new ones. However, for
   severe incidents, the operator may want to stop everything including
   in-flight work.

4. **Freeze permissions.** Who can create and thaw freezes? The proposal assumes
   a `deployment_freeze.create` and `deployment_freeze.thaw` permission. Should
   thawing require higher privileges than freezing (to prevent accidental
   thaws)? Should there be a "break glass" thaw that requires admin approval?

5. **Notification timing.** Should the system send warning notifications before
   a freeze auto-expires (e.g., "Freeze expires in 30 minutes — extend or thaw
   manually")? This prevents surprise resumption of deployments if the operator
   intended to extend.

6. **Interaction with deployment windows.** If a freeze is active during a
   deployment window's allow period, the freeze takes precedence (denied). When
   the freeze lifts, should the system check if the deployment window is still
   open? The evaluator pipeline handles this naturally — after the freeze
   evaluator allows, the deployment window evaluator runs next and checks the
   current time. But this means a freeze that lifts 5 minutes before a window
   closes gives only 5 minutes of deployment time. Should the window be
   extended to compensate?

7. **Terraform / IaC representation.** Should freezes be expressible as
   Terraform resources? Freezes are inherently imperative and transient, which
   maps poorly to Terraform's declarative model. A `ctrlplane_deployment_freeze`
   resource would be created on `apply` and destroyed on `destroy`, which
   technically works but feels semantically odd for an incident response action.

8. **Cascading thaw.** When a workspace freeze is thawed, should all
   narrower-scope freezes within that workspace also be thawed? Or should they
   remain active independently? The current proposal treats each freeze as
   independent — thawing the workspace freeze does not affect the environment
   freeze.

## Future Considerations

### PagerDuty integration

The most natural extension of deployment freezes is automatic activation from an
incident management system. PagerDuty is the primary target, with the pattern
generalizing to Opsgenie, Grafana OnCall, and similar tools.

**Auto-freeze on incident creation.** A PagerDuty webhook listener receives
incident events and creates a deployment freeze when an incident is triggered.
The mapping from incident to freeze scope could be configured per-service:

```json theme={null}
{
  "pagerduty": {
    "serviceMapping": [
      {
        "pdServiceId": "PABC123",
        "freezeScope": "environment",
        "scopeEntityId": "<prod-env-id>",
        "minSeverity": "P1"
      },
      {
        "pdServiceId": "PXYZ789",
        "freezeScope": "deployment",
        "scopeEntityId": "<payment-deployment-id>",
        "minSeverity": "P2"
      }
    ],
    "defaultTtl": "PT4H",
    "defaultScope": "workspace",
    "minSeverity": "P1"
  }
}
```

A `minSeverity` threshold prevents low-priority alerts from triggering freezes.
P1 incidents could freeze the workspace, P2 freeze the affected environment,
P3/P4 are informational only.

The freeze's `reason` and `incident_url` are populated automatically from the
PagerDuty incident title and URL. The `created_by` is set to a service account
representing the PagerDuty integration, with the PagerDuty incident responder
recorded in event metadata.

**Auto-thaw on incident resolution.** When PagerDuty sends a `resolved` webhook,
the integration finds freezes linked to that incident (via `incident_url` or a
`pagerduty_incident_id` metadata field) and thaws them. The thaw reason is
populated from the PagerDuty resolution note.

A configurable `thaw_delay` (e.g., 15 minutes after resolution) provides a
buffer — the incident may be resolved in PagerDuty before the system is fully
stable. During the delay, the freeze remains active but a notification warns
that auto-thaw is imminent.

**Bidirectional timeline.** The PagerDuty integration posts timeline entries on
the incident when freezes are activated, extended, or thawed. This gives
incident responders visibility into deployment state directly from their
incident management tool:

```
10:03 AM — Deployment freeze activated (scope: production)
10:45 AM — Freeze extended to 6 hours
11:30 AM — Hotfix v2.3.2 bypassed freeze
 1:15 PM — Deployment freeze thawed (incident resolved)
```

### Slack integration

Beyond notifications (covered in the main proposal), Slack could provide
interactive freeze management:

* **Slash commands.** `/ctrlplane freeze production "Payment service incident"`
  creates a freeze. `/ctrlplane thaw <freeze-id>` lifts one. Useful during
  incident response when switching to the ctrlplane UI is a context switch.
* **Interactive messages.** Freeze notifications include "Extend" and "Thaw"
  buttons that trigger API calls directly from Slack.
* **Incident channel binding.** When a freeze is created with an incident URL
  that maps to a Slack channel (e.g., `#inc-4521`), freeze lifecycle
  notifications are posted to that channel specifically, not just the default
  notification channel.

### Statuspage integration

Active workspace-wide or environment-wide freezes could automatically update an
external status page (Atlassian Statuspage, Instatus, etc.) to reflect that
deployments are paused. This is relevant for platform teams that publish
deployment status to internal consumers:

* Freeze activated → status component set to "Degraded Performance" or
  "Maintenance" with the freeze reason.
* Freeze thawed → status component restored to "Operational."

### CI/CD pipeline gating

Freezes could be exposed as a check endpoint that CI/CD systems query before
proceeding with deployment steps:

```
GET /v1/workspaces/{id}/freezes/check?scope=environment&entityId={envId}

200 OK: { "frozen": false }
200 OK: { "frozen": true, "freezeId": "...", "reason": "..." }
```

GitHub Actions, GitLab CI, and Jenkins pipelines could poll this endpoint as a
gate step, failing the pipeline early rather than pushing a version that the
evaluator will deny. This provides faster feedback to developers.

### Calendar-based planned freezes

While the current proposal focuses on emergency freezes, the same primitive
could support planned change freezes (e.g., end-of-quarter code freezes, holiday
freezes). These would be created ahead of time with a future `created_at`
(or a separate `effective_at` field) and a known `expires_at`. The integration
with Google Calendar or Outlook could auto-create freezes from calendar events
tagged with a specific label.

### Incident management post-mortem

On freeze thaw, the system could auto-create a post-mortem template in the
configured project management tool (Jira, Linear, etc.) pre-populated with:

* Freeze duration and scope.
* Which deployments were blocked and for how long.
* Which versions bypassed the freeze.
* Timeline of freeze events.

This reduces the manual effort of gathering deployment context for incident
retrospectives.


# RFC 0009: Global Variable Sets
Source: https://docs.ctrlplane.dev/rfc/0009-global-variable-sets



| Category  | Status               | Created    | Author        |
| --------- | -------------------- | ---------- | ------------- |
| Variables | <Badge>Draft</Badge> | 2026-03-13 | Justin Brooks |

## Summary

Add global variable sets — named, reusable collections of key-value pairs that
can be scoped to a workspace, system, or environment and are automatically
injected into deployment variable resolution. Variable sets eliminate the need
to duplicate shared configuration across deployments and provide a single place
to manage cross-cutting variables like database endpoints, feature flags, region
metadata, and shared credentials references.

## Motivation

### Shared configuration is duplicated across deployments

A typical workspace has configuration that spans many deployments: database
connection strings, message queue endpoints, feature flags, regional settings,
cloud account IDs, and API keys. Today, each deployment must define its own
deployment variable for each of these values.

Consider a workspace with 15 deployments that all need `DATABASE_URL`,
`REDIS_URL`, `LOG_LEVEL`, and `REGION`. The operator must:

1. Create 4 deployment variables on each of the 15 deployments (60 variables).
2. For each variable, create deployment variable values with the correct
   resource selectors to differentiate production from staging.
3. When the staging database endpoint rotates, update the value across all 15
   deployments individually.

This is tedious, error-prone, and scales poorly. A forgotten update leaves one
deployment pointing at a stale endpoint. There is no mechanism to express "these
variables are the same across all deployments in this system."

### No hierarchy for variable inheritance

The current variable model is flat. Deployment variables live on deployments.
Resource variables live on resources. There is no intermediate layer where an
operator can say "all deployments in this system inherit these variables" or
"all deployments in this workspace get these defaults unless overridden."

Other deployment platforms solve this with variable groups (Azure DevOps),
variable sets (Terraform Cloud), environment variables (Vercel), or config maps
(Kubernetes). Ctrlplane's closest analog is resource variables, but those are
scoped to the resource — they express "this resource has this property," not
"this configuration should flow to all deployments targeting this resource."

### Variable changes need consistent rollout

When a shared value changes, the operator wants all affected deployments to
pick up the change atomically. Today, updating 15 deployment variables is 15
separate mutations, each triggering its own release reconciliation. There is no
way to batch the update and have all affected release targets reconcile with the
new values simultaneously.

### The deployment variable model is the wrong abstraction for shared config

Deployment variables answer the question "what configuration does this
deployment need?" They are owned by the deployment and vary per deployment.
Shared configuration answers a different question: "what configuration exists
in this environment/system/workspace that multiple deployments consume?"

Overloading deployment variables for shared config creates problems:

* **Ownership ambiguity.** Who owns the `DATABASE_URL` deployment variable — the
  deployment owner or the platform team that manages the database? When it lives
  on every deployment, there is no single owner.
* **Drift.** Without a single source of truth, values drift between deployments.
  Deployment A gets updated, deployment B does not.
* **Onboarding cost.** Adding a new deployment requires copying all shared
  variables from an existing deployment. There is no template or inheritance.
* **Audit difficulty.** Answering "which deployments use this database endpoint?"
  requires scanning every deployment's variables.

## Proposal

### Variable sets as a first-class entity

A **variable set** is a named collection of key-value pairs with a scope
(workspace, system, or environment) and an optional selector that further
narrows which release targets receive the variables. Variable sets are resolved
during the variable evaluation phase alongside deployment variables and resource
variables.

Variable sets are **not** deployment variables. They are a separate entity with
their own lifecycle, ownership, and API surface. They are injected into the
variable resolution pipeline as an additional source of values, sitting between
deployment variable defaults and deployment variable values in the resolution
priority chain.

### Schema

```sql theme={null}
CREATE TABLE variable_set (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,

    name TEXT NOT NULL,
    description TEXT,

    -- Scope determines at which level this set applies.
    -- 'workspace' = all deployments in the workspace.
    -- 'system' = all deployments in a specific system.
    -- 'environment' = all deployments targeting a specific environment.
    scope TEXT NOT NULL CHECK (scope IN ('workspace', 'system', 'environment')),

    -- The entity ID for system/environment scopes. NULL for workspace scope.
    scope_entity_id UUID,

    -- Optional CEL selector for fine-grained filtering within the scope.
    -- Evaluated against the release target context (resource, deployment,
    -- environment metadata).
    selector TEXT,

    -- Priority for ordering when multiple sets define the same key.
    -- Higher priority wins. Default 0.
    priority INTEGER NOT NULL DEFAULT 0,

    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),

    CONSTRAINT valid_scope_entity CHECK (
        (scope = 'workspace' AND scope_entity_id IS NULL) OR
        (scope != 'workspace' AND scope_entity_id IS NOT NULL)
    ),
    CONSTRAINT unique_name_per_workspace UNIQUE (workspace_id, name)
);

CREATE INDEX idx_variable_set_workspace ON variable_set (workspace_id);
CREATE INDEX idx_variable_set_scope ON variable_set (workspace_id, scope);
```

Each variable set contains variables:

```sql theme={null}
CREATE TABLE variable_set_variable (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    variable_set_id UUID NOT NULL
        REFERENCES variable_set(id) ON DELETE CASCADE,

    key TEXT NOT NULL,
    value JSONB NOT NULL,

    -- Whether this variable is sensitive (masked in UI and API responses).
    sensitive BOOLEAN NOT NULL DEFAULT FALSE,

    CONSTRAINT unique_key_per_set UNIQUE (variable_set_id, key)
);

CREATE INDEX idx_variable_set_variable_set
    ON variable_set_variable (variable_set_id);
```

The `value` column uses the same JSONB format as existing deployment variable
values, supporting both literal values and reference values:

```json theme={null}
// Literal string
"us-east-1"

// Literal number
42

// Literal boolean
true

// Reference value
{
    "reference": "workspace",
    "path": ["metadata", "database_url"]
}
```

### Resolution priority

The variable resolution priority is extended to include variable sets. The
full priority chain, from highest to lowest:

1. **Resource variable** — a variable defined directly on the resource with a
   matching key. This is the most specific override and always wins.
2. **Deployment variable value** — a deployment variable value whose resource
   selector matches the target resource, sorted by priority (highest first).
3. **Variable set (environment scope)** — a variable set scoped to the
   specific environment in the release target, sorted by set priority.
4. **Variable set (system scope)** — a variable set scoped to the system
   containing the deployment, sorted by set priority.
5. **Variable set (workspace scope)** — a variable set scoped to the
   workspace, sorted by set priority.
6. **Deployment variable default** — the default value defined on the
   deployment variable.

This ordering follows the principle of specificity: the most specific source
wins. Resource variables are the most specific (set on the exact resource),
variable sets widen from environment to system to workspace, and deployment
variable defaults are the fallback.

Within the same scope level, multiple variable sets may define the same key.
The set with the highest `priority` value wins. If two sets at the same scope
have the same priority, the one created most recently wins (deterministic
tiebreaker).

**Only keys declared as deployment variables are resolved.** Variable sets do
not introduce new keys into the release — they provide values for keys that
the deployment has declared via deployment variables. A variable set with
`DATABASE_URL = "postgres://..."` has no effect on a deployment that does not
declare a `DATABASE_URL` deployment variable. This preserves the deployment's
contract: the deployment declares what variables it needs, and variable sets
(along with other sources) provide the values.

### Variable manager changes

The `Manager.Evaluate` method in the workspace-engine is extended to query
variable sets after deployment variable values and before deployment variable
defaults:

```go theme={null}
func (m *Manager) Evaluate(
    ctx context.Context,
    releaseTarget *oapi.ReleaseTarget,
    relatedEntities map[string][]*oapi.EntityRelation,
) (map[string]*oapi.LiteralValue, error) {
    ctx, span := tracer.Start(ctx, "VariableManager.Evaluate")
    defer span.End()

    resolvedVariables := make(map[string]*oapi.LiteralValue)

    resource, exists := m.store.Resources.Get(releaseTarget.ResourceId)
    if !exists {
        return nil, fmt.Errorf("resource %q not found", releaseTarget.ResourceId)
    }
    entity := relationships.NewResourceEntity(resource)

    resourceVariables := m.store.Resources.Variables(releaseTarget.ResourceId)
    deploymentVariables := m.store.Deployments.Variables(releaseTarget.DeploymentId)

    // Load variable sets in scope order (environment > system > workspace).
    variableSets := m.store.VariableSets.ForReleaseTarget(ctx, releaseTarget)

    for key, deploymentVar := range deploymentVariables {
        // 1. Resource variable
        resolved := m.tryResolveResourceVariable(
            ctx, key, resourceVariables, entity, relatedEntities,
        )
        if resolved != nil {
            resolvedVariables[key] = resolved
            continue
        }

        // 2. Deployment variable value (selector + priority)
        resolved = m.tryResolveDeploymentVariableValue(
            ctx, deploymentVar, resource, entity, relatedEntities,
        )
        if resolved != nil {
            resolvedVariables[key] = resolved
            continue
        }

        // 3. Variable sets (environment > system > workspace)
        resolved = m.tryResolveFromVariableSets(
            ctx, key, variableSets, entity, relatedEntities,
        )
        if resolved != nil {
            resolvedVariables[key] = resolved
            continue
        }

        // 4. Deployment variable default
        if deploymentVar.DefaultValue != nil {
            resolvedVariables[key] = deploymentVar.DefaultValue
        }
    }

    return resolvedVariables, nil
}
```

The `tryResolveFromVariableSets` method iterates through variable sets in
scope order. Sets are pre-sorted: environment-scoped first, then system-scoped,
then workspace-scoped. Within each scope level, sets are sorted by priority
(descending), then by `created_at` (descending):

```go theme={null}
func (m *Manager) tryResolveFromVariableSets(
    ctx context.Context,
    key string,
    variableSets []*VariableSetWithVariables,
    entity *oapi.RelatableEntity,
    relatedEntities map[string][]*oapi.EntityRelation,
) *oapi.LiteralValue {
    for _, vs := range variableSets {
        variable, exists := vs.Variables[key]
        if !exists {
            continue
        }

        result, err := m.store.Variables.ResolveValue(
            ctx, entity, &variable.Value, relatedEntities,
        )
        if err != nil {
            continue
        }

        return result
    }

    return nil
}
```

### Reconciliation on variable set changes

When a variable set is created, updated, or deleted, the system must
re-evaluate all release targets that could be affected. The scope determines
the blast radius:

* **Workspace scope**: all release targets in the workspace.
* **System scope**: all release targets for deployments in the system.
* **Environment scope**: all release targets in the environment.

If the variable set has a `selector`, only release targets matching the
selector are re-evaluated. This is the same reconciliation pattern used when
deployment variables change — the workspace-engine detects that variable
inputs have changed, computes the new resolved variables, and creates a new
release if the resolved values differ from the current release.

```go theme={null}
func (w *Worker) handleVariableSetChange(
    ctx context.Context,
    event VariableSetChangeEvent,
) error {
    affectedTargets := w.store.ReleaseTargets.InScope(
        ctx,
        event.VariableSet.WorkspaceId,
        event.VariableSet.Scope,
        event.VariableSet.ScopeEntityId,
        event.VariableSet.Selector,
    )

    for _, target := range affectedTargets {
        w.enqueueReleaseTargetEvaluation(ctx, target)
    }

    return nil
}
```

### API

#### REST

```
POST   /v1/workspaces/{workspaceId}/variable-sets                Create a variable set
GET    /v1/workspaces/{workspaceId}/variable-sets                List variable sets
GET    /v1/workspaces/{workspaceId}/variable-sets/{id}           Get variable set with variables
PATCH  /v1/workspaces/{workspaceId}/variable-sets/{id}           Update variable set metadata
DELETE /v1/workspaces/{workspaceId}/variable-sets/{id}           Delete a variable set

PUT    /v1/workspaces/{workspaceId}/variable-sets/{id}/variables Upsert variables (bulk)
DELETE /v1/workspaces/{workspaceId}/variable-sets/{id}/variables/{key} Delete a variable
```

**Create:**

```json theme={null}
POST /v1/workspaces/{workspaceId}/variable-sets

{
  "name": "production-database",
  "description": "Database connection details for production",
  "scope": "environment",
  "scopeEntityId": "<prod-env-id>",
  "priority": 10,
  "variables": [
    {
      "key": "DATABASE_URL",
      "value": "postgres://prod-db.internal:5432/app",
      "sensitive": true
    },
    {
      "key": "DATABASE_POOL_SIZE",
      "value": 20
    },
    {
      "key": "DATABASE_SSL_MODE",
      "value": "verify-full"
    }
  ]
}
```

**Upsert variables:**

```json theme={null}
PUT /v1/workspaces/{workspaceId}/variable-sets/{id}/variables

{
  "variables": [
    { "key": "DATABASE_URL", "value": "postgres://new-db.internal:5432/app", "sensitive": true },
    { "key": "DATABASE_POOL_SIZE", "value": 25 }
  ]
}
```

This is a partial upsert — keys included in the request are created or updated,
keys not included are left unchanged. To remove a key, use the DELETE endpoint.

**List with scope filtering:**

```
GET /v1/workspaces/{workspaceId}/variable-sets?scope=environment&scopeEntityId=<env-id>
```

Returns all variable sets that apply to the given scope, including
workspace-scoped sets that apply everywhere.

#### tRPC

```typescript theme={null}
variableSet.create
variableSet.list
variableSet.get
variableSet.update
variableSet.delete
variableSet.upsertVariables
variableSet.deleteVariable
```

### Terraform provider

```hcl theme={null}
resource "ctrlplane_variable_set" "production_db" {
  workspace_id = ctrlplane_workspace.main.id
  name         = "production-database"
  description  = "Database connection details for production"
  scope        = "environment"
  scope_entity_id = ctrlplane_environment.production.id
  priority     = 10
}

resource "ctrlplane_variable_set_variable" "db_url" {
  variable_set_id = ctrlplane_variable_set.production_db.id
  key             = "DATABASE_URL"
  literal_value   = "postgres://prod-db.internal:5432/app"
  sensitive       = true
}

resource "ctrlplane_variable_set_variable" "db_pool" {
  variable_set_id = ctrlplane_variable_set.production_db.id
  key             = "DATABASE_POOL_SIZE"
  literal_value   = "20"
}
```

### UI

#### Variable sets management page

A new page at `/workspaces/{id}/settings/variable-sets` lists all variable sets
in the workspace:

| Name                  | Scope       | Target          | Variables | Priority |
| --------------------- | ----------- | --------------- | --------- | -------- |
| production-database   | Environment | production      | 3         | 10       |
| staging-database      | Environment | staging         | 3         | 10       |
| shared-feature-flags  | Workspace   | All deployments | 8         | 0        |
| payment-system-config | System      | payment         | 5         | 5        |

Clicking a variable set opens an editor showing:

* Name, description, scope, priority.
* A table of variables with key, value (masked if sensitive), and actions.
* An "Add variable" form.
* A "Used by" section showing which deployments declare variables with matching
  keys and would receive values from this set.

#### Variable resolution preview

The deployment detail page gains a "Variable Resolution" panel that shows, for
each deployment variable, where its value comes from for a selected resource:

| Variable         | Value                                | Source                      |
| ---------------- | ------------------------------------ | --------------------------- |
| DATABASE\_URL    | `postgres://prod-db.internal:5432/…` | Variable Set: production-db |
| REPLICA\_COUNT   | `5`                                  | Deployment Variable Value   |
| LOG\_LEVEL       | `info`                               | Variable Set: defaults      |
| FEATURE\_NEW\_UI | `true`                               | Resource Variable           |
| CACHE\_TTL       | `300`                                | Deployment Variable Default |

This makes the resolution chain visible and debuggable. Each source is a link
to the entity that provided the value.

#### System and environment detail pages

The system detail page and environment detail page show variable sets scoped
to them, with a quick-add button to create a new set at that scope.

## Examples

### Shared database configuration

A platform team manages database endpoints. They create variable sets per
environment:

```bash theme={null}
# Production database config
curl -X POST ".../workspaces/{id}/variable-sets" \
  -d '{
    "name": "production-database",
    "scope": "environment",
    "scopeEntityId": "<prod-env-id>",
    "priority": 10,
    "variables": [
      { "key": "DATABASE_URL", "value": "postgres://prod-db:5432/app", "sensitive": true },
      { "key": "DATABASE_POOL_SIZE", "value": 20 },
      { "key": "DATABASE_SSL_MODE", "value": "verify-full" }
    ]
  }'

# Staging database config
curl -X POST ".../workspaces/{id}/variable-sets" \
  -d '{
    "name": "staging-database",
    "scope": "environment",
    "scopeEntityId": "<staging-env-id>",
    "priority": 10,
    "variables": [
      { "key": "DATABASE_URL", "value": "postgres://staging-db:5432/app", "sensitive": true },
      { "key": "DATABASE_POOL_SIZE", "value": 5 },
      { "key": "DATABASE_SSL_MODE", "value": "prefer" }
    ]
  }'
```

Every deployment that declares a `DATABASE_URL` deployment variable
automatically receives the correct value based on which environment the
release target is in. When the production database endpoint changes, the
operator updates one variable set and all deployments pick up the change.

### Workspace-wide defaults

A workspace admin sets sensible defaults that apply everywhere:

```bash theme={null}
curl -X POST ".../workspaces/{id}/variable-sets" \
  -d '{
    "name": "workspace-defaults",
    "scope": "workspace",
    "priority": 0,
    "variables": [
      { "key": "LOG_LEVEL", "value": "info" },
      { "key": "METRICS_ENABLED", "value": true },
      { "key": "OTEL_EXPORTER_ENDPOINT", "value": "https://otel.internal:4317" }
    ]
  }'
```

Individual deployments can override these by setting deployment variable values
or deployment variable defaults, which have higher priority. A deployment that
needs `LOG_LEVEL=debug` sets its own deployment variable value — the
workspace-wide default is ignored for that deployment.

### System-specific configuration

A payment system has configuration shared across its deployments (payment-api,
payment-worker, payment-webhook):

```bash theme={null}
curl -X POST ".../workspaces/{id}/variable-sets" \
  -d '{
    "name": "payment-system-config",
    "scope": "system",
    "scopeEntityId": "<payment-system-id>",
    "priority": 5,
    "variables": [
      { "key": "STRIPE_API_VERSION", "value": "2025-12-01" },
      { "key": "PAYMENT_TIMEOUT_MS", "value": 30000 },
      { "key": "IDEMPOTENCY_KEY_TTL", "value": 86400 }
    ]
  }'
```

All three deployments in the payment system receive these variables without
any per-deployment configuration.

### Layered overrides

Multiple variable sets at different scopes can coexist. The resolution
chain handles precedence naturally:

```
Workspace set (priority 0):   LOG_LEVEL = "warn"
System set (priority 5):      LOG_LEVEL = "info"
Environment set (priority 10): LOG_LEVEL = "debug"
```

For a release target in the matching environment, the environment-scoped set
wins (it is more specific). For a release target in a different environment
within the same system, the system-scoped set provides `LOG_LEVEL = "info"`.
For a release target in a different system entirely, the workspace-scoped set
provides `LOG_LEVEL = "warn"`.

If a resource has a resource variable `LOG_LEVEL = "trace"`, that overrides
everything — resource variables are always the highest priority.

### Variable set with selector

A variable set can be further narrowed with a CEL selector:

```bash theme={null}
curl -X POST ".../workspaces/{id}/variable-sets" \
  -d '{
    "name": "gpu-cluster-config",
    "scope": "workspace",
    "selector": "resource.metadata[\"gpu_enabled\"] == \"true\"",
    "priority": 10,
    "variables": [
      { "key": "GPU_MEMORY_LIMIT", "value": "16Gi" },
      { "key": "CUDA_VERSION", "value": "12.4" }
    ]
  }'
```

Only release targets whose resource has `gpu_enabled: true` in metadata
receive these variables. Other release targets are unaffected.

## Migration

* The `variable_set` and `variable_set_variable` tables are new. No data
  migration required.
* Existing deployment variables, resource variables, and their resolution logic
  are unchanged. The variable set layer is additive.
* The workspace-engine's variable manager gains a new resolution step between
  deployment variable values and deployment variable defaults. Existing
  resolution behavior is preserved — variable sets only provide values when
  higher-priority sources (resource variables, deployment variable values) do
  not.
* No changes to existing API endpoints. New endpoints are additive.

## Open Questions

1. **Variable set assignment vs. scope-based matching.** The current proposal
   uses scope-based matching: a variable set with `scope = environment` and
   `scope_entity_id = <prod>` automatically applies to all release targets in
   production. An alternative is explicit assignment: the operator attaches
   variable sets to deployments, systems, or environments manually. Assignment
   gives more control but requires more configuration. Scope-based matching is
   simpler but less flexible. Should we support both?

2. **Key collision across sets.** When two variable sets at the same scope level
   define the same key, priority determines the winner. Should collisions be
   surfaced as warnings in the UI? Should there be a strict mode that rejects
   ambiguous resolutions?

3. **Sensitive variable handling.** The `sensitive` flag masks values in the UI
   and API list responses. Should sensitive variables also be excluded from
   audit logs? Should they require a separate permission to read the plaintext
   value? How does this interact with the secret provider integration from
   RFC 0006?

4. **Variable set versioning.** Should variable sets support versioning or
   change history? When a variable is updated, should the system record the
   previous value for audit purposes? This adds complexity but is valuable
   for debugging "what changed."

5. **Bulk update atomicity.** When updating multiple variables in a set, should
   the operation be atomic (all-or-nothing)? The current proposal uses upsert
   semantics where each key is updated independently. An atomic bulk update
   would prevent partial updates but requires transactional semantics.

6. **Cross-workspace variable sets.** Should variable sets be shareable across
   workspaces? This is relevant for organizations with multiple workspaces that
   share infrastructure. The current proposal scopes sets to a single workspace.

7. **Interaction with variable set selectors and deployment variable
   selectors.** A variable set with a selector and a deployment variable value
   with a resource selector are conceptually similar. Should we unify the
   filtering mechanism or keep them separate?

## Future Considerations

### Variable set templates

Pre-built variable set templates for common patterns:

* **Cloud provider defaults**: AWS region, account ID, VPC settings.
* **Observability stack**: OTEL endpoints, log levels, metrics configuration.
* **Database connection**: URL, pool size, SSL mode, timeout.

Templates could be shared across workspaces or published as community
templates.

### Environment cloning

When creating a new environment (e.g., a new staging environment for a feature
branch), variable sets scoped to a reference environment could be cloned
automatically with modified values. This supports dynamic environment workflows
where environments are created and destroyed frequently.

### Secret provider integration

RFC 0006 proposes secret provider integration for resolving secrets from
external stores (Vault, AWS Secrets Manager, etc.). Variable sets are a natural
place to reference external secrets:

```json theme={null}
{
  "key": "DATABASE_PASSWORD",
  "value": {
    "secretRef": {
      "provider": "vault",
      "path": "secret/data/production/database",
      "key": "password"
    }
  },
  "sensitive": true
}
```

The variable resolution pipeline would resolve the secret reference at
evaluation time, fetching the current value from the secret provider. This
keeps secrets out of the database while allowing variable sets to manage
which secrets each deployment receives.

### Variable set policies

Policies could enforce rules on variable sets:

* **Required variables**: a policy that denies deployment if certain keys are
  not provided by any variable set (e.g., "all deployments must have
  `LOG_LEVEL` defined").
* **Value constraints**: a policy that validates variable values against a
  schema (e.g., "`DATABASE_POOL_SIZE` must be between 1 and 100").
* **Sensitive enforcement**: a policy that requires certain keys to be marked
  sensitive (e.g., any key containing `PASSWORD`, `SECRET`, or `TOKEN`).

### Variable set drift detection

Detect and alert when variable sets that should be consistent across
environments have diverged. For example, if the staging and production database
variable sets should have the same keys (but different values), drift detection
would flag when production has a key that staging does not. This prevents
configuration gaps from reaching production.


# RFC 0010: Unified Variable & Secret Resolution System
Source: https://docs.ctrlplane.dev/rfc/0010-variable-storage



| Category       | Status               | Created    | Author     |
| -------------- | -------------------- | ---------- | ---------- |
| Infrastructure | <Badge>Draft</Badge> | 2026-04-01 | Mike Leone |

## **Summary**

This RFC proposes a unified data model for managing variables across resources, deployments, and deployment job agents. It consolidates the current fragmented schema into a single, extensible system that supports:

* Multiple scopes (resource, deployment, deployment job agent)
* Multiple value types (literal, reference, secret reference)
* Override semantics via selectors and priority
* First-class support for secret providers without duplicating schema

The proposal replaces multiple duplicated tables with two core tables: variable and variable\_value.

***

## **Motivation**

The current schema exhibits significant duplication across two dimensions:

1. **Scope duplication**
   * Separate handling for resource, deployment, and job-agent variables
2. **Value-type duplication**
   * Separate tables for literal values and reference values

This results in:

* Schema explosion and maintenance overhead
* Repeated logic in queries and resolution code
* Increased risk of inconsistency and bugs
* Difficulty extending the system (e.g., adding secrets)

Additionally, introducing secrets under the current model would require duplicating the entire table structure again, further compounding complexity.

We need a model that:

* Treats scope and value type as data, not schema
* Supports extensibility without table proliferation
* Centralizes resolution logic

***

## **Goals**

* Eliminate duplicated tables across scopes and value types
* Provide a single resolution model for all variable types
* Support secret references without storing raw secrets
* Maintain strong data integrity constraints
* Enable future extensibility (new value types, new scopes)

***

## **Non-Goals**

* Implementing secret storage (this system references external providers)
* Defining a full selector language
* Enforcing cross-variable resolution correctness at the database level

***

## **Proposal**

```jsx theme={null}
CREATE TYPE variable_scope as ENUM (
            'resource',
            'deployment',
            'deployment_job_agent'
);

CREATE TYPE variable_value_kind as ENUM (
            'literal',
            'ref',
            'secret_ref'
);

CREATE TABLE IF NOT EXISTS variable (
    id uuid PRIMARY KEY DEFAULT uuid_generate_v4(),

    scope variable_scope not null,

    resource_id uuid references resource(id) on delete cascade,
    deployment_id uuid references deployment(id) on delete cascade,
    deployment_job_agent_id uuid references deployment_job_agent(id) on delete cascade,

    key text not null,

    -- metadata
    is_sensitive boolean not null default false,
    description text,

    created_at timestamptz not null default now(),
    updated_at timestamptz not null default now(),

    -- exactly one owner target must be set, and it must match scope
    CONSTRAINT variable_scope_target_check check (
        (
            scope = 'resource'
            and resource_id is not null
            and deployment_id is null
            and deployment_job_agent_id is null
        )
        or
        (
            scope = 'deployment'
            and deployment_id is not null
            and resource_id is null
            and deployment_job_agent_id is null
        )
        or
        (
            scope = 'deployment_job_agent'
            and deployment_job_agent_id is not null
            and resource_id is null
            and deployment_id is null
        )
    )
);

create unique index if not exists variable_resource_key_uniq
    on variable(resource_id, key)
    where resource_id is not null;

create unique index if not exists variable_deployment_key_uniq
    on variable(deployment_id, key)
    where deployment_id is not null;

create unique index if not exists variable_dja_key_uniq
    on variable(deployment_job_agent_id, key)
    where deployment_job_agent_id is not null;

create index if not exists variable_scope_idx
    on variable(scope);

create index if not exists variable_resource_lookup_idx
    on variable(resource_id, key)
    where resource_id is not null;

create index if not exists variable_deployment_lookup_idx
    on variable(deployment_id, key)
    where deployment_id is not null;

create index if not exists variable_dja_lookup_idx
    on variable(deployment_job_agent_id, key)
    where deployment_job_agent_id is not null;


CREATE TABLE IF NOT EXISTS variable_value (
    id uuid primary key default uuid_generate_v4(),

    variable_id uuid not null references variable(id) on delete cascade,

    resource_selector text,

    priority bigint not null default 0,

    kind variable_value_kind not null,

    literal_value jsonb,

    ref_key text,
    ref_path text[],

    secret_provider text,
    secret_key text,
    secret_path text[],

    created_at timestamptz not null default now(),
    updated_at timestamptz not null default now(),

    CONSTRAINT variable_value_kind_shape_check check (
        (
            kind = 'literal'
            and literal_value is not null
            and ref_key is null
            and ref_path is null
            and secret_provider is null
            and secret_key is null
            and secret_path is null
        )
        or
        (
            kind = 'ref'
            and literal_value is null
            and ref_key is not null
            and secret_provider is null
            and secret_key is null
            and secret_path is null
        )
        or
        (
            kind = 'secret_ref'
            and literal_value is null
            and ref_key is null
            and ref_path is null
            and secret_provider is not null
            and secret_key is not null
        )
    )
);

create index if not exists variable_value_variable_priority_idx
    on variable_value(variable_id, priority desc, id);

create index if not exists variable_value_selector_idx
    on variable_value(variable_id, resource_selector, priority desc);

create index if not exists variable_value_kind_idx
    on variable_value(kind);

create unique index if not exists variable_value_resolution_uniq
    on variable_value (
        variable_id,
        coalesce(resource_selector, ''),
        priority
    );


```

### **Core Concepts**

The system is built around two primary entities:

1. **Variable**
   * Defines a key within a specific scope
2. **Variable Value**
   * Defines one or more candidate values for a variable
   * Supports override semantics via priority and selectors

***

### **Variable**

Represents a named configuration key scoped to a specific owner.

Key properties:

* scope: one of resource, deployment, deployment\_job\_agent
* Exactly one owner reference is set
* key: variable identifier
* is\_sensitive: indicates whether the variable contains sensitive data

***

### **Variable Value**

Represents a candidate value for a variable.

Supports three value types:

* literal: JSON value stored directly
* ref: reference to another variable
* secret\_ref: reference to an external secret provider

Also includes:

* resource\_selector: optional matching condition
* priority: determines precedence

***

### **Value Types**

### **Literal**

Stores a JSON value directly in the database.

Example:

```
{ "host": "db.internal", "port": 5432 }
```

### **Reference**

References another variable by key, optionally with a path.

Example:

```
{ "ref": "db.config", "path": ["host"] }
```

### **Secret Reference**

References a value stored in an external secret manager.

Example:

```
{
  "provider": "vault",
  "key": "kv/data/prod/db",
  "path": ["password"]
}
```

***

### **Resolution Model**

To resolve a variable:

1. Identify the variable by scope and key
2. Retrieve all associated variable\_value rows
3. Filter by selector (if applicable)
4. Sort by priority (descending)
5. Select the highest-priority match
6. Resolve based on value type:
   * literal → return value
   * ref → recursively resolve referenced variable
   * secret\_ref → fetch from external provider

***

### **Why This Design**

### **Eliminates Duplication**

* One table for variables instead of per-scope tables
* One table for values instead of per-type tables

### **Extensible**

Adding a new value type (e.g., computed, templated) requires:

* Adding a new enum value
* Adding optional columns or extending logic

No new tables required.

### **Unified Resolution Logic**

All variables follow the same resolution pipeline regardless of scope or type.

### **Secret Handling**

Secrets are treated as a value source, not a separate system:

* Avoids duplicating schema
* Keeps resolution consistent
* Prevents storing sensitive data directly in the DB

***

## **Alternatives Considered**

### **1. Separate Tables per Scope**

Rejected because:

* Leads to schema duplication
* Requires duplicating logic
* Hard to extend

### **2. Separate Tables per Value Type**

Rejected because:

* Introduces join complexity
* Makes adding new types expensive

### **3. Separate Secret Tables**

Rejected because:

* Secrets participate in the same resolution semantics
* Only the value source differs
* Duplication would increase system complexity

***

## **Tradeoffs**

### **Pros**

* Dramatically simpler schema
* Centralized resolution logic
* Easier to extend
* Reduces duplication

### **Cons**

* More nullable columns in variable\_value
* Some validation shifts to application logic
* Slightly more complex constraints

***

## **Future Work**

* Replace ref\_key with referenced\_variable\_id for stronger integrity
* Introduce structured selector model (e.g., JSON-based matching)
* Add expression-based value model (single JSON expression column)
* Add audit logging and versioning

***

## **Migration Strategy**

1. Create new tables alongside existing schema
2. Backfill variables and values
3. Update read paths to use new schema
4. Deprecate old tables
5. Remove old schema after validation

***

## **Conclusion**

This proposal replaces a fragmented and duplicated schema with a unified, extensible model for variable management.

By treating scope and value type as data rather than schema, the system becomes:

* Easier to maintain
* Easier to extend
* More consistent in behavior

It also provides a clean path to integrate secrets without introducing additional structural complexity.


# RFC 0011: Auto-create Versions from GitHub Releases
Source: https://docs.ctrlplane.dev/rfc/0011-github-releases



| Category     | Status               | Created    | Author           |
| ------------ | -------------------- | ---------- | ---------------- |
| Integrations | <Badge>Draft</Badge> | 2026-04-20 | Aditya Choudhari |

**Issue:** [#993](https://github.com/ctrlplanedev/ctrlplane/issues/993)

## Problem

How do we know which Ctrlplane deployment a GitHub release belongs to?

`owner/repo` alone fails for monorepos (e.g. `ctrlplanedev/deployments` produces releases for `wandb`, `shared-tenant`, `k8s`, etc.). Naming-convention approaches (`<service>/v<version>`) fight existing tools — release-please emits `<service>-v<ver>`, changesets emits `<pkg>@<ver>`, semantic-release emits `v<ver>`.

## Proposal

A single CEL selector on deployment metadata.

```
git/release-selector: "repository.full_name == 'ctrlplanedev/deployments' && changedPaths.exists(p, p.startsWith('wandb/'))"
```

### Flow

1. `release` webhook hits `apps/api/src/routes/github/release.ts`.
2. Verify signature, resolve installation.
3. Build CEL context.
4. For every deployment with `git/release-selector` in metadata, evaluate.
5. Match → create deployment version with `release.tag_name` as the version.

### CEL context

```
action: string
release: ReleaseObject           // webhook payload
repository: RepositoryObject     // webhook payload (topics, custom_properties included)
sender: User                     // webhook payload

previousTag: string | null       // compare API
changedPaths: string[]           // compare API
commits: { sha, message, author }[]  // compare API
```

Only the bottom block requires an API call (`GET /repos/{o}/{r}/compare/{prev}...{tag}`).

### Metadata keys

| Key                    | Required | Purpose            |
| ---------------------- | -------- | ------------------ |
| `git/release-selector` | yes      | CEL returning bool |

No `git/repo` key — repo check lives inside the selector.

### GitHub App permissions

* Metadata: Read
* Contents: Read
* Events: `Release`, `Installation`, `Installation repositories`

### Error handling

* Selector throws → log, skip deployment.
* Compare API fails → evaluate without enriched fields (selectors referencing them evaluate false).
* Zero matches → log + 200.

## Out of scope (v1)

* Selector prefilter / indexing
* Draft / prerelease handling beyond what selectors express
* Backfilling historical releases


# RFC 0012: First-Class GitHub Installations
Source: https://docs.ctrlplane.dev/rfc/0012-github-installations



| Category     | Status               | Created    | Author           |
| ------------ | -------------------- | ---------- | ---------------- |
| Integrations | <Badge>Draft</Badge> | 2026-04-20 | Aditya Choudhari |

## Summary

Introduce a first-class `github_installation` table that binds a GitHub App
installation to a ctrlplane workspace, with creation restricted to a verified
OAuth flow. This closes a multi-tenancy gap where any workspace on a shared
ctrlplane instance can currently impersonate any GitHub installation by typing
its numeric ID into a job agent config. The RFC covers the schema, the linking
flow, and the migration of the existing job agent integration to reference
installations by foreign key.

## Motivation

### The current state

Ctrlplane runs a single GitHub App (`GITHUB_BOT_APP_ID`, `GITHUB_BOT_PRIVATE_KEY`)
that users install on their GitHub org. The installation is represented inside
`job_agent.config` as an untyped JSON blob:

```json theme={null}
{
  "type": "github-app",
  "installationId": 12345678,
  "owner": "acme-corp"
}
```

There is no dedicated schema for GitHub installations. The zod validator in
`packages/trpc/src/routes/job-agents.ts` accepts any `installationId:
z.number()` without cross-checking it against the calling workspace or the
calling user's GitHub identity. At runtime, the workspace engine mints an
installation token using the App's private key and whatever `installationId`
the job agent config carries — the server has no way to know whether the
workspace should legitimately have access to that installation.

### The multi-tenancy gap

Installation IDs are not secret. They are visible in the URL of any GitHub App
installation settings page (`https://github.com/organizations/<org>/settings/installations/<id>`)
and are sequential integers.

On a shared ctrlplane instance, the following attack is trivial:

1. A user in Workspace B discovers (or guesses) the installation ID of
   Workspace A's GitHub org.
2. The user creates a job agent in Workspace B with that installation ID.
3. Ctrlplane's server, holding the App's private key, successfully mints an
   installation token for Workspace A's org.
4. Workspace B can now list repos, dispatch workflows, and read repo metadata
   for Workspace A's org.

The only implicit defense today is "you have to know the installation ID,"
which is not an auth boundary.

### Why the fix has to happen at link-time

The workspace engine's GitHub dispatcher at
`apps/workspace-engine/pkg/jobagents/github/workflow_dispatcher.go` is a thin
wrapper that calls `gh.CreateClientForInstallation(ctx, cfg.InstallationId)`
with whatever `InstallationId` is in the config. Runtime validation in the
engine is the wrong layer — by then, the config has already been persisted and
the damage is done. The check must happen at the point of creation, i.e.
when a workspace first claims an installation.

## Goals

* Give GitHub installations a first-class table scoped to a workspace.
* Eliminate free-text `installationId` input from every user-facing surface.
* Require GitHub-side proof of admin rights (via OAuth + `GET /user/installations`)
  before a link is accepted.
* Allow one installation to be linked by multiple workspaces, as long as each
  link is independently verified.
* Migrate the existing GitHub Actions job agent to reference installations by
  FK instead of duplicating the ID in JSON.

## Non-Goals

* A CLI flow for linking. The UI flow is the v1 surface; CLI is a follow-up
  that plugs into the same backend handler.
* Supporting user-level GitHub OAuth as a general-purpose login mechanism. The
  OAuth token acquired during linking is used only to verify installation
  access for that single request and is discarded immediately after.
* Terraform-based creation of installation rows. Linking requires a human
  browser session; a data-source or CLI-assisted workflow will follow.

## Proposal

### Schema

```sql theme={null}
CREATE TABLE github_installation (
    id                 uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id       uuid NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,

    installation_id    bigint NOT NULL,   -- GitHub's numeric ID
    owner              text   NOT NULL,   -- GitHub org or user login
    account_type       text   NOT NULL,   -- 'Organization' | 'User'

    created_by_user_id uuid NOT NULL REFERENCES "user"(id),
    created_at         timestamptz NOT NULL DEFAULT now(),
    updated_at         timestamptz NOT NULL DEFAULT now(),

    UNIQUE (workspace_id, installation_id)
);

CREATE INDEX github_installation_installation_id_idx
    ON github_installation(installation_id);
```

Key points:

* **`UNIQUE (workspace_id, installation_id)`** — one installation can be
  linked to many workspaces, but each `(workspace, installation)` pair exists
  at most once. A shared-org scenario (e.g., dev workspace and prod workspace
  both watching the same repos) is supported natively.
* **No global uniqueness on `installation_id`** — multiple workspaces
  legitimately share a real-world org installation.
* **`created_by_user_id`** — records which ctrlplane user completed the link.
  Used for audit and for the per-user OAuth verification flow.

### Linking flow

The **only** backend entrypoint that creates rows in `github_installation` is a
single Setup URL handler. No tRPC mutation, no REST endpoint, no job-agent
config path accepts a raw `installationId`.

```
apps/api/src/routes/github/setup.ts
```

**Step 1 — User initiates the link from the UI.**
A "Connect GitHub" action in workspace settings opens:

```
https://github.com/apps/<app-slug>/installations/new?state=<signedJWT>
```

The `state` is a short-lived (≤5 min), single-use, HMAC-signed token containing
`{ workspaceId, userId, nonce }`. This binds the subsequent redirect to the
initiating ctrlplane session and prevents CSRF.

**Step 2 — GitHub install UX happens on github.com.**
The user picks an org, selects which repos to grant access to, and installs
(or reconfigures) the App. GitHub redirects the browser to ctrlplane's Setup
URL with `?installation_id=<id>&setup_action=install&state=<jwt>`.

**Step 3 — Ctrlplane verifies state and requires GitHub user OAuth.**

1. Verify `state` JWT signature, expiry, and single-use nonce.
2. Confirm the signed-in ctrlplane session matches `userId` in the state.
3. Redirect the user through the GitHub OAuth authorize endpoint to obtain a
   GitHub user access token (scope: minimal, enough to call
   `GET /user/installations`). The token is held only for the duration of the
   current linking request — it is **not persisted**. Once verification
   completes (success or failure), the token is discarded along with the
   request context.
4. Call `GET /user/installations` with the user's OAuth token. If the
   `installation_id` from the redirect is not in the returned list, **reject
   the link**. This is the teeth of the check: even if a malicious user
   replays a redirect with someone else's `installation_id`, the API will not
   list an installation they don't admin.

**Step 4 — Fetch installation metadata and insert.**
Call `GET /app/installations/<id>` with App JWT auth to fetch `account.login`
(`owner`), `account.type` (`accountType`), and other metadata. Insert one row
into `github_installation` scoped to the workspace.

### Why this flow is sufficient

* **No free-text input path exists.** The API never accepts `installationId`
  as user input. There is nothing to spoof.
* **`state` handles CSRF.** A user in Workspace B cannot mint a valid `state`
  for Workspace A.
* **User-level OAuth handles authorization.** Even if a `state` is somehow
  intercepted, the final `GET /user/installations` check fails unless the
  signed-in user is an admin of the target org — in which case they could
  install the App on that org themselves anyway.
* **`UNIQUE (workspace_id, installation_id)`** blocks duplicate claims within a
  workspace and permits legitimate cross-workspace sharing.

### Migration of the existing job agent

Today, `job_agent.config` for type `github-app` contains
`{ installationId, owner }` inline. After this RFC:

1. Add column `github_installation_id uuid NULL REFERENCES github_installation(id)`
   to `job_agent` (or continue storing it inside `config` as a UUID — TBD
   during implementation).
2. Update the zod schema in `packages/trpc/src/routes/job-agents.ts` to accept
   `{ type: "github-app", githubInstallationId: z.string().uuid() }` instead
   of `{ installationId, owner }`. The dropdown in the job agent creation UI
   becomes "pick from your workspace's linked installations" rather than a
   free-text ID field.
3. Update the workspace-engine dispatcher to resolve the `installationId`
   through a getter keyed on `githubInstallationId`, scoped to the job's
   workspace. The engine never sees a raw installation ID supplied by user
   input.

### UI changes

* **Workspace settings → Integrations → GitHub**: a new page listing linked
  installations, with "Connect GitHub" triggering the flow above and
  "Disconnect" removing the workspace's row (without uninstalling the App on
  GitHub).
* **Job agent creation (GitHub App type)**: the `installationId` free-text
  input is removed. Replace with a dropdown populated from
  `github_installation` rows in the current workspace. If the list is empty,
  link to the GitHub integrations page.

### Webhook handling

Unchanged by this RFC. The existing `workflow_run` handler at
`apps/api/src/routes/github/workflow_run.ts` continues to operate on
`installation.id` from the webhook payload. Since `installation.id` comes from
GitHub (not from a workspace), no trust change is needed there.

Lifecycle events (`installation.deleted`, `installation_repositories.*`) will
be handled alongside this RFC to keep `github_installation` rows in sync when
users uninstall the App or change repo access on GitHub.

## Alternatives Considered

### 1. Keep installation data in `job_agent.config`, add a validation step

Rejected. Layering validation on top of a free-text schema means every new
caller must remember to apply the check. A malformed or forgotten validation
in any future code path reopens the gap. Making the schema itself the source
of truth eliminates the class of bug.

### 2. Use installation ID as a globally unique primary key

Rejected. Forbids the legitimate case of one GitHub org serving multiple
ctrlplane workspaces (dev/staging/prod separation, multi-team monorepos).
`UNIQUE (workspace_id, installation_id)` captures the right invariant.

### 3. Rely on the `state` CSRF token alone, skip GitHub OAuth

Rejected. The `state` parameter proves the redirect was initiated from a
ctrlplane session but does not prove the ctrlplane user has admin rights on
the target org. Without the `GET /user/installations` check, a determined
user can still initiate an install flow, swap the `installation_id` in the
redirect (if they intercept it), and claim an installation they don't own.
OAuth verification is what makes the boundary real.

### 4. Allow Terraform / API creation of installation rows

Rejected for v1. The verification fundamentally requires a browser session
for GitHub OAuth; any programmatic path would need to either skip the check
(defeating the purpose) or accept pre-issued OAuth tokens (significant added
complexity and credential-handling surface). IaC users can still manage
everything *downstream* of the installation (job agents, deployments,
deployment sources) via Terraform once the installation row exists.

## Tradeoffs

### Pros

* Closes the cross-workspace installation access gap.
* Existing `job_agent.config` shape is simplified.
* Clear audit trail: `created_by_user_id` + timestamp on every link.

### Cons

* Introduces a user-level GitHub OAuth flow that ctrlplane did not previously
  need. Additional client ID/secret management for the App's OAuth surface.
* Terraform and API users cannot create installations programmatically; they
  must complete the UI flow once per workspace per installation.

## Migration Strategy

Ctrlplane is still in an internal-only phase, so the set of existing GitHub
installations in the wild is small and known. There is no need for an
automated backfill.

1. Add the `github_installation` table as a purely additive schema change.
   Nothing else in the system references it yet.
2. Ship the UI + Setup URL handler so new links go through the verified flow.
3. **Manually re-link each existing installation through the UI.** For the
   handful of existing internal workspaces that have a GitHub installation
   in their `job_agent.config` today, an authorized user in each workspace
   clicks "Connect GitHub" and completes the same verified flow. This
   produces a `github_installation` row with correct `created_by_user_id`
   and audit metadata — no raw SQL inserts, no nullable-column carve-outs.
4. Once all known installations have corresponding `github_installation`
   rows, flip the job agent config schema to require `githubInstallationId`
   (the FK) and update the workspace-engine dispatcher to resolve via getter.
   At this point the raw `installationId` free-text path is removed.
5. Remove the old `{ installationId, owner }` JSON shape from `job_agent.config`.

Manual re-linking is acceptable precisely because the population is tiny and
each re-link takes less than a minute. If ctrlplane were public or had many
installations, we'd want an automated backfill instead; at current scale,
going through the UI is simpler than writing migration code.

## Open Questions

1. **OAuth client.** Does the GitHub App already have OAuth credentials
   configured (`GITHUB_BOT_CLIENT_ID`/`_CLIENT_SECRET`), or do we need a
   separate OAuth app registration? If the App's OAuth surface is sufficient,
   we avoid an extra credential.
2. **Unlink semantics.** When a workspace disconnects an installation, should
   linked job agents be disabled, deleted, or left in a broken state pointing
   at a now-missing FK? Soft-disable seems safest.
3. **Reinstall handling.** If a user uninstalls the App on GitHub and
   reinstalls it later, GitHub may issue a new `installation_id`. Should
   ctrlplane detect this (via `installation.deleted` + subsequent
   `installation.created` events from the same `account.id`) and offer a
   "relink" UX, or require the user to re-run the flow manually?

## Conclusion

The GitHub integration today conflates "the App can reach this installation"
with "this workspace is entitled to use this installation." Adding a
first-class `github_installation` table, with a verified link flow as the
only creation path, separates the two and closes a real multi-tenancy gap on
shared instances. The work is small, contained, and additive.


# RFC 0013: Multi-Kind Plan Results
Source: https://docs.ctrlplane.dev/rfc/0013-multi-kind-plan-results



| Category | Status               | Created    | Author           |
| -------- | -------------------- | ---------- | ---------------- |
| Engine   | <Badge>Draft</Badge> | 2026-05-12 | Aditya Choudhari |

## Summary

A job agent's `Plan` returns a single `(current, proposed)` diff today, stored
in one result row per agent invocation. Generalize so an agent can return
multiple labeled diffs per invocation, distinguished by a new `kind` column.
This unblocks #1075 (surface the rendered ArgoCD Application CR alongside the
existing rendered manifest diff) and gives future agents a uniform shape for
shipping multiple kinds of output.

## Motivation

Issue #1075 asks for the rendered ArgoCD Application CR to be shown in plan
output. The CR is already computed by the planner today (`proposedApp` at
`argocd_plan.go:111`) — used to create a temp Application, then discarded
after the downstream manifest diff is extracted.

More broadly, there's no shape in the current model for an agent to express
"I have multiple distinct diffs the deployer should review." The `Plannable`
interface and `deployment_plan_target_result` are 1:1.

## Proposal

### Storage

Add a `kind` column. Each agent invocation produces N rows, one per kind.

```sql theme={null}
ALTER TABLE deployment_plan_target_result
  ADD COLUMN kind TEXT NOT NULL DEFAULT '';
```

Single statement. Existing rows get `''`; new rows specify a kind explicitly
or fall back to `''`. Section vocabulary is agent-defined — the schema
doesn't enumerate kinds.

```text theme={null}
deployment_plan_target  (one RT)
  ├─ result (kind="manifest", agent=argo-cd)   current/proposed = rendered manifests
  ├─ result (kind="cr",       agent=argo-cd)   current/proposed = Application CR YAML
  └─ result (kind="plan",     agent=tfc)       current/proposed = TF plan output
```

### Plan interface

`Plannable.Plan` returns `[]PlanResult`. Each result carries a `Kind`.

```go theme={null}
type PlanResult struct {
    Kind        string
    Current     string
    Proposed    string
    HasChanges  bool
    ContentHash string
    Status, Message, State, CompletedAt   // unchanged
}

Plan(ctx, dispatchCtx, state) ([]PlanResult, error)
```

The ArgoCD planner emits two results — `manifest` (existing flow) and `cr`
(marshal `proposedApp` + fetch current Application from ArgoCD by name). TFC
emits one (`plan`). Agents that don't implement `Plannable` are skipped as
today. All kinds for an invocation complete together; if any is incomplete,
the agent returns `CompletedAt == nil` and the worker requeues per the
existing pattern.

### Stage-2 controller

The work-queue item still represents one agent invocation. Stage-1 inserts
one row (the work item's anchor). Stage-2 calls `Plan`, gets `[]PlanResult`,
writes the first result into the anchor row and inserts additional rows for
the remaining kinds. Validation runs once across all rows for the invocation.

### Validation: one run per invocation against flat input

Validation runs once per agent invocation, not per row. The OPA input is
built by combining all kind-rows into a flat shape:

```json theme={null}
{
  "manifest":   { "current": <parsed>, "proposed": <parsed>, "has_changes": true },
  "cr":         { "current": <parsed>, "proposed": <parsed>, "has_changes": true },
  "agent_type": "argo-cd",
  "deployment": { ... }, "environment": { ... }, "resource": { ... },
  "proposed_version": { ... }, "current_version": { ... }
}
```

Rules self-select via `input.manifest.proposed` / `input.cr.proposed`. No
DB-layer routing, no per-rule `applies_to_kind` declaration. Reserved
top-level keys — `agent_type`, `deployment`, `environment`, `resource`,
`proposed_version`, `current_version` — are documented; agents shouldn't use
these as section names.

Violations attach to the anchor row (the original stage-1 row). The
`deployment_plan_target_result_validation` schema is unchanged.

### Aggregation: kind → agent → target

Rows are kind-level. Aggregates are agent-level. Target totals roll up over
agents — same as today, just one extra rollup step.

```text theme={null}
target
  └─ agent
       ├─ kind "manifest"   ← row
       └─ kind "cr"         ← row
```

Per-agent rollup:

| Field        | Rule across the agent's kinds                                           |
| ------------ | ----------------------------------------------------------------------- |
| `status`     | worst kind wins (`errored` > `computing` > `unsupported` > `completed`) |
| `hasChanges` | OR across kinds                                                         |

`aggregateResults` then runs over agent states. Existing target-level counts
(`Total`, `Completed`, `Errored`, `Changed`, …) keep their semantics — they
just source from agent rollups instead of raw rows.

### UI

* **Plan results table:** the existing Changes column shows the total
  `+N -M` summed across all kinds for the row. A new column surfaces the
  number of diff kinds for that row (e.g. `2` for an ArgoCD row producing
  manifest + CR), so the deployer knows there's more than one diff behind
  the row before clicking in.
* **Detail modal:** when a row is opened, the modal contains a select whose
  options are populated dynamically from whatever kinds the agent returned
  for that release target. Picking an option renders that kind's diff.
  Single-kind rows show the select with one option (or hide it entirely).

### GitHub check rendering

`formatAgentSection` iterates the agent's kinds and renders each as its own
labeled `diff` block. `aggregate.checkTitle` follows the worst-kind / OR
rules above. Existing `MaybeUpdateTargetCheck` flow is otherwise unchanged.

## Migration

* One ALTER adds `kind TEXT NOT NULL DEFAULT ''`.
* Existing rows keep `kind=''`. The renderer maps `''` → "Manifest" for
  legacy display (existing rows are all manifest diffs by construction).
* No production validation rules depend on the current flat OPA input shape,
  so the input restructure has no rule breakage.
* Other Plannable agents (TFC, TestRunner) wrap their existing single
  `PlanResult` in a one-element slice and set `Kind` to a chosen string.

## Out of scope

* Plan triggers other than version publish (no `deployment_plan` snapshot
  rework, no resource/environment plan kinds).
* Stage-1 controller fan-out, variable resolution, release-target snapshot.
* Anything beyond plan-result diff content.

## Open Questions

1. **Per-kind vs invocation-level violation attribution.** Validation runs
   once per invocation with all sections visible, so a violation logically
   describes the whole invocation. UI displays violations at the agent
   level. Could be revisited if rule authors want to tag violations with a
   specific kind for per-section display (e.g. "this denial is about the CR,
   not the manifest"). Default: invocation-level for v1.

2. **Backfill legacy `kind=''` rows or leave them.** Backfilling existing
   rows to `kind='manifest'` is more honest but costs a single UPDATE.
   Leaving them as `''` works but requires the renderer to special-case
   legacy. Lean: leave as `''`; rows drain quickly via `expires_at`.


# RFC 0014: Pull-Based Job Agent API
Source: https://docs.ctrlplane.dev/rfc/0014-pull-based-job-agent



| Category   | Status               | Created    | Author           |
| ---------- | -------------------- | ---------- | ---------------- |
| Job Agents | <Badge>Draft</Badge> | 2026-06-02 | Aditya Choudhari |

## Summary

Provide a REST API that lets an external provider act as a job agent by
**pulling** the jobs assigned to it, executing them, and reporting status back —
rather than ctrlplane pushing work into the provider's environment.

The work is split into two parts:

* **V1** delivers the pull contract: an agent polls for queued jobs, claims one
  atomically (at most once), runs it, and reports status. A new `queued` job
  status marks a job as claimable. Polling is a side-effect-free list; a separate
  claim call transitions the job and returns its execution context.
* **V2** adds crash recovery: a lease, a heartbeat endpoint, and a reaper that
  returns abandoned jobs to the queue. V2 is purely additive — V1 is shippable
  and useful on its own.

## Motivation

Ctrlplane's existing job agents are **push / dispatch-style**. The workspace
engine initiates execution inside the agent's system: ArgoCD syncs an
Application, GitHub Actions runs a workflow, Terraform Cloud applies a plan. In
each case ctrlplane reaches outbound into the agent's environment.

This does not fit an external provider that:

* cannot (or should not) be reached inbound by ctrlplane, and
* wants to integrate generically over HTTP rather than through a bespoke,
  per-system integration.

There is currently no generic way for such a provider to pull the jobs assigned
to its job agent and run them. This RFC adds that path while reusing the
existing job model, status-reporting endpoint, and verification flow.

## Proposal

### Model: producer / consumer

A push agent's dispatch step both *produces* the job and *delivers* it (fires
the workflow). A pull agent splits these:

* ctrlplane **produces** the job and marks it claimable.
* the external agent **consumes** it by polling, claiming, and running it.

The job row in Postgres is the queue. The dispatch controller is the producer;
the agent's poll discovers work and its claim takes delivery.

### Job status: `queued`

A new value `queued` is added to the `job_status` enum
(`packages/db/src/schema/job.ts`). It means: ctrlplane has finished preparing
the job, and it is available for an agent to claim.

```sql theme={null}
ALTER TYPE job_status ADD VALUE 'queued' AFTER 'pending';
```

The lifecycle for a pull-agent job:

```text theme={null}
queued ───claim (poll)───► in_progress ───report───► successful / failure
```

`queued` is semantically distinct from the existing states:

* `pending` — created, not yet processed by the dispatch controller.
* `queued` — prepared, waiting for an agent to claim it.
* `in_progress` — claimed by an agent and executing.

The new value must be mirrored everywhere the enum is represented: the
`@ctrlplane/validators` job statuses, the `dbToOapiStatus` / `oapiToDbStatus`
maps in `apps/api/src/routes/v1/workspaces/jobs.ts`, the OpenAPI `JobStatus`
schema, and the workspace-engine `oapi` enum plus its sqlc mappings.

### Agent type: `http-pull`

A new agent type `http-pull` is registered in the workspace engine's job agent
registry (`apps/workspace-engine/pkg/jobagents/`, registered in
`apps/workspace-engine/svc/controllers/jobdispatch/controller.go`). It
implements `types.Dispatchable`. Its `Dispatch` does not push to an external
system; it transitions the job to `queued`:

```go theme={null}
package httppull

var _ types.Dispatchable = &HttpPull{}

func (a *HttpPull) Type() string { return "http-pull" }

func (a *HttpPull) Dispatch(ctx context.Context, job *oapi.Job) error {
    return a.setter.UpdateJob(ctx, job.Id, oapi.JobStatusQueued, "", nil)
}
```

This keeps the dispatch pipeline uniform. Eligibility and the dispatch
controller are otherwise **unchanged**: a job is created `pending`, enqueued for
dispatch, the controller creates verification specs as it does for every agent,
and the `Dispatch` call marks the job `queued` instead of pushing.

### Verifications

Verifications are created by the dispatch controller at dispatch time, exactly
as they are for the ArgoCD and Terraform Cloud agents. No change is made to the
verification flow. As with those agents, verification metrics begin measuring
when created rather than when execution starts. For `http-pull` this means
measurements can begin before an agent claims the job; this matches existing
behavior and is accepted for V1. See Open Questions.

### Poll endpoint (V1)

```
GET /v1/workspaces/{workspaceId}/job-agents/{jobAgentId}/jobs?status=queued
```

Returns all jobs for the agent in the requested status. This is a plain,
side-effect-free poll: it lists what is claimable but claims nothing. The agent
picks a job from the list and claims it with a separate call. Added to
`apps/api/src/routes/v1/workspaces/job-agents.ts`.

The list response is intentionally lightweight — job id, deployment,
environment, resource, and `created_at` — and **omits `dispatch_context`**.
Resolved variables (including secret-flagged ones) are not returned here, so a
poll never broadcasts secrets for every queued job to every agent. Context is
returned only on claim, and only to the agent that wins it.

### Claim endpoint (V1)

```
POST /v1/workspaces/{workspaceId}/job-agents/{jobAgentId}/jobs/{jobId}/claim
```

Atomically transitions a specific job `queued → in_progress` and returns the
full job, including `dispatch_context`. Because the poll has no side effects,
this is the single mutating step that hands a job to an agent.

The claim is a conditional update guarded on the current status. Postgres row
locking — not the transaction boundary — provides the at-most-once guarantee:

```sql theme={null}
UPDATE job
SET status = 'in_progress', started_at = now()
WHERE id = $1 AND status = 'queued' AND job_agent_id = $2
RETURNING *;
```

If two agents claim the same job id concurrently, the row lock serializes them
and only the first still sees `status = 'queued'`; the second matches zero rows
and receives `409 Conflict`. No `SELECT ... FOR UPDATE SKIP LOCKED` scan is
needed because the agent names the job id explicitly — the `status = 'queued'`
predicate does the work the locking scan did in the next-job design. The
reconcile work queue uses the same conditional-claim shape
(`ClaimReconcileWorkItems`).

### Job payload

The **claim** response returns the job as-is; the **poll** response omits it.
The job's `dispatch_context` column is a self-contained execution snapshot
already populated at job creation — deployment, environment, resource, release,
version, resolved inputs, and variables. No joins or additional assembly are
required; the existing `toJobResponse` shape already emits `jobAgentConfig` and
`dispatchContext`.

Note: `dispatch_context` includes resolved variable values, so secret-flagged
variables are returned to the external agent. Returning context only on claim —
not on poll — limits this exposure to the one job the agent actually runs,
rather than every queued job a poll would list. This data otherwise never leaves
ctrlplane for push agents. The endpoint must be served over TLS; per-agent
authentication is addressed under V2.

### Status reporting

Status reporting reuses the existing endpoint:

```
PUT /v1/workspaces/{workspaceId}/jobs/{jobId}/status
```

It already records the status, sets `completed_at` on terminal states, and
enqueues a desired-release evaluation to advance the release. No new endpoint is
required for V1.

### Authentication (V1)

V1 uses the existing `x-api-key` authentication and verifies that the target job
agent belongs to the authenticated workspace. Per-agent credentials are
addressed under V2.

### Concurrency

The issue identifies two failure modes. V1 addresses the first; V2 addresses the
second.

1. **Double-pickup** — handled by the conditional claim above. A job is handed
   out at most once, even under overlapping claims or client retries; losers get
   `409 Conflict`.
2. **Crash mid-job** — not handled in V1. If an agent claims a job and dies, the
   job remains `in_progress`. Recovery is a manual transition back to `queued`
   (the same transition V2 automates). V2 adds automatic recovery.

### V1 implementation surface

| Area              | Change                                                                                                    |
| ----------------- | --------------------------------------------------------------------------------------------------------- |
| `job_status` enum | add `queued` (schema + migration, validators, API status maps, OpenAPI, oapi/sqlc)                        |
| Agent type        | new `http-pull` package; `Dispatch` sets `queued`; register in `jobdispatch`                              |
| Poll endpoint     | `GET .../job-agents/{id}/jobs?status=queued`; lists queued jobs, no context; OpenAPI                      |
| Claim endpoint    | `POST .../job-agents/{id}/jobs/{jobId}/claim`; conditional `queued→in_progress`, returns context; OpenAPI |
| Status reporting  | reuse `PUT .../jobs/{jobId}/status`                                                                       |
| Eligibility       | unchanged                                                                                                 |
| Dispatch flow     | unchanged except the `http-pull` `Dispatch` body                                                          |
| Auth              | reuse `x-api-key` + workspace ownership check                                                             |

## V2: Lease, Heartbeat, and Reclaim (add-on)

V2 adds crash recovery. It is additive in the strongest sense: a new table, two
endpoints, and a periodic sweep. **The `job` table is not modified at all** —
the `queued` enum value was already added in V1.

### Claim table

Lease state lives in a dedicated `job_claim` table rather than as columns on
`job`:

```sql theme={null}
CREATE TABLE job_claim (
  job_id           uuid PRIMARY KEY REFERENCES job(id) ON DELETE CASCADE,
  job_agent_id     uuid NOT NULL,
  claimed_at       timestamptz NOT NULL DEFAULT now(),
  claim_expires_at timestamptz NOT NULL,
  claim_id         uuid NOT NULL DEFAULT gen_random_uuid()
);
```

The job's `status` remains the state machine — the claim still flips
`queued → in_progress` on `job` — but the lease lifecycle and the high-frequency
heartbeat writes are isolated to this narrow table. The motivation is write
locality: heartbeats are the most frequent write in this feature (every in-flight
job, every interval), and `job` is a hot, heavily-joined table with several
indexes and an `updated_at` trigger. Keeping heartbeats off `job` avoids index
churn and MVCC bloat on the read path. `claim_id` is a fencing token, populated
for free.

### Lease

The claim records lease state in `job_claim` in the same statement that flips
the job to `in_progress`, using a CTE so it remains a single atomic operation:

```sql theme={null}
WITH claimed AS (
  UPDATE job SET status = 'in_progress', started_at = now()
  WHERE id = $1 AND status = 'queued' AND job_agent_id = $2
  RETURNING id
)
INSERT INTO job_claim (job_id, job_agent_id, claim_expires_at)
SELECT id, $2, now() + make_interval(secs => $lease_seconds) FROM claimed
RETURNING *;
```

The lease is a liveness window, not an execution deadline. A job may run far
longer than the lease as long as the agent keeps the claim alive. The claim
response advertises `lease_seconds` so the agent can choose a heartbeat interval.

### Heartbeat

```
POST /v1/workspaces/{workspaceId}/jobs/{jobId}/heartbeat
```

Extends the lease. This touches only `job_claim`, never `job`:

```sql theme={null}
UPDATE job_claim
SET claim_expires_at = now() + make_interval(secs => $lease_seconds)
WHERE job_id = $1;
```

The agent calls this periodically while executing. The interval is the agent's
choice (a fraction of the advertised lease); the server does not store it.

### Reaper

A periodic sweep returns abandoned claims to the queue — deleting the expired
claim and flipping the job back to `queued` in one statement:

```sql theme={null}
WITH expired AS (
  DELETE FROM job_claim WHERE claim_expires_at < now() RETURNING job_id
)
UPDATE job SET status = 'queued'
WHERE id IN (SELECT job_id FROM expired) AND status = 'in_progress';
```

Expiry is detected by this sweep, not by an event at the exact expiry time. The
sweep mirrors the reconcile queue's `CleanupExpiredClaims`.

Reclaim is **opt-in by construction**: only jobs that have a `job_claim` row are
ever swept. A job claimed without recording lease state — or any V1-era agent
that never engages the lease protocol — has no claim row and is never reclaimed,
preserving V1 behavior after V2 ships. When a job reaches a terminal status, its
`job_claim` row is removed.

### Reclaim and double-run

When a lease expires, the job returns to `queued` and becomes claimable again.
The reaper cannot distinguish a crashed agent from one that is alive but quiet
for longer than the lease, so a long pause can cause a job to be reclaimed and
run twice. A generous lease relative to the heartbeat interval reduces this
window but does not close it.

If exactly-once execution is required, the `claim_id` fencing token is returned
on claim, echoed by the agent on heartbeat and status, and a write carrying a
stale `claim_id` (one whose claim row was already reclaimed and superseded) is
rejected. The token exists in the schema from the start; enforcing it is
optional.

### V2 implementation surface

| Area            | Change                                                                       |
| --------------- | ---------------------------------------------------------------------------- |
| `job` schema    | none — `job` is not modified                                                 |
| `job_claim`     | new table (single `CREATE TABLE`, no change to `job`)                        |
| Claim           | record `job_claim` row in the claim CTE; return `lease_seconds` + `claim_id` |
| Heartbeat       | new `POST .../jobs/{jobId}/heartbeat`, writes only `job_claim`; OpenAPI path |
| Reaper          | periodic sweep deleting expired claims and returning jobs to `queued`        |
| Terminal status | remove the `job_claim` row when a job reaches a terminal state               |
| Optional        | per-agent lease config, `claim_id` fencing enforcement, per-agent tokens     |

## Migration

* V1 adds the `queued` value to the `job_status` enum. V2 adds a new `job_claim`
  table and does not modify `job`. Both are additive; existing jobs are
  unaffected.
* The dispatch controller, eligibility logic, and promotion lifecycle are
  unchanged except for recognizing the `queued` status and the `http-pull`
  agent's `Dispatch` body.
* The status-reporting endpoint is reused unchanged. The poll and claim
  endpoints (V1) and heartbeat endpoint (V2) are new and do not alter existing
  endpoints.
* The V2 reaper only acts on jobs that have a `job_claim` row, so introducing it
  does not change the behavior of any agent that does not heartbeat.

## Open Questions

1. **Long-poll vs. plain poll.** V1 uses a plain poll: the list endpoint returns
   immediately with the current set of queued jobs (possibly empty). A long-poll
   variant (hold the request open until a job appears or a timeout elapses,
   bounded by a server-enforced maximum) reduces idle polling and is a candidate
   for V2. Backpressure and fairness limits on held connections are open.

2. **Verification timing.** Verifications begin measuring when created (at
   dispatch), which for a pull agent can precede the claim by an unbounded
   queue wait. For long verification windows this is harmless; a short window
   could complete before the agent claims the job. If this becomes a problem,
   verification creation can be moved to the claim transition, or measurement
   can be gated on the job reaching `in_progress`. Deferred until needed.

3. **Lease configuration.** Should the lease duration be per-agent
   (`job_agent.config`, bounded) or a single global default? A global default is
   the V2 starting point; per-agent is a later refinement for agents with
   different reliability characteristics.

## AI Generated Questions

1. **Agent registration.** A job is only routed to an agent that already exists
   and is matched by a deployment's `jobAgentSelector`. Should an external agent
   be able to self-register its `job_agent` row and credentials via the API, or
   must agents be pre-provisioned by an operator?

2. **Per-agent authentication.** V1 reuses `x-api-key`. V2 should issue a
   per-agent credential at registration so an agent authenticates as itself and
   can claim only its own jobs. What is the token model and rotation story?

3. **Fencing.** Should V2 include a fencing token from the start, or add it only
   if double-run under lease expiry proves to be a real problem for the
   workloads pull agents run?


# Deployment Verification
Source: https://docs.ctrlplane.dev/use-cases/deployment-verification

Automatically verify deployments are healthy before proceeding

## The Scenario

Your deployment pipeline says "success" but the service is actually broken. You want:

* Automatic health checks after every deployment
* Real metrics from Datadog, Prometheus, or your monitoring stack
* Automatic rollback if verification fails
* No more "deploy succeeded, service is down" situations

## Without Ctrlplane

**The typical flow:**

1. CI/CD deploys the service
2. Pipeline says ✅ because kubectl apply succeeded
3. Service is actually broken (bad config, missing dependency, etc.)
4. Someone notices 30 minutes later
5. Frantic rollback ensues

**What goes wrong:**

* "Deployed successfully" ≠ "Working correctly"
* Health checks are bolted on, not built in
* Rollback is manual and error-prone
* No verification between regions during multi-cluster deploys

## With Ctrlplane

Add a verification policy:

```yaml theme={null}
type: Policy
name: Deployment Health Check
selectors:
  - environment: environment.name in ["Staging", "Production"]
rules:
  - verification:
      metrics:
        - name: error-rate
          provider:
            type: datadog
            apiKey: "{{.secrets.DD_API_KEY}}"
            appKey: "{{.secrets.DD_APP_KEY}}"
            query: "sum:trace.http.request.errors{service:{{.deployment.name}},env:{{.environment.name}}}.as_rate()"
          successCondition: result.value < 0.01
          failureThreshold: 2
          intervalSeconds: 60
          count: 5

        - name: latency-p99
          provider:
            type: datadog
            apiKey: "{{.secrets.DD_API_KEY}}"
            appKey: "{{.secrets.DD_APP_KEY}}"
            query: "p99:trace.http.request.duration{service:{{.deployment.name}},env:{{.environment.name}}}"
          successCondition: result.value < 500  # 500ms
          failureThreshold: 2
          intervalSeconds: 60
          count: 5
```

## What Happens

```mermaid theme={null}
flowchart LR
    Deploy["Deploy"] --> M1["Check 1"]
    M1 --> M2["Check 2"]
    M2 --> M3["Check 3"]
    M3 --> M4["Check 4"]
    M4 --> M5["Check 5"]
    M5 -->|"all pass"| Success["Success"]
    M3 -->|"2 failures"| Rollback["Rollback"]
```

1. **Deployment completes** — Job agent reports success
2. **Verification starts** — First metric check at 60 seconds
3. **Metrics queried** — Datadog returns error rate and latency
4. **Success condition evaluated** — Is error rate \< 1%? Is p99 \< 500ms?
5. **Continue or fail** — If 2+ checks fail, trigger rollback
6. **Rollback executes** — Previous version is deployed automatically

## Key Benefits

| Benefit                     | How It Works                                          |
| --------------------------- | ----------------------------------------------------- |
| **Real metrics**            | Use your actual monitoring data, not synthetic checks |
| **Automatic rollback**      | No manual intervention when verification fails        |
| **Configurable thresholds** | Define what "healthy" means for your service          |
| **Multiple metrics**        | Check error rate AND latency AND custom metrics       |
| **Per-environment rules**   | Stricter verification in production                   |

## Verification Providers

### Datadog

```yaml theme={null}
provider:
  type: datadog
  apiKey: "{{.secrets.DD_API_KEY}}"
  appKey: "{{.secrets.DD_APP_KEY}}"
  query: "sum:errors{service:api}.as_rate()"
```

### Prometheus

```yaml theme={null}
provider:
  type: prometheus
  address: "http://prometheus.monitoring:9090"
  query: "rate(http_requests_total{status=~\"5..\"}[5m])"
```

### HTTP Endpoint

```yaml theme={null}
provider:
  type: http
  url: "https://{{.resource.config.host}}/health"
  method: GET
  headers:
    Authorization: "Bearer {{.secrets.HEALTH_TOKEN}}"
```

### Custom Script

```yaml theme={null}
provider:
  type: http
  url: "https://internal-api/verify"
  method: POST
  body: |
    {
      "deployment": "{{.deployment.name}}",
      "version": "{{.version.tag}}",
      "resource": "{{.resource.identifier}}"
    }
```

## Variations

### Progressive Verification

Start with quick checks, then extend to thorough verification:

```yaml theme={null}
metrics:
  # Quick smoke test (first 2 minutes)
  - name: health-endpoint
    provider:
      type: http
      url: "https://{{.resource.config.host}}/health"
    successCondition: result.statusCode == 200
    intervalSeconds: 30
    count: 4
    
  # Extended error rate check (next 5 minutes)
  - name: error-rate
    provider:
      type: datadog
      query: "sum:errors{service:api}"
    successCondition: result.value < 0.01
    intervalSeconds: 60
    count: 5
```

### Environment-Specific Thresholds

```yaml theme={null}
# Staging: Lenient
- name: Staging Verification
  selectors:
    - environment: environment.name == "Staging"
  rules:
    - verification:
        metrics:
          - name: error-rate
            successCondition: result.value < 0.05  # 5% error rate OK

# Production: Strict
- name: Production Verification
  selectors:
    - environment: environment.name == "Production"
  rules:
    - verification:
        metrics:
          - name: error-rate
            successCondition: result.value < 0.01  # Only 1% allowed
            failureThreshold: 1  # Fail fast
```

### Multi-Metric Verification

```yaml theme={null}
metrics:
  - name: error-rate
    successCondition: result.value < 0.01
    
  - name: latency-p99
    successCondition: result.value < 500
    
  - name: saturation
    successCondition: result.value < 0.8  # CPU < 80%
    
  - name: availability
    successCondition: result.value > 0.999  # 99.9% uptime
```

## Rollback Behavior

When verification fails:

1. **Rollback triggered** — Ctrlplane initiates rollback
2. **Previous version deployed** — The last successful version is redeployed
3. **Verification runs again** — Confirms rollback was successful
4. **Release marked failed** — Full audit trail preserved

You can also configure rollback behavior:

```yaml theme={null}
rules:
  - verification:
      rollback:
        enabled: true
        toVersion: previous  # or specific version tag
```

## Next Steps

<CardGroup>
  <Card title="Datadog Verification" icon="chart-line" href="../policies/verification/providers/datadog">
    Configure Datadog metrics
  </Card>

  <Card title="HTTP Verification" icon="globe" href="../policies/verification/providers/http">
    Set up HTTP health checks
  </Card>

  <Card title="Environment Promotion" icon="arrow-right" href="./environment-promotion">
    Gate promotions on verification
  </Card>

  <Card title="Multi-Region" icon="globe" href="./multi-region">
    Verify between regions
  </Card>
</CardGroup>


# Dynamic Environments
Source: https://docs.ctrlplane.dev/use-cases/dynamic-environments

Automatically include new infrastructure in deployments

## The Scenario

You're adding infrastructure regularly—new clusters, new regions, new services. You want:

* New clusters to automatically receive deployments
* No config file updates when infrastructure changes
* Environment membership based on resource attributes
* Different policies for different resource types

## Without Ctrlplane

**The typical approach:**

1. Add new cluster to Kubernetes
2. Update CI/CD pipeline config
3. Update deployment manifests
4. Update ArgoCD ApplicationSet
5. Update monitoring config
6. Forget one thing, wonder why deploys don't work

**What goes wrong:**

* Adding infrastructure requires multiple config changes
* Easy to miss something
* Config files become the source of truth (and drift)
* Manual process doesn't scale

## With Ctrlplane

### Define Environments with Selectors

Instead of listing clusters, define what "Production" means:

```yaml theme={null}
type: Environment
name: Production
resourceSelector: resource.metadata["env"] == "production"
```

### Tag Resources Appropriately

When you add a new cluster:

```yaml theme={null}
type: Resource
name: prod-ap-south-1
kind: KubernetesCluster
metadata:
  env: production      # ← This makes it "Production"
  region: ap-south-1
  cloud: aws
```

That's it. The new cluster:

* Is automatically part of the "Production" environment
* Receives deployments for all services targeting Production
* Inherits policies that apply to Production
* Shows up in the inventory immediately

## What Happens

```mermaid theme={null}
flowchart TB
    subgraph Before["Before: 3 Clusters"]
        R1["us-east-1<br/>env: production"]
        R2["us-west-2<br/>env: production"]
        R3["eu-west-1<br/>env: production"]
    end

    Env["Production Environment<br/>selector: env == production"]

    R1 -.->|"matches"| Env
    R2 -.->|"matches"| Env
    R3 -.->|"matches"| Env

    subgraph After["After: 4 Clusters"]
        R4["ap-south-1<br/>env: production"]
    end

    R4 -.->|"auto-joins"| Env
```

1. **Environment defined** — "Production" = all resources where `env == production`
2. **3 clusters exist** — All have `env: production`, all are in Production
3. **New cluster added** — ap-south-1 tagged with `env: production`
4. **Automatic membership** — New cluster is now part of Production
5. **Deployments flow** — Next release automatically targets all 4 clusters

## Key Benefits

| Benefit                    | How It Works                                   |
| -------------------------- | ---------------------------------------------- |
| **Zero config changes**    | Tag the resource, it joins automatically       |
| **Declarative membership** | Environment definition is the source of truth  |
| **Immediate effect**       | New resources are included in next deployment  |
| **Scales infinitely**      | 4 clusters or 400, same environment definition |
| **Consistent policies**    | New resources inherit existing policies        |

## Selector Patterns

### By Environment Label

```yaml theme={null}
# Simple environment matching
name: Production
resourceSelector: resource.metadata["env"] == "production"

name: Staging
resourceSelector: resource.metadata["env"] == "staging"
```

### By Region

```yaml theme={null}
# US regions only
name: Production US
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["region"].startsWith("us-")

# EU regions only
name: Production EU
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["region"].startsWith("eu-")
```

### By Team

```yaml theme={null}
# Platform team's infrastructure
name: Platform Production
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["team"] == "platform"
```

### By Resource Type

```yaml theme={null}
# Only Kubernetes clusters
name: K8s Production
resourceSelector: |
  resource.kind == "KubernetesCluster" &&
  resource.metadata["env"] == "production"

# Only Lambda functions
name: Lambda Production
resourceSelector: |
  resource.kind == "AWS/Lambda" &&
  resource.metadata["env"] == "production"
```

### By Tier

```yaml theme={null}
# Critical infrastructure only
name: Critical Production
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["tier"] == "critical"
```

### Compound Selectors

```yaml theme={null}
# Critical K8s clusters in US production
name: Critical US K8s
resourceSelector: |
  resource.kind == "KubernetesCluster" &&
  resource.metadata["env"] == "production" &&
  resource.metadata["region"].startsWith("us-") &&
  resource.metadata["tier"] == "critical"
```

## Variations

### Hierarchical Environments

```yaml theme={null}
# Global production
name: Production
resourceSelector: resource.metadata["env"] == "production"

# Regional sub-environments (inherit from Production)
name: Production/US
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["region"].startsWith("us-")
directory: Production

name: Production/EU  
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["region"].startsWith("eu-")
directory: Production
```

### Canary Environment

```yaml theme={null}
# Canary gets deploys first
name: Canary
resourceSelector: resource.metadata["canary"] == "true"

# Production excludes canary
name: Production
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["canary"] != "true"
```

### Deployment-Specific Targeting

Deployments can also filter which resources they target:

```yaml theme={null}
type: Deployment
name: api-gateway
# Only deploy to clusters with api workloads
resourceSelector: resource.metadata["workloads"].contains("api")
```

Combined with environment:

```
Release Target = Deployment Filter ∩ Environment Filter
```

## Common Metadata Schema

Standardize your metadata for consistent environment targeting:

```yaml theme={null}
metadata:
  # Required for environment membership
  env: production | staging | development
  
  # Geographic
  region: us-east-1 | eu-west-1 | ap-south-1
  cloud: aws | gcp | azure | onprem
  
  # Organizational
  team: platform | backend | data
  cost-center: engineering | infrastructure
  
  # Operational
  tier: critical | standard | experimental
  canary: "true" | "false"
  
  # Workload hints
  workloads: ["api", "worker", "cron"]
```

## Next Steps

<CardGroup>
  <Card title="Selectors" icon="filter" href="../concepts/selectors">
    Full selector syntax reference
  </Card>

  <Card title="Environments" icon="layer-group" href="../concepts/environments">
    Environment configuration details
  </Card>

  <Card title="Infrastructure Inventory" icon="server" href="./infrastructure-inventory">
    Sync and manage resources
  </Card>

  <Card title="Multi-Region" icon="globe" href="./multi-region">
    Deploy across dynamic environments
  </Card>
</CardGroup>


# Environment Promotion
Source: https://docs.ctrlplane.dev/use-cases/environment-promotion

Automatically promote releases from staging to production

## The Scenario

You have a staging environment and a production environment. You want:

* Every version to deploy to staging first
* Production to only receive versions that passed staging
* Optional approval before production deploys
* No manual "okay, staging looks good, now deploy to prod" steps

## Without Ctrlplane

**The typical flow:**

1. CI deploys to staging
2. Someone checks if staging is healthy (maybe)
3. Someone remembers to trigger the prod deploy
4. Hopefully they deploy the right version
5. Hopefully staging actually passed

**What goes wrong:**

* "Staging passed" is based on vibes, not metrics
* The manual promotion step gets forgotten
* Wrong version deployed to prod
* No audit trail of what was verified

## With Ctrlplane

Define your environments:

```yaml theme={null}
type: Environment
name: Staging
resourceSelector: resource.metadata["env"] == "staging"
---
type: Environment
name: Production
resourceSelector: resource.metadata["env"] == "production"
```

Add an environment progression policy:

```yaml theme={null}
type: Policy
name: Staging Before Production
selectors:
  - environment: environment.name == "Production"
rules:
  - environmentProgression:
      waitFor: Staging
```

Add verification to staging:

```yaml theme={null}
type: Policy
name: Staging Verification
selectors:
  - environment: environment.name == "Staging"
rules:
  - verification:
      metrics:
        - name: health-check
          provider:
            type: http
            url: "https://{{.resource.config.host}}/health"
          successCondition: result.statusCode == 200
          intervalSeconds: 30
          count: 10  # 5 minutes of health checks
```

Add approval for production:

```yaml theme={null}
type: Policy
name: Production Approval
selectors:
  - environment: environment.name == "Production"
rules:
  - anyApproval:
      minApprovals: 1
```

## What Happens

```mermaid theme={null}
flowchart LR
    V["v1.2.3"] --> S["Staging"]
    S --> SV["Verify 5min"]
    SV -->|"pass"| PA["Approval"]
    PA -->|"approved"| P["Production"]
    P --> PV["Verify"]
    
    SV -->|"fail"| Block["Blocked"]
```

1. **CI creates v1.2.3** — Version is marked ready
2. **Staging release created** — Deploys immediately (no gates)
3. **Staging verification runs** — 5 minutes of health checks
4. **Staging passes** → **Production is unblocked**
5. **Approval requested** — Notification sent to approvers
6. **Approval granted** — Team lead approves
7. **Production deploys** — Version goes live
8. **Production verification** — Confirms health in prod

If staging verification **fails**, production never receives the version. No manual intervention needed to block it.

## Key Benefits

| Benefit                | How It Works                                       |
| ---------------------- | -------------------------------------------------- |
| **Enforced ordering**  | Production physically cannot deploy before staging |
| **Verified promotion** | Only versions that pass verification can promote   |
| **Audit trail**        | Full history of what was verified and when         |
| **Optional approval**  | Add human gates where needed                       |
| **Automatic flow**     | No manual "deploy to prod" step                    |

## Variations

### Multiple Pre-Production Environments

```yaml theme={null}
# QA → Staging → Production
- name: Staging After QA
  selectors:
    - environment: environment.name == "Staging"
  rules:
    - environmentProgression:
        waitFor: QA

- name: Production After Staging  
  selectors:
    - environment: environment.name == "Production"
  rules:
    - environmentProgression:
        waitFor: Staging
```

### Different Verification per Environment

```yaml theme={null}
# Light verification in QA
- name: QA Verification
  selectors:
    - environment: environment.name == "QA"
  rules:
    - verification:
        metrics:
          - name: smoke-test
            count: 3  # Quick check

# Thorough verification in Staging
- name: Staging Verification
  selectors:
    - environment: environment.name == "Staging"
  rules:
    - verification:
        metrics:
          - name: integration-tests
            count: 10  # Longer check
          - name: performance-baseline
            count: 5
```

### Skip Staging for Hotfixes

Use version metadata to bypass staging for critical fixes:

```yaml theme={null}
- name: Hotfix Direct to Prod
  selectors:
    - environment: environment.name == "Production"
    - version: version.metadata["hotfix"] == "true"
  rules:
    - anyApproval:
        minApprovals: 2  # Require more approvals for hotfixes
    # No environment progression rule = skip staging
```

## Next Steps

<CardGroup>
  <Card title="Environment Progression" icon="arrow-right" href="../policies/environment-progression">
    Configure progression rules
  </Card>

  <Card title="Verification" icon="check" href="../policies/verification/overview">
    Set up health checks
  </Card>

  <Card title="Approvals" icon="user-check" href="../policies/approval">
    Configure approval workflows
  </Card>

  <Card title="Multi-Region" icon="globe" href="./multi-region">
    Add gradual rollouts within production
  </Card>
</CardGroup>


# Infrastructure Inventory
Source: https://docs.ctrlplane.dev/use-cases/infrastructure-inventory

A single source of truth for what exists and what's running

## The Scenario

You need to answer questions like:

* "What version of the API is running in eu-west-1 prod?"
* "How many clusters do we have? What's deployed to each?"
* "Which services are running on the new cluster we added last week?"
* "Show me all resources owned by the platform team"

## Without Ctrlplane

**The typical approach:**

1. Check Kubernetes dashboard (but which cluster?)
2. Check AWS console (but which region?)
3. Check the team's spreadsheet (but is it up to date?)
4. Ask in Slack (but who knows?)

**What goes wrong:**

* No single source of truth
* Information is scattered across tools
* Spreadsheets become stale immediately
* "What version is running?" requires multiple lookups

## With Ctrlplane

### Sync Resources Automatically

Resources are synced from your infrastructure via resource providers:

```yaml theme={null}
# Kubernetes provider syncs all clusters
type: ResourceProvider
name: kubernetes-clusters
provider:
  type: kubernetes
  config:
    kubeconfig: "{{.secrets.KUBECONFIG}}"
sync:
  interval: 5m
```

Or register resources via API:

```bash theme={null}
curl -X POST "https://api.ctrlplane.dev/resources" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "name": "prod-us-east-1",
    "kind": "KubernetesCluster",
    "identifier": "k8s-prod-use1",
    "metadata": {
      "env": "production",
      "region": "us-east-1",
      "team": "platform",
      "tier": "critical"
    },
    "config": {
      "server": "https://k8s-prod-use1.example.com",
      "namespace": "default"
    }
  }'
```

### Query Your Infrastructure

Once synced, query resources with selectors:

```yaml theme={null}
# All production clusters
resource.metadata["env"] == "production"

# Critical services in us-east
resource.metadata["tier"] == "critical" && 
resource.metadata["region"] == "us-east-1"

# Everything owned by platform team
resource.metadata["team"] == "platform"
```

### See What's Deployed

Each resource shows its current deployed versions:

```
prod-us-east-1 (KubernetesCluster)
├── api-gateway: v1.2.3 (deployed 2h ago)
├── user-service: v2.0.1 (deployed 1d ago)
└── payment-service: v3.1.0 (deployed 3h ago)

prod-eu-west-1 (KubernetesCluster)
├── api-gateway: v1.2.2 (deployed 1d ago)  ← behind!
├── user-service: v2.0.1 (deployed 1d ago)
└── payment-service: v3.1.0 (deployed 3h ago)
```

## What You Get

```mermaid theme={null}
flowchart TB
    subgraph Sources["Infrastructure Sources"]
        K8s["Kubernetes"]
        AWS["AWS"]
        GCP["GCP"]
        Custom["Custom"]
    end

    subgraph Inventory["Ctrlplane Inventory"]
        Resources["Resources"]
        Metadata["Metadata"]
        Versions["Current Versions"]
        Relations["Relationships"]
    end

    subgraph Views["Query & View"]
        Env["By Environment"]
        Team["By Team"]
        Region["By Region"]
        Service["By Service"]
    end

    Sources -->|"sync"| Inventory
    Inventory -->|"query"| Views
```

## Key Benefits

| Benefit                    | How It Works                             |
| -------------------------- | ---------------------------------------- |
| **Single source of truth** | All resources in one place               |
| **Real-time sync**         | Providers keep inventory up to date      |
| **Rich metadata**          | Tag resources with any attributes        |
| **Version tracking**       | See what's deployed where                |
| **Cross-provider**         | Kubernetes, AWS, GCP, custom—all unified |

## Resource Metadata

Tag resources with metadata for powerful querying:

```yaml theme={null}
metadata:
  # Environment classification
  env: production
  
  # Geographic location
  region: us-east-1
  cloud: aws
  
  # Ownership
  team: platform
  cost-center: engineering
  
  # Operational
  tier: critical
  on-call: platform-oncall@company.com
  
  # Custom
  compliance: soc2
  data-classification: pii
```

## Dynamic Environments

Environments use selectors to automatically group resources:

```yaml theme={null}
type: Environment
name: Production US
resourceSelector: |
  resource.metadata["env"] == "production" &&
  resource.metadata["region"].startsWith("us-")
```

When you add a new cluster with matching metadata, it automatically joins the environment:

```yaml theme={null}
# New cluster added
type: Resource
name: prod-us-west-2
metadata:
  env: production      # ← matches "production"
  region: us-west-2    # ← starts with "us-"
# Automatically part of "Production US" environment!
```

## Resource Relationships

Model dependencies between resources:

```yaml theme={null}
type: Resource
name: api-gateway
relationships:
  - type: depends-on
    target: user-database
  - type: depends-on
    target: cache-cluster
```

Query relationships:

```yaml theme={null}
# Find all resources that api-gateway depends on
resource.relationships.any(r, r.type == "depends-on")
```

## Variations

### Multi-Cloud Inventory

```yaml theme={null}
# AWS resources
- type: ResourceProvider
  name: aws-resources
  provider:
    type: aws
    config:
      region: us-east-1

# GCP resources  
- type: ResourceProvider
  name: gcp-resources
  provider:
    type: gcp
    config:
      project: my-project

# On-prem Kubernetes
- type: ResourceProvider
  name: onprem-k8s
  provider:
    type: kubernetes
    config:
      kubeconfig: "{{.secrets.ONPREM_KUBECONFIG}}"
```

### Custom Resource Provider

Sync from any source using HTTP:

```yaml theme={null}
type: ResourceProvider
name: custom-inventory
provider:
  type: http
  config:
    url: "https://internal-api/resources"
    method: GET
    headers:
      Authorization: "Bearer {{.secrets.INTERNAL_TOKEN}}"
sync:
  interval: 10m
```

### Version Tracking Queries

Find resources running old versions:

```yaml theme={null}
# Resources where api-gateway is behind latest
resource.deployments["api-gateway"].version != "v1.2.3"

# Resources not updated in 7 days
resource.deployments["api-gateway"].deployedAt < now() - 7d
```

## Next Steps

<CardGroup>
  <Card title="Resources" icon="server" href="../concepts/resources">
    Deep dive into resource configuration
  </Card>

  <Card title="Resource Providers" icon="rotate" href="../integrations/resource-providers">
    Set up automatic syncing
  </Card>

  <Card title="Selectors" icon="filter" href="../concepts/selectors">
    Learn the selector query language
  </Card>

  <Card title="Dynamic Environments" icon="layer-group" href="./dynamic-environments">
    Group resources automatically
  </Card>
</CardGroup>


# Multi-Region Deployments
Source: https://docs.ctrlplane.dev/use-cases/multi-region

Deploy to multiple clusters and regions with confidence

## The Scenario

You have 8 Kubernetes clusters across 3 regions (us-east, us-west, eu-west). When you release a new version, you need to:

* Deploy to all 8 clusters
* Not break everything at once
* Verify each cluster before moving to the next
* Roll back if something goes wrong

## Without Ctrlplane

**Option A: Deploy to all at once**

* Fast, but risky
* One bad deploy breaks all regions simultaneously
* Rollback is chaotic

**Option B: Manual sequential deploys**

* Safe, but slow
* Someone has to babysit each deployment
* Easy to forget a cluster
* Inconsistent timing between regions

**Option C: Complex CI/CD matrix**

* Build intricate pipeline logic
* Hard to maintain
* Verification is bolted on, not built in

## With Ctrlplane

Define your resources with region metadata:

```yaml theme={null}
# Resources are synced from your infrastructure
type: Resource
name: prod-us-east-1
metadata:
  env: production
  region: us-east-1
---
type: Resource
name: prod-us-west-2
metadata:
  env: production
  region: us-west-2
---
type: Resource
name: prod-eu-west-1
metadata:
  env: production
  region: eu-west-1
```

Create an environment that matches all production clusters:

```yaml theme={null}
type: Environment
name: Production
resourceSelector: resource.metadata["env"] == "production"
```

Add a gradual rollout policy:

```yaml theme={null}
type: Policy
name: Production Gradual Rollout
selectors:
  - environment: environment.name == "Production"
rules:
  - gradualRollout:
      rolloutType: linear
      timeScaleInterval: 600  # 10 minutes between clusters
  - verification:
      metrics:
        - name: error-rate
          provider:
            type: datadog
            query: "sum:errors{env:prod,cluster:{{.resource.name}}}"
          successCondition: result.value < 0.01
          intervalSeconds: 60
          count: 5
```

## What Happens

```mermaid theme={null}
flowchart LR
    V["v1.2.3"] --> C1["us-east-1"]
    C1 -->|"verify"| W1["wait 10m"]
    W1 --> C2["us-west-2"]
    C2 -->|"verify"| W2["wait 10m"]
    W2 --> C3["eu-west-1"]
    C3 -->|"verify"| Done["Done"]
```

1. **Version created** — CI builds v1.2.3 and tells Ctrlplane
2. **First cluster deploys** — us-east-1 receives the deployment
3. **Verification runs** — Datadog metrics are checked for 5 minutes
4. **Wait interval** — 10 minutes pass before next cluster
5. **Next cluster deploys** — us-west-2 receives the deployment
6. **Repeat** — Continue until all clusters are updated
7. **Auto-rollback** — If any verification fails, roll back that cluster

## Key Benefits

| Benefit                   | How It Works                                    |
| ------------------------- | ----------------------------------------------- |
| **Automatic sequencing**  | No manual intervention between clusters         |
| **Built-in verification** | Each cluster is verified before proceeding      |
| **Configurable timing**   | Control how fast the rollout proceeds           |
| **Auto-rollback**         | Failed verification triggers immediate rollback |
| **Full visibility**       | See progress across all clusters in real-time   |

## Variations

### By Region First

Deploy to all clusters in us-east first, then us-west, then eu-west:

```yaml theme={null}
rules:
  - gradualRollout:
      rolloutType: linear
      sortBy:
        - resource.metadata["region"]
```

### Canary Then Full

Deploy to 1 cluster first, verify for an hour, then deploy to the rest:

```yaml theme={null}
# First policy: canary cluster
- name: Canary
  selectors:
    - resource: resource.metadata["canary"] == "true"
  rules:
    - verification:
        # Extended verification for canary
        metrics:
          - name: error-rate
            intervalSeconds: 300
            count: 12  # 1 hour of checks

# Second policy: remaining clusters wait for canary
- name: Post-Canary Rollout
  selectors:
    - resource: resource.metadata["canary"] != "true"
  rules:
    - deploymentDependency:
        waitFor: canary-cluster
    - gradualRollout:
        rolloutType: linear
```

## Next Steps

<CardGroup>
  <Card title="Gradual Rollouts" icon="chart-line" href="../policies/gradual-rollouts">
    Configure rollout timing and ordering
  </Card>

  <Card title="Verification" icon="check" href="../policies/verification/overview">
    Set up health checks with Datadog, Prometheus, HTTP
  </Card>
</CardGroup>


# Why Ctrlplane?
Source: https://docs.ctrlplane.dev/why-ctrlplane

The deployment problems you're already dealing with—and how to solve them

## Sound Familiar?

<AccordionGroup>
  <Accordion title="What version is running in prod-eu-west-1?">
    **The pain**: You check Kubernetes, then AWS console, then your team's spreadsheet. 15 minutes later, you're still not 100% sure.

    **With Ctrlplane**: Single inventory showing every resource and its current version, updated in real-time.
  </Accordion>

  <Accordion title="Can someone approve this deploy?">
    **The pain**: Slack message sent. Thread dies. Deploy sits for hours. You ping again. Finally someone approves at 6pm.

    **With Ctrlplane**: Built-in approval workflows with notifications. Approvers see pending releases in one place.
  </Accordion>

  <Accordion title="Staging passed—now deploy to prod">
    **The pain**: Manual step someone forgets. Or worse, staging didn't actually pass but prod got deployed anyway.

    **With Ctrlplane**: Auto-promote to production only when staging verification succeeds. No manual intervention needed.
  </Accordion>

  <Accordion title="We added a new cluster">
    **The pain**: Update 6 config files, 3 CI pipelines, and remember to tell the team. Miss one and deploys fail silently.

    **With Ctrlplane**: New clusters auto-join environments via selectors. Tag it `env: production` and it starts receiving deployments.
  </Accordion>

  <Accordion title="Rollback NOW!">
    **The pain**: Scramble to find the right commands. Which version was stable? Did we roll back all regions? Did we miss one?

    **With Ctrlplane**: Automatic rollback when verification fails. One-click manual rollback when needed.
  </Accordion>

  <Accordion title="Deploy to 10 regions without breaking everything">
    **The pain**: Either deploy to all at once (risky) or manually babysit each region (slow and error-prone).

    **With Ctrlplane**: Gradual rollouts deploy one region at a time, verify health, then continue—automatically.
  </Accordion>
</AccordionGroup>

## The Problems at Scale

When you have 5 services and 2 environments, everything is manageable. When you hit 20+ services across 4 environments and 8 clusters, you start feeling the pain:

| Challenge                     | What Teams Do Today                           | What Goes Wrong                                      |
| ----------------------------- | --------------------------------------------- | ---------------------------------------------------- |
| **Environment promotion**     | Manual deploy after checking staging          | Someone forgets, or staging wasn't actually verified |
| **Deployment verification**   | "It deployed successfully!"                   | Service is broken, but pipeline says green           |
| **Infrastructure visibility** | Spreadsheets + multiple dashboards            | Outdated info, no source of truth                    |
| **Adding new infrastructure** | Update configs everywhere                     | Miss something, deploys fail                         |
| **Multi-region deploys**      | Sequential manual deploys or YOLO all-at-once | Too slow or too risky                                |
| **Rollbacks**                 | Ad-hoc scripts and hope                       | Inconsistent, slow, error-prone                      |

## Two Core Systems

<CardGroup>
  <Card title="Deployment Orchestration" icon="rocket">
    Controls **when** and **where** releases happen:

    * Auto-promote after verification
    * Approval gates for production
    * Gradual rollouts across regions
    * Automatic rollback on failure
  </Card>

  <Card title="Infrastructure Inventory" icon="server">
    Tracks **what exists** and **what's running**:

    * Real-time resource inventory
    * Dynamic environment membership
    * Version tracking across all targets
    * Works with K8s, AWS, GCP, custom
  </Card>
</CardGroup>

## What Ctrlplane Is NOT

Ctrlplane doesn't replace your existing tools—it coordinates them:

* **Not a CI system** — Your CI (GitHub Actions, GitLab, Jenkins) still builds code
* **Not a GitOps engine** — ArgoCD/Flux still syncs manifests to clusters
* **Not infrastructure provisioning** — Terraform still creates your resources

Ctrlplane decides *when* deployments should happen, *where* they should go, and *whether* they passed verification.

## Next Steps

<CardGroup>
  <Card title="5-Minute Overview" icon="clock" href="./overview">
    Understand the mental model
  </Card>

  <Card title="Quickstart" icon="rocket" href="./quickstart">
    Set up your first deployment pipeline
  </Card>

  <Card title="Use Cases" icon="lightbulb" href="./use-cases/multi-region">
    See specific scenarios solved
  </Card>

  <Card title="Comparisons" icon="scale-balanced" href="./comparisons">
    How Ctrlplane compares to other tools
  </Card>
</CardGroup>


# Workflows
Source: https://docs.ctrlplane.dev/workflows

Run ad-hoc operations across your infrastructure using parameterized, multi-agent workflows

Workflows let you run on-demand operations — restarts, migrations, backfills, data exports — across one or more job agents. Unlike deployments, workflows are not tied to versions or release targets; they accept typed inputs at runtime and fan out jobs to whichever agents match the conditions you specify.

```
User triggers run
  → Inputs resolved (provided + defaults)
  → For each job agent whose selector matches:
      → Job created with dispatch context
      → Job agent executes with access to inputs
      → Status tracked
```

## Core Concepts

| Concept          | Description                                                                              |
| ---------------- | ---------------------------------------------------------------------------------------- |
| **Workflow**     | A named template with input definitions and a list of job agents                         |
| **Input**        | A typed parameter declared on the workflow (string, number, boolean, object, or array)   |
| **Job Agent**    | An executor (GitHub Actions, ArgoCD, Terraform Cloud, etc.) referenced by ID             |
| **Selector**     | A CEL expression evaluated against the dispatch context to decide whether the agent runs |
| **Workflow Run** | A single execution of a workflow with a specific set of input values                     |
| **Workflow Job** | A job dispatched to one agent during a run                                               |

## Specifying Inputs

Inputs are declared in the `inputs` array of a workflow. Each input has a `key`, a `type`, and an optional `default`.

### Scalar inputs

```json theme={null}
{
  "inputs": [
    { "key": "environment", "type": "string", "default": "staging" },
    { "key": "replicas",    "type": "number", "default": 2 },
    { "key": "dry_run",     "type": "boolean", "default": false },
    { "key": "config",      "type": "object",  "default": { "timeout": 30 } }
  ]
}
```

| Type      | JSON type | Notes                 |
| --------- | --------- | --------------------- |
| `string`  | `string`  |                       |
| `number`  | `number`  | Integer or float      |
| `boolean` | `boolean` |                       |
| `object`  | `object`  | Arbitrary JSON object |
| `array`   | `array`   | See below             |

### Array inputs

There are two kinds of array inputs:

**Manual array** — the caller provides a list of items directly:

```json theme={null}
{ "key": "hosts", "type": "array" }
```

**Selector array** — dynamically resolved from your inventory using a CEL expression. Instead of the caller listing items, Ctrlplane queries entities matching the selector:

```json theme={null}
{
  "key": "targets",
  "type": "array",
  "selector": {
    "entityType": "resource",
    "default": "resource.metadata['environment'] == 'staging'"
  }
}
```

`entityType` can be `resource`, `environment`, or `deployment`. The `default` is a CEL expression used when the caller does not override the selector.

### Input resolution

When a run is created, Ctrlplane merges the caller's provided values with the workflow's defaults:

1. Start with the values provided by the caller.
2. For each input defined on the workflow, if no value was provided, apply the `default` (if set).
3. The final merged map is stored on the `WorkflowRun` record and passed to every dispatched job.

If the caller omits an input and no default is defined, that key will not appear in the resolved inputs.

## Wiring in Job Agents

The `jobAgents` array defines which executors the workflow uses. Each entry has:

```json theme={null}
{
  "jobAgents": [
    {
      "name": "my-github-agent",
      "ref": "<job-agent-id>",
      "config": {
        "installationId": "12345678",
        "owner": "my-org",
        "repo": "my-repo",
        "workflowId": "deploy.yml"
      },
      "selector": "true"
    }
  ]
}
```

| Field      | Required | Description                                                                        |
| ---------- | -------- | ---------------------------------------------------------------------------------- |
| `name`     | Yes      | Display name for the agent within this workflow                                    |
| `ref`      | Yes      | The UUID of a registered job agent                                                 |
| `config`   | Yes      | Agent-specific configuration (see [Templates](#templating-inputs-into-job-agents)) |
| `selector` | Yes      | CEL expression — the agent only runs if this evaluates to `true`                   |

### Selector evaluation

Before dispatching, Ctrlplane evaluates each agent's `selector` against the **dispatch context**:

```
{
  "workflow": { ... },    // workflow definition
  "inputs":   { ... }     // resolved input values
}
```

Use `"true"` to always dispatch to this agent, or write a CEL expression to make it conditional:

```json theme={null}
{ "selector": "inputs.environment == 'production'" }
```

This lets a single workflow definition fan out differently depending on what the caller provides — e.g., only trigger a manual approval agent for production.

Multiple agents can run in the same workflow run: every agent whose selector evaluates to `true` receives its own job.

## Templating Inputs into Job Agents

Both `config` values and the `selector` field support **Go `text/template`** syntax. Templates are rendered before dispatch using the same dispatch context:

| Variable              | Description                                                  |
| --------------------- | ------------------------------------------------------------ |
| `{{.inputs.<key>}}`   | Resolved input values                                        |
| `{{.workflow}}`       | The workflow definition                                      |
| `{{.jobAgentConfig}}` | This agent's rendered config (available in nested templates) |

### Example: GitHub Actions

```json theme={null}
{
  "name": "github-deploy",
  "ref": "<github-agent-id>",
  "config": {
    "installationId": "12345678",
    "owner": "my-org",
    "repo": "{{.inputs.repo}}",
    "workflowId": "workflow-dispatch.yml",
    "ref": "{{.inputs.branch}}"
  },
  "selector": "true"
}
```

When the run is created with `{ "repo": "api-service", "branch": "main" }`, the rendered config becomes:

```json theme={null}
{
  "installationId": "12345678",
  "owner": "my-org",
  "repo": "api-service",
  "workflowId": "workflow-dispatch.yml",
  "ref": "main"
}
```

This rendered config is stored on the `Job` record and passed to the job agent dispatcher. The agent then uses it to construct the actual dispatch call (e.g., the GitHub API `workflow_dispatch` event).

### Conditional agent selection

```json theme={null}
{
  "name": "prod-approvals",
  "ref": "<approval-agent-id>",
  "config": {},
  "selector": "inputs.environment == 'production'"
}
```

The approval agent only runs when `environment` is `"production"`.

## Full Example: Database Migration Workflow

This workflow runs a database migration via GitHub Actions, with a dry-run option and environment targeting.

### Workflow definition (API)

```json theme={null}
{
  "name": "Run Database Migration",
  "inputs": [
    { "key": "environment",  "type": "string",  "default": "staging" },
    { "key": "migration_id", "type": "string" },
    { "key": "dry_run",      "type": "boolean", "default": true }
  ],
  "jobAgents": [
    {
      "name": "migration-runner",
      "ref": "<github-agent-id>",
      "config": {
        "installationId": "12345678",
        "owner": "my-org",
        "repo": "my-app",
        "workflowId": "migrate.yml",
        "ref": "main"
      },
      "selector": "true"
    }
  ]
}
```

### GitHub Actions workflow

The GitHub Actions workflow receives a `job_id` from Ctrlplane. Use the `ctrlplanedev/get-job-inputs` action to fetch the resolved inputs:

```yaml theme={null}
# .github/workflows/migrate.yml
name: Database Migration

on:
  workflow_dispatch:
    inputs:
      job_id:
        description: "Ctrlplane Job ID"
        required: true

jobs:
  migrate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Fetch workflow inputs
        uses: ctrlplanedev/get-job-inputs@v1
        id: job
        with:
          base_url: ${{ secrets.CTRLPLANE_BASE_URL }}
          job_id: ${{ inputs.job_id }}
          api_key: ${{ secrets.CTRLPLANE_API_KEY }}

      - name: Run migration
        env:
          DB_URL: ${{ secrets[format('DB_URL_{0}', steps.job.outputs.inputs_environment)] }}
        run: |
          echo "Environment: ${{ steps.job.outputs.inputs_environment }}"
          echo "Migration:   ${{ steps.job.outputs.inputs_migration_id }}"
          echo "Dry run:     ${{ steps.job.outputs.inputs_dry_run }}"

          if [ "${{ steps.job.outputs.inputs_dry_run }}" = "true" ]; then
            make migrate-dry-run ID=${{ steps.job.outputs.inputs_migration_id }}
          else
            make migrate ID=${{ steps.job.outputs.inputs_migration_id }}
          fi
```

<Note>
  Workflow inputs are surfaced on the job as `inputs_<key>` outputs by `get-job-inputs`.
</Note>

### Triggering a run (API)

```bash theme={null}
curl -X POST "https://ctrlplane.example.com/api/v1/workspaces/{workspaceId}/workflows/{workflowId}/runs" \
  -H "Authorization: Bearer $CTRLPLANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": {
      "environment": "production",
      "migration_id": "20240115_add_user_index",
      "dry_run": false
    }
  }'
```

If `dry_run` were omitted, Ctrlplane would fall back to the default `true`.

## Example: Multi-Agent Workflow (GitHub + Terraform Cloud)

A single workflow can dispatch to multiple agents simultaneously. Here, a restart workflow notifies Slack via GitHub Actions and triggers a Terraform Cloud run to scale down and back up:

```json theme={null}
{
  "name": "Rolling Restart",
  "inputs": [
    { "key": "service",     "type": "string" },
    { "key": "environment", "type": "string", "default": "staging" }
  ],
  "jobAgents": [
    {
      "name": "notify",
      "ref": "<github-agent-id>",
      "config": {
        "installationId": "12345678",
        "owner": "my-org",
        "repo": "ops-tooling",
        "workflowId": "notify-restart.yml",
        "ref": "main"
      },
      "selector": "true"
    },
    {
      "name": "terraform-restart",
      "ref": "<tfe-agent-id>",
      "config": {
        "organization": "my-org",
        "address": "https://app.terraform.io",
        "token": "<tfe-token>",
        "webhookUrl": "https://ctrlplane.example.com/api/tfe/webhook",
        "template": "name: restart-{{.inputs.service}}-{{.inputs.environment}}\nauto_apply: true\n"
      },
      "selector": "inputs.environment == 'production'"
    }
  ]
}
```

The `notify` agent runs for every environment. The `terraform-restart` agent only runs when targeting production.

## Terraform Provider

Use the `ctrlplane` Terraform provider to manage workflows as code.

### Create a workflow

```hcl theme={null}
resource "ctrlplane_workflow" "db_migration" {
  name = "Run Database Migration"

  input {
    key     = "environment"
    type    = "string"
    default = "staging"
  }

  input {
    key  = "migration_id"
    type = "string"
  }

  input {
    key     = "dry_run"
    type    = "boolean"
    default = "true"
  }

  job_agent {
    name     = "migration-runner"
    ref      = ctrlplane_job_agent.github.id
    selector = "true"

    config = {
      installationId = var.github_installation_id
      owner          = "my-org"
      repo           = "my-app"
      workflowId     = "migrate.yml"
      ref            = "main"
    }
  }
}
```

### Reference other resources

```hcl theme={null}
resource "ctrlplane_job_agent" "github" {
  name = "github-actions"

  github_app {
    installation_id = var.github_installation_id
    owner           = "my-org"
  }
}

resource "ctrlplane_workflow" "rolling_restart" {
  name = "Rolling Restart"

  input {
    key  = "service"
    type = "string"
  }

  input {
    key     = "environment"
    type    = "string"
    default = "staging"
  }

  job_agent {
    name     = "notify"
    ref      = ctrlplane_job_agent.github.id
    selector = "true"

    config = {
      installationId = var.github_installation_id
      owner          = "my-org"
      repo           = "ops-tooling"
      workflowId     = "notify-restart.yml"
      ref            = "main"
    }
  }

  job_agent {
    name     = "terraform-restart"
    ref      = ctrlplane_job_agent.tfc.id
    selector = "inputs.environment == 'production'"

    config = {
      organization = var.tfc_org
      address      = "https://app.terraform.io"
      token        = var.tfc_token
      webhookUrl   = "https://ctrlplane.example.com/api/tfe/webhook"
      template     = <<-EOT
        name: restart-{{"{{"}.inputs.service{{"}}"}}-{{"{{"}.inputs.environment{{"}}"}}
        auto_apply: true
      EOT
    }
  }
}
```

## API Reference

| Method   | Endpoint                                                   | Description            |
| -------- | ---------------------------------------------------------- | ---------------------- |
| `GET`    | `/v1/workspaces/{workspaceId}/workflows`                   | List all workflows     |
| `POST`   | `/v1/workspaces/{workspaceId}/workflows`                   | Create a workflow      |
| `GET`    | `/v1/workspaces/{workspaceId}/workflows/{workflowId}`      | Get a workflow         |
| `PUT`    | `/v1/workspaces/{workspaceId}/workflows/{workflowId}`      | Update a workflow      |
| `DELETE` | `/v1/workspaces/{workspaceId}/workflows/{workflowId}`      | Delete a workflow      |
| `POST`   | `/v1/workspaces/{workspaceId}/workflows/{workflowId}/runs` | Trigger a workflow run |

### Create workflow run request body

```json theme={null}
{
  "inputs": {
    "environment": "production",
    "migration_id": "20240115_add_index",
    "dry_run": false
  }
}
```

Only `inputs` is required. Omit keys to use their defaults; keys with no default and no provided value will be absent from the dispatch context.

## How It Works Internally

```mermaid theme={null}
sequenceDiagram
    participant U as Caller
    participant A as API
    participant E as Workspace Engine
    participant J as Job Agent

    U->>A: POST /workflows/{id}/runs { inputs }
    A->>E: Forward run request
    E->>E: Resolve inputs (provided + defaults)
    E->>E: For each jobAgent: evaluate CEL selector
    E->>E: For matching agents: create Job + enqueue dispatch
    E->>J: Dispatch job (rendered config + dispatch context)
    J->>J: Execute operation
    J-->>A: Status update (webhook or API)
    A-->>U: Job status visible in UI
```

1. **Input resolution** — user-provided values are merged with defaults declared on the workflow.
2. **Selector evaluation** — each agent's `selector` CEL expression is evaluated against `{ workflow, inputs }`. Only agents whose selector returns `true` receive a job.
3. **Job creation** — a `Job` is inserted with the rendered `jobAgentConfig` and the full `DispatchContext` (which includes the resolved inputs).
4. **Job dispatch** — the workspace engine picks up the job and calls the agent-specific dispatcher (GitHub, TFC, ArgoCD, etc.).
5. **Status tracking** — agents report back via webhooks or the Ctrlplane API, updating job status in real time.

## Next Steps

<CardGroup>
  <Card title="GitHub Actions" icon="github" href="/integrations/job-agents/github">
    Wire up GitHub Actions as a job agent
  </Card>

  <Card title="Terraform Cloud" icon="cloud" href="/integrations/job-agents/terraform-cloud">
    Trigger Terraform Cloud runs from workflows
  </Card>

  <Card title="ArgoCD" icon="ship" href="/integrations/job-agents/argocd">
    Sync ArgoCD applications on demand
  </Card>

  <Card title="Argo Workflows" icon="diagram-project" href="/integrations/job-agents/argo-workflows">
    Execute Argo Workflow templates
  </Card>
</CardGroup>


# Domain Matching
Source: https://docs.ctrlplane.dev/workspaces/domain-matching

Automatically assign workspace roles to users based on their email domain.

**Domain matching** lets you configure rules that automatically assign roles to
users when they sign in with an email address matching a specified domain. This
simplifies onboarding for teams by removing the need to manually invite every
user.

## Overview

```mermaid theme={null}
flowchart TD
    A[User Signs In] --> B[Extract Email Domain]
    B --> C{Domain Rule Exists?}
    C -->|No| D[No Action]
    C -->|Yes| E[Assign Role in Workspace]
    E --> F[User Has Workspace Access]
```

When a user signs in, Ctrlplane extracts the domain portion of their email
address (e.g., `acme.com` from `alice@acme.com`) and checks it against all
configured domain matching rules. If a match is found, the user is automatically
assigned the specified role in the corresponding workspace.

## Why Use Domain Matching?

Domain matching helps you:

* **Automate onboarding** - New team members get workspace access instantly on
  first sign-in
* **Enforce consistency** - Everyone from the same domain gets the same baseline
  role
* **Reduce admin work** - No need to send individual invitations for every user
* **Support multiple workspaces** - A single domain can map to multiple
  workspaces with different roles

## Configuration

Domain matching is configured in **Workspace Settings > General**.

### Adding a Rule

To create a domain matching rule, provide:

| Field                  | Description                                             |
| ---------------------- | ------------------------------------------------------- |
| **Domain**             | The email domain to match (e.g., `acme.com`)            |
| **Role**               | The workspace role to assign to matching users          |
| **Verification Email** | An email address used to verify ownership of the domain |

<Steps>
  <Step title="Navigate to Settings">
    Go to your workspace settings by clicking the gear icon, then select
    **General**.
  </Step>

  <Step title="Find Domain Matching">
    Scroll to the **Domain Matching** card.
  </Step>

  <Step title="Add a Rule">
    Enter the domain (e.g., `acme.com`), select a role, and provide a
    verification email address. Click **Add**.
  </Step>
</Steps>

### Verification

Each domain matching rule includes a verification step to confirm ownership of
the domain. When a rule is created, a verification code is generated. The domain
owner must verify the rule before it is considered fully active.

### Deleting a Rule

To remove a domain matching rule, click the trash icon next to the rule in the
Domain Matching card. Users who were previously assigned via the rule will retain
their existing role assignments.

## How It Works

1. **User signs in** via any configured authentication provider (Google, email
   and password, or custom OAuth).
2. **Domain extraction** - The domain is extracted from the user's email address
   and converted to lowercase.
3. **Rule lookup** - Ctrlplane queries all domain matching rules for the
   extracted domain.
4. **Role assignment** - For each matching rule, the user is assigned the
   configured role in the corresponding workspace. If the user already has the
   role, no duplicate is created.

<Note>
  Domain matching runs on every sign-in, so if you add a new rule, existing
  users will be assigned the role the next time they sign in.
</Note>

## Constraints

* Each workspace can only have **one rule per domain**. You cannot create two
  rules for the same domain within the same workspace.
* Domain matching is **case-insensitive** - `Acme.com` and `acme.com` are
  treated as the same domain.
* Role assignments created by domain matching behave the same as manually
  assigned roles and can be removed individually if needed.

## Common Patterns

### Company-Wide Access

Give everyone at your company viewer access:

| Domain     | Role   |
| ---------- | ------ |
| `acme.com` | Viewer |

### Team-Based Roles

Assign different roles based on team domains:

| Domain           | Role   |
| ---------------- | ------ |
| `engineering.co` | Admin  |
| `contractors.co` | Viewer |

### Multi-Workspace Setup

The same domain can be configured across different workspaces with different
roles. For example, `acme.com` could map to an **Admin** role in the
`infrastructure` workspace and a **Viewer** role in the `production` workspace.

## Best Practices

* Use domain matching for broad organizational access and supplement with
  individual role assignments for elevated permissions.
* Verify domain rules promptly to ensure they are active.
* Review domain matching rules periodically as teams and organizational
  structures change.
* Combine domain matching with workspace-level RBAC for fine-grained access
  control.

## Next Steps

* [Concepts Overview](/concepts/overview) - Understand core Ctrlplane concepts
* [Quickstart](/quickstart) - Get started with Ctrlplane