> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ctrlplane.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# RFC 0006: Secret Provider Integration

| Category       | Status                            | Created    | Author        |
| -------------- | --------------------------------- | ---------- | ------------- |
| Infrastructure | <Badge color="gray">Draft</Badge> | 2026-03-06 | Justin Brooks |

## Summary

Add a pluggable secret provider system that lets workspaces connect to external
secret managers (Doppler, HashiCorp Vault, AWS Secrets Manager, etc.) and
reference secrets by path rather than storing sensitive values in ctrlplane. A
new workspace-scoped `secret_provider` entity holds encrypted provider
credentials. A `SecretReference` type describes a secret's location. The
workspace-engine resolves references at variable resolution or job dispatch time
by calling the external provider.

## Motivation

Ctrlplane currently stores sensitive credentials at two levels:

**Job agent configs** — When an operator creates an ArgoCD runner, the API token
is stored in the `job_agent.config` JSON column. The same applies to Terraform
Cloud tokens.

**Deployment variables** — The `SensitiveValue` type exists in the OpenAPI
schema and the variable resolver recognizes it as a distinct value type, but
resolution is explicitly rejected:

```go theme={null}
case "sensitive":
    return nil, fmt.Errorf("sensitive values are not resolved by the variable resolver")
```

The `release_variable.encrypted` column, `job_variable.sensitive` flag, and
`@ctrlplane/secrets` AES-256 service all exist but are not wired end-to-end. The
infrastructure for handling secrets was scaffolded but never completed.

### Why this matters

1. **Compliance** — SOC 2, ISO 27001, and similar frameworks require that
   secrets are encrypted at rest and access-controlled.

2. **Secret rotation** — When credentials are stored directly in ctrlplane,
   rotation requires updating every agent config and variable value that
   references the credential. With an external provider, rotation happens in
   Doppler/Vault/AWS and ctrlplane picks up the new value on the next
   resolution.

3. **Separation of concerns** — Platform teams manage secrets in their existing
   secret management infrastructure. Application teams reference secrets by name
   in ctrlplane without needing access to the actual values.

4. **Multi-workspace** — Each workspace may use a different secret provider or
   account. The provider credentials themselves need per-workspace encrypted
   storage.

### Existing mechanisms and their limitations

**`@ctrlplane/secrets` AES-256** — A TypeScript encryption service exists in
`packages/secrets/` but is not imported anywhere in the application code. It
provides encrypt/decrypt with a 256-bit key from `VARIABLES_AES_256_KEY`. This
could handle encryption at rest but does not solve external provider
integration.

**`SensitiveValue` type** — Defined in the OpenAPI schema with a `valueHash`
field. The variable resolver detects it but refuses to resolve it, returning an
error. The intent was that a separate decryption path would handle these values,
but that path was never built.

**Go template interpolation** — The ArgoCD agent config already supports Go
templates, and the docs show `apiKey: "{{.variables.argocd_token}}"`. This means
credentials can technically flow through the variable system into agent configs
at dispatch time. The missing piece is a way to resolve variables whose values
come from an external source.

## Proposal

### New entity: `secret_provider`

A workspace-scoped entity that holds the credentials for connecting to an
external secret management service:

```sql theme={null}
CREATE TABLE secret_provider (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    config BYTEA NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(workspace_id, name)
);
```

| Column   | Description                                                                  |
| -------- | ---------------------------------------------------------------------------- |
| `type`   | Provider type: `doppler`, `vault`, `aws-secretsmanager`, `kubernetes`, `env` |
| `config` | AES-256 encrypted JSON blob with provider-specific credentials               |
| `name`   | Human-readable name, unique per workspace                                    |

The `config` column uses `BYTEA` for the encrypted payload rather than `JSON`,
since the ciphertext is opaque binary. The workspace-engine decrypts it in
memory using the instance-level `VARIABLES_AES_256_KEY` when a secret resolution
is needed.

Example configs (before encryption):

```json theme={null}
// Doppler
{ "serviceToken": "dp.st.xxxxxxxxxxxx" }

// Vault
{ "address": "https://vault.internal.company.com", "authMethod": "kubernetes", "role": "ctrlplane" }

// AWS Secrets Manager
{ "region": "us-east-1", "accessKeyId": "AKIA...", "secretAccessKey": "..." }

// Kubernetes
{ "namespace": "default" }

// Environment variables (instance-level, no config needed)
{}
```

### `SecretReference` type

A value object that describes where a secret lives:

```go theme={null}
type SecretReference struct {
    Provider string `json:"provider"`
    Path     string `json:"path,omitempty"`
    Key      string `json:"key"`
}
```

| Field      | Description                                                                                        |
| ---------- | -------------------------------------------------------------------------------------------------- |
| `Provider` | Matches `secret_provider.name` within the workspace                                                |
| `Path`     | Provider-specific path (e.g., `my-project/production` for Doppler, `secret/data/argocd` for Vault) |
| `Key`      | The specific secret key within the path                                                            |

The `Provider` field matches by `name`, not by `type`. This allows a workspace
to have multiple providers of the same type (e.g., two Doppler connections for
different teams).

### Go interfaces

A new package `pkg/secrets/` in the workspace-engine:

```go theme={null}
// Provider resolves secret references against an external secret store.
type Provider interface {
    Name() string
    Resolve(ctx context.Context, workspaceID string, ref SecretReference) (string, error)
}

// ProviderConfig holds the decrypted configuration for a workspace's
// secret provider integration.
type ProviderConfig struct {
    ID          string         `json:"id"`
    WorkspaceID string         `json:"workspaceId"`
    Type        string         `json:"type"`
    Name        string         `json:"name"`
    Config      map[string]any `json:"config"`
}

// ProviderConfigStore retrieves decrypted provider configuration for a
// workspace. The implementation handles AES-256 decryption transparently.
type ProviderConfigStore interface {
    GetProviderConfig(ctx context.Context, workspaceID string, providerName string) (*ProviderConfig, error)
    ListProviderConfigs(ctx context.Context, workspaceID string) ([]*ProviderConfig, error)
}

// Resolver holds a registry of provider implementations and resolves
// references by dispatching to the appropriate one.
type Resolver struct {
    store         ProviderConfigStore
    implementations map[string]ProviderFactory
}

// ProviderFactory creates a Provider instance from decrypted config.
type ProviderFactory func(config map[string]any) (Provider, error)
```

The `Resolver` uses the `ProviderConfigStore` to look up the workspace's
provider credentials, then passes them to the appropriate `ProviderFactory` to
construct a `Provider`, then calls `Resolve`. Provider instances can be cached
per workspace with a TTL to avoid repeated decryption and construction.

### Resolution points

The secret resolver integrates at two points in the workspace-engine:

#### 1. Variable resolution (deployment variables)

The `SensitiveValue` type is extended to carry a `SecretReference`:

```jsonnet theme={null}
SensitiveValue: {
  type: 'object',
  required: ['valueHash'],
  properties: {
    valueHash: { type: 'string' },
    secretRef: {
      type: 'object',
      required: ['provider', 'key'],
      properties: {
        provider: { type: 'string' },
        path: { type: 'string' },
        key: { type: 'string' },
      },
    },
  },
},
```

In `variableresolver/value.go`, the `"sensitive"` case changes from an error to
a resolution call:

```go theme={null}
case "sensitive":
    return resolveSensitive(ctx, secretResolver, workspaceID, value)
```

The resolved value becomes a `LiteralValue` and flows into the release like any
other variable. The variable's key is added to `release.EncryptedVariables` so
downstream consumers know it originated from a sensitive source.

#### 2. Job agent config resolution

For agent configs that use Go template interpolation (the existing mechanism),
no changes are needed. The credential flows through the variable system:

1. Deployment variable `argocd_token` is a `SensitiveValue` with a
   `SecretReference` pointing to Doppler
2. Variable resolver calls the secret provider, gets the plaintext token
3. Token lands in `release.Variables` and then `DispatchContext.Variables`
4. The agent config template `{{.variables.argocd_token}}` renders the value

This reuses the existing template interpolation rather than adding a second
resolution path in the agent config.

### Provider implementations

Each provider is a Go package under `pkg/secrets/<provider>/`:

**Doppler** (`pkg/secrets/doppler/`)

```
Path format: <project>/<config>
Key: secret name
API: GET /v3/configs/config/secret?project=X&config=Y&name=Z
Auth: Bearer <serviceToken> from ProviderConfig
```

**HashiCorp Vault** (`pkg/secrets/vault/`)

```
Path format: <mount>/<path> (e.g., secret/data/argocd)
Key: field within the secret data
API: Vault KV v2 read
Auth: Kubernetes auth, AppRole, or token from ProviderConfig
```

**AWS Secrets Manager** (`pkg/secrets/awssm/`)

```
Path format: secret ARN or name
Key: JSON field within the secret string
API: GetSecretValue
Auth: accessKeyId/secretAccessKey from ProviderConfig, or IAM role
```

**Kubernetes** (`pkg/secrets/kubernetes/`)

```
Path format: <namespace>/<secret-name>
Key: data key within the Secret
API: K8s client-go, in-cluster or kubeconfig
Auth: service account or kubeconfig from ProviderConfig
```

**Environment** (`pkg/secrets/env/`)

```
Path: unused
Key: environment variable name
Auth: none (reads from the workspace-engine process env)
```

The `env` provider is special — it does not use `ProviderConfigStore` and does
not require a `secret_provider` row. It reads directly from the process
environment and is available in every workspace by default. This is useful for
development and for instance-level secrets that are the same across all
workspaces.

### Bootstrap chain

The bootstrap dependency chain for secrets is:

```
Operator deploys workspace-engine
  → sets VARIABLES_AES_256_KEY as env var (one key per instance)

Workspace admin connects a secret provider via the UI
  → enters Doppler service token / Vault address / AWS credentials
  → API encrypts with AES-256, stores in secret_provider.config

User creates a deployment variable as SensitiveValue
  → references provider by name + path + key
  → no secret value touches the API or database

Workspace-engine resolves at release time
  → decrypts secret_provider.config in memory
  → calls external provider to fetch the actual secret
  → resolved value exists only in memory during resolution
```

One symmetric key on the process (`VARIABLES_AES_256_KEY`) gates access to all
provider credentials. This key is the same one that `@ctrlplane/secrets` in the
TypeScript layer uses, so both sides can encrypt/decrypt interchangeably.

### API

**REST endpoints:**

```
PUT  /v1/workspaces/{workspaceId}/secret-providers/{providerId}
GET  /v1/workspaces/{workspaceId}/secret-providers
GET  /v1/workspaces/{workspaceId}/secret-providers/{providerId}
DELETE /v1/workspaces/{workspaceId}/secret-providers/{providerId}
```

The `PUT` body:

```json theme={null}
{
  "name": "doppler-production",
  "type": "doppler",
  "config": {
    "serviceToken": "dp.st.xxxxxxxxxxxx"
  }
}
```

The API layer encrypts `config` before storing. The `GET` response never
includes the decrypted config — it returns the provider metadata only:

```json theme={null}
{
  "id": "...",
  "name": "doppler-production",
  "type": "doppler",
  "createdAt": "...",
  "updatedAt": "..."
}
```

### Web UI

**Settings > Secret Providers** — A workspace-level settings page to manage
provider connections. CRUD for `secret_provider` entities. The config form is
type-specific (Doppler shows a service token field, Vault shows address + auth
method, etc.).

**Variable editor** — When creating a deployment variable value, a "Secret"
option is available alongside "Literal" and "Reference". Selecting it shows a
provider dropdown (populated from the workspace's `secret_provider` entries) and
path/key inputs. The submitted value is a `SensitiveValue` with a `secretRef`.

**Runner creation** — The ArgoCD dialog can suggest using a variable reference
for the API key instead of a direct value, linking to the variable editor
workflow.

### Caching

External provider calls add latency to variable resolution. The resolver should
cache resolved values with a configurable TTL (default: 5 minutes). The cache is
keyed by `(workspaceID, providerName, path, key)` and held in memory on the
workspace-engine instance. Cache entries are invalidated when:

* The `secret_provider` config is updated (provider credentials changed)
* The TTL expires
* The workspace-engine restarts

This means secret rotation in the external provider takes effect within the TTL
window. For immediate rotation, the operator can update the `secret_provider`
entity to flush the cache.

### Audit

Secret resolution events should be recorded in the existing `event` table:

```json theme={null}
{
  "action": "secret.resolved",
  "workspaceId": "...",
  "payload": {
    "provider": "doppler-production",
    "path": "backend/production",
    "key": "ARGOCD_TOKEN",
    "releaseId": "...",
    "releaseTargetId": "..."
  }
}
```

The resolved value is never included in the audit event — only the reference
metadata.

## Examples

### ArgoCD API token from Doppler

1. Workspace admin creates a secret provider:

```
PUT /v1/workspaces/{wsId}/secret-providers/{id}
{
  "name": "doppler-platform",
  "type": "doppler",
  "config": { "serviceToken": "dp.st.xxxx" }
}
```

2. Deployment author creates a variable:

```
PUT /v1/workspaces/{wsId}/deployment-variable-values/{id}
{
  "deploymentVariableId": "...",
  "value": {
    "valueHash": "sha256:abcdef...",
    "secretRef": {
      "provider": "doppler-platform",
      "path": "backend/production",
      "key": "ARGOCD_TOKEN"
    }
  }
}
```

3. The ArgoCD agent config template uses the variable:

```yaml theme={null}
serverUrl: argocd.example.com:443
apiKey: "{{.variables.argocd_token}}"
template: |
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  ...
```

4. At release time, the variable resolver hits the `SensitiveValue`, calls the
   Doppler provider, and the resolved token flows through `release.Variables`
   into the dispatch context.

### Vault for database credentials

```json theme={null}
{
  "valueHash": "sha256:...",
  "secretRef": {
    "provider": "vault-prod",
    "path": "database/creds/api-service",
    "key": "password"
  }
}
```

The Vault provider reads the dynamic credential, and it resolves as a regular
`LiteralValue` in the release variables.

### Environment variable fallback (development)

For local development where no external provider is configured, the `env`
provider reads from the workspace-engine process:

```json theme={null}
{
  "valueHash": "sha256:...",
  "secretRef": {
    "provider": "env",
    "path": "",
    "key": "ARGOCD_TOKEN"
  }
}
```

## Migration

* The `secret_provider` table is additive — no existing tables are modified.
* Existing job agent configs with plaintext credentials continue to work. The
  migration path is to create a secret provider, create a deployment variable
  with a `SensitiveValue` reference, update the agent config template to use
  `{{.variables.xxx}}`, and remove the plaintext credential from the agent
  config.
* The `SensitiveValue` schema extension (`secretRef` field) is additive.
  Existing `SensitiveValue` entries without a `secretRef` will fail resolution
  with a clear error message.
* The `EncryptedVariables` field on releases, currently always `[]string{}`,
  will begin to be populated for variables resolved from secret providers.

## Open questions

1. **Should the `env` provider require a `secret_provider` row?** The current
   proposal makes it available by default without configuration. This is
   convenient for development but means any variable can read arbitrary
   environment variables from the workspace-engine process.

2. **Secret versioning** — Should the `SecretReference` support pinning to a
   specific secret version (e.g., Vault lease ID, AWS version stage)? This would
   improve reproducibility but complicates the reference format.

3. **Cross-workspace providers** — Should an instance admin be able to define
   providers at the instance level that are available to all workspaces? This
   avoids duplicating provider configs across workspaces but adds a
   multi-tenancy concern.

4. **Release snapshotting** — Should the resolved secret value be stored
   (encrypted) in `release_variable` for reproducibility, or should it always be
   re-resolved from the external provider? Snapshotting means a release is fully
   reproducible; re-resolving means secrets are never in the database but a
   provider outage blocks dispatch.

5. **RBAC** — Should creating `SensitiveValue` variables that reference a
   provider require a specific permission? This would prevent unprivileged users
   from reading arbitrary secrets from the workspace's providers.