Skip to main content
CategoryStatusCreatedAuthor
InfrastructureDraft2026-03-06Justin Brooks

​
Summary

Add a pluggable secret provider system that lets workspaces connect to external secret managers (Doppler, HashiCorp Vault, AWS Secrets Manager, etc.) and reference secrets by path rather than storing sensitive values in ctrlplane. A new workspace-scoped secret_provider entity holds encrypted provider credentials. A SecretReference type describes a secret’s location. The workspace-engine resolves references at variable resolution or job dispatch time by calling the external provider.

​
Motivation

Ctrlplane currently stores sensitive credentials at two levels: Job agent configs — When an operator creates an ArgoCD runner, the API token is stored in the job_agent.config JSON column. The same applies to Terraform Cloud tokens. Deployment variables — The SensitiveValue type exists in the OpenAPI schema and the variable resolver recognizes it as a distinct value type, but resolution is explicitly rejected: 
case "sensitive":
    return nil, fmt.Errorf("sensitive values are not resolved by the variable resolver")
The release_variable.encrypted column, job_variable.sensitive flag, and @ctrlplane/secrets AES-256 service all exist but are not wired end-to-end. The infrastructure for handling secrets was scaffolded but never completed.

​
Why this matters

  1. Compliance — SOC 2, ISO 27001, and similar frameworks require that secrets are encrypted at rest and access-controlled.
  2. Secret rotation — When credentials are stored directly in ctrlplane, rotation requires updating every agent config and variable value that references the credential. With an external provider, rotation happens in Doppler/Vault/AWS and ctrlplane picks up the new value on the next resolution.
  3. Separation of concerns — Platform teams manage secrets in their existing secret management infrastructure. Application teams reference secrets by name in ctrlplane without needing access to the actual values.
  4. Multi-workspace — Each workspace may use a different secret provider or account. The provider credentials themselves need per-workspace encrypted storage.

​
Existing mechanisms and their limitations

@ctrlplane/secrets AES-256 — A TypeScript encryption service exists in packages/secrets/ but is not imported anywhere in the application code. It provides encrypt/decrypt with a 256-bit key from VARIABLES_AES_256_KEY. This could handle encryption at rest but does not solve external provider integration. SensitiveValue type — Defined in the OpenAPI schema with a valueHash field. The variable resolver detects it but refuses to resolve it, returning an error. The intent was that a separate decryption path would handle these values, but that path was never built. Go template interpolation — The ArgoCD agent config already supports Go templates, and the docs show apiKey: "{{.variables.argocd_token}}". This means credentials can technically flow through the variable system into agent configs at dispatch time. The missing piece is a way to resolve variables whose values come from an external source.

​
Proposal

​
New entity: secret_provider

A workspace-scoped entity that holds the credentials for connecting to an external secret management service: 
CREATE TABLE secret_provider (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
    name TEXT NOT NULL,
    type TEXT NOT NULL,
    config BYTEA NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    UNIQUE(workspace_id, name)
);
ColumnDescription
typeProvider type: doppler, vault, aws-secretsmanager, kubernetes, env
configAES-256 encrypted JSON blob with provider-specific credentials
nameHuman-readable name, unique per workspace
The config column uses BYTEA for the encrypted payload rather than JSON, since the ciphertext is opaque binary. The workspace-engine decrypts it in memory using the instance-level VARIABLES_AES_256_KEY when a secret resolution is needed. Example configs (before encryption): 
// Doppler
{ "serviceToken": "dp.st.xxxxxxxxxxxx" }

// Vault
{ "address": "https://vault.internal.company.com", "authMethod": "kubernetes", "role": "ctrlplane" }

// AWS Secrets Manager
{ "region": "us-east-1", "accessKeyId": "AKIA...", "secretAccessKey": "..." }

// Kubernetes
{ "namespace": "default" }

// Environment variables (instance-level, no config needed)
{}

​
SecretReference type

A value object that describes where a secret lives: 
type SecretReference struct {
    Provider string `json:"provider"`
    Path     string `json:"path,omitempty"`
    Key      string `json:"key"`
}
FieldDescription
ProviderMatches secret_provider.name within the workspace
PathProvider-specific path (e.g., my-project/production for Doppler, secret/data/argocd for Vault)
KeyThe specific secret key within the path
The Provider field matches by name, not by type. This allows a workspace to have multiple providers of the same type (e.g., two Doppler connections for different teams).

​
Go interfaces

A new package pkg/secrets/ in the workspace-engine: 
// Provider resolves secret references against an external secret store.
type Provider interface {
    Name() string
    Resolve(ctx context.Context, workspaceID string, ref SecretReference) (string, error)
}

// ProviderConfig holds the decrypted configuration for a workspace's
// secret provider integration.
type ProviderConfig struct {
    ID          string         `json:"id"`
    WorkspaceID string         `json:"workspaceId"`
    Type        string         `json:"type"`
    Name        string         `json:"name"`
    Config      map[string]any `json:"config"`
}

// ProviderConfigStore retrieves decrypted provider configuration for a
// workspace. The implementation handles AES-256 decryption transparently.
type ProviderConfigStore interface {
    GetProviderConfig(ctx context.Context, workspaceID string, providerName string) (*ProviderConfig, error)
    ListProviderConfigs(ctx context.Context, workspaceID string) ([]*ProviderConfig, error)
}

// Resolver holds a registry of provider implementations and resolves
// references by dispatching to the appropriate one.
type Resolver struct {
    store         ProviderConfigStore
    implementations map[string]ProviderFactory
}

// ProviderFactory creates a Provider instance from decrypted config.
type ProviderFactory func(config map[string]any) (Provider, error)
The Resolver uses the ProviderConfigStore to look up the workspace’s provider credentials, then passes them to the appropriate ProviderFactory to construct a Provider, then calls Resolve. Provider instances can be cached per workspace with a TTL to avoid repeated decryption and construction.

​
Resolution points

The secret resolver integrates at two points in the workspace-engine:

​
1. Variable resolution (deployment variables)

The SensitiveValue type is extended to carry a SecretReference: 
SensitiveValue: {
  type: 'object',
  required: ['valueHash'],
  properties: {
    valueHash: { type: 'string' },
    secretRef: {
      type: 'object',
      required: ['provider', 'key'],
      properties: {
        provider: { type: 'string' },
        path: { type: 'string' },
        key: { type: 'string' },
      },
    },
  },
},
In variableresolver/value.go, the "sensitive" case changes from an error to a resolution call: 
case "sensitive":
    return resolveSensitive(ctx, secretResolver, workspaceID, value)
The resolved value becomes a LiteralValue and flows into the release like any other variable. The variable’s key is added to release.EncryptedVariables so downstream consumers know it originated from a sensitive source.

​
2. Job agent config resolution

For agent configs that use Go template interpolation (the existing mechanism), no changes are needed. The credential flows through the variable system:
  1. Deployment variable argocd_token is a SensitiveValue with a SecretReference pointing to Doppler
  2. Variable resolver calls the secret provider, gets the plaintext token
  3. Token lands in release.Variables and then DispatchContext.Variables
  4. The agent config template {{.variables.argocd_token}} renders the value
This reuses the existing template interpolation rather than adding a second resolution path in the agent config.

​
Provider implementations

Each provider is a Go package under pkg/secrets/<provider>/: Doppler (pkg/secrets/doppler/) 
Path format: <project>/<config>
Key: secret name
API: GET /v3/configs/config/secret?project=X&config=Y&name=Z
Auth: Bearer <serviceToken> from ProviderConfig
HashiCorp Vault (pkg/secrets/vault/) 
Path format: <mount>/<path> (e.g., secret/data/argocd)
Key: field within the secret data
API: Vault KV v2 read
Auth: Kubernetes auth, AppRole, or token from ProviderConfig
AWS Secrets Manager (pkg/secrets/awssm/) 
Path format: secret ARN or name
Key: JSON field within the secret string
API: GetSecretValue
Auth: accessKeyId/secretAccessKey from ProviderConfig, or IAM role
Kubernetes (pkg/secrets/kubernetes/) 
Path format: <namespace>/<secret-name>
Key: data key within the Secret
API: K8s client-go, in-cluster or kubeconfig
Auth: service account or kubeconfig from ProviderConfig
Environment (pkg/secrets/env/) 
Path: unused
Key: environment variable name
Auth: none (reads from the workspace-engine process env)
The env provider is special — it does not use ProviderConfigStore and does not require a secret_provider row. It reads directly from the process environment and is available in every workspace by default. This is useful for development and for instance-level secrets that are the same across all workspaces.

​
Bootstrap chain

The bootstrap dependency chain for secrets is: 
Operator deploys workspace-engine
  → sets VARIABLES_AES_256_KEY as env var (one key per instance)

Workspace admin connects a secret provider via the UI
  → enters Doppler service token / Vault address / AWS credentials
  → API encrypts with AES-256, stores in secret_provider.config

User creates a deployment variable as SensitiveValue
  → references provider by name + path + key
  → no secret value touches the API or database

Workspace-engine resolves at release time
  → decrypts secret_provider.config in memory
  → calls external provider to fetch the actual secret
  → resolved value exists only in memory during resolution
One symmetric key on the process (VARIABLES_AES_256_KEY) gates access to all provider credentials. This key is the same one that @ctrlplane/secrets in the TypeScript layer uses, so both sides can encrypt/decrypt interchangeably.

​
API

REST endpoints: 
PUT  /v1/workspaces/{workspaceId}/secret-providers/{providerId}
GET  /v1/workspaces/{workspaceId}/secret-providers
GET  /v1/workspaces/{workspaceId}/secret-providers/{providerId}
DELETE /v1/workspaces/{workspaceId}/secret-providers/{providerId}
The PUT body: 
{
  "name": "doppler-production",
  "type": "doppler",
  "config": {
    "serviceToken": "dp.st.xxxxxxxxxxxx"
  }
}
The API layer encrypts config before storing. The GET response never includes the decrypted config — it returns the provider metadata only: 
{
  "id": "...",
  "name": "doppler-production",
  "type": "doppler",
  "createdAt": "...",
  "updatedAt": "..."
}

​
Web UI

Settings > Secret Providers — A workspace-level settings page to manage provider connections. CRUD for secret_provider entities. The config form is type-specific (Doppler shows a service token field, Vault shows address + auth method, etc.). Variable editor — When creating a deployment variable value, a “Secret” option is available alongside “Literal” and “Reference”. Selecting it shows a provider dropdown (populated from the workspace’s secret_provider entries) and path/key inputs. The submitted value is a SensitiveValue with a secretRef. Runner creation — The ArgoCD dialog can suggest using a variable reference for the API key instead of a direct value, linking to the variable editor workflow.

​
Caching

External provider calls add latency to variable resolution. The resolver should cache resolved values with a configurable TTL (default: 5 minutes). The cache is keyed by (workspaceID, providerName, path, key) and held in memory on the workspace-engine instance. Cache entries are invalidated when:
  • The secret_provider config is updated (provider credentials changed)
  • The TTL expires
  • The workspace-engine restarts
This means secret rotation in the external provider takes effect within the TTL window. For immediate rotation, the operator can update the secret_provider entity to flush the cache.

​
Audit

Secret resolution events should be recorded in the existing event table: 
{
  "action": "secret.resolved",
  "workspaceId": "...",
  "payload": {
    "provider": "doppler-production",
    "path": "backend/production",
    "key": "ARGOCD_TOKEN",
    "releaseId": "...",
    "releaseTargetId": "..."
  }
}
The resolved value is never included in the audit event — only the reference metadata.

​
Examples

​
ArgoCD API token from Doppler

  1. Workspace admin creates a secret provider:
PUT /v1/workspaces/{wsId}/secret-providers/{id}
{
  "name": "doppler-platform",
  "type": "doppler",
  "config": { "serviceToken": "dp.st.xxxx" }
}
  1. Deployment author creates a variable:
PUT /v1/workspaces/{wsId}/deployment-variable-values/{id}
{
  "deploymentVariableId": "...",
  "value": {
    "valueHash": "sha256:abcdef...",
    "secretRef": {
      "provider": "doppler-platform",
      "path": "backend/production",
      "key": "ARGOCD_TOKEN"
    }
  }
}
  1. The ArgoCD agent config template uses the variable:
serverUrl: argocd.example.com:443
apiKey: "{{.variables.argocd_token}}"
template: |
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  ...
  1. At release time, the variable resolver hits the SensitiveValue, calls the Doppler provider, and the resolved token flows through release.Variables into the dispatch context.

​
Vault for database credentials

{
  "valueHash": "sha256:...",
  "secretRef": {
    "provider": "vault-prod",
    "path": "database/creds/api-service",
    "key": "password"
  }
}
The Vault provider reads the dynamic credential, and it resolves as a regular LiteralValue in the release variables.

​
Environment variable fallback (development)

For local development where no external provider is configured, the env provider reads from the workspace-engine process: 
{
  "valueHash": "sha256:...",
  "secretRef": {
    "provider": "env",
    "path": "",
    "key": "ARGOCD_TOKEN"
  }
}

​
Migration

  • The secret_provider table is additive — no existing tables are modified.
  • Existing job agent configs with plaintext credentials continue to work. The migration path is to create a secret provider, create a deployment variable with a SensitiveValue reference, update the agent config template to use {{.variables.xxx}}, and remove the plaintext credential from the agent config.
  • The SensitiveValue schema extension (secretRef field) is additive. Existing SensitiveValue entries without a secretRef will fail resolution with a clear error message.
  • The EncryptedVariables field on releases, currently always []string{}, will begin to be populated for variables resolved from secret providers.

​
Open questions

  1. Should the env provider require a secret_provider row? The current proposal makes it available by default without configuration. This is convenient for development but means any variable can read arbitrary environment variables from the workspace-engine process.
  2. Secret versioning — Should the SecretReference support pinning to a specific secret version (e.g., Vault lease ID, AWS version stage)? This would improve reproducibility but complicates the reference format.
  3. Cross-workspace providers — Should an instance admin be able to define providers at the instance level that are available to all workspaces? This avoids duplicating provider configs across workspaces but adds a multi-tenancy concern.
  4. Release snapshotting — Should the resolved secret value be stored (encrypted) in release_variable for reproducibility, or should it always be re-resolved from the external provider? Snapshotting means a release is fully reproducible; re-resolving means secrets are never in the database but a provider outage blocks dispatch.
  5. RBAC — Should creating SensitiveValue variables that reference a provider require a specific permission? This would prevent unprivileged users from reading arbitrary secrets from the workspace’s providers.