Documentation Index
Fetch the complete documentation index at: https://docs.ctrlplane.dev/llms.txt
Use this file to discover all available pages before exploring further.
| Category | Status | Created | Author |
|---|
| Infrastructure | Draft | 2026-03-06 | Justin Brooks |
Summary
Add a pluggable secret provider system that lets workspaces connect to external
secret managers (Doppler, HashiCorp Vault, AWS Secrets Manager, etc.) and
reference secrets by path rather than storing sensitive values in ctrlplane. A
new workspace-scoped secret_provider entity holds encrypted provider
credentials. A SecretReference type describes a secret’s location. The
workspace-engine resolves references at variable resolution or job dispatch time
by calling the external provider.
Motivation
Ctrlplane currently stores sensitive credentials at two levels:
Job agent configs — When an operator creates an ArgoCD runner, the API token
is stored in the job_agent.config JSON column. The same applies to Terraform
Cloud tokens.
Deployment variables — The SensitiveValue type exists in the OpenAPI
schema and the variable resolver recognizes it as a distinct value type, but
resolution is explicitly rejected:
case "sensitive":
return nil, fmt.Errorf("sensitive values are not resolved by the variable resolver")
The release_variable.encrypted column, job_variable.sensitive flag, and
@ctrlplane/secrets AES-256 service all exist but are not wired end-to-end. The
infrastructure for handling secrets was scaffolded but never completed.
Why this matters
-
Compliance — SOC 2, ISO 27001, and similar frameworks require that
secrets are encrypted at rest and access-controlled.
-
Secret rotation — When credentials are stored directly in ctrlplane,
rotation requires updating every agent config and variable value that
references the credential. With an external provider, rotation happens in
Doppler/Vault/AWS and ctrlplane picks up the new value on the next
resolution.
-
Separation of concerns — Platform teams manage secrets in their existing
secret management infrastructure. Application teams reference secrets by name
in ctrlplane without needing access to the actual values.
-
Multi-workspace — Each workspace may use a different secret provider or
account. The provider credentials themselves need per-workspace encrypted
storage.
Existing mechanisms and their limitations
@ctrlplane/secrets AES-256 — A TypeScript encryption service exists in
packages/secrets/ but is not imported anywhere in the application code. It
provides encrypt/decrypt with a 256-bit key from VARIABLES_AES_256_KEY. This
could handle encryption at rest but does not solve external provider
integration.
SensitiveValue type — Defined in the OpenAPI schema with a valueHash
field. The variable resolver detects it but refuses to resolve it, returning an
error. The intent was that a separate decryption path would handle these values,
but that path was never built.
Go template interpolation — The ArgoCD agent config already supports Go
templates, and the docs show apiKey: "{{.variables.argocd_token}}". This means
credentials can technically flow through the variable system into agent configs
at dispatch time. The missing piece is a way to resolve variables whose values
come from an external source.
Proposal
New entity: secret_provider
A workspace-scoped entity that holds the credentials for connecting to an
external secret management service:
CREATE TABLE secret_provider (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
workspace_id UUID NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
name TEXT NOT NULL,
type TEXT NOT NULL,
config BYTEA NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(workspace_id, name)
);
| Column | Description |
|---|
type | Provider type: doppler, vault, aws-secretsmanager, kubernetes, env |
config | AES-256 encrypted JSON blob with provider-specific credentials |
name | Human-readable name, unique per workspace |
The config column uses BYTEA for the encrypted payload rather than JSON,
since the ciphertext is opaque binary. The workspace-engine decrypts it in
memory using the instance-level VARIABLES_AES_256_KEY when a secret resolution
is needed.
Example configs (before encryption):
// Doppler
{ "serviceToken": "dp.st.xxxxxxxxxxxx" }
// Vault
{ "address": "https://vault.internal.company.com", "authMethod": "kubernetes", "role": "ctrlplane" }
// AWS Secrets Manager
{ "region": "us-east-1", "accessKeyId": "AKIA...", "secretAccessKey": "..." }
// Kubernetes
{ "namespace": "default" }
// Environment variables (instance-level, no config needed)
{}
SecretReference type
A value object that describes where a secret lives:
type SecretReference struct {
Provider string `json:"provider"`
Path string `json:"path,omitempty"`
Key string `json:"key"`
}
| Field | Description |
|---|
Provider | Matches secret_provider.name within the workspace |
Path | Provider-specific path (e.g., my-project/production for Doppler, secret/data/argocd for Vault) |
Key | The specific secret key within the path |
The Provider field matches by name, not by type. This allows a workspace
to have multiple providers of the same type (e.g., two Doppler connections for
different teams).
Go interfaces
A new package pkg/secrets/ in the workspace-engine:
// Provider resolves secret references against an external secret store.
type Provider interface {
Name() string
Resolve(ctx context.Context, workspaceID string, ref SecretReference) (string, error)
}
// ProviderConfig holds the decrypted configuration for a workspace's
// secret provider integration.
type ProviderConfig struct {
ID string `json:"id"`
WorkspaceID string `json:"workspaceId"`
Type string `json:"type"`
Name string `json:"name"`
Config map[string]any `json:"config"`
}
// ProviderConfigStore retrieves decrypted provider configuration for a
// workspace. The implementation handles AES-256 decryption transparently.
type ProviderConfigStore interface {
GetProviderConfig(ctx context.Context, workspaceID string, providerName string) (*ProviderConfig, error)
ListProviderConfigs(ctx context.Context, workspaceID string) ([]*ProviderConfig, error)
}
// Resolver holds a registry of provider implementations and resolves
// references by dispatching to the appropriate one.
type Resolver struct {
store ProviderConfigStore
implementations map[string]ProviderFactory
}
// ProviderFactory creates a Provider instance from decrypted config.
type ProviderFactory func(config map[string]any) (Provider, error)
The Resolver uses the ProviderConfigStore to look up the workspace’s
provider credentials, then passes them to the appropriate ProviderFactory to
construct a Provider, then calls Resolve. Provider instances can be cached
per workspace with a TTL to avoid repeated decryption and construction.
Resolution points
The secret resolver integrates at two points in the workspace-engine:
1. Variable resolution (deployment variables)
The SensitiveValue type is extended to carry a SecretReference:
SensitiveValue: {
type: 'object',
required: ['valueHash'],
properties: {
valueHash: { type: 'string' },
secretRef: {
type: 'object',
required: ['provider', 'key'],
properties: {
provider: { type: 'string' },
path: { type: 'string' },
key: { type: 'string' },
},
},
},
},
In variableresolver/value.go, the "sensitive" case changes from an error to
a resolution call:
case "sensitive":
return resolveSensitive(ctx, secretResolver, workspaceID, value)
The resolved value becomes a LiteralValue and flows into the release like any
other variable. The variable’s key is added to release.EncryptedVariables so
downstream consumers know it originated from a sensitive source.
2. Job agent config resolution
For agent configs that use Go template interpolation (the existing mechanism),
no changes are needed. The credential flows through the variable system:
- Deployment variable
argocd_token is a SensitiveValue with a
SecretReference pointing to Doppler
- Variable resolver calls the secret provider, gets the plaintext token
- Token lands in
release.Variables and then DispatchContext.Variables
- The agent config template
{{.variables.argocd_token}} renders the value
This reuses the existing template interpolation rather than adding a second
resolution path in the agent config.
Provider implementations
Each provider is a Go package under pkg/secrets/<provider>/:
Doppler (pkg/secrets/doppler/)
Path format: <project>/<config>
Key: secret name
API: GET /v3/configs/config/secret?project=X&config=Y&name=Z
Auth: Bearer <serviceToken> from ProviderConfig
HashiCorp Vault (pkg/secrets/vault/)
Path format: <mount>/<path> (e.g., secret/data/argocd)
Key: field within the secret data
API: Vault KV v2 read
Auth: Kubernetes auth, AppRole, or token from ProviderConfig
AWS Secrets Manager (pkg/secrets/awssm/)
Path format: secret ARN or name
Key: JSON field within the secret string
API: GetSecretValue
Auth: accessKeyId/secretAccessKey from ProviderConfig, or IAM role
Kubernetes (pkg/secrets/kubernetes/)
Path format: <namespace>/<secret-name>
Key: data key within the Secret
API: K8s client-go, in-cluster or kubeconfig
Auth: service account or kubeconfig from ProviderConfig
Environment (pkg/secrets/env/)
Path: unused
Key: environment variable name
Auth: none (reads from the workspace-engine process env)
The env provider is special — it does not use ProviderConfigStore and does
not require a secret_provider row. It reads directly from the process
environment and is available in every workspace by default. This is useful for
development and for instance-level secrets that are the same across all
workspaces.
Bootstrap chain
The bootstrap dependency chain for secrets is:
Operator deploys workspace-engine
→ sets VARIABLES_AES_256_KEY as env var (one key per instance)
Workspace admin connects a secret provider via the UI
→ enters Doppler service token / Vault address / AWS credentials
→ API encrypts with AES-256, stores in secret_provider.config
User creates a deployment variable as SensitiveValue
→ references provider by name + path + key
→ no secret value touches the API or database
Workspace-engine resolves at release time
→ decrypts secret_provider.config in memory
→ calls external provider to fetch the actual secret
→ resolved value exists only in memory during resolution
One symmetric key on the process (VARIABLES_AES_256_KEY) gates access to all
provider credentials. This key is the same one that @ctrlplane/secrets in the
TypeScript layer uses, so both sides can encrypt/decrypt interchangeably.
API
REST endpoints:
PUT /v1/workspaces/{workspaceId}/secret-providers/{providerId}
GET /v1/workspaces/{workspaceId}/secret-providers
GET /v1/workspaces/{workspaceId}/secret-providers/{providerId}
DELETE /v1/workspaces/{workspaceId}/secret-providers/{providerId}
The PUT body:
{
"name": "doppler-production",
"type": "doppler",
"config": {
"serviceToken": "dp.st.xxxxxxxxxxxx"
}
}
The API layer encrypts config before storing. The GET response never
includes the decrypted config — it returns the provider metadata only:
{
"id": "...",
"name": "doppler-production",
"type": "doppler",
"createdAt": "...",
"updatedAt": "..."
}
Web UI
Settings > Secret Providers — A workspace-level settings page to manage
provider connections. CRUD for secret_provider entities. The config form is
type-specific (Doppler shows a service token field, Vault shows address + auth
method, etc.).
Variable editor — When creating a deployment variable value, a “Secret”
option is available alongside “Literal” and “Reference”. Selecting it shows a
provider dropdown (populated from the workspace’s secret_provider entries) and
path/key inputs. The submitted value is a SensitiveValue with a secretRef.
Runner creation — The ArgoCD dialog can suggest using a variable reference
for the API key instead of a direct value, linking to the variable editor
workflow.
Caching
External provider calls add latency to variable resolution. The resolver should
cache resolved values with a configurable TTL (default: 5 minutes). The cache is
keyed by (workspaceID, providerName, path, key) and held in memory on the
workspace-engine instance. Cache entries are invalidated when:
- The
secret_provider config is updated (provider credentials changed)
- The TTL expires
- The workspace-engine restarts
This means secret rotation in the external provider takes effect within the TTL
window. For immediate rotation, the operator can update the secret_provider
entity to flush the cache.
Audit
Secret resolution events should be recorded in the existing event table:
{
"action": "secret.resolved",
"workspaceId": "...",
"payload": {
"provider": "doppler-production",
"path": "backend/production",
"key": "ARGOCD_TOKEN",
"releaseId": "...",
"releaseTargetId": "..."
}
}
The resolved value is never included in the audit event — only the reference
metadata.
Examples
ArgoCD API token from Doppler
- Workspace admin creates a secret provider:
PUT /v1/workspaces/{wsId}/secret-providers/{id}
{
"name": "doppler-platform",
"type": "doppler",
"config": { "serviceToken": "dp.st.xxxx" }
}
- Deployment author creates a variable:
PUT /v1/workspaces/{wsId}/deployment-variable-values/{id}
{
"deploymentVariableId": "...",
"value": {
"valueHash": "sha256:abcdef...",
"secretRef": {
"provider": "doppler-platform",
"path": "backend/production",
"key": "ARGOCD_TOKEN"
}
}
}
- The ArgoCD agent config template uses the variable:
serverUrl: argocd.example.com:443
apiKey: "{{.variables.argocd_token}}"
template: |
apiVersion: argoproj.io/v1alpha1
kind: Application
...
- At release time, the variable resolver hits the
SensitiveValue, calls the
Doppler provider, and the resolved token flows through release.Variables
into the dispatch context.
Vault for database credentials
{
"valueHash": "sha256:...",
"secretRef": {
"provider": "vault-prod",
"path": "database/creds/api-service",
"key": "password"
}
}
The Vault provider reads the dynamic credential, and it resolves as a regular
LiteralValue in the release variables.
Environment variable fallback (development)
For local development where no external provider is configured, the env
provider reads from the workspace-engine process:
{
"valueHash": "sha256:...",
"secretRef": {
"provider": "env",
"path": "",
"key": "ARGOCD_TOKEN"
}
}
Migration
- The
secret_provider table is additive — no existing tables are modified.
- Existing job agent configs with plaintext credentials continue to work. The
migration path is to create a secret provider, create a deployment variable
with a
SensitiveValue reference, update the agent config template to use
{{.variables.xxx}}, and remove the plaintext credential from the agent
config.
- The
SensitiveValue schema extension (secretRef field) is additive.
Existing SensitiveValue entries without a secretRef will fail resolution
with a clear error message.
- The
EncryptedVariables field on releases, currently always []string{},
will begin to be populated for variables resolved from secret providers.
Open questions
-
Should the
env provider require a secret_provider row? The current
proposal makes it available by default without configuration. This is
convenient for development but means any variable can read arbitrary
environment variables from the workspace-engine process.
-
Secret versioning — Should the
SecretReference support pinning to a
specific secret version (e.g., Vault lease ID, AWS version stage)? This would
improve reproducibility but complicates the reference format.
-
Cross-workspace providers — Should an instance admin be able to define
providers at the instance level that are available to all workspaces? This
avoids duplicating provider configs across workspaces but adds a
multi-tenancy concern.
-
Release snapshotting — Should the resolved secret value be stored
(encrypted) in
release_variable for reproducibility, or should it always be
re-resolved from the external provider? Snapshotting means a release is fully
reproducible; re-resolving means secrets are never in the database but a
provider outage blocks dispatch.
-
RBAC — Should creating
SensitiveValue variables that reference a
provider require a specific permission? This would prevent unprivileged users
from reading arbitrary secrets from the workspace’s providers.