Skip to main content

Summary

This RFC proposes a unified data model for managing variables across resources, deployments, and deployment job agents. It consolidates the current fragmented schema into a single, extensible system that supports:
  • Multiple scopes (resource, deployment, deployment job agent)
  • Multiple value types (literal, reference, secret reference)
  • Override semantics via selectors and priority
  • First-class support for secret providers without duplicating schema
The proposal replaces multiple duplicated tables with two core tables: variable and variable_value.

Motivation

The current schema exhibits significant duplication across two dimensions:
  1. Scope duplication
    • Separate handling for resource, deployment, and job-agent variables
  2. Value-type duplication
    • Separate tables for literal values and reference values
This results in:
  • Schema explosion and maintenance overhead
  • Repeated logic in queries and resolution code
  • Increased risk of inconsistency and bugs
  • Difficulty extending the system (e.g., adding secrets)
Additionally, introducing secrets under the current model would require duplicating the entire table structure again, further compounding complexity. We need a model that:
  • Treats scope and value type as data, not schema
  • Supports extensibility without table proliferation
  • Centralizes resolution logic

Goals

  • Eliminate duplicated tables across scopes and value types
  • Provide a single resolution model for all variable types
  • Support secret references without storing raw secrets
  • Maintain strong data integrity constraints
  • Enable future extensibility (new value types, new scopes)

Non-Goals

  • Implementing secret storage (this system references external providers)
  • Defining a full selector language
  • Enforcing cross-variable resolution correctness at the database level

Proposal

Core Concepts

The system is built around two primary entities:
  1. Variable
    • Defines a key within a specific scope
  2. Variable Value
    • Defines one or more candidate values for a variable
    • Supports override semantics via priority and selectors

Variable

Represents a named configuration key scoped to a specific owner. Key properties:
  • scope: one of resource, deployment, deployment_job_agent
  • Exactly one owner reference is set
  • key: variable identifier
  • is_sensitive: indicates whether the variable contains sensitive data

Variable Value

Represents a candidate value for a variable. Supports three value types:
  • literal: JSON value stored directly
  • ref: reference to another variable
  • secret_ref: reference to an external secret provider
Also includes:
  • resource_selector: optional matching condition
  • priority: determines precedence

Value Types

Literal

Stores a JSON value directly in the database. Example:

Reference

References another variable by key, optionally with a path. Example:

Secret Reference

References a value stored in an external secret manager. Example:

Resolution Model

To resolve a variable:
  1. Identify the variable by scope and key
  2. Retrieve all associated variable_value rows
  3. Filter by selector (if applicable)
  4. Sort by priority (descending)
  5. Select the highest-priority match
  6. Resolve based on value type:
    • literal → return value
    • ref → recursively resolve referenced variable
    • secret_ref → fetch from external provider

Why This Design

Eliminates Duplication

  • One table for variables instead of per-scope tables
  • One table for values instead of per-type tables

Extensible

Adding a new value type (e.g., computed, templated) requires:
  • Adding a new enum value
  • Adding optional columns or extending logic
No new tables required.

Unified Resolution Logic

All variables follow the same resolution pipeline regardless of scope or type.

Secret Handling

Secrets are treated as a value source, not a separate system:
  • Avoids duplicating schema
  • Keeps resolution consistent
  • Prevents storing sensitive data directly in the DB

Alternatives Considered

1. Separate Tables per Scope

Rejected because:
  • Leads to schema duplication
  • Requires duplicating logic
  • Hard to extend

2. Separate Tables per Value Type

Rejected because:
  • Introduces join complexity
  • Makes adding new types expensive

3. Separate Secret Tables

Rejected because:
  • Secrets participate in the same resolution semantics
  • Only the value source differs
  • Duplication would increase system complexity

Tradeoffs

Pros

  • Dramatically simpler schema
  • Centralized resolution logic
  • Easier to extend
  • Reduces duplication

Cons

  • More nullable columns in variable_value
  • Some validation shifts to application logic
  • Slightly more complex constraints

Future Work

  • Replace ref_key with referenced_variable_id for stronger integrity
  • Introduce structured selector model (e.g., JSON-based matching)
  • Add expression-based value model (single JSON expression column)
  • Add audit logging and versioning

Migration Strategy

  1. Create new tables alongside existing schema
  2. Backfill variables and values
  3. Update read paths to use new schema
  4. Deprecate old tables
  5. Remove old schema after validation

Conclusion

This proposal replaces a fragmented and duplicated schema with a unified, extensible model for variable management. By treating scope and value type as data rather than schema, the system becomes:
  • Easier to maintain
  • Easier to extend
  • More consistent in behavior
It also provides a clean path to integrate secrets without introducing additional structural complexity.