Skip to main content

Summary

Add two new policy rule types — deployment bracket and resource concurrency — that together enable coordinated multi-deployment rollouts with cluster-wide capacity limits. Deployment brackets group related deployments so they execute as a unit against each resource (drain a node once, apply all upgrades, uncordon). Resource concurrency limits how many resources in a group can be simultaneously undergoing deployment (only 20% of nodes offline at a time). Both are policy rules, not new entity types. They slot into the existing evaluator pipeline alongside gradual rollout, deployment dependency, and the other rule types.

Motivation

When deploying software to Kubernetes nodes, two operational constraints exist that ctrlplane’s current policy engine cannot express:

Drain cost amortization

Deploying to a node requires draining pods first — evicting workloads, respecting PodDisruptionBudgets, waiting for graceful termination. This is slow (minutes to tens of minutes) and disruptive. If multiple deployments target the same node (kubelet upgrade, containerd upgrade, OS patch), each deployment today triggers its own independent drain/uncordon cycle because each release target is evaluated independently. For 3 deployments across 10 nodes, this means 30 drain cycles instead of 10. The overhead scales linearly with the number of bracketed deployments.

Cluster-wide concurrency limits

Only a percentage of nodes in a cluster can be offline simultaneously. If 20% of a 10-node cluster goes down, the remaining 8 nodes must absorb all workloads. Exceeding this threshold risks cascading failures. The existing concurrency control (ReleaseTargetConcurrencyEvaluator) enforces a hard limit of 1 active job per release target:
This prevents overlapping jobs on the same release target but provides no cross-resource limit. There is no way to say “at most 2 of these 10 nodes can be draining at once.”

Why existing rules are insufficient

Gradual rollout staggers deployments over time using hash-based position ordering. It controls when each target’s turn arrives but not how many can be active simultaneously. If multiple targets’ rollout times pass while earlier targets are still executing (e.g., a slow drain), all of them proceed at once. Deployment dependency checks whether an upstream deployment has succeeded for the same resource. It controls ordering (drain before upgrade, upgrade before uncordon) but not grouping. Each deployment’s policy pipeline runs independently — there is no mechanism to hold all deployments for a resource until a collection of new versions is ready. Version cooldown batches frequent releases for a single deployment. It does not coordinate across deployments. These three rules address different dimensions (time, order, frequency) but none addresses the two missing dimensions: cross-deployment grouping and cross-resource capacity limits.

Proposal

New policy rule: deployment bracket

A deployment bracket groups deployments that should execute as a coordinated unit per resource. The bracket rule gates deployment until all member deployments have versions ready, using a configurable collection window.

Schema

readiness_mode controls when a bracket starts executing: unchanged_member_strategy controls members with no new version when the window closes: overlap_strategy controls what happens when a new version group becomes ready while a previous group is still executing on a resource:

Version group state

The bracket evaluator needs to track which versions belong to an active collection window or executing group. A lightweight state table supports this:

Evaluator

The bracket evaluator implements the standard Evaluator interface:
The Evaluate method performs three checks:
  1. Overlap check — if a version group is executing for this bracket + resource, and the candidate version is not part of that group, return Pending (for queue strategy) or Allowed (for merge, if pre-hooks are complete).
  2. Collection window check — if no active group exists, find or create a bracket_version_group. If the window hasn’t closed, return Pending with NextEvaluationTime set to collection_ends_at.
  3. Readiness check — if the window has closed, verify the candidate version matches the locked version for this deployment in the version group. If so, return Allowed.

New policy rule: resource concurrency

A resource concurrency rule limits how many resources in a group can simultaneously be undergoing deployment. The group is defined by a CEL selector, and the limit can be a percentage or absolute count.

Schema

Evaluator

The “active” count includes any resource that has at least one job in a processing state across any deployment matched by the parent policy’s selector. For brackets, this means a resource in any phase (pre-hook, deploying, post-hook) counts against the limit.

Enhancement: scoped deployment dependencies

Today, a deployment dependency rule on a policy applies to every release target matched by the policy’s selector. This forces separate policies for each ordering constraint in a bracket. Adding an optional applies_to field lets multiple dependency rules coexist on one policy:
When NULL, the rule applies to all matched release targets (current behavior). When set, the rule is only evaluated for release targets whose deployment matches the CEL expression:

Bracket-aware gradual rollout

When a bracket rule exists on the same policy as a gradual rollout rule, the rollout evaluator should hash on resourceId + bracketRuleId instead of the individual release target key. This ensures all deployments in a bracket for a given resource receive the same rollout position. The gradual rollout evaluator already receives the full policy context through the store. The change is in the hash input for rollout position calculation:
Without this, kubelet-upgrade and containerd-upgrade on the same node would get different rollout positions and arrive at different times, defeating the purpose of bracketing.

Evaluator chain integration

Both new evaluators slot into the existing factory:
For any release target, all evaluators must return Allowed for a job to be created. The evaluators gate at different levels:

Pre/post hooks as deployments

Rather than introducing a separate hook mechanism, lifecycle hooks (drain, uncordon) are modeled as regular deployments that are bracket members with dependency ordering. A “drain” deployment’s job agent runs kubectl drain. An “uncordon” deployment’s job agent runs kubectl uncordon. Deployment dependency rules control execution order within the bracket. This reuses the existing job agent, job dispatch, retry, rollback, and verification infrastructure. Hooks get observability (traces, job status) and policy controls (retry on failure, rollback) for free.

Examples

Node upgrade with ordered bracket

A cluster has 10 nodes. Three workload deployments (kubelet, containerd, os-patch) plus two lifecycle deployments (drain, uncordon) are tagged with metadata.layer = "node". os-patch must run before kubelet and containerd. All five share a 24-hour collection window and a 20% concurrency limit. Policy configuration:
This expresses the execution DAG:
Execution trace for node-3 (position 1 in rollout):

Different clusters, different policies

The same deployments can have different bracket configurations per cluster by using the policy selector to scope rules:
Staging deploys immediately with 50% concurrency. Production waits 24 hours and limits to 20%. Same deployments, different operational posture.

Partial readiness

Only kubelet gets a new version within the 24-hour window. containerd and os-patch have no updates.

Overlapping version groups

A new version group becomes ready while the previous group is still executing.

Failure mid-bracket

kubelet fails on node-3 while containerd succeeds.
This is why retry and/or rollback rules should always accompany bracket policies. A stuck resource holds a concurrency slot indefinitely.

Migration

  • The policy_rule_deployment_bracket and policy_rule_resource_concurrency tables are new. No data migration required.
  • The bracket_version_group and bracket_version_group_member tables are new.
  • The applies_to column on policy_rule_deployment_dependency is additive and nullable. Existing rules have applies_to = NULL, preserving current behavior (rule applies to all matched release targets).
  • The new evaluators return nil from their factory functions when the policy rule does not contain the relevant configuration, following the same pattern as all existing evaluators.
  • Agents that execute lifecycle hooks (drain, uncordon) are standard job agents. No new agent interfaces are needed.

Open Questions

  1. Collection window trigger semantics. Should the collection window start when the first member’s version is published or when the first member’s version passes other policy rules (approval, version selector)? Starting at publication is simpler but means the window runs concurrently with approval — a 24h window with a 20h approval process only leaves 4h of actual collection time.
  2. Gradual rollout interaction. The proposal changes the hash input when a bracket rule is present. This means adding or removing a bracket rule changes the rollout order for all targets. Should the bracket-aware hashing be opt-in to avoid surprising rollout order changes?
  3. Bracket membership dynamism. The deploymentSelector on the bracket rule is evaluated dynamically. If a new deployment is added mid-rollout that matches the selector, should in-progress version groups absorb it? The simplest behavior is to only affect future version groups.
  4. Merge strategy completeness. The merge overlap strategy avoids double drain cycles but requires the dependency evaluator to distinguish between “upstream succeeded with Group A’s version” and “upstream succeeded with Group B’s version.” The current evaluator checks success status but not which version succeeded. This may need a version-aware dependency check for merge to work correctly.
  5. Failure blast radius. A bracket failure on one resource holds a concurrency slot. With 20% concurrency on a 10-node cluster, 2 stuck nodes block the entire rollout. Should there be a configurable timeout that auto-releases concurrency slots after a bracket has been stuck for too long, even if the resource is in an unknown state?
  6. Hook idempotency. The queue overlap strategy runs drain → uncordon → drain → uncordon for consecutive groups. This assumes drain and uncordon are idempotent. The merge strategy assumes the resource remains in a drained state between groups. Should the bracket rule have an explicit field declaring whether hooks are idempotent and/or whether the prepared state persists?
  7. Auto rollback interaction. If a rollback policy triggers for one member deployment mid-bracket (e.g., kubelet fails health checks and auto-rolls back to v1.28.x), the resource is in a partially-upgraded state — some bracket members succeeded with new versions, others rolled back. Several sub-questions arise:
    • Should a rollback of any bracket member trigger a rollback of all bracket members on that resource to restore a consistent version set? This is the safe default for tightly coupled components (kubelet + containerd), but overly aggressive for loosely coupled ones.
    • If only the failed member rolls back, do post-hooks (uncordon) still run? The dependency graph may be satisfied (kubelet “completed” via rollback, containerd succeeded), but the resource is in a mixed state that the operator may not have intended to uncordon.
    • Should the bracket rule have a rollback_strategy field (e.g., individual, all_members, halt_and_notify) that controls whether rollback is scoped to the failing member, cascaded to the full bracket, or paused for manual intervention?
    • How does a bracket-wide rollback interact with the concurrency limit? Rolling back N members on a resource means N additional jobs — does each count against the concurrency slot, or does the resource’s existing slot cover the rollback work?