| Category | Status | Created | Author |
|---|---|---|---|
| Policies | Draft | 2026-03-13 | Justin Brooks |
Summary
Add two new policy rule types — deployment bracket and resource concurrency — that together enable coordinated multi-deployment rollouts with cluster-wide capacity limits. Deployment brackets group related deployments so they execute as a unit against each resource (drain a node once, apply all upgrades, uncordon). Resource concurrency limits how many resources in a group can be simultaneously undergoing deployment (only 20% of nodes offline at a time). Both are policy rules, not new entity types. They slot into the existing evaluator pipeline alongside gradual rollout, deployment dependency, and the other rule types.Motivation
When deploying software to Kubernetes nodes, two operational constraints exist that ctrlplane’s current policy engine cannot express:Drain cost amortization
Deploying to a node requires draining pods first — evicting workloads, respecting PodDisruptionBudgets, waiting for graceful termination. This is slow (minutes to tens of minutes) and disruptive. If multiple deployments target the same node (kubelet upgrade, containerd upgrade, OS patch), each deployment today triggers its own independent drain/uncordon cycle because each release target is evaluated independently. For 3 deployments across 10 nodes, this means 30 drain cycles instead of 10. The overhead scales linearly with the number of bracketed deployments.Cluster-wide concurrency limits
Only a percentage of nodes in a cluster can be offline simultaneously. If 20% of a 10-node cluster goes down, the remaining 8 nodes must absorb all workloads. Exceeding this threshold risks cascading failures. The existing concurrency control (ReleaseTargetConcurrencyEvaluator) enforces
a hard limit of 1 active job per release target:
Why existing rules are insufficient
Gradual rollout staggers deployments over time using hash-based position ordering. It controls when each target’s turn arrives but not how many can be active simultaneously. If multiple targets’ rollout times pass while earlier targets are still executing (e.g., a slow drain), all of them proceed at once. Deployment dependency checks whether an upstream deployment has succeeded for the same resource. It controls ordering (drain before upgrade, upgrade before uncordon) but not grouping. Each deployment’s policy pipeline runs independently — there is no mechanism to hold all deployments for a resource until a collection of new versions is ready. Version cooldown batches frequent releases for a single deployment. It does not coordinate across deployments. These three rules address different dimensions (time, order, frequency) but none addresses the two missing dimensions: cross-deployment grouping and cross-resource capacity limits.Proposal
New policy rule: deployment bracket
A deployment bracket groups deployments that should execute as a coordinated unit per resource. The bracket rule gates deployment until all member deployments have versions ready, using a configurable collection window.Schema
readiness_mode controls when a bracket starts executing:
| Mode | Behavior |
|---|---|
collection_window | Wait readiness_window_seconds after the first member version is published. Take the latest version of each member at window close. |
wait_for_all | Wait until every member deployment has a new version. Fall back to unchanged_member_strategy after readiness_window_seconds timeout. |
immediate | Proceed as soon as any member has a new version. Use latest available version for all members. |
unchanged_member_strategy controls members with no new version when the
window closes:
| Strategy | Behavior |
|---|---|
skip_unchanged | Only deploy members that have new versions. Others are no-ops. |
redeploy_current | Re-deploy the currently running version for all members. Useful when hooks have side effects. |
require_all | Don’t close the window until all members have a new version. Fall back to skip_unchanged after timeout. |
overlap_strategy controls what happens when a new version group becomes
ready while a previous group is still executing on a resource:
| Strategy | Behavior |
|---|---|
queue | Wait for the active group to fully complete (including post-hooks), then start a fresh cycle. Safe default. |
merge | If the resource is already in a pre-hook state (e.g., drained), deploy the new group’s versions before running post-hooks. Avoids double drain cycles. |
Version group state
The bracket evaluator needs to track which versions belong to an active collection window or executing group. A lightweight state table supports this:Evaluator
The bracket evaluator implements the standardEvaluator interface:
Evaluate method performs three checks:
-
Overlap check — if a version group is executing for this bracket +
resource, and the candidate version is not part of that group, return
Pending(forqueuestrategy) orAllowed(formerge, if pre-hooks are complete). -
Collection window check — if no active group exists, find or create a
bracket_version_group. If the window hasn’t closed, returnPendingwithNextEvaluationTimeset tocollection_ends_at. -
Readiness check — if the window has closed, verify the candidate version
matches the locked version for this deployment in the version group. If so,
return
Allowed.
New policy rule: resource concurrency
A resource concurrency rule limits how many resources in a group can simultaneously be undergoing deployment. The group is defined by a CEL selector, and the limit can be a percentage or absolute count.Schema
Evaluator
Enhancement: scoped deployment dependencies
Today, a deployment dependency rule on a policy applies to every release target matched by the policy’sselector. This forces separate policies for each
ordering constraint in a bracket. Adding an optional applies_to field lets
multiple dependency rules coexist on one policy:
NULL, the rule applies to all matched release targets (current behavior).
When set, the rule is only evaluated for release targets whose deployment
matches the CEL expression:
Bracket-aware gradual rollout
When a bracket rule exists on the same policy as a gradual rollout rule, the rollout evaluator should hash onresourceId + bracketRuleId instead of the
individual release target key. This ensures all deployments in a bracket for a
given resource receive the same rollout position.
The gradual rollout evaluator already receives the full policy context through
the store. The change is in the hash input for rollout position calculation:
Evaluator chain integration
Both new evaluators slot into the existing factory:Allowed for a job to be
created. The evaluators gate at different levels:
| Evaluator | Gates at | Question it answers |
|---|---|---|
| Deployment bracket | Version group | Are all sibling deployments ready? |
| Gradual rollout | Resource ordering | Is it this resource’s turn? |
| Resource concurrency | Cluster capacity | Is there a concurrency slot? |
| Deployment dependency | Execution ordering | Have upstream deployments succeeded? |
Pre/post hooks as deployments
Rather than introducing a separate hook mechanism, lifecycle hooks (drain, uncordon) are modeled as regular deployments that are bracket members with dependency ordering. A “drain” deployment’s job agent runskubectl drain. An
“uncordon” deployment’s job agent runs kubectl uncordon. Deployment dependency
rules control execution order within the bracket.
This reuses the existing job agent, job dispatch, retry, rollback, and
verification infrastructure. Hooks get observability (traces, job status) and
policy controls (retry on failure, rollback) for free.
Examples
Node upgrade with ordered bracket
A cluster has 10 nodes. Three workload deployments (kubelet, containerd, os-patch) plus two lifecycle deployments (drain, uncordon) are tagged withmetadata.layer = "node".
os-patch must run before kubelet and containerd. All five share a 24-hour
collection window and a 20% concurrency limit.
Policy configuration:
Different clusters, different policies
The same deployments can have different bracket configurations per cluster by using the policy selector to scope rules:Partial readiness
Only kubelet gets a new version within the 24-hour window. containerd and os-patch have no updates.Overlapping version groups
A new version group becomes ready while the previous group is still executing.Failure mid-bracket
kubelet fails on node-3 while containerd succeeds.Migration
- The
policy_rule_deployment_bracketandpolicy_rule_resource_concurrencytables are new. No data migration required. - The
bracket_version_groupandbracket_version_group_membertables are new. - The
applies_tocolumn onpolicy_rule_deployment_dependencyis additive and nullable. Existing rules haveapplies_to = NULL, preserving current behavior (rule applies to all matched release targets). - The new evaluators return
nilfrom their factory functions when the policy rule does not contain the relevant configuration, following the same pattern as all existing evaluators. - Agents that execute lifecycle hooks (drain, uncordon) are standard job agents. No new agent interfaces are needed.
Open Questions
- Collection window trigger semantics. Should the collection window start when the first member’s version is published or when the first member’s version passes other policy rules (approval, version selector)? Starting at publication is simpler but means the window runs concurrently with approval — a 24h window with a 20h approval process only leaves 4h of actual collection time.
- Gradual rollout interaction. The proposal changes the hash input when a bracket rule is present. This means adding or removing a bracket rule changes the rollout order for all targets. Should the bracket-aware hashing be opt-in to avoid surprising rollout order changes?
-
Bracket membership dynamism. The
deploymentSelectoron the bracket rule is evaluated dynamically. If a new deployment is added mid-rollout that matches the selector, should in-progress version groups absorb it? The simplest behavior is to only affect future version groups. -
Merge strategy completeness. The
mergeoverlap strategy avoids double drain cycles but requires the dependency evaluator to distinguish between “upstream succeeded with Group A’s version” and “upstream succeeded with Group B’s version.” The current evaluator checks success status but not which version succeeded. This may need a version-aware dependency check for merge to work correctly. - Failure blast radius. A bracket failure on one resource holds a concurrency slot. With 20% concurrency on a 10-node cluster, 2 stuck nodes block the entire rollout. Should there be a configurable timeout that auto-releases concurrency slots after a bracket has been stuck for too long, even if the resource is in an unknown state?
-
Hook idempotency. The
queueoverlap strategy runs drain → uncordon → drain → uncordon for consecutive groups. This assumes drain and uncordon are idempotent. Themergestrategy assumes the resource remains in a drained state between groups. Should the bracket rule have an explicit field declaring whether hooks are idempotent and/or whether the prepared state persists? -
Auto rollback interaction. If a rollback policy triggers for one member
deployment mid-bracket (e.g., kubelet fails health checks and auto-rolls
back to v1.28.x), the resource is in a partially-upgraded state — some
bracket members succeeded with new versions, others rolled back. Several
sub-questions arise:
- Should a rollback of any bracket member trigger a rollback of all bracket members on that resource to restore a consistent version set? This is the safe default for tightly coupled components (kubelet + containerd), but overly aggressive for loosely coupled ones.
- If only the failed member rolls back, do post-hooks (uncordon) still run? The dependency graph may be satisfied (kubelet “completed” via rollback, containerd succeeded), but the resource is in a mixed state that the operator may not have intended to uncordon.
- Should the bracket rule have a
rollback_strategyfield (e.g.,individual,all_members,halt_and_notify) that controls whether rollback is scoped to the failing member, cascaded to the full bracket, or paused for manual intervention? - How does a bracket-wide rollback interact with the concurrency limit? Rolling back N members on a resource means N additional jobs — does each count against the concurrency slot, or does the resource’s existing slot cover the rollback work?