RFC 0014: Pull-Based Job Agent API

Documentation Index

Fetch the complete documentation index at: https://docs.ctrlplane.dev/llms.txt

Use this file to discover all available pages before exploring further.

​

Summary

• V1 delivers the pull contract: an agent polls for the next job, claims it atomically (at most once), runs it, and reports status. A new queued job status marks a job as claimable.
• V2 adds crash recovery: a lease, a heartbeat endpoint, and a reaper that returns abandoned jobs to the queue. V2 is purely additive — V1 is shippable and useful on its own.

​

Motivation

• cannot (or should not) be reached inbound by ctrlplane, and
• wants to integrate generically over HTTP rather than through a bespoke, per-system integration.

​

Proposal

​

Model: producer / consumer

• ctrlplane produces the job and marks it claimable.
• the external agent consumes it by polling, claiming, and running it.

​

Job status: queued

ALTER TYPE job_status ADD VALUE 'queued' AFTER 'pending';

queued ───claim (poll)───► in_progress ───report───► successful / failure

• pending — created, not yet processed by the dispatch controller.
• queued — prepared, waiting for an agent to claim it.
• in_progress — claimed by an agent and executing.

​

Agent type: http-pull

package httppull

var _ types.Dispatchable = &HttpPull{}

func (a *HttpPull) Type() string { return "http-pull" }

func (a *HttpPull) Dispatch(ctx context.Context, job *oapi.Job) error {
    return a.setter.UpdateJob(ctx, job.Id, oapi.JobStatusQueued, "", nil)
}

​

Verifications

​

Claim endpoint (V1)

GET /v1/workspaces/{workspaceId}/job-agents/{jobAgentId}/jobs/next

UPDATE job
SET status = 'in_progress', started_at = now()
WHERE id = (
  SELECT id FROM job
  WHERE status = 'queued' AND job_agent_id = $1
  ORDER BY created_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING *;

​

Job payload

​

Status reporting

PUT /v1/workspaces/{workspaceId}/jobs/{jobId}/status

​

Authentication (V1)

​

Concurrency

1. Double-pickup — handled by the atomic claim above. A job is handed out at most once, even under overlapping requests or client retries.
2. Crash mid-job — not handled in V1. If an agent claims a job and dies, the job remains in_progress. Recovery is a manual transition back to queued (the same transition V2 automates). V2 adds automatic recovery.

​

V1 implementation surface

​

V2: Lease, Heartbeat, and Reclaim (add-on)

​

Claim table

CREATE TABLE job_claim (
  job_id           uuid PRIMARY KEY REFERENCES job(id) ON DELETE CASCADE,
  job_agent_id     uuid NOT NULL,
  claimed_at       timestamptz NOT NULL DEFAULT now(),
  claim_expires_at timestamptz NOT NULL,
  claim_id         uuid NOT NULL DEFAULT gen_random_uuid()
);

​

Lease

WITH claimed AS (
  UPDATE job SET status = 'in_progress', started_at = now()
  WHERE id = (
    SELECT id FROM job
    WHERE status = 'queued' AND job_agent_id = $1
    ORDER BY created_at LIMIT 1
    FOR UPDATE SKIP LOCKED
  )
  RETURNING id
)
INSERT INTO job_claim (job_id, job_agent_id, claim_expires_at)
SELECT id, $1, now() + make_interval(secs => $lease_seconds) FROM claimed
RETURNING *;

​

Heartbeat

POST /v1/workspaces/{workspaceId}/jobs/{jobId}/heartbeat

UPDATE job_claim
SET claim_expires_at = now() + make_interval(secs => $lease_seconds)
WHERE job_id = $1;

​

Reaper

WITH expired AS (
  DELETE FROM job_claim WHERE claim_expires_at < now() RETURNING job_id
)
UPDATE job SET status = 'queued'
WHERE id IN (SELECT job_id FROM expired) AND status = 'in_progress';

​

Reclaim and double-run

​

V2 implementation surface

​

Migration

• V1 adds the queued value to the job_status enum. V2 adds a new job_claim table and does not modify job. Both are additive; existing jobs are unaffected.
• The dispatch controller, eligibility logic, and promotion lifecycle are unchanged except for recognizing the queued status and the http-pull agent’s Dispatch body.
• The status-reporting endpoint is reused unchanged. The claim endpoint (V1) and heartbeat endpoint (V2) are new and do not alter existing endpoints.
• The V2 reaper only acts on jobs that have a job_claim row, so introducing it does not change the behavior of any agent that does not heartbeat.

​

Open Questions

1. Long-poll vs. plain poll. V1 can return immediately when no job is available. A long-poll variant (hold the request open until a job appears or a timeout elapses, bounded by a server-enforced maximum) reduces idle polling and is a candidate for V2. Backpressure and fairness limits on held connections are open.
2. Verification timing. Verifications begin measuring when created (at dispatch), which for a pull agent can precede the claim by an unbounded queue wait. For long verification windows this is harmless; a short window could complete before the agent claims the job. If this becomes a problem, verification creation can be moved to the claim transition, or measurement can be gated on the job reaching in_progress. Deferred until needed.
3. Lease configuration. Should the lease duration be per-agent (job_agent.config, bounded) or a single global default? A global default is the V2 starting point; per-agent is a later refinement for agents with different reliability characteristics.

​

AI Generated Questions

1. Agent registration. A job is only routed to an agent that already exists and is matched by a deployment’s jobAgentSelector. Should an external agent be able to self-register its job_agent row and credentials via the API, or must agents be pre-provisioned by an operator?
2. Per-agent authentication. V1 reuses x-api-key. V2 should issue a per-agent credential at registration so an agent authenticates as itself and can claim only its own jobs. What is the token model and rotation story?
3. Fencing. Should V2 include a fencing token from the start, or add it only if double-run under lease expiry proves to be a real problem for the workloads pull agents run?