Running Cue on a Server

​

Before you start

• Agents run unattended with full permissions. Cue always runs agents in YOLO mode, and on a server nobody watches the run. Give the engine its own machine or container, the dedicated maestro user both installs create, and workspaces that hold nothing else.
• Secrets never travel in a bundle. A bundle carries the names of the secrets its agents and webhooks need, never the values. You supply each value on the server.
• Sign agents in on the server. Each provider is signed in on the server, either with an API key supplied as a secret or by running that provider’s own login there, as the maestro user. Do not copy a login from your desktop.
• Claude agents run in API mode. The server bundle does not include the driver for Claude’s TUI mode, so a Claude agent exported in TUI or Dynamic mode runs claude --print on the server.
• Webhooks stay on loopback. The webhook listener is never exposed directly. Public delivery goes through a reverse proxy or a tunnel (see Webhooks behind a proxy).
• One engine per data directory. Do not point two servers or containers at the same data directory or volume.

maestro-cli bundle export --pipeline "Nightly Review" --output pipeline.zip
maestro-cli bundle export --agent "Coder" --output agent.zip

Error: Refusing to export: the Cue config would fail bundle validation with 1 error. Fix it and export again.
  [cue-config-invalid] workspace "review": Subscription "gather": "source_sub" is required for agent.completed subscriptions when action is "command"

​

Build the server bundle

npm run build:server

​

Install

​

Option 1: Linux VM with systemd

scp dist/maestro-server-<version>.tgz my-server:
ssh my-server
tar -xzf maestro-server-<version>.tgz
sudo ./maestro-server/install.sh

sudo ./maestro-server/install.sh --agent-cli @anthropic-ai/claude-code --agent-cli opencode-ai

error npm cannot fetch from the registry https://registry.npmjs.org/ (ENOTFOUND). Check the network, the proxy (HTTPS_PROXY, or npm config get https-proxy) and the registry (npm config get registry)

​

Option 2: Container

docker build -f packaging/server/Dockerfile -t maestro-cue .

AGENT_CLIS="@anthropic-ai/claude-code opencode-ai" docker compose up -d --build

docker compose stop
for v in home work; do
  docker run --rm --user 0 --entrypoint sh \
    -v server_maestro-$v:/from -v maestro-$v:/to maestro-cue -c 'cp -a /from/. /to/'
done
docker compose up -d

​

Sign agents in

​

Declared secrets

1. $CREDENTIALS_DIRECTORY/<NAME>, a systemd credential (LoadCredential=)
2. /run/secrets/<NAME>, a Docker or Compose secret
3. the environment variable <NAME>

sudo install -m 0600 /dev/null /etc/maestro/credentials/ANTHROPIC_API_KEY
sudo nano /etc/maestro/credentials/ANTHROPIC_API_KEY
sudo systemctl edit maestro-cue

[Service]
LoadCredential=ANTHROPIC_API_KEY:/etc/maestro/credentials/ANTHROPIC_API_KEY

chmod 0700 secrets && chmod 0644 secrets/GH_WEBHOOK_SECRET

​

Variables server mode lets through

• system variables (PATH, HOME, USER, LANG, TMPDIR and the like), proxy and TLS trust variables (HTTPS_PROXY, NODE_EXTRA_CA_CERTS and the like), and common toolchain homes;
• every name starting with LC_, XDG_, SSH_ or MAESTRO_;
• the provider names Maestro knows: ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ANTHROPIC_BASE_URL, ANTHROPIC_MODEL, CLAUDE_CONFIG_DIR, CLAUDE_CODE_MAX_OUTPUT_TOKENS, MAX_THINKING_TOKENS, CLAUDE_CODE_USE_BEDROCK, CLAUDE_CODE_USE_VERTEX, CODEX_HOME, OPENAI_API_KEY, OPENAI_BASE_URL, OPENCODE_CONFIG, OPENCODE_CONFIG_CONTENT, FACTORY_API_KEY, FACTORY_PROJECT_DIR, FACTORY_DISABLE_KEYRING, FACTORY_LOG_FILE, COPILOT_HOME, COPILOT_GITHUB_TOKEN, GH_TOKEN, GITHUB_TOKEN;
• the names you list, comma separated, in MAESTRO_SERVER_ENV_ALLOW;
• the agent’s declared secrets.

MAESTRO_SERVER_ENV_ALLOW=GEMINI_API_KEY,AWS_PROFILE,AWS_REGION

​

Provider logins

sudo -iu maestro
claude          # then type /login and follow the link it prints
exit

docker run --rm -it -v maestro-home:/var/lib/maestro --entrypoint claude maestro-cue
docker exec -it maestro-cue codex login --device-auth

​

Per provider

• Claude Code. An API key takes precedence over a login. For Bedrock or Vertex, set CLAUDE_CODE_USE_BEDROCK=1 or CLAUDE_CODE_USE_VERTEX=1 and add the cloud credentials’ names to MAESTRO_SERVER_ENV_ALLOW. The installer and the image install Claude Code by default.
• Codex. Whether a given Codex version reads OPENAI_API_KEY from the environment on its own is not verified here. Codex documents storing a key with its login instead, printenv OPENAI_API_KEY | codex login --with-api-key, run as the maestro user (not verified here). Either way the result is kept in ~/.codex/auth.json.
• Copilot-CLI. GH_TOKEN and GITHUB_TOKEN are also what gh and Cue’s GitHub triggers use, and in maestro.env they reach every agent. Give Copilot its own COPILOT_GITHUB_TOKEN, and keep Cue’s GitHub token in a credential file (next section).
• Rows marked “Not verified”. Maestro knows the login command, but not which variable the CLI reads or where it stores the login. Check that provider’s documentation, then prove the sign-in with one turn (see Run an exported agent on demand).

​

GitHub for Cue triggers

• A token file keeps the token away from agents: /etc/maestro/credentials/GH_TOKEN with LoadCredential=GH_TOKEN:/etc/maestro/credentials/GH_TOKEN, or a GH_TOKEN Docker secret. It is read again on every poll, so a rotated file takes effect without a restart, and it takes precedence over gh auth login.
• A login: sudo -iu maestro then gh auth login, or docker exec -it maestro-cue gh auth login. It is kept in the maestro user’s home.
• GH_TOKEN in maestro.env works too, but every agent receives it.

​

Import

​

Import on a VM

# The workspace the agents work in
sudo -H -u maestro git clone https://github.com/acme/app.git /srv/maestro/app

# A pipeline
sudo -H -u maestro maestro-cli bundle import /tmp/pipeline.zip --workspace app=/srv/maestro/app

# An agent, with its playbooks and Auto Run documents
sudo -H -u maestro maestro-cli bundle import /tmp/agent.zip --workspace app=/srv/maestro/app

​

Import in a container

docker run --rm -v maestro-work:/srv/maestro --entrypoint git \
  maestro-cue clone https://github.com/acme/app.git /srv/maestro/app

docker run --rm \
  -v maestro-home:/var/lib/maestro -v maestro-work:/srv/maestro \
  -v "$PWD/pipeline.zip:/tmp/pipeline.zip:ro" \
  maestro-cue bundle import /tmp/pipeline.zip --workspace app=/srv/maestro/app

docker run --rm \
  -v maestro-home:/var/lib/maestro -v maestro-work:/srv/maestro \
  -v "$PWD/agent.zip:/tmp/agent.zip:ro" \
  maestro-cue bundle import /tmp/agent.zip --workspace app=/srv/maestro/app

sudo -H -u maestro maestro-cli list agents
sudo -H -u maestro maestro-cli list playbooks

docker run --rm -v maestro-home:/var/lib/maestro maestro-cue list agents

​

Check

# VM, with the service's env file loaded
sudo -H -u maestro sh -c 'set -a; . /etc/maestro/maestro.env; set +a; exec maestro-cli cue engine check'

# Container
docker run --rm --env-file maestro.env \
  -v maestro-home:/var/lib/maestro -v maestro-work:/srv/maestro \
  maestro-cue cue engine check

• Secrets in credential files. On a VM, LoadCredential= files are visible to the service alone, so a check from a shell reports them as missing. The service runs the same check when it starts (--require-ready) and logs every gap it still finds. To run the check with the service’s credentials before you start it, use systemd-run with the unit’s user, env file and one LoadCredential= per drop-in line:
  sudo systemd-run --quiet --pipe --wait --collect \
    -p User=maestro -p Group=maestro -p WorkingDirectory=/var/lib/maestro \
    -p Environment=HOME=/var/lib/maestro -p Environment=MAESTRO_SERVER_MODE=1 \
    -p EnvironmentFile=-/etc/maestro/maestro.env \
    -p LoadCredential=GH_WEBHOOK_SECRET:/etc/maestro/credentials/GH_WEBHOOK_SECRET \
    /usr/local/bin/maestro-cli cue engine check --data-dir /var/lib/maestro/data
  
• Disabled subscriptions are ignored. A subscription with enabled: false adds no gap: it needs no gh, webhook secret, provider binary or target agent until it is enabled. A cue.yaml that does not parse is still reported.
• One owner per workspace. When several agents share a workspace, only the owner runs its unpinned subscriptions, so only the owner is checked for them: the agent named by settings.owner_agent_id (id, or display name), else the first agent listed. If owner_agent_id matches no agent there, or more than one by name, the engine runs none of them and the check reports it as a cue-config gap naming the setting and the workspace. A subscription with agent_id runs only on that agent, and only when the agent’s own workspace holds it; one pinned to an agent in another workspace is reported as unknown-agent. An exported agent that does not own its workspace brings only the subscriptions pinned to it and no owner_agent_id, so it runs on the server what it ran on the desktop (see Agents that share a workspace).
• Nothing to run. A data directory with no agents, or with agents but no enabled subscription on any of them, has a nothing-to-run gap: the check exits 1, --require-ready refuses to start, and /readyz answers 503. That is almost always a bundle that was never imported. An exported agent brings its own subscriptions, if it had any. A server that only runs agents on demand needs no engine: leave the service off.
• install.sh --enable runs the check before it enables the service. On nothing-to-run it still installs everything, leaves the service disabled, says why, and exits 3. Any other gap is left to the service, which judges secrets in its own environment.
• Not checked: whether a provider or gh is signed in. Run one turn to prove a sign-in (see Run an exported agent on demand).

​

Start

​

Start on a VM

sudo systemctl enable --now maestro-cue
journalctl -u maestro-cue -f

• It is ready (systemctl status shows active) once agents are loaded, the lock is held and triggers are armed. A missing agent binary, secret or gh stops the start, and the log lists every gap. Restart=on-failure then tries again every 5 seconds, logging the gaps each time, until you fix them or run sudo systemctl stop maestro-cue. Start-up may take up to 60 seconds.
• A watchdog restarts it if the engine stops answering for 30 seconds or loses its lock to another engine (the journal shows Watchdog timeout and the engine is killed with SIGABRT), and it restarts after a crash, 5 seconds later. When the engine dies either way, systemd kills the rest of the service with it, including agent turns and shell steps still running; the restarted engine marks those runs failed and runs what was queued.
• It never writes a core dump (LimitCORE=0), and neither do its agents, which inherit the limit. The engine holds webhook secrets, tokens and provider keys in memory, and a core file would put them on disk; the watchdog’s SIGABRT above would otherwise leave one.
• It runs with a read-only system. Only /var/lib/maestro and /srv/maestro are writable. For workspaces elsewhere, add a drop-in with sudo systemctl edit maestro-cue:

[Service]
ReadWritePaths=/home/me/projects

​

Start in a container

docker run -d --name maestro-cue --restart unless-stopped --stop-timeout 120 \
  --ulimit core=0 --env-file maestro.env \
  -v maestro-home:/var/lib/maestro -v maestro-work:/srv/maestro \
  -p 127.0.0.1:17997:17997 \
  maestro-cue

• Stop timeout. The engine drains active runs for up to 90 seconds on stop, but docker stop waits only 10 seconds unless the container was created with a longer stop timeout. The --stop-timeout 120 above and Compose’s stop_grace_period: 120s set one, so a plain docker stop of either container drains in full; for a container created without it, use docker stop -t 120. After 10 seconds Docker kills the engine in the middle of the drain (exit code 137): the run in flight is marked failed at the next start, and the lock is left behind (see “Restart after a hard kill”). The image runs tini without -g, so the stop signal reaches the engine alone and runs in flight can finish.
• No core dumps. --ulimit core=0 (ulimits: core: 0 in Compose) keeps a crash from writing the engine’s memory, with its secrets and keys, to disk. The image cannot set it; Docker’s default comes from the daemon and often allows core files. Add it to any other docker run that is given secrets, such as an on-demand run.
• One engine per volume. A second engine started on the same data volume, in another container, sees the first one’s lock while its heartbeat is fresh, logs that a Cue engine in another PID namespace (another container) holds the lock, and exits 1.
• Restart after a hard kill. If the engine is killed without stopping (docker kill, out of memory, a host crash), its lock stays behind. When the new engine can tell the lock was left by an earlier process with its own process id in its own PID namespace, it takes the lock at once. A container restarted by its restart policy normally gets both back (same namespace number, same process id), so that is the usual case. When it cannot tell, the lock may belong to a live engine in another container, so it waits for that lock to go quiet: it refuses to start for up to 3 minutes, says when the lock frees, and the restart policy brings it back after that.
• Health. The image’s health check calls /healthz on the engine’s status port, 7433, inside the container. docker ps shows healthy once it answers. If the engine stops answering, the container turns unhealthy after three failed checks (about 90 seconds), but Docker does not restart it: a restart policy acts only when the container exits. Restart it yourself (docker restart -t 120 maestro-cue), or run it under something that acts on health.
• Webhook port. Inside the container the webhook listener listens on all interfaces so the port can be published. Publish it to the host’s loopback only (127.0.0.1:17997:17997), as above.

​

Run an exported agent on demand

mrun() { sudo -H -u maestro sh -c 'set -a; . /etc/maestro/maestro.env; set +a; exec maestro-cli "$@"' maestro-cli "$@"; }

mrun send <agent> "Reply with the word OK" --read-only
mrun send <agent> "Now list the files you would change" --session <sessionId>
mrun list playbooks --agent <agent>
mrun playbook <playbook-id> --json
mrun run-doc /srv/maestro/app/docs/release-checklist.md --agent <agent> --json
mrun goal-run <agent> "Make the test suite pass" --exit-criteria "npm test exits 0" --max-iterations 5 --json

docker exec maestro-cue maestro-cli send <agent> "Reply with the word OK" --read-only
docker exec maestro-cue maestro-cli playbook <playbook-id> --json

docker run --rm --ulimit core=0 --env-file maestro.env \
  -v maestro-home:/var/lib/maestro -v maestro-work:/srv/maestro \
  maestro-cue send <agent> "Reply with the word OK" --read-only

• Environment. An on-demand run passes the command’s whole environment to the agent; server mode filters only what the engine passes. Declared secrets are looked up the same way, but on a VM the service’s LoadCredential= files are not visible to the command, so supply those as variables or use a provider login.
• Records. Each turn is recorded in the data directory’s run ledger with its token usage. playbook, run-doc and goal-run also write history entries; send writes none.
• No triggers. Nothing fires on its own. To fire a subscription by hand while the engine runs, use maestro-cli cue trigger <subscription-name> (in a container, docker exec maestro-cue maestro-cli cue trigger <subscription-name>). Like a real tick, it also fires every subscription in the same pipeline with identical trigger settings; give a subscription a pipeline of its own, or different trigger settings, to fire it alone (see Triggering a Subscription).

​

Webhooks behind a proxy

• forwards POST /cue/... to http://127.0.0.1:17997 with the path unchanged;
• passes the body byte for byte, since signatures are computed over the raw body;
• passes the signature header (X-Hub-Signature-256 for GitHub, or the signature_header the subscription names), X-Maestro-Cue-Secret or Authorization for senders that present the secret, and the delivery id headers X-GitHub-Delivery, X-Request-Id and X-Maestro-Delivery;
• does not set X-Request-Id itself. The listener uses it as the delivery id when the sender sends no X-GitHub-Delivery, and a new id on every request defeats the protection against repeats;
• allows a body of 1 MiB, the listener’s cap (it answers 413 above that);
• waits long enough for the answer. The listener answers only after it has handled the delivery (see What a 2xx means). 60 seconds is ample; GitHub itself gives up after 10.

​

nginx

server {
    listen 443 ssl;
    server_name hooks.example.com;
    ssl_certificate     /etc/letsencrypt/live/hooks.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/hooks.example.com/privkey.pem;

    location /cue/ {
        proxy_pass http://127.0.0.1:17997;
        proxy_http_version 1.1;
        client_max_body_size 1m;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
    }

    location / {
        return 404;
    }
}

​

Caddy

hooks.example.com {
	handle /cue/* {
		request_body {
			max_size 1MiB
		}
		reverse_proxy 127.0.0.1:17997 {
			transport http {
				response_header_timeout 60s
			}
		}
	}
	handle {
		respond 404
	}
}

​

A tunnel

cloudflared tunnel --url http://127.0.0.1:17997
ngrok http 127.0.0.1:17997

​

The container case

​

Test it

SECRET='the value of the secret_env variable'
body='{"hello":"world"}'
sig=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/^.* //')
curl -i https://hooks.example.com/cue/<path> \
  -H 'Content-Type: application/json' \
  -H "X-Hub-Signature-256: sha256=$sig" \
  -H 'X-Maestro-Delivery: test-1' \
  --data-binary "$body"

​

What a 2xx means

• A crash (kill -9, out of memory, a host reboot) followed by a restart loses nothing. A queued delivery runs after the restart; a run that was in progress is marked failed at the next start.
• systemctl stop and docker stop drain: a delivery still being checked when the stop begins is finished and queued for the next start, within the drain timeout.
• A redelivery of the same id fires nothing for 24 hours, and a GitHub change seen by both a webhook and a poll fires once.
• A cue.yaml change while the delivery waits (a hand edit, maestro-cli cue model) keeps it queued while its subscription is still in the file and switched on. If the change removes, renames or switches off that subscription, the delivery is dropped, logged (Dropped N queued event(s) ... on a cue.yaml reload) and recorded as never run. A change that fails to parse keeps the queue until the next good save.

​

Limits

• From the stop signal until the engine exits, including after a second stop signal cuts the drain short, the listener stays open: a delivery to a path it listens on (with the right secret or signature) is answered 503 with Retry-After: 30 and nothing is recorded, so the sender should retry. The listener closes as the engine exits. A stopped engine has no listener at all, and the connection is refused (a proxy answers 502).
• The SusFactor check runs before the answer, so a slow 0DIN call delays it (each call gives up after 3 to 5 seconds). A sender that stops waiting first (GitHub waits 10 seconds) records a failure although the delivery runs; its retry is then a duplicate and fires nothing.
• A full queue drops its oldest event, and queue_size: 0 drops a delivery that arrives while the agent is busy. A subscription with no prompt, or a fan-out target that does not exist, does not run. Each of these is logged, and the delivery still counts as handled.
• After a crash (not a drain), a queued delivery that has waited longer than its subscription’s timeout_minutes by the time Maestro is back is dropped as stale and shown as timed out in the activity log. A drain stamps the queue so that rule does not apply.
• The delivery id is recorded in the same step that hands the delivery on. A crash at exactly that instant can let one redelivery fire again.
• A sender that sends no delivery id header (X-GitHub-Delivery, X-Request-Id or X-Maestro-Delivery) gets no protection against repeats.

​

Operate

​

Health and status

# VM
curl -s http://127.0.0.1:7433/readyz

# Container (the image has no curl)
docker exec maestro-cue node -e "fetch('http://127.0.0.1:7433/status').then(r=>r.text()).then(console.log)"

​

Logs

​

Stop

​

Sleep and pause

​

Add or replace agents and pipelines

# VM
sudo systemctl stop maestro-cue
sudo -H -u maestro maestro-cli bundle import /tmp/new.zip --workspace app=/srv/maestro/app
sudo -H -u maestro sh -c 'set -a; . /etc/maestro/maestro.env; set +a; exec maestro-cli cue engine check'
sudo systemctl start maestro-cue

# Container
docker stop -t 120 maestro-cue
docker run --rm \
  -v maestro-home:/var/lib/maestro -v maestro-work:/srv/maestro \
  -v "$PWD/new.zip:/tmp/new.zip:ro" \
  maestro-cue bundle import /tmp/new.zip --workspace app=/srv/maestro/app
docker start maestro-cue

​

Do not open the desktop app on the server’s data directory

​

Upgrade

docker build -f packaging/server/Dockerfile -t maestro-cue .
docker stop -t 120 maestro-cue
docker rm maestro-cue

​

Known limits

• Run history without output. maestro-cli cue activity works on a server: with the desktop app not running it reads cue.db read-only and lists the most recent runs (up to the last 1,000), newest first. The database keeps no run output, so for that read the engine’s log (each run logs Run started and Run finished with its runId). A run that was still in flight when its engine stopped is listed as failed. While an engine is running on that directory, a run that has not finished yet is listed as running, with how long it has run so far (no duration in --json). After a hard kill, a read from outside the stopped container (a docker run on its volumes) can still list that run as running for up to 3 minutes, as cue engine status still says Running, until the dead engine’s heartbeat goes stale; once the engine restarts the run reads failed, with the downtime counted in its duration. On a VM, run it as the maestro user with --data-dir /var/lib/maestro/data (the container sets the directory already). It refuses to read a cue.db-wal left without its cue.db-shm by a process that was killed, until an engine has started on that directory again. With no engine running, the read copies cue.db into memory; one that overlaps an engine starting on the same directory could fail or show wrong rows (not seen so far), so run it again.
• No log level in JSON mode. Debug output is left out of the JSON logs and cannot be turned on there (see Logs).
• After a hard kill. The restarted engine normally takes the lock at once, on a VM and in a container restarted by Docker. It refuses to start for up to 3 minutes, while the old lock goes stale, only when it cannot tell the lock is its predecessor’s (see Start in a container). Its refusal says whether the lock belongs to a different running process, to an engine that had this process’s own id and is gone, or to another container, and how long until the lock frees. Once it starts it logs one Engine was not running for Nm line for the gap.
• GitHub triggers quiet for 30 days. What a GitHub trigger has already seen is kept for 30 days, so after 30 days with no new items a subscription treats its next poll as its first: it records the items it finds and runs none of them.
• Stop over SSH for shell steps. Stopping a Cue action: command step that runs on an SSH remote stops only the local ssh client, not the command on the remote host. Agent runs over SSH are stopped on the remote.
• GitHub triggers need repo:. A github.pull_request, github.issue or github.label subscription without repo: owner/name never polls or receives webhooks: the engine does not detect the repository from the checkout. cue engine check (and --require-ready) reports each such enabled subscription as a github-repo-missing gap. A manual run (cue trigger, or Play in the desktop dashboard) still dispatches it, with no GitHub item. Set repo: on every GitHub trigger.
• One runner per data directory. Do not run maestro-cli host start and the maestro-cue service (or cue engine start) on the same data directory. They do not keep each other out in every case, and cue engine stop there stops the whole host, not just Cue. Use the service for Cue on a server.