Documentation
Explicit shell sessions in worker pools.
Overview
The server allocates a worker to a session in a named profile. A session has one persistent Bash shell, one identity, and one worker until release. Stateful commands run in that shell, one at a time, and keep its directory, variables, and exports. Parallel commands run in child shells that overlap other commands; they cannot change the shell's state. Commands start in submission order, and every command waits for the stateful commands submitted before it (see Command ordering). A terminal can attach, detach, and attach again without releasing the worker.
Workers stream stdout and stderr over their authenticated dispatch WebSocket. The server spools output locally for HTTP range reads. No capture volume is shared between server and workers. The worker uses temporary files inside its own container while a command runs and removes them when the command finishes.
Quickstart
Set the API and dispatch tokens on the server, and the dispatch URL, token, unique worker id, and profile on each worker. Then create a session, finish a command, and release it:
set -e
base=http://localhost:8080/api/v1
until curl -fsS "$base/ready" >/dev/null 2>&1; do sleep 1; done
sid=$(curl -fsS -H "Authorization: Bearer $SHIITAKE_AUTH_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"profile":"default","env":{"PATH":"/usr/bin:/bin"}}' \
"$base/sessions" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_id"])')
handle=$(curl -fsS -H "Authorization: Bearer $SHIITAKE_AUTH_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"command":"echo hello","mode":"stateful"}' \
"$base/sessions/$sid/commands" |
python3 -c 'import json,sys; print(json.load(sys.stdin)["handle"])')
while :; do
state=$(curl -fsS -H "Authorization: Bearer $SHIITAKE_AUTH_TOKEN" \
"$base/sessions/$sid/commands/$handle" |
python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')
[ "$state" = running ] || break
sleep 0.2
done
if [ "$state" = completed ]; then
curl -fsS -H "Authorization: Bearer $SHIITAKE_AUTH_TOKEN" \
"$base/sessions/$sid/commands/$handle/stdout"
else
echo "command ended as $state"
fi
curl -fsS -X DELETE -H "Authorization: Bearer $SHIITAKE_AUTH_TOKEN" \
"$base/sessions/$sid"
Release each session after its commands finish; release cancels any jobs still running and resets the worker. An idle session is reclaimed after one hour as a backstop for abandoned clients. The worker clears its environment before the shell starts, so supply PATH when using external binaries.
HTTP API
All public routes are under /api/v1. Only /health and /ready are unauthenticated.
| Method | Path | What it does |
|---|---|---|
| GET | /profiles | Connected profiles and idle/allocated counts. |
| POST | /sessions | Allocate a worker and start its shell. Returns 201 on success, 400 for invalid input, 404 for an unknown profile, 429 when the selected profile is busy, or 502 if the worker fails during creation. |
| GET, DELETE | /sessions/{id} | Inspect or release a session. |
| POST | /sessions/{id}/commands | Submit a stateful or parallel command. Returns a handle. |
| GET | /sessions/{id}/commands | List retained commands in the session, oldest first, with status and output byte counts. |
| GET, DELETE | /sessions/{id}/commands/{handle} | Inspect or cancel a command. |
| GET | /sessions/{id}/commands/{handle}/stdout | Read stdout, including HTTP Range requests. |
| GET | /sessions/{id}/commands/{handle}/stderr | Read stderr by byte range. |
| GET | /sessions/{id}/terminal | Attach an interactive WebSocket terminal. |
Session creation accepts profile (default default), workdir, env, drop_to, cols, and rows. The environment is limited to 128 entries and 64 KiB across keys and values; workdir is limited to 4096 bytes. The drop_to object accepts uid, gid, supplementary_gids, umask, and optionally name and create_home. Identity cannot change during a session. Session IDs begin with s- and command handles with c-; each has 16 lowercase letters or digits from an unambiguous alphabet. Treat them as opaque random IDs.
A command request is {"command":"echo hello","mode":"stateful"}. The mode defaults to stateful; parallel starts a separate child shell. Commands are shell strings with no wall-time deadline. For a long job of unknown duration, submit {"command":"echo started; sleep 3600; echo finished","mode":"parallel"}, list handles with GET /sessions/{id}/commands, poll GET /sessions/{id}/commands/{handle}, and read /stdout while it runs. Send DELETE /sessions/{id}/commands/{handle} to cancel, poll until terminal, then release the session. Handle status includes exit code, exit cause, and output byte counters. Output reads support Range: bytes=0-1023 and Range: bytes=-1024.
Command ordering
Commands in a session start in the order they are submitted:
- A stateful command runs in the session shell. It starts once every stateful command submitted before it has finished. Its stdin is the session terminal.
- A parallel command runs in its own child shell. It also waits for every stateful command submitted before it, then starts with the working directory and exported variables that command left. It does not wait for earlier parallel commands, and later commands do not wait for it. Its stdin is empty.
- A command waiting to start reports status
running.DELETEon it cancels it before it starts. - A stateful command that does not finish, such as a foreground server,
cat, or a command waiting at a prompt, holds back every command submitted after it, parallel ones included, until it exits or is cancelled. Start long-lived processes with"mode":"parallel"or in the background (server > log 2>&1 &), and answer prompts through the terminal. - At most 32 commands per session can be running or waiting. A submission beyond that ends with status
error.
For example, submitting cd /repo (stateful) and then make test (parallel) without waiting runs make test in /repo, after the cd finishes.
Interactive terminal
Upgrade GET /sessions/{id}/terminal with the API bearer token. Binary WebSocket messages carry raw terminal bytes in both directions. A text message {"op":"resize","cols":100,"rows":40} resizes the terminal. Invalid text control frames close the attachment with WebSocket code 1007; the session remains active. The server pings attached clients and detaches a connection that stops responding (about 30 seconds with the default keepalive). Disconnecting detaches the terminal but keeps the session and shell alive. Only one terminal attachment may be active for a session. The terminal is the same shell that runs stateful commands, so it sees their directory and variables; stateful commands and terminal attachment cannot run together.
Clients
clients/shiitake-py provides async and synchronous context-managed sessions. Its optional [pty] extra adds terminal attachment. clients/shiitake-rs provides an async HTTP client for sessions and command output.
Topologies and worker profiles
In single-pod mode, the server and workers share a pod and the dispatcher listens on loopback. In two-pod mode, worker profiles run in separate Deployments and connect through a Service. The server has a local output spool in either mode. Profiles are advertised by workers on connection; the server selects the least recently used idle worker in the requested profile.
For two-pod startup, set publishNotReadyAddresses: true on the dispatch Service. Workers register through that Service while the server remains unready until workers connect; without this setting, startup deadlocks. Give the server ServiceAccount a Role with get permission on pods so it can distinguish container OOM kills from other worker exits. The test chart shows the complete deployment.
The test chart offers a default profile with /shared and an isolated profile without that application mount. In two-pod mode they also have separate pod boundaries. The caller manages file permissions in shared folders. A single-pod layout shares a network namespace, so use separate pods when pod-level network policy matters.
Configuration
Server: SHIITAKE_AUTH_TOKEN, SHIITAKE_DISPATCH_TOKEN, SHIITAKE_PROFILE_TOKENS (JSON map of additional profile credentials), SHIITAKE_DISPATCH_HOST/PORT, SHIITAKE_CAPTURE_ROOT (server-local output spool), SHIITAKE_DEFAULT_WORKDIR, SHIITAKE_MIN_READY_WORKERS, and SHIITAKE_PTY_KEEPALIVE_SECS (terminal client ping interval, default 10). Worker: SHIITAKE_WORKER_ID, SHIITAKE_PROFILE, SHIITAKE_DISPATCH_URL, SHIITAKE_DISPATCH_TOKEN, SHIITAKE_CAPTURE_ROOT (worker-local temporary output), SHIITAKE_RESET_PATHS, SHIITAKE_RESTART_AFTER, and SHIITAKE_LEASE_TIMEOUT.
/health stays healthy while the server process is serving; /ready needs the configured number of registered workers, including allocated workers. The server exports OpenTelemetry traces and metrics when OTEL_EXPORTER_OTLP_ENDPOINT is set. CPU and memory figures for parallel jobs are omitted because jobs share a container cgroup.
Dispatch protocol
Workers send a JSON Hello frame with a unique worker id, protocol_version: 3, profile, and optional pod location. The server acknowledges version 3 with HelloAck before assigning work. It rejects older workers, and a worker rejects commands from a server that does not acknowledge the version. The server sends SessionOpen, SessionRun, SessionResize, and SessionClose; workers respond with SessionReady, Result, and SessionClosed. Tagged binary chunks carry stdout, stderr, and terminal bytes before a command's Result. A chunk is a stream byte, a two-byte big-endian id length, a UTF-8 id, then raw bytes. Streams 0 and 1 identify command stdout/stderr, 2 terminal output, and 3 terminal input. The server spools command output and the worker purges its temporary capture after delivery. The canonical types and codec are in shiitake-worker-api.
Security
Use distinct API and dispatch bearer tokens. Worker commands run under the session's drop_to identity, and the worker process is not dumpable, so commands, root included, cannot read its dispatch credential from /proc or its memory unless the container has CAP_SYS_PTRACE. Do not grant that capability to worker containers. Configure a distinct dispatch token for each additional profile with SHIITAKE_PROFILE_TOKENS. The server binds the token used on the WebSocket upgrade to the profile in Hello, so a worker cannot claim another profile. Give each profile its intended mounts and network policy; workers within one profile share its trust domain.
Development
cargo +nightly fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
bash tests/build.sh && bash tests/setup.sh && bash tests/run.sh
SHIITAKE_E2E_TOPOLOGY=two-pod bash tests/setup.sh
SHIITAKE_E2E_TOPOLOGY=two-pod bash tests/run.sh