Self-hosted runner

Four claims. Each is something you can check yourself rather than take on faith.

• Lithora holds no key to your machine. There is no SSH key, password or certificate for your host anywhere in Lithora. The runner authenticates to us; we never authenticate to you. A total compromise of Lithora yields no credential that logs into your server.
• No inbound port is opened. The runner makes outbound HTTPS requests and never listens on a socket. No firewall change, no bastion, no inbound allowlist. Check it: ss -lntp shows nothing for it, and the shipped systemd unit declares no ListenStream and no socket unit — the only mention of one in the file is the comment saying there must never be one.
• Your local manifest is the final authority. The server checks a job against the manifest it holds — and the runner then checks it again against the file on your disk and refuses anything absent from it. Two independent checks, because a server that has been compromised or has simply gone wrong is exactly what the second one is for.
• You can revoke instantly, on your own. The credential is checked on every poll, so revoking in Dashboard → Runners takes effect on the next one. Stopping the service and deleting the state file works too, and does not involve us at all.

A Linux host with systemd and Python 3.7 or newer. Nothing else — no pip install, no virtualenv, no packages. That is deliberate: this runs on your production hosts, and its supply chain should be auditable in zero minutes. You will also need a team admin account, because registering a runner defines what Lithora may execute on your infrastructure and is an administrative act.

A runner release is five files: runner.py, lithora-runner.service, manifest.example.json, install.sh, SHA256SUMS. The checksum file is generated from the tagged source at publish time, so it describes exactly the runner.pypublished with it — not a hand-maintained list that drifts. Ask your Lithora contact for the current release if you do not already have it; a self-serve download is not open yet, and we would rather say so than link you somewhere that does not resolve.

Verify it before anything else. Every line must say OK; if any says FAILED, stop — install.sh runs the same check itself and refuses to install on a mismatch.

sha256sum --check SHA256SUMS
# macOS / BSD:  shasum -a 256 --check SHA256SUMS

runner.py: OK
lithora-runner.service: OK
manifest.example.json: OK
install.sh: OK

sudo useradd --system --no-create-home --shell /usr/sbin/nologin lithora-runner

No home directory and no login shell: the account exists to own a process. Do not run the runner as root. If a manifest command genuinely needs privilege, give that command an explicit sudoers rule — a narrow, reviewable grant beats a daemon that already has everything.

sudo install -d -m 0755 /opt/lithora-runner /etc/lithora-runner
sudo install -m 0644 -o root -g root runner.py /opt/lithora-runner/runner.py

# The credential and the job journal live here, so it is 0700 and runner-owned.
sudo install -d -m 0700 -o lithora-runner -g lithora-runner /var/lib/lithora-runner

runner.py and the manifest stay root-owned and read-only to the runner. A daemon that can rewrite its own manifest decides what it is allowed to run, which would dismantle the whole design.

This is the file that matters. It is the complete statement of your exposure, so keep it in your own repository and review changes to it the way you review code.

{
  "commands": [
    {
      "name": "status",
      "risk": "read",
      "run": ["/usr/bin/systemctl", "is-active", "api"],
      "timeout": 30
    },
    {
      "name": "deploy",
      "risk": "write",
      "run": ["/opt/app/bin/deploy.sh", "{ref}"],
      "args": [{ "name": "ref", "type": "string", "required": true }],
      "working_dir": "/opt/app",
      "env": { "DEPLOY_ENV": "production", "API_TOKEN": "$APP_API_TOKEN" },
      "timeout": 900
    },
    {
      "name": "restart-api",
      "risk": "destructive",
      "run": ["/usr/bin/systemctl", "restart", "api"],
      "confirm_phrase": "restart-api",
      "timeout": 60
    }
  ]
}

A token in run of the exact form {name} is replaced by that argument as a whole argv element. It cannot expand into two arguments, and it cannot become shell syntax because there is no shell. Arguments containing shell metacharacters are refused on both sides.

# /etc/systemd/system/lithora-runner.service.d/env.conf
# A drop-in, so reinstalling the unit does not take it away.
[Service]
EnvironmentFile=/etc/lithora-runner/env

# The file itself: KEY=value per line, root-owned, mode 0600.
sudo install -m 0600 -o root -g root /dev/null /etc/lithora-runner/env
printf 'APP_API_TOKEN=%s\n' "$TOKEN" | sudo tee /etc/lithora-runner/env >/dev/null
sudo systemctl daemon-reload && sudo systemctl restart lithora-runner

Validate it

sudo cp manifest.example.json /etc/lithora-runner/manifest.json
sudo nano /etc/lithora-runner/manifest.json

python3 /opt/lithora-runner/runner.py check --manifest /etc/lithora-runner/manifest.json

3 command(s) declared in /etc/lithora-runner/manifest.json:

  deploy  [write]
      run:     ['/opt/app/bin/deploy.sh', '{ref}']
      cwd:     /opt/app
      timeout: 900s
  restart-api  [destructive]
      run:     ['/usr/bin/systemctl', 'restart', 'api']
      timeout: 60s
  status  [read]
      run:     ['/usr/bin/systemctl', 'is-active', 'api']
      timeout: 30s

This is the COMPLETE set of things Lithora can run on this host.

Run check before every restart, and put it in the CI for whichever repository holds the manifest.

In Lithora, go to Dashboard → Runners → New runner and copy the enrollment token. It is shown once, expires in 30 minutes and is single-use — only its hash is stored, so it cannot be recovered afterwards.

# Enrol AS the runner user. Enrolling as root writes a root-owned state file
# and the service then cannot read its own credential.
sudo runuser -u lithora-runner -- python3 /opt/lithora-runner/runner.py enroll \
    --token lrn_enroll_xxxxx \
    --state /var/lib/lithora-runner/state.json

sudo install -m 0644 lithora-runner.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now lithora-runner

The token is exchanged for a long-lived credential written to state.json with mode 0600. The shipped unit runs as the lithora-runner user with NoNewPrivileges, ProtectSystem=strict, PrivateTmp, an empty capability bounding set and a start limit that stops a broken install from restart-looping.

Self-hosted Lithora: add --api https://lithora.example.com to the enroll command. The URL is stored in state.json beside the credential and is what the daemon polls from then on, which is why the unit does not repeat the flag. --api on run overrides it for that one invocation.

systemctl status lithora-runner
journalctl -u lithora-runner -n 20

[lithora-runner 2026-08-11T09:14:02Z] manifest declares 3 command(s): ['deploy', 'restart-api', 'status']
[lithora-runner 2026-08-11T09:14:02Z] polling https://api.lithora.app — outbound only, no inbound port is opened

Then dispatch the lowest-risk command in your manifest from the dashboard and watch it appear. The runner shows as active with a recent last-seen time.

install.shdoes steps 1–5 and refuses to proceed on a checksum mismatch.

sudo ./install.sh --enroll-token lrn_enroll_xxxxx

It is short and touches nothing outside the paths above. Read it before running it as root — advice that applies to every install script, including ours.

Dashboard → Runners lists every job with its command, arguments, exit code, a redacted output excerpt and a SHA-256 of the output the runner captured. That capture is capped at 256 KB on the host, so for a command that prints more than that the digest covers the first 256 KB — the part that was hashed is the part that was sent. Every enrolment, dispatch, claim and completion also writes to an append-only audit trail recording actor, action, arguments and timestamps.

journalctl -u lithora-runner -f              # live
journalctl -u lithora-runner --since today   # everything the runner did today

Results that never reached us

sudo runuser -u lithora-runner -- python3 /opt/lithora-runner/runner.py jobs \
    --journal /var/lib/lithora-runner/jobs.json

This is the one case the dashboard cannot cover. If a command ran on the host but the report never got through, that host is the only witness. Normally it reports nothing outstanding. Three phases can appear:

• pending — the result is still being retried. Nothing is lost.
• abandoned — it ran, and Lithora does not have the output. The entry keeps the exit code and the digest, and says whether that exit code is still being offered hourly or never will be, because Lithora rejected the result outright.
• executing — the command started on this host and the runner was killed before it finished. Whether it completed is unknown here and in Lithora, and it is never re-run. It is listed even though there is no result to deliver: it is the entry the dashboard can least afford to be missing.

Pass --journalexplicitly, as above: its default is a name relative to the working directory, while the daemon is given the absolute path in its unit. “There is no ledger here” is not “nothing is outstanding”, so the exit status says which one you got.

• Exit 0 — the host answered, because it read the ledger you named: here is what is outstanding, or the ledger is empty and every result reached Lithora.
• Exit 2 — the host could not answer, and the printed reason says which: a ledger this process may not read, which says nothing at all about what is in it; or nothing at that path at all. /var/lib/lithora-runner is 0700 and owned by the runner user, which is why the command above goes through runuser.

Nothing else in the directory turns exit 2 into exit 0. A credential sitting beside a missing ledger proves a runner was enrolled in that directory — not that the path you asked about is the one the daemon writes, and the shipped unit puts state.json beside a ledger named jobs.json while the default --journal is lithora-runner-jobs.json. So the command prints what it found, including a ledger-shaped file worth asking about instead, and still reports that it cannot tell you. A freshly enrolled host whose daemon has never started lands here too: the ledger is written at startup, so its absence at the daemon’s own path means the daemon has never run — worth a look rather than an all-clear.

If the runner dies between executing a command and reporting it, the server returns the job to the queue after 30 minutes so the work is not stranded. That means the job can be handed back to the same host — and the runner never runs it a second time. A job id is written to a local ledger before the command starts, so a redelivered job is recognised and reported as interrupted (exit 125) rather than re-executed. Running your deploy twice is the one failure you cannot undo, so it is the one the ledger is built to prevent.

If reporting fails, the result is kept on disk and retried with per-job backoff, a few reports per poll at most. Reporting is idempotent, so a retry after a lost response is harmless. After 40 failed attempts — roughly three hours of outage — the runner stops retrying at that rate: it logs the exit code and digest at error priority, drops the output so the ledger stays bounded, and tombstones the job so it still cannot be re-executed.

journalctl -p err -u lithora-runner   # only the results that never got through

The exit code is still owed, though. The server requeues a job it has no terminal result for every 30 minutes, so a tombstone that stayed silent would leave that job circling the queue forever. It does not: the exit code is offered again hourly, and immediately if the server hands that job back — being handed it back proves the server is reachable and still waiting. Once it is accepted, the requeueing stops. A result the server actively rejectsis never offered again, and then the host’s own ledger is the only record of it.

Once 32 results are undelivered the runner stops claiming new work rather than piling up commands whose outcomes nobody can see. Tombstones do not count toward that limit — one permanently undeliverable result must not leave the runner idle for good.

• Change the manifest. Edit it, run runner.py check, then systemctl restart lithora-runner. It is read at startup.
• Rotate the credential. Revoke in the dashboard, create a new runner, and enrol with --force.
• Revoke. Dashboard → Runners → Revoke, or systemctl stop lithora-runner and delete state.json.
• Uninstall. systemctl disable --now lithora-runner, remove /opt/lithora-runner, /etc/lithora-runner, /var/lib/lithora-runner and the unit file, then revoke in the dashboard.

• not enrolled — run ... enroll --token ... firstThere is no credential at the --state path the daemon was given. Either enrolment never ran, or it wrote somewhere else: the unit passes /var/lib/lithora-runner/state.json, and the quick-start default is a file in the working directory.
• state file ... is unreadable: [Errno 13] Permission deniedThe credential exists and the service may not read it. Almost always because enrolment ran as root rather than through runuser: ls -l /var/lib/lithora-runner/state.json should show lithora-runner and mode 0600. The same mistake on the directory makes runner.py jobs exit 2 rather than report a clean ledger — refusing to answer is the point, since a file it cannot open tells it nothing.
• credential rejected (HTTP 401)The runner was revoked, or another host enrolled with the same one-time token. The daemon exits deliberately rather than hammering the API. Re-enrol to resume — anything undelivered in the journal is sent once it is back.
• REFUSED: job ... is not in this host’s manifestThe manifest registered in Lithora and the file on the host disagree. The local file wins, by design. Reconcile them and restart the service.
• The unit is failed and will not restartAfter 5 failures in 5 minutes systemd stops retrying, so the real error stays visible instead of scrolling past. systemctl status lithora-runner shows it — usually a missing or invalid manifest.
• REFUSED: <command>: the manifest asks for host environment variable(s) this runner process does not haveA $NAME in that command’s env is not set for the service — nothing was executed, and the job is reported as exit 126. Set it as shown under “Where $APP_API_TOKEN comes from”; systemctl show -p Environment lithora-runner shows what the daemon actually has.
• A command works in your shell but not through the runnerIt is not inheriting your environment, and that is intended: the command gets PATH, HOME and exactly what the manifest’s env grants. Use absolute paths in run. Failing that the unit’s sandbox may be responsible — ProtectProc=invisible hides other processes from pgrep, and ProtectSystem=strict makes the filesystem read-only outside /var/lib/lithora-runner.

It is not a shell, a web terminal, or remote command execution. There is no free-form string that reaches a shell on either side, and there is no Lithora admin surface that can dispatch to your runner — an internal shell into customer infrastructure would make every Lithora administrator a standing privileged user in every customer’s production, which is precisely what this shape was chosen to avoid. If you want a shell, this is not it, and that is the deliberate answer rather than a missing feature.