systemg

Search docs

/
Install

Configuration

Configuration

systemg uses YAML files to define services and their relationships.

Complete example

version: "2"
projects:
  myapp:
    name: My App
    services:
      # ... service definitions (see below)

A single file can declare many projects under the projects: map, keyed by project id. The example below shows one project's full set of sections. Project entries may override env and logs; top-level metrics and status settings apply across the loaded projects.

version: "2"
project:
  id: myapp
  name: My App
env:
  vars:
    APP_ENV: "production"
logs:
  sink: file
  max_bytes: 10485760
  max_files: 5
status:
  snapshot_mode: summary
  snapshot_interval_secs: 5
services:
  postgres:
    command: "postgres -D /var/lib/postgresql/data"
    restart_policy: "always"
  redis:
    command: "redis-server /etc/redis/redis.conf"
    restart_policy: "always"
  api:
    command: >
      gunicorn app:application
      --bind 0.0.0.0:8000
    env:
      file: "/etc/myapp/production.env"
      vars:
        PORT: "8000"
        DATABASE_URL: "postgres://localhost/myapp"
    depends_on:
      - postgres
      - redis
    restart_policy: "always"
    backoff: "10s"
    deployment:
      pre_start: "python manage.py migrate"
      health_check:
        command: "curl --fail http://localhost:8000/health"
    hooks:
      onstart:
        command: "echo 'API started'"
      onerr:
        command: "curl --request POST https://alerts.example.com/api/crash"
  worker:
    command: >
      celery -A tasks worker
      --loglevel=info
    depends_on:
      - redis
    restart_policy: "on-failure"
    max_restarts: 5
  backup:
    command: >
      pg_dump mydb >
      /backups/db-$(date +%Y%m%d).sql
    cron:
      expression: "0 0 2 * * *"

Supervisor configuration

Project manifests describe workloads. Supervisor-wide defaults live separately in ~/.local/share/systemg/supervisor.xml, or /var/lib/systemg/supervisor.xml with --sys:

<supervisor>
  <logs>
    <max_bytes>10485760</max_bytes>
    <max_files>5</max_files>
  </logs>
  <timeouts>
    <pre_start_secs>300</pre_start_secs>
    <startup_stability_ms>250</startup_stability_ms>
    <stop_verify_secs>10</stop_verify_secs>
    <start_settle_secs>360</start_settle_secs>
    <command_wait_secs>900</command_wait_secs>
  </timeouts>
  <start>
    <max_concurrent>-1</max_concurrent>
  </start>
</supervisor>
  • pre_start_secs: default execution budget for deployment pre_start commands.
  • startup_stability_ms: survival window for services without a health check.
  • stop_verify_secs: time allowed to confirm that a terminated process is gone.
  • start_settle_secs: maximum wait for an unresolved queued project start.
  • command_wait_secs: how long a CLI command waits for the supervisor's reply before reporting SG0111. 0 waits indefinitely. Raise it when a project is large enough that a restart runs longer than the default; the command is never cancelled by the wait ending.
  • max_concurrent: how many services a bulk start runs at once. -1 (the default) starts every service whose dependencies have resolved, 1 starts one service at a time, and any other positive number is a cap. Applies to sysg start and to project boot; restart remains one service at a time.

The file is created on first supervisor start. Existing compact XML remains compatible and is rewritten in the indented form after it parses successfully. SYSG_PRE_START_TIMEOUT_SECS remains a higher-precedence compatibility override.

Info

Health-check attempt_timeout and total_timeout remain in the project manifest because readiness is workload-specific. The IPC poll slice, liveness probe window, unresponsive grace and live-upgrade deadlines are protocol invariants and are not operator settings.

Registered manifests

When you run a command with -c <file>, the resident supervisor remembers that manifest's resolved path for each project it registers. Later commands can omit -c and target the registered project from any working directory.

Info

Ad-hoc units use the same manifest pipeline. sysg start -- <command...> first stages the command as a generated version 2 manifest. If a supervisor is already running, that file is not registered or started until you run the explicit sysg start --config ... command systemg prints. See Units.

sysg restart treats the files at those paths as the source of truth. It reads and validates every registered manifest before touching a process, then re-registers added, changed, and removed projects and services. An invalid manifest returns SG0301 and leaves the running workloads unchanged.

# submit the manifest once
$ sysg start -c stack.yaml --daemonize

# later, -c is optional even after editing stack.yaml
$ sysg status

# adopt the current manifest and reconcile its changes
$ sysg restart

Configuration sections

version

Required. Specifies the configuration schema version. The current schema is 2. Older version: "1" manifests are no longer accepted. For a legacy singular-project: manifest, run sysg migrate, which converts the shape and emits version: "2". For a loose or existing projects: manifest, update the version field directly. When upgrading from 0.54.x or older, also follow the state-layout migration.

version: "2"

projects

The canonical way to declare projects. A map keyed by project id, where each entry carries that project's name, services, and optional env / logs sections. One file can hold as many projects as you like:

version: "2"
projects:
  arbitration:
    name: Arbitration
    services:
      worker:
        command: "python worker.py"
  gamecast:
    services:
      api:
        command: "python api.py"

Each key is the project.id. When a name is omitted, the id doubles as the display name. Every project gets its own state directory — see State — and you target each one by id with -p/--project at runtime.

Warning

Treat the project id (the map key) as durable runtime identity. Changing it does not rename a project — it creates a new namespace, and the old one's running services become orphaned state. Rename freely via name; never rename by editing the id.

Loose (project-less) services

Top-level services: with no project form a loose bundle. They still run, and their state persists under projects/__loose__/:

version: "2"
services:
  web:
    command: "python app.py"

Use this for single-service or quick setups where a project id adds no value.

Singular project: (deprecated)

The older singular block still parses, so existing single-project manifests keep working — but it emits a deprecation warning. Prefer projects:.

project:
  id: arbitration
  name: Arbitration

The shorthand project: arbitration (which sets both id and name) is also still accepted. Convert old-shape manifests with sysg migrate.

See Projects for how one supervisor hosts many projects at once and how -p/--project targets them at runtime.

env

Optional environment variables shared by all services.

env:
  vars:
    LOG_LEVEL: "info"
    APP_ENV: "production"
  file: "/etc/myapp/common.env"

logs

Optional defaults for service stdout/stderr handling.

logs:
  sink: file
  max_bytes: 10485760
  max_files: 5

Fields:

  • sink: file captures service output to systemg-managed log files. none discards service output without creating log-writer threads or files.
  • max_bytes: active log-file size before rotation for the file sink.
  • max_files: number of rotated files to retain per active log.

Use sink: none for noisy production services when service output is already collected by another logging pipeline.

status

Optional defaults for status and inspect runtime detail.

status:
  snapshot_mode: summary
  snapshot_interval_secs: 5

Fields:

  • snapshot_mode: off, summary, or detailed.
  • snapshot_interval_secs: seconds between background snapshot refreshes, clamped between 1 and 300.

Modes:

  • summary: default. Tracks service state, pid, health, last exit, cron state, and sampled metric summaries while skipping expensive process tree expansion.
  • detailed: includes runtime command details and process/spawn descendants for richer inspect output.
  • off: disables background runtime snapshot refresh and uses persisted state plus pid files.

For large deployments, keep summary globally and use focused inspect --service workflows when deeper investigation is needed.

metrics

Optional tuning for the CPU/memory sampling that powers status and inspect.

metrics:
  retention_minutes: 720
  sample_interval_secs: 1
  max_memory_bytes: 10485760
  spillover_path: ".state/metrics"

Fields:

  • retention_minutes: minutes of in-memory samples to keep (default 720).
  • sample_interval_secs: seconds between samples, clamped 1-60 (default 1).
  • max_memory_bytes: memory cap across all sample buffers (default 10 MiB).
  • spillover_path: optional directory for spilling older samples to disk, with spillover_max_bytes and spillover_segment_bytes controlling disk usage.

services

Defines the services to manage. Each entry under projects: requires its own services: map. A top-level services: map is optional and defines the loose bundle.

services:
  web:
    command: "python app.py"

!include

Splits a manifest across files: an !include <path> tag at any node is replaced by the parsed content of the referenced file. Relative paths resolve against the directory of the file doing the including, and included files can include further files.

version: "2"
projects:
  api: !include projects/api.yaml
  worker:
    services: !include services/worker.yaml
# projects/api.yaml — a fragment is a plain replacement value;
# only the root manifest declares `version:`
services:
  server:
    command: "python app.py"

Info

!include uses YAML's standard local tag syntax, part of both the YAML 1.1 and YAML 1.2 specs, so an include-bearing manifest is valid YAML to any spec-conformant parser. The inclusion behavior itself is sysg-specific: the spec deliberately leaves local tag semantics to the application, so generic tools (yq, linters) parse the tag but do not resolve it, and strict loaders that reject unknown tags (such as PyYAML's safe_load) refuse to construct it.

Fragments are held to the same trust bar as the root manifest, ${VAR} expansion applies to included content, and every command (validate, status, restart, upgrade) sees the assembled result. A missing or broken fragment is always a hard error carrying the include chain (SG0207) — never a partially loaded manifest — and cyclic includes (SG0208) or includes past the depth/size caps (SG0209) are refused. Include paths themselves cannot use ${VAR} expansion. sysg migrate preserves !include tags unresolved.

Service configuration

command

The command to execute, run through sh -c. Required unless the service declares exec instead.

services:
  web:
    command: "python app.py"

The shell stays alive for as long as the service does, and it is the process systemg tracks: signals, CPU and memory readings, and the recorded exit status all describe the shell rather than the program inside it. On most Linux systems /bin/sh is dash, which does not replace itself with the command, so a shell form service costs one extra process and one extra entry in status.

Use it when the service genuinely needs a shell - a pipeline, &&, a glob, a variable expansion. Reach for exec when it does not.

exec

The program and its arguments as a list. systemg runs it directly, with no shell in between, so the tracked process is the workload itself.

services:
  web:
    exec: ["python", "app.py"]

A service declares command or exec, never both. Nothing is quoted, split, or expanded: each list entry is passed through as one argument, so a value containing spaces stays one argument.

working_dir

The directory the service runs in, relative to the manifest's directory or absolute. Without it, a service that needs another directory has to say cd elsewhere && ..., which forces the shell form.

services:
  web:
    working_dir: "services/api"
    exec: ["python", "app.py"]

depends_on

Services that must start before this one.

services:
  api:
    command: "python app.py"
    depends_on:
      - postgres
      - redis

depends_on is the only thing that orders a start. Services with no dependency between them start at the same time, and a service waits for the dependencies it declared and for nothing else — an unrelated slow service never holds it back. Position in the manifest has never ordered anything and does not now.

Warning

If a service needs another one up first, declare it. A service that relied on a manifest's ordering without saying so will now start alongside what it used to follow. Set max_concurrent to 1 in supervisor.xml to restore one-at-a-time startup while you add the missing depends_on entries.

env

Service-specific environment configuration.

services:
  api:
    command: "python app.py"
    env:
      vars:
        PORT: "8000"
        DATABASE_URL: "postgres://localhost/myapp"
      file: "/etc/myapp/production.env"

restart_policy

Control how services recover from crashes.

services:
  api:
    command: "python app.py"
    restart_policy: "always"
    backoff: "5s"
    max_restarts: 10

Service logs

Override global logging settings for one service.

services:
  api:
    command: "python app.py"
    logs:
      sink: file
      max_bytes: 5242880
      max_files: 3
  noisy_worker:
    command: "worker --verbose"
    logs:
      sink: none

Policies:

  • always - Restart on non-zero exit codes
  • on-failure - Restart on non-zero exit codes
  • never - Don't restart

A clean (zero) exit is treated as intentional and never triggers a restart, regardless of policy. Restarts respect backoff between attempts and stop after max_restarts (unlimited when unset).

hooks

Run commands after successful starts or unsuccessful exits.

services:
  api:
    command: "python app.py"
    hooks:
      onstart:
        command: "curl --request POST https://status.example.com/api/up"
      onerr:
        command: "/usr/local/bin/report-crash api"

cron

Run services on a schedule instead of continuously.

services:
  backup:
    command: >
      pg_dump mydb >
      /backups/db-$(date +%Y%m%d).sql
    cron:
      expression: "0 0 2 * * *"

deployment

Control how services update during restarts.

services:
  api:
    command: "python app.py"
    deployment:
      strategy: "rolling"
      pre_start: "python manage.py migrate"
      health_check:
        command: "curl --fail http://localhost:8000/health"
        interval: "5s"
        attempt_timeout: "30s"
        total_timeout: "5m"
        retries: 3
      grace_period: "5s"
      blue_green:
        env_var: "PORT"
        slots: ["8000", "8001"]
        candidate_health_check:
          command: "curl --fail http://127.0.0.1:{slot}/health"
          interval: "2s"
        switch_command: "/usr/local/bin/switch-upstream {candidate_slot}"
        switch_verify:
          command: "curl --fail http://localhost:8000/health"
        state_path: ".state/api-slot.xml"

Rolling deployments start the new instance, wait for health checks, then stop the old instance. For single-host zero-downtime with fixed ports, use blue_green so traffic can be switched between two slots. A blue-green deployment uses two identical slots, starts the new version in the idle slot, verifies it, and then switches traffic only after the candidate is ready.

Field reference

Service fields

Primary keys available on each service definition.

FieldTypeDescription
commandstringCommand to execute through sh -c (required unless exec is set)
execarrayProgram and arguments run directly, with no shell
working_dirstringDirectory the service runs in
depends_onarrayServices that must start first
envobjectEnvironment configuration
restart_policystringalways, on-failure, or never
backoffstringTime between restart attempts
max_restartsnumberMaximum restart attempts
hooksobjectLifecycle event handlers
cronobjectCron schedule (expression, optional timezone)
deploymentobjectUpdate strategy configuration
logsobjectService stdout/stderr capture and rotation settings
skipbool or stringSkip this service, or a command whose success skips it
spawnobjectDynamic child-process policy (mode, limits)
user / groupstringRun the service as this user/group (privileged mode)
supplementary_groupsarrayExtra groups applied before dropping privileges
capabilitiesarrayLinux capabilities retained after the privilege drop
limitsobjectResource limits (nofile, nproc, memlock, nice, cpu_affinity, cgroup)
isolationobjectNamespace isolation (network, mount, pid, user)

user, group, supplementary_groups, capabilities, limits, and isolation only take effect in privileged mode - see System mode for details and examples.

Health checks are configured under deployment.health_check, not as a top-level service key.

Environment object

Environment sources and inline overrides merged into the service process environment.

FieldTypeDescription
varsobjectKey-value environment variables
filestringPath to env file
inherit_envboolLet a privilege-dropped service inherit the supervisor's environment instead of starting clean (default false)
clear_session_varsboolStrip session-scoped variables like SSH_* and DISPLAY (default true)
striparrayAdditional variable names to remove from the service environment

Hooks object

Commands triggered by service outcomes.

FieldTypeDescription
onstartobjectCommand run after readiness or successful one-shot completion
onerrobjectCommand run after an unsuccessful service exit

Each hook supports:

  • command - Command to execute
  • timeout - Maximum execution time

Health check object

Probe configuration used to determine readiness/health during deployment workflows.

FieldTypeDescription
commandstringCheck command
urlstringHTTP endpoint (alternative to command)
intervalstringTime between attempts (default 2s); must be greater than zero
attempt_timeoutstringMaximum time for a single probe (default 30s)
total_timeoutstringMinimum total readiness window before giving up; timeout is accepted as a compatibility alias
retriesnumberMinimum attempts before giving up (default 3)

Note

attempt_timeout bounds one probe. total_timeout controls the whole readiness window, so connection refusals that return immediately do not exhaust a slow-starting service's budget. A check fails only after both retries and total_timeout are exhausted. The failure carries a code by cause: SG0022 (could not reach), SG0023 (a probe timed out), or SG0104 (ran but reported unhealthy).

Durations

Every duration-valued field — backoff, grace_period, hook timeout, and the health-check windows — is a whole number with an optional unit:

UnitMeaningExample
msmillisecondsinterval: "100ms"
ssecondsbackoff: "10s"
mminutestotal_timeout: "5m"
hhourstotal_timeout: "1h"

A bare number is seconds, so 15 and 15s are the same value. Fractions (0.5s) are not accepted — write them in a smaller unit (500ms).

Durations are checked when the manifest is loaded, so sysg validate refuses exactly what sysg start refuses, naming the offending field's path (SG0210).

Deployment object

Controls how restarts are performed and what validation happens before cutover.

FieldTypeDescription
strategystringrolling or immediate
pre_startstringCommand that must exit successfully before starting; its process tree is terminated after the configured supervisor command budget (SG0108)
health_checkobjectHealth check configuration
grace_periodstringTime before stopping old instance
blue_greenobjectSingle-host blue/green rollout settings

Note

pre_start runs from the manifest directory with the service environment. Its output is captured in the service log. A non-zero exit is SG0103; exceeding the supervisor.xml pre_start_secs budget is SG0108, and the service is not launched.

Blue/green deployment object

Single-host zero-downtime options for alternating between two rollout slots (typically ports).

FieldTypeDescription
env_varstringEnv var injected with slot value (PORT default)
slotsarrayExactly two slot values to alternate between
switch_commandstringCommand to switch traffic to candidate slot
candidate_health_checkobjectOptional candidate verification check ({slot} supported in url or command)
switch_verifyobjectOptional post-switch verification check
state_pathstringOptional persisted active-slot state file path

Manifest schema compatibility

The top-level version field declares the manifest schema version. The current schema is 2, accepted as either a string or integer, so version: "2" and version: 2 are equivalent. version: "1" is no longer accepted.

systemg reads the declared version before the rest of the manifest. Version 1 is rejected rather than silently reinterpreted; version 2 is the only current runtime schema. systemg never rewrites a manifest as a side effect of starting services.

When downgrading, the older binary can only parse versions it knows. Keep the previous manifest or convert it before starting an older release.

Schema validation is separate from manifest shape conversion. To rewrite an old singular-project: manifest into the canonical projects: map, run sysg migrate — it prints the converted YAML to stdout unless --in-place is requested.

Commands