API

DistSSHKit — meta-package for the run and the queue.

Users add this package. The command is julia -m DistSSHKit. DistSSHRun is the run. DistSSHQueue is the queue, started on a queue host. This module reexports their public names.

go!(script, workers...; kwargs...)
go!(script, workers::AbstractVector; kwargs...)

Run an as-is complete job on one or more slots (local and/or remote).

go!("job.jl")                          # one parent slot
go!("job.jl"; repeat=100)              # 100 independent runs on parent
go!("job.jl", "parent:2"; args=["8"])
go!("job.jl", "child:user@h1:1", "child:user@h2:1"; remote="/path/to/project")

Each slot gets DISTRIBUTED_OUTPUT_DIR pointing at {script}/.distsshkit/go/<stem>_<UTC>/<slot>/. Override the batch root with output_dir (CLI: --output-dir). For backward compatibility collect_spec::AbstractString also sets the batch root, but passing both output_dir and collect_spec::String is an error. collect_spec === false means "skip collect" and is orthogonal to output_dir.

Default sync is false (no pre-run sync; remotes are checked first — setup is assumed done). Pass sync=:sync or sync=:rsync to copy, then check. After :rsync, hosts that still lack Manifest deps get instantiate! before the check. Use sync=:rsync only onto a missing/empty remote path (or setup --delete / setup!(session, :delete) first). go! has no git-parity gate; use drive! with skip_hash_check=false (CLI: drive --require-git) when you need that.

julia sets the Julia binary for each slot (nothing / "auto" → detect; same as CLI --julia).

When workers is empty and hosts_file is omitted, DISTSSHKIT_HOSTS_FILE is still read (API go!("job.jl")). Non-empty workers (CLI host_tokens) does not re-read that ENV.

parent:N and child:NAME:N mean N independent full-job runs (not Distributed workers), started together. repeat=N (CLI --repeat N) is the total number of those runs, spread round-robin across listed hosts (no tokens → all on parent). :N on a token is a per-host cap when repeat is set; omit it to leave that host uncapped. Without repeat, every listed token needs :N (omitting it is not 1 and not size!). Empty tokens stay one parent slot. path_anchor shortens displayed paths (CLI passes kit project root).

Outcome of go!. On failure, failed_step is "sync", "run", or "collect".

report_go_errors(result::GoResult; io=stderr)

Print a short summary when go! failed. Returns result.ok.

Shared run outcome (ok, kind, dirs, failed_step, exit_code, hosts).

kind is :go, :drive, :ride, or :pipeline. Convert with kit_run_result. hosts is post-run collect (HostRunResult); empty for go, hung wait, or a drive that never collected.

Build KitRunResult from a kit outcome (:go / :ride / :drive / :pipeline).

report_run_errors(result; io=stderr)

Print a short summary when a kit run failed. Accepts KitRunResult or a typed outcome (GoResult / RideResult / DriveResult / PipelineResult). Returns result.ok.

Runtime context for sync!, size!, drive!, collect!.

setup!(session::KitSession, mode::Symbol; kwargs...) -> SyncResult
setup!(session::KitSession, modes::Symbol...) -> SyncResult

Prepare SSH hosts — same jobs as julia -m DistSSHKit setup --….

Confirmations follow session.yes (CLI -y). A matching confirm_path on :delete or :prune deletes even when yes=false. Any other mode rejects confirm_path. Multiple modes run in order and stop on the first failure:

session = KitSession(workers=["child:user@h1"], remote="~/proj", yes=true)
setup!(session, :rsync, :instantiate)
setup!(session, :check; ignore_julia_version=true)

sync! / instantiate! remain as thin aliases for the common deploy steps. Prefer setup! when you want the full CLI vocabulary in one place.

sync!(session::KitSession; mode=:sync)

Sync local project to SSH hosts (call explicitly; go/drive do not call this by default).

• mode=:sync — git push + pull on remotes (confirm unless session.yes)
• mode=:rsync — rsync working tree (no git; confirm unless session.yes). Refuses nonempty remote paths; use setup!(session, :delete) / setup --delete first.

Also available as setup!(session, :rsync) / setup!(session, :sync).

Returns SyncResult.

instantiate!(session::KitSession; julia="auto") -> SyncResult

Run Pkg.instantiate() on each SSH host in session (parallel).

julia is the remote Julia path ("auto" detects per host, same as setup --instantiate / drive --julia). Returns SyncResult.

Typical first-time remote prep:

session = KitSession(workers=["child:user@h1"], remote="/path/to/project", yes=true)
setup!(session, :rsync, :instantiate)  # or sync!(…; mode=:rsync) then instantiate!

Also available as setup!(session, :instantiate; julia=…).

Per-host outcome for sync and similar operations.

Outcome of sync! (rsync or git sync). cancelled is set when a confirm prompt is aborted.

drive!(session::KitSession, script; plan=nothing, args=[], ...)
drive!(script, workers...; kwargs...)
drive!(script, workers::AbstractVector; kwargs...)

Run a driver script on workers. Tokens match the CLI (parent:2, child:user@host:1). A file with no Distributed vocabulary warns (this-file scan only) and still runs. Load is the master include. Publish sends definitions to workers (not top-level work). Run is main() when defined. Use plan / go! / ride! when that matches the script.

drive!("job.jl", "parent:2"; args=["8"])
drive!(session, "job.jl")  # uses `session.tokens`

With sync omitted or false, prepare remotes with setup! or CLI setup first; remotes are checked before execution. Optional sync=:sync / sync=:rsync copies, then checks. After :rsync, hosts that still lack Manifest deps get instantiate! (same as CLI drive --rsync). Use sync=:rsync only onto a missing/empty remote path (or setup --delete / setup!(session, :delete) first). Git parity is off by default (skip_hash_check=true). With sync=:rsync, parity stays off even if skip_hash_check=false (no remote .git/). require_all_hosts=true (the default; CLI --require-all-hosts) fails unless every explicit parent:N / child:NAME:N joined, stayed through the run, and collect succeeded. Pass require_all_hosts=false or CLI --best-effort for a partial run.

mem_headroom / parent_gb are the RAM budget for drive preflight (same as size! / CLI --mem-headroom / --parent-gb). pipeline! passes config.mem_headroom and config.parent_gb.

julia sets the remote Julia binary (nothing / "auto" → detect; same as CLI --julia). plan is an optional explicit WorkerPlan. pipeline! syncs separately and does not pass sync= into drive!, so it does not instantiate after rsync.

sync_script=true (CLI --sync-script) re-includes the full driver on workers. The default publishes definitions / using / import / include only, so top-level work on Load is not repeated on workers.

Outcome of drive! (and similar CLI steps that return an exit code).

ok is true only when the run finished with exit 0. By default that includes the placement contract: listed parent / child hosts joined, stayed, and collect succeeded (require_all_hosts=true). require_all_hosts=false keeps a partial run as ok=true.

output_dir / log_dir are the directories actually used for this run — resolved the same way drive reports Results: / writes its log, even when drive! was not called with output_dir= / log_dir=. nothing when no real run happened (e.g. built by hand) or, for log_dir, when logging was disabled.

hosts is one HostRunResult per host that joined as a worker, in the order results were collected. Empty when no host-collection step ran (e.g. built by hand, or a run with no remote/local-only hosts).

Per-host outcome of the drive result-collection step.

ok is false when SSH / find / rsync raises while collecting from host. An empty file list after a successful probe is not an error. error is the message string, or nothing on success. An AbstractString is stored as-is so kit.result round-trips; other values use sprint(showerror, error).

collect!(
    session::KitSession,
    local_root::AbstractString;
    merge=false,
    hosts=nothing,
)

Rsync result files from SSH hosts into local_root.

Collect modes:

• merge=false → collect-missing (CLI drive --collect-missing)
• merge=true → collect-overwrite (CLI drive --collect-overwrite)

Distinct from drive's automatic post-run-new (sentinel / newer-than-run) and go's slot-overwrite. Returns CollectResult.

Outcome of collect!.

ride!(script, workers...; args=[], spi_check=false, output_dir=nothing, project=pwd())

Experimental. Rewrite map / filter / simple comprehensions / independent indexed for (dest[i] = …) and run the script on Distributed workers (parent and optional SSH child:). Rejects Distributed vocabulary (use drive!).

Worker add for SSH children is the same add_drive_workers! path as drive. The job project is Pkg.activated on the parent first (drive does this before using the app package on every process). The script still runs on the parent (no driver include on workers). Named functions used by map / filter are sent to workers as a prelude.

--spi-check (default off) compares the distributed result to a sequential map / filter. Unknown syntax stays sequential. Inspect first with plan. Listed child: hosts are fail-closed (require_all_hosts=true). Queue: execute! :ride (detached=true writes kit.result like go; SSH child: also writes kit.hosts for terminate!).

Outcome of ride!. Experimental.

Print a RideResult. Field names match go / drive (Script:, Workers:).

stored_path(rel; project=pwd(), output=nothing) -> String

Where rel is stored. output=nothing reads DISTRIBUTED_OUTPUT_DIR. A passed string is used as given and does not read that variable. "" means no output directory.

An absolute rel, after expanding a leading ~ on this machine, is returned as a canonical local path. The output directory is not consulted.

A relative rel uses the first match:

1. The output directory is set and rel already exists there.
2. rel already exists under project, even when the output directory is set.
3. The output directory is set, including when the file does not exist yet.
4. project, when the output directory is empty.

SHA-256 hex digest of file contents at path.

cache_file(src; project=pwd()) -> String

Copy src into .distsshkit/cache/sha256/<digest> if that blob is missing or stale. Same contents share one file (no second copy). Returns the cache path. Does not rsync by itself; .distsshkit/ is excluded from project sync.

Absolute cache path under project for hash.

Project-relative cache path for digest hash (64 hex chars).

push_cache!(session; hashes=nothing) -> SyncResult

Rsync local .distsshkit/cache/sha256/ blobs onto each SSH host under the same relative path on the session remote project root. Does not use --delete. Hosts run at most DISTSSHKIT_JOBS at a time (default 1). Host result order matches session.hosts. Project setup --rsync still excludes .distsshkit/; this call is the dedicated transfer.

hashes limits the push to those digests. nothing pushes every local blob. No SSH hosts → error. Empty cache → success with no host rows.

cache_remote_dir(remote_root) -> String

Join remote_root with .distsshkit/cache/sha256 (same layout as local cache_relpath).

plan(script; session=nothing, workers=[], project=pwd(),
     gb_per_worker=nothing, probe=nothing, mem_headroom, parent_gb) -> KitPlan

Inspect script without starting a job. Syntax only: Distributed vocabulary, map / filter / comprehensions, and independent indexed for (dest[i] = … with loop-var indices and no return in the RHS). Accumulating, stencil, or multi-statement for is out of scope. Effect analysis is not applied at this stage.

Suggests :drive, :ride, or :go. The caller still chooses the command.

Slot estimates (slots::WorkerPlan) run only when session is given, or when workers / gb_per_worker / probe request sizing. That path calls size! internally. Default is syntax only (no SSH probe).

Result of plan. Does not start a job.

One syntax finding from plan.

Print a KitPlan (CLI and tests).

pool!(session; gb_per_worker=nothing, mem_headroom, parent_gb) -> ResourcePool

Probe listed hosts for cores and RAM, then a slot hint via the same RAM/CPU formula as size! (no RSS measure_rss; that stays on size!). Unreachable hosts stay in the list with ok=false. The pool is ok only when every listed host probed successfully.

Does not start workers or rewrite go / drive membership.

Cluster view of listed hosts: totals plus per-host health.

ok is false if any listed host failed the probe (fail-closed). cores, gb, and slots sum reachable hosts only; failed rows stay in hosts. Does not change go / drive --best-effort.

One host in a ResourcePool. slots is 0 when ok is false.

Print cluster totals first, then per-host health.

size!(
    session::KitSession;
    gb_per_worker=nothing, probe=nothing,
    mem_headroom=DEFAULT_MEM_HEADROOM, parent_gb=DEFAULT_PARENT_GB,
)

Estimate worker counts for hosts in session. When gb_per_worker is omitted, probes each host via measure_rss (package-load baseline, optional warm-up probe script for peak RSS). Counts use effective_worker_gb.

Returns WorkerPlan.

Sized worker counts per host (parent_workers + child_workers).

pipeline!(driver, workers...; kwargs...) -> PipelineResult
pipeline!(driver, workers::AbstractVector; kwargs...) -> PipelineResult
pipeline!(config::PipelineConfig) -> PipelineResult

Run the usual remote workflow: optional sync, driver, optional collect. Does not call setup!; prepare remotes first.

Worker tokens match the CLI (parent:2, child:user@host:1). Listed tokens need :N. Keyword args are passed to the driver; remote is the remote project path. Default yes=true skips confirm prompts.

pipeline!(driver, "parent:2"; args=["8"])
pipeline!(driver, "child:user@h1:1", "child:user@h2:1"; remote="/path/to/project", args=["8"], collect=true)

Remote hosts default to no pre-run sync; set sync=:sync / :rsync explicitly. Collect defaults on when remotes are present (collect=false to skip). Use sync=:rsync only onto a missing/empty remote path (or setup --delete / setup!(session, :delete) first). pipeline! sync=:rsync copies only; it does not instantiate missing Manifest deps. Use drive! with sync=:rsync, or setup! :instantiate, for that.

Returns PipelineResult; check result.ok or use report_pipeline_errors.

PipelineConfig

Settings for pipeline!: sync, worker tokens, driver run, and optional collect.

Worker placement uses CLI-style tokens (parent:2, child:user@host:1). Listed tokens need :N. Set sync=false to skip sync. Set collect=false to skip rsync-back. Git parity is off by default; pass skip_hash_check=false (or CLI --require-git) to require matching remote commits.

julia sets the remote Julia binary (nothing / "auto" → detect; same as CLI --julia). Prefer pipeline!(driver, workers...; …) for day-to-day use.

pipeline_config_from_env(; driver=...)

Build PipelineConfig from environment variables.

Combined outcome of pipeline!. On failure, failed_step names the step that stopped.

report_pipeline_errors(result::PipelineResult; io=stderr)

Print a short summary when pipeline! failed. Returns result.ok.

Whether host denotes this job's parent in drive/go/size.

Only parent matches. parenthost is rejected by parse_placement_token. A token local / localhost / l is an ordinary SSH child (child:local).

Examples

julia> using DistSSHRun

julia> DistSSHRun.is_parent_host_name("parent")
true

julia> DistSSHRun.is_parent_host_name("parenthost")
false

julia> DistSSHRun.is_parent_host_name("local")
false

julia> DistSSHRun.is_parent_host_name("worker1")
false

host_tokens(hosts::AbstractVector{<:AbstractString}) -> Vector{String}
host_tokens(hosts::AbstractVector{Tuple{String,Union{Int,Nothing}}}; parent_workers=0)
host_tokens(parsed; kind::Symbol) -> Vector{String}

Rebuild CLI host tokens for execute!.

Go tokens are the parser strings. Ride uses the same shape. Drive tuples plus parent_workers emit parent:N then child:NAME:N. kind must be :go, :drive, or :ride.

execute!(kind, script, tokens=String[]; output_dir=nothing, args=String[], project=pwd(), sync=nothing, julia=nothing, detached=false, kwargs...) -> KitRunResult or KitProcess

One seam over go! / ride! / drive! for callers that pick the kind at runtime (kind ∈ (:go, :ride, :drive)), returning the shared KitRunResult instead of GoResult / RideResult / DriveResult.

execute!(:go, "job.jl", ["parent:2"]; args=["8"])
execute!(:ride, "job.jl", ["parent:2"])
execute!(:drive, "job.jl", ["parent:2"]; args=["8"])
wait(execute!(:go, "job.jl", ["parent:1"]; detached=true, args=["8"]))

output_dir, args, project, sync, julia are the keywords go! and drive! already share (ride! ignores sync). With detached=false (default), any other keyword is forwarded to the chosen function.

detached=true spawns julia -m DistSSHKit go|ride|drive and returns a KitProcess. --project= is project= when that tree lists DistSSHRun in Project.toml [deps] (julia -m needs a direct dep); otherwise pkgdir(DistSSHRun). A Manifest-only / transitive DistSSHRun does not count. Keywords are then an allow-list (unknown names throw): output_dir, args, project, sync, julia, quiet, verbosity, yes, remote, hosts_file, job_id, and drive-only log_dir, enable_log, package, require_all_hosts, skip_hash_check, mem_headroom, parent_gb, workers, sync_script. Go-only repeat. Ride: spi_check (default false). Detached argv passes --spi-check when true and --no-spi-check when false, so an older child whose default is on still skips the compare. Neither go nor ride takes size! flags. yes must be true (the default): an unattended child cannot answer a prompt. remote that starts with ~ is stored in DISTRIBUTED_REMOTE_PROJECT_ROOT as a layout path (not expanduser on the kit parent). Child stdio defaults to kit.out / kit.err in run_dir. Pass stdout / stderr (IO) to override; stdout=stdout inherits the parent. Parent redirect_stdout does not apply to the subprocess. Detached :drive omits --output-dir unless output_dir= or inherited DISTRIBUTED_OUTPUT_DIR is set, so init_output_dir! can choose the artifact root.

job_id, if given, is passed to the child as DISTSSHKIT_JOB_ID, which adds job=<id> to every progress: log line. DISTSSHKIT_PROGRESS=1 is --progress verbosity, not a watcher; read lines with DistSSHRun.parse_progress_line / DistSSHRun.kit_progress_latest. Omitted entirely when unset.

allocate_output_dir(kind, script; project=pwd(), job_id=nothing) -> String

Create a unique output directory for a later detached execute! and return its path. kind is :go, :drive, or :ride (the same values as execute!). The directory is created; pass it as output_dir=.

Layout is {script dir}/.distsshkit/<kind>/<script-stem>_<UTC-stamp>/. When job_id is set it is appended after the stamp (same charset as execute! job_id). The leaf is created with exclusive mkdir; if it already exists (same-second collision), a nanosecond suffix is retried so two allocations do not share a directory.

This matches omitted in-process defaults for go, ride, and drive: {script}/.distsshkit/<kind>/<stem>_<UTC>/. Drive still honors output_dir / --output-dir and a driver's init_output_dir! (DISTRIBUTED_OUTPUT_DIR) when those are set. Detached execute!(:drive) does not pin --output-dir unless output_dir= or inherited DISTRIBUTED_OUTPUT_DIR is set.

allocate_run_dir(kind, script; project=pwd(), job_id=nothing) -> String

Create a unique Kit run directory (cancel / logs / run.toml), not the artifact output_dir.

Layout: {script dir}/.distsshkit/runs/<kind>/<script-stem>_<UTC-stamp>/. When job_id is set it is appended after the stamp (same charset as execute! job_id).

If that parent tree is not writable (read-only pkgdir / Registry install), the same leaf is created under {project}/.distsshkit/runs/…, then under tempdir()/distsshkit-runs/….

ENV["DISTSSHKIT_RUN_DIR"] when set and non-blank, else nothing.

Read run.toml from a run directory (or the file itself). nothing if missing.

execute_detached_accepts(kw; kind) -> Bool

Whether execute!(kind, ...; detached=true) accepts keyword kw. Uses the same tables as the detached throw path, plus the named parameters (output_dir, args, project, sync, julia, detached).

execute_kwargs_from_parsed(parsed; kind) -> Dict{Symbol,Any}

Keywords for execute! (detached=true) from parse_go_args / parse_drive_args / DistSSHRun.parse_ride_args. Keys are a subset of execute_detached_accepts.

Does not include hosts_file / --hosts: those tokens belong in host_tokens (kind required). Does not set project, detached, yes, job_id, remote, or stdio. Drive --workers is :workers only when the parser set default_workers.

Handle to a detached execute! child (detached=true).

process is the julia -m DistSSHKit go|drive|ride subprocess. run_dir is always allocated in the parent before spawn (pid / kit.out / run.toml). output_dir is the artifact root when known at spawn (go / ride, explicit output_dir=, or inherited DISTRIBUTED_OUTPUT_DIR). Detached :drive without those leaves output_dir as nothing so the child can honor init_output_dir!. log_dir is nothing for :go / :ride. wait returns a KitRunResult; it does not fill output_dir on this handle.

wait(kp::KitProcess; timeout=nothing) -> KitRunResult

Block until the detached child exits, then return a KitRunResult.

Does not mutate kp.output_dir / kp.log_dir. After a detached :drive that deferred the artifact leaf, read the returned result (or kit_result_from_dir / read_kit_run_toml on kp.run_dir).

timeout is wall-clock seconds until the child process exits (not the drive worker heartbeat). nothing waits forever. On timeout the child is left running: failed_step is "hung", exit_code is 124, and owned stdio stays open. Call terminate! if the hang is fatal.

If the child wrote kit.result under run_dir or output_dir, that file is the source of truth (including failed_step from go! / ride!). Otherwise failed_step is "go" / "ride" / "drive" on a non-zero exit — the parent cannot recover a more specific in-process step name.

Best-effort: remove kit.pid if it still names this child (pid captured before wait on the OS process; after reap getpid can throw ESRCH). Not on a hung timeout.

kit_pid_file_running(output_dir) -> Bool

Whether kit.pid names a still-running child of this run. Requires DistSSHRun.kit_pid_alive and, when a start key is in the file, a match with kit_process_start_key. A leftover after SIGKILL whose pid was reused is false. One-line (pid-only) files keep the old probe. Missing file is false.

terminate!(kp::KitProcess; grace=10, interrupt_grace=0) -> KitRunResult

Stop a detached run. interrupt_grace == 0 (the default) does not send SIGINT. A positive interrupt_grace sends SIGINT and waits that many seconds first: a drive child runs its Ctrl-C path (interrupt workers, rmprocs, exit 130). On Julia 1.13, a driver waiting in pmap does not run that path, and SIGTERM stops it. A process-group SIGINT, as from a terminal Ctrl-C, also misses that path on Julia 1.13: the driver exits 1 and does not send kill -INT. Measured on macOS. On Julia 1.14 the driver exits 130, but a worker can be removed before it sees the interrupt. A go or ride child is julia -m with exit-on-sigint, so SIGINT runs atexit and does not enter catch or finally. Then SIGTERM, wait up to grace seconds (0 sends SIGTERM and does not wait), then SIGKILL if the process is still alive. Then pkill only processes tagged with this run's job_id (from kit.job), never julia.*--worker. Without job_id, only the child is signaled.

terminate_run!(output_dir; grace=10, interrupt_grace=0, log_dir=nothing, kind=:go) -> KitRunResult

Like terminate! after losing KitProcess: read kit.pid / kit.job / kit.hosts under output_dir. interrupt_grace is the same SIGINT stage. kind is only used when kit.result is missing.

kit_result_from_dir(output_dir) -> Union{Nothing,KitRunResult}

Read output_dir/kit.result written by a finished go / ride / drive child. nothing when the file is missing or unreadable (still running, or a hard death).

drive_host_status(output_dir) -> Vector{DriveHostStatus}
drive_host_status(kp::KitProcess) -> Vector{DriveHostStatus}

Live per-host membership for a running (or recently collecting) drive or ride. Reads kit.hosts.status. Empty when the file is missing (too early, a go run, or a hard death before join). This is not DriveResult hosts (post-run collect); that vector is stored in kit.result as hosts.

Live per-host membership during a drive (not the post-run collect outcome).

state is :joined, :alive, :left, or :collect_pending. last_seen is Unix time when this host last appeared in Distributed.workers(), or nothing if it has not been observed since join.

Parse go CLI arguments (same shape as other kit CLI parsers).

Parse drive CLI arguments (same shape as other kit CLI parsers).

Print go --help (same chrome as julia -m DistSSHKit go -h).

Print drive --help (same chrome as julia -m DistSSHKit drive -h).

Version of this meta-package, from its Project.toml.

println_kit_version(io::IO=stdout)

Print two lines: this package, then DistSSHRun and DistSSHQueue.

ssh_opts(; request_tty=false) -> Vector{String}

SSH argv flags for ssh / scp / rsync -e.

Reads DISTRIBUTED_SSH_OPTS live (not frozen at package precompile). Prefer this over a const so E2E / ProxyJump overrides apply in the same process.

A non-empty DISTRIBUTED_SSH_OPTS replaces the default vector (it is not merged). It is split with Base.shell_split, so a quoted value that contains spaces stays one argument. An unmatched quote throws ArgumentError. request_tty only changes the default vector: false includes -o RequestTTY=no; true omits it so a caller may put ssh -t before or after these flags. The ENV replacement is unchanged (if it contains RequestTTY=no, put -t after the vector). Use -o RequestTTY=no, not ssh -T: these flags are also passed to scp (scp -T is unrelated). Does not insert -t (that flag is ssh-only).

run_on_host(host, argv; julia=nothing, detect=true, tty=false, wait=true) -> Base.Process

One SSH connection: resolve remote Julia the same way as resolve_remote_julia / detect_julia_path, then exec it with argv (Julia flags / script / args). Does not replace resolve_remote_julia when the caller only needs the path.

julia=nothing / "auto" with detect=true probes candidates on the remote. detect=false requires an explicit path. tty=true adds ssh -t. Process-local detect cache is not used (a new CLI process never hits it).

SSH is ignorestatus: a non-zero remote or ssh exit returns the Process (.exitcode) instead of throwing ProcessFailedException. Missing ssh on PATH throws ArgumentError.

Resolve the Julia binary on this kit parent (nothing / "auto" / empty → the running process). Explicit paths are kept as given after usability check.

Throws ArgumentError when the binary is missing or --version does not parse.

Absolute local path with ~ expanded (canonical form for file I/O and comparisons).

Shorten absolute paths by replacing the home directory prefix with ~.

Walk upward from start_dir to find the directory that should be passed to Pkg.activate on workers.

If the first Project.toml found is a vendored DistSSHKit, DistSSHQueue, or DistSSHRun stub (name is one of those) and the parent directory also has a Project.toml, skip it and keep walking. Scripts next to that stub then use the application project root. The folder name does not matter.

Pkg environment for project (a directory, or a Project.toml path).

project_dir is the directory of that Project.toml (--project). manifest is Base.active_manifest of that file, or nothing when Pkg has no lock yet. env_dir is the directory of manifest, or project_dir when there is no lock. That is the tree setup --rsync sends.

Queue: default_queue_env (no dedicated ~/.distsshqueue/env) should return env_dir. stage_job_tree! should rsync env_dir, and set DISTRIBUTED_PROJECT_ROOT to the staged project_dir (relative to that tree). Serve --queue-env is env_dir. A job's --project stays project_dir, because env_dir as --project misses [deps] that exist only on the member.

Full script/driver-not-found message (headline + optional demo-related hint).

The demo tip stays here because it knows the bundled examples.

CLI user error on stderr (no stacktrace).

Kit help chrome — every overview / --help starts here:

DistSSHQueue setup
──────────────────

Exported for family callers. The signature is stable. Glyphs and colors are not.

Section heading (dim) with a trailing blank line so body lines follow immediately.

Section
<blank>
  body…

One or more verbatim help body lines.

One blank line in kit --help output.

Print msg with color when the kit would use ANSI (TTY, no NO_COLOR).

Frames for the kit TTY spinner (kit_spin!) and matching queue chrome.

worker_pmap(f, collection)

Like Distributed.pmap, but evaluates f with Base.invokelatest on each element.

After drive syncs your driver to workers and loads the project package, plain pmap is usually enough. Use worker_pmap when a callback still hits Julia 1.12+ world-age errors (for example, a function reference or closure captured before package load).

FIFO table: one DistSSHKit job at a time. Placement tokens are Kit's, not a Queue ceiling.

One table row. hosts are DistSSHKit placement tokens (parent[:N] / child:NAME[:N]). kwargs is an opaque bag for DistSSHRun execute! (project is the Kit project).

Enqueue. kind is a DistSSHRun execute! kind in KIT_EXECUTE_KINDS (:go, :ride, :drive).

hosts must be DistSSHKit 0.4 placement tokens (parent[:N] / child:NAME[:N]). Missing project uses job_project() (cwd / DISTRIBUTED_PROJECT_ROOT), not the serve env (enable --queue-env / Julia --project= that loaded Queue). Same verb as CLI submit.

q.allowed is the inventory (Queue(; allowed=…)), names plus optional max :N. It does not read config.toml. CLI submit uses Queue(; follow_config=true) so each enqueue re-reads hosts without restarting serve. step! does not drop :queued rows when a name is later removed, and does not stop a Kit job that is already :running.

Cancel :queued, or :running via DistSSHKit terminate_run! when the Kit sidecar dir is known.

Copy of the table.

Copy of one row. id may be a unique prefix of the stored UUID.

Load store (stale :running → :failed).

Start the FIFO head if nothing is :running. At most one Kit job.

Load store (stale :running → :failed), then step! until interrupt. Ctrl-C stops serve, not Kit.

A second serve on the same store prints Already running and returns.

serve(; store=default_store_path(), interval=0.2) -> Nothing

Run serve! on store. Ctrl-C stops this process, not a job that is already running. The CLI rejects a non-finite --interval. interval == 0 still runs and warns once; a negative finite value warns once, then sleep fails.

Jobs file for a queue host: ~/.distsshqueue/jobs.toml under home.

Kit project: DISTRIBUTED_PROJECT_ROOT, else the Project.toml above cwd, else cwd.

Not the Julia --project= that loaded DistSSHQueue.