API
Julia entry points when you embed DistSSHKit in a notebook or your own package. Day-to-day work stays on the CLI (julia --project=. -m DistSSHKit …); see Introduction, First Steps, and the User Guide. REPL help also works (?DistSSHKit.go!).
The shape mirrors the CLI: go! for as-is scripts, drive! (and friends) for Distributed drivers. pipeline! is optional sugar that runs the usual remote order in one call. Worker placement uses the same tokens as the CLI (local:2, user@host:1):
pipeline!(driver, "local:2"; args=["8"])
pipeline!(driver, "user@h1:1", "user@h2:1"; remote="/path/to/project", args=["8"])
go!("job.jl", "local:2"; args=["8"])
drive!("job.jl", "local:2"; args=["8"])Run a script as-is — go!
No Kit imports in the job file. Each local:N / host:N slot is one full run, concurrent.
DistSSHKit.go! — Function
go!(script, workers...; kwargs...)
go!(script, workers::AbstractVector; kwargs...)Run an as-is complete job on one or more slots (local and/or remote).
go!("job.jl") # one local slot
go!("job.jl", "local:2"; args=["8"])
go!("job.jl", "user@h1:1", "user@h2:1"; remote="/path/to/project")Each slot gets DISTRIBUTED_OUTPUT_DIR pointing at <project>/.distsshkit/go/<stem>_<UTC>/<slot>/. Setup on remotes is assumed done. Override the batch root with output_dir (CLI: --output-dir). For backward compatibility collect_spec::AbstractString also sets the batch root, but passing both output_dir and collect_spec::String is an error. collect_spec === false means "skip collect" and is orthogonal to output_dir.
Default sync is false (no pre-run sync; prepare remotes with setup! or CLI setup first — :rsync / --rsync or :clone / --clone, then :instantiate / --instantiate). Pass sync=:sync or sync=:rsync to sync before running. Use sync=:rsync only onto a missing/empty remote path (or setup --delete / setup!(session, :delete) first). go! has no git-parity gate; use drive! with skip_hash_check=false (CLI: drive --require-git) when you need that.
julia sets the Julia binary for each slot (nothing / "auto" → detect; same as CLI --julia).
local:N and host:N mean N independent full-job runs (not Distributed workers), started together. path_anchor shortens displayed paths (CLI passes kit project root).
DistSSHKit.GoResult — Type
Outcome of go!. On failure, failed_step is "sync", "run", or "collect".
DistSSHKit.report_go_errors — Function
report_go_errors(result::GoResult; io=stderr)Print a short summary when go! failed. Returns result.ok.
DistSSHKit.KitRunResult — Type
Shared run outcome for the queue layer (ok, kind, dirs, failed_step, exit_code).
kind is :go, :drive, or :pipeline. Convert with kit_run_result.
DistSSHKit.kit_run_result — Function
Build KitRunResult from a kit outcome (:go / :drive / :pipeline).
DistSSHKit.report_run_errors — Function
report_run_errors(result; io=stderr)Print a short summary when a kit run failed. Accepts KitRunResult or a typed outcome (GoResult / DriveResult / PipelineResult). Returns result.ok.
Drive work across workers
When the script is a driver (init_output_dir! / main, pmap, …), build a KitSession, then call the steps you need:
(optional setup! / sync! / instantiate!) → size! → drive! → (optional collect!)First-time remotes usually look like:
session = KitSession(workers=["user@h1"], remote="/path/to/project", yes=true)
setup!(session, :delete, :rsync, :instantiate)
setup!(session, :check; ignore_julia_version=true) # optional
setup!(session, :runtest) # optional: job Pkg.test() on remotes
# git trees: setup!(session, :clone; repo="https://…") instead of :rsyncsetup! mirrors julia -m DistSSHKit setup --… (:delete, :rsync, :clone, :sync, :pull, :instantiate, :check, :runtest, :cleanup). Confirmations follow session.yes. :clone requires repo= — no silent origin lookup; clone runs on the remote. sync! / instantiate! remain as short aliases for the common deploy steps.
A few points that carry over from the CLI:
go!/drive!/pipeline!do not pre-run sync or require git parity by default. Passsync=:sync/:rsyncfor a one-shot deploy- Git parity (
skip_hash_check=false, CLI:drive --require-git) is drive / pipeline only —go!stays simpler - Prefer positional worker tokens over building a
WorkerPlanby hand (WorkerPlanis the return type ofsize!) - Pass
julia=ongo!/drive!/pipeline!to pin the remote Julia binary (same as CLI--julia)
Or call pipeline! for optional sync → size! → drive! → collect in one shot (pipeline! does not call setup!). pipeline_config_from_env reads DISTSSHKIT_HOSTS / DISTSSHKIT_HOSTS_FILE, SYNC_MODE (rsync / sync / off; unset → off for remotes too), JULIA_DISTRIBUTED_EXE (same as CLI --julia), and the usual quiet / progress / yes flags — same vocabulary as the CLI.
DistSSHKit.KitSession — Type
Runtime context for sync!, size!, drive!, collect!.
DistSSHKit.setup! — Function
setup!(session::KitSession, mode::Symbol; kwargs...) -> SyncResult
setup!(session::KitSession, modes::Symbol...) -> SyncResultPrepare SSH hosts — same jobs as julia -m DistSSHKit setup --….
mode | CLI | Notes |
|---|---|---|
:delete | --delete | Destructive; confirm unless session.yes |
:rsync | --rsync | Refuses nonempty remote; delete first to replace |
:clone | --clone | Requires repo=; clone runs on the remote |
:sync | --sync | Local push + remote pull (git remotes); confirm unless session.yes |
:pull | --pull | Local pull then remote pull; confirm unless session.yes |
:instantiate | --instantiate | julia= (default "auto") |
:check | --check | ignore_julia_version=, check_code_sync= |
:runtest | --runtest | job Pkg.test() on remotes; julia= |
:cleanup | --cleanup | Kill stale workers (no confirm) |
Confirmations follow session.yes (CLI -y). Multiple modes run in order and stop on the first failure:
session = KitSession(workers=["user@h1"], remote="~/proj", yes=true)
setup!(session, :delete, :rsync, :instantiate)
setup!(session, :check; ignore_julia_version=true)sync! / instantiate! remain as thin aliases for the common deploy steps. Prefer setup! when you want the full CLI vocabulary in one place.
DistSSHKit.sync! — Function
sync!(session::KitSession; mode=:sync)Sync local project to SSH hosts (call explicitly; go/drive do not call this by default).
mode=:sync— git push + pull on remotes (confirm unlesssession.yes)mode=:rsync— rsync working tree (no git; confirm unlesssession.yes). Refuses nonempty remote paths; usesetup!(session, :delete)/setup --deletefirst.
Also available as setup!(session, :rsync) / setup!(session, :sync).
Returns SyncResult.
DistSSHKit.instantiate! — Function
instantiate!(session::KitSession; julia="auto") -> SyncResultRun Pkg.instantiate() on each SSH host in session (parallel).
julia is the remote Julia path ("auto" detects per host, same as setup --instantiate / drive --julia). Returns SyncResult.
Typical first-time remote prep:
session = KitSession(workers=["user@h1"], remote="/path/to/project", yes=true)
setup!(session, :delete, :rsync, :instantiate) # or sync!(…; mode=:rsync) then instantiate!Also available as setup!(session, :instantiate; julia=…).
DistSSHKit.HostResult — Type
Per-host outcome for sync and similar operations.
DistSSHKit.SyncResult — Type
Outcome of sync! (rsync or git sync). cancelled is set when a confirm prompt is aborted.
DistSSHKit.size! — Function
size!(
session::KitSession;
gb_per_worker=nothing, probe=nothing,
mem_headroom=DEFAULT_MEM_HEADROOM, master_gb=DEFAULT_MASTER_GB,
)Estimate worker counts for hosts in session. When gb_per_worker is omitted, probes each host via measure_rss (package-load baseline, optional warm-up probe script for peak RSS). Counts use effective_worker_gb.
Returns WorkerPlan.
DistSSHKit.WorkerPlan — Type
Sized worker counts per host (local_workers + remote_workers).
DistSSHKit.drive! — Function
drive!(session::KitSession, script; plan=nothing, args=[], ...)
drive!(script, workers...; kwargs...)
drive!(script, workers::AbstractVector; kwargs...)Run a driver script on workers. Tokens match the CLI (local:2, user@host:1).
drive!("job.jl", "local:2"; args=["8"])
drive!(session, "job.jl") # uses `session.tokens`Prepare remotes with setup! or CLI setup first. Optional sync=:sync / sync=:rsync runs sync! (same as setup!(session, :sync) / :rsync) immediately before workers. Git parity is off by default (skip_hash_check=true). With sync=:rsync, parity stays off even if skip_hash_check=false (no remote .git/). require_all_hosts=true (CLI --require-all-hosts) fails if a listed SSH host did not join, or if collect reported an error (default: best-effort, exit 0).
julia sets the remote Julia binary (nothing / "auto" → detect; same as CLI --julia). plan is an optional explicit WorkerPlan. pipeline! syncs separately and does not pass sync= into drive!.
DistSSHKit.DriveResult — Type
Outcome of drive! (and similar CLI steps that return an exit code).
output_dir / log_dir are the directories actually used for this run — resolved the same way drive reports Results: / writes its log, even when drive! was not called with output_dir= / log_dir=. nothing when no real run happened (e.g. built by hand) or, for log_dir, when logging was disabled.
DistSSHKit.collect! — Function
collect!(
session::KitSession,
local_root::AbstractString;
merge=false,
hosts=nothing,
)Rsync result files from SSH hosts into local_root.
Collect modes:
merge=false→ collect-missing (CLIdrive --collect-missing)merge=true→ collect-overwrite (CLIdrive --collect-overwrite)
Distinct from drive's automatic post-run-new (sentinel / newer-than-run) and go's slot-overwrite. Returns CollectResult.
DistSSHKit.CollectResult — Type
Outcome of collect!.
DistSSHKit.pipeline! — Function
pipeline!(driver, workers...; kwargs...) -> PipelineResult
pipeline!(driver, workers::AbstractVector; kwargs...) -> PipelineResult
pipeline!(config::PipelineConfig) -> PipelineResultRun the usual remote workflow: optional sync, size!, driver, optional collect. Does not call setup!; prepare remotes first.
Worker tokens match the CLI (local:2, user@host:1). Bare hosts are sized with size!. Keyword args are passed to the driver; remote is the remote project path. Default yes=true skips confirm prompts.
pipeline!(driver, "local:2"; args=["8"])
pipeline!(driver, "user@h1:1", "user@h2:1"; remote="/path/to/project", args=["8"], collect=true)Remote hosts default to no pre-run sync; set sync=:sync / :rsync explicitly. Collect defaults on when remotes are present (collect=false to skip). Use sync=:rsync only onto a missing/empty remote path (or setup --delete / setup!(session, :delete) first).
Returns PipelineResult; check result.ok or use report_pipeline_errors.
DistSSHKit.PipelineConfig — Type
PipelineConfigSettings for pipeline!: sync, worker tokens, driver run, and optional collect.
Worker placement uses CLI-style tokens (local:2, user@host:1). Bare hosts are sized via size!. Set sync=false to skip sync. Set collect=false to skip rsync-back. Git parity is off by default; pass skip_hash_check=false (or CLI --require-git) to require matching remote commits.
julia sets the remote Julia binary (nothing / "auto" → detect; same as CLI --julia). Prefer pipeline!(driver, workers...; …) for day-to-day use.
DistSSHKit.pipeline_config_from_env — Function
pipeline_config_from_env(; driver=...)Build PipelineConfig from environment variables.
| Variable | Role |
|---|---|
DISTSSHKIT_HOSTS | Comma-separated worker tokens (host / host:N) |
DISTSSHKIT_HOSTS_FILE | Hosts file (appended after DISTSSHKIT_HOSTS) |
DISTRIBUTED_REMOTE_PROJECT_ROOT | Remote repo root |
DISTRIBUTED_PROJECT_ROOT | Local project root |
DRIVER | Driver script path |
GB_PER_WORKER | Skip RSS probe when set |
DISTSSHKIT_SIZE_PROBE | Optional warm-up script for peak RSS (see size --probe) |
SYNC_MODE | rsync, sync, or off |
JULIA_DISTRIBUTED_EXE | Remote Julia path (same as CLI --julia; auto / unset → detect) |
DISTSSHKIT_YES / DISTSSHKIT_QUIET / DISTSSHKIT_PROGRESS / DISTSSHKIT_VERBOSE | Same as CLI -y / -q / --progress / --verbose |
DistSSHKit.PipelineResult — Type
Combined outcome of pipeline!. On failure, failed_step names the step that stopped.
DistSSHKit.report_pipeline_errors — Function
report_pipeline_errors(result::PipelineResult; io=stderr)Print a short summary when pipeline! failed. Returns result.ok.
Worker tokens — local:N / host:N
Use this surface when callers need to classify tokens or decide whether size! is needed before building workers (e.g. the queue layer's own occupancy math), instead of re-parsing the grammar or reaching into private internals.
parse_worker_tokens validates and classifies the grammar. worker_tokens_fully_specified says whether every token has an explicit :N. remote_hosts_from_tokens extracts only SSH host names. worker_plan_from_tokens resolves to a concrete WorkerPlan. split_worker_token and is_local_host_name are the low-level primitives.
DistSSHKit.parse_worker_tokens — Function
parse_worker_tokens(tokens) -> ParsedWorkerTokensClassify CLI-style tokens. Explicit :N is fixed; bare hosts are sized later. local / l / localhost are local; everything else is remote SSH.
DistSSHKit.ParsedWorkerTokens — Type
Parsed drive/go worker tokens (counts may still need size!).
Build with parse_worker_tokens; the keyword constructor here only coerces types (same convenience shape as DriveResult / PipelineResult), it does not re-validate cross-field consistency.
Field name matches WorkerPlan: a host with an explicit :N is remote_workers[host] = N, same key as WorkerPlan.remote_workers.
DistSSHKit.worker_tokens_fully_specified — Function
True when every token has an explicit worker/slot count (:N).
DistSSHKit.remote_hosts_from_tokens — Function
SSH host names from tokens (local tokens omitted).
DistSSHKit.worker_plan_from_tokens — Function
Build a WorkerPlan from tokens.
When any host lacks :N, run size! on session (must list those hosts; set include_local_for_size for bare local). Explicit :N wins over size.
DistSSHKit.split_worker_token — Function
Parse host or host:N into (hostname, workers).
N is a worker/slot count for drive / go. setup / size keep the hostname only (split_worker_token(…)[1]).
DistSSHKit.is_local_host_name — Function
Whether host denotes this machine in drive/go (local / localhost / l).
Examples
julia> using DistSSHKit
julia> DistSSHKit.is_local_host_name("local")
true
julia> DistSSHKit.is_local_host_name("worker1")
falseOne seam over go! / drive! — execute!
For callers that pick the kind at runtime (the queue layer): one function, one result type, instead of branching on kind yourself.
execute!(:go, "job.jl", ["local:2"]; args=["8"])
execute!(:drive, "job.jl", ["local:2"]; args=["8"])
wait(execute!(:go, "job.jl", ["local:1"]; detached=true, args=["8"]))detached=true:
- Spawns a child
julia -m DistSSHKit go|drive(not in-processgo!/drive!) - Keywords are an allow-list;
yesmust staytrue - Child stdio inherits the parent —
redirect_stdoutin the caller does not apply to the subprocess. Passstdout/stderrto capture it instead KitProcessholds theBase.Processand the dirs resolved before spawnwaitconverts it toKitRunResult; on a non-zero child exit,failed_stepis"go"/"drive"only
DistSSHKit.execute! — Function
execute!(kind, script, tokens=String[]; output_dir=nothing, args=String[], project=pwd(), sync=nothing, julia=nothing, detached=false, kwargs...) -> KitRunResult or KitProcessOne seam over go! / drive! for callers that pick the kind at runtime (kind ∈ (:go, :drive)), returning the shared KitRunResult instead of GoResult / DriveResult.
execute!(:go, "job.jl", ["local:2"]; args=["8"])
execute!(:drive, "job.jl", ["local:2"]; args=["8"])
wait(execute!(:go, "job.jl", ["local:1"]; detached=true, args=["8"]))output_dir, args, project, sync, julia are the keywords go! and drive! already share. With detached=false (default), any other keyword (remote, hosts_file, quiet, verbosity, yes, collect_spec, path_anchor, skip_hash_check, require_all_hosts, plan, …) is forwarded verbatim to the chosen function.
detached=true spawns julia -m DistSSHKit go|drive and returns a KitProcess. Keywords are then an allow-list (unknown names throw): output_dir, args, project, sync, julia, quiet, verbosity, yes, remote, hosts_file, and drive-only log_dir, enable_log, package, require_all_hosts, skip_hash_check. yes must be true (the default): an unattended child cannot answer a prompt. Child stdio inherits the parent; pass stdout / stderr (IO) to capture. Parent redirect_stdout does not apply to the subprocess.
DistSSHKit.KitProcess — Type
Handle to a detached execute! child (detached=true).
process is the julia -m DistSSHKit go|drive subprocess. output_dir / log_dir are resolved in the parent before spawn so they match the child (log_dir is nothing for :go, matching kit_run_result on GoResult). Convert with wait.
Inside a driver — worker_pmap
World-age escape hatch when a driver needs pmap-like fan-out after defining methods in the same session.
DistSSHKit.worker_pmap — Function
worker_pmap(f, collection)Like Distributed.pmap, but evaluates f with Base.invokelatest on each element.
After drive syncs your driver to workers and loads the project package, plain pmap is usually enough. Use worker_pmap when a callback still hits Julia 1.12+ world-age errors (for example, a function reference or closure captured before package load).