DistSSHRun

DistSSHRun is one run on shared machines over SSH: setup, go, ride, drive, hostinfo, pool, demo, and progress. Supported on macOS, Linux, and WSL2 Ubuntu (not native Windows).

The command is julia -m DistSSHRun.

pkg> app add DistSSHRun still installs distsshrun. That command is experimental. The binary is the Apps copy, not --project=.. Apps fixes JULIA_LOAD_PATH, so drive, hostinfo, and pool still need julia -m DistSSHRun. Worker processes (local and SSH) set JULIA_LOAD_PATH=@:@stdlib and --project= of the job tree; they do not see @v#.#. Local workers also put the parent project, then DistSSHRun's package directory, on that path so ride can load DistSSHRun when the job tree does not. Parent LOAD_PATH mutations do not otherwise affect workers.

DistSSHKit does not install a terminal command. When that package is installed, julia -m DistSSHKit is the same entry.

The longer guide is the DistSSHKit manual.

Install

pkg> add DistSSHRun

Julia 1.13+. Hosts need ssh. Collect, collect!, and go slot pull-back use stdlib Tar over SSH. The remote Julia executable is --julia, or auto-detect. Original mtimes are not kept. Pull-back touches extracted files. Ship does not touch; files keep the extract-time mtime. A non-executable file arrives as mode 0644, and an executable file as 0755. A symlink that points outside the copied tree is refused. push_cache! uses stdlib Tar over SSH and needs a remote Julia executable (--julia, or auto-detect) that can load stdlib Tar and SHA. Ship uses stdlib Tar over SSH. Git clone, sync, and pull also need git.

setup --ship-tree, go --ship-tree, and drive --ship-tree walk the ship root and apply each directory's .gitignore (case-sensitive). .git/info/exclude, core.excludesFile, and a .gitignore above the ship root are not used. No git binary is run. The ship root is the project directory that was passed, not a workspace Manifest directory above it. A Manifest or path dependency outside that directory is left out with a warning. A gitignored Project.toml, JuliaProject.toml, Manifest.toml, or JuliaManifest.toml is left out with one warning. A gitignored LocalPreferences.toml is left out too; add !LocalPreferences.toml to ship it. A file over 50 MiB, or a send list over 100 MiB, prints a warning (DISTSSHKIT_SHIP_WARN_FILE_BYTES, DISTSSHKIT_SHIP_WARN_TOTAL_BYTES; an invalid value is an error). The 256 MiB cap still stops the ship. Declared inputs are left out. ship_send_list(root) is that path list for the directory it is given. It is public and is not exported. rsync_send_list remains the git list for clone, sync, and pull. Extract is into <remote_root>/runs/<run_id>/src/ (run_id is UTC plus 6 hex). The same leaf also gets an empty <remote_root>/runs/<run_id>/out/. A repeated host is refused. A missing remote_root is created when safe: a ~/… layout may create components under the remote home with mode 700; an absolute path requires its parent and creates only the final component (same rule as setup --remote-root). A symlink at the target, unsafe ownership or permissions, /, a single directory under / (for example /scratch), a path with . or .., this machine's home, and the remote home itself are refused. Existing ancestor directory symlinks are followed; components this operation creates must not be symlinks. Existing directories are not chmod'd. Transfer failure after create leaves the safe directory. go / drive --ship-tree and sync!(mode=:ship_tree) use the same ensure path; they still instantiate after ship when those entry points do today. Later --instantiate / --check / go use the latest src/ when the remote root has no Project.toml (or pass --remote-path to that tree). --rsync was removed in 0.2.0. Workers --project= that tree. snapshot(local_root) builds that tar in a temporary directory and records content_tree_hash (git-sha1). The caller deletes snapshot_directory. The tar is snapshot_tar (ship.tar in that directory). snapshot does not read depot preferences. ship(snapshot, hosts, remote_root) sends that tar and does not pack again or delete it. It refuses depot preferences, hashes the tar again immediately before sending, and throws ArgumentError when the hash differs or the tar is missing. confirm defaults to false, so a library ship does not prompt. record=false does not keep the ship for controller run.toml in this process. setup --ship-tree still confirms unless -y is used, and that path deletes its temporary directory. setup --remote-root --remote-path /absolute/path creates that one directory with mode 700. Group and other write are cleared again after mkdir. The parent must already exist. ~ and DISTRIBUTED_REMOTE_PROJECT_ROOT are not a source for this command. Confirmation is required unless -y is used. The directory must be owned by the SSH user and must not be group-writable or world-writable. No marker file is written, and runs/ is not created. Controller provenance lives under DISTSSHKIT_RUN_DIR/run.toml (schema = 1, writer = "DistSSHRun x.y.z"). When the run that wrote this file shipped a tree, provenance.shipped records content_tree_hash (git-sha1) and run_id (the remote runs/<run_id>/ leaf). An absent or empty key means that ship was not recorded. result_content_tree_hash and result_remote_run_id read those strings from a DriveResult, GoResult, or KitRunResult. They also read a :ship_tree SyncResult and a ShipTreeResult. A SyncResult from any other mode returns nothing for both. result_run_dir reads DISTSSHKIT_RUN_DIR from those results and from KitProcess. Each remote runs/<run_id>/ also gets a minimal run.toml with only schema, writer, and kind = "ship" so setup --delete can refuse a Ship leaf that is not DistSSHRun-owned. That stamp is not controller provenance. setup --prune searches .distsshkit candidates and applies the same ownership check there; it does not inspect those Ship leaves. Older leaves without that file still use structural checks only. DistSSHQueue teardown is a different command; it may reuse the same remote stamp later. Unknown keys are ignored when reading. A ship that sees .distsshkit/inputs.toml writes those inputs into this run's run.toml. With hashing on, it also writes a manifest under inputs/<sha256>.manifest. The file name is the SHA-256 of ncodeunits(name):name, a newline, and the host. Each line is hex *path. A regular file is the SHA-256 of its bytes. A symlink is the SHA-256 of its link text and is not followed. sha256sum -c follows the symlink and normally reports FAILED; use check_input_manifest to verify symlink rows. A declared path that is a symlink to a file is recorded as that link, not as the target bytes. A link target with a tab or newline is refused, including when the declared path itself is a symlink. A path with a backslash is refused. check_input_manifest(manifest, root) checks both and resolves each path component by NFC form when the filesystem kept NFD. sha256sum -c opens paths literally, so use check_input_manifest for those trees. hash = false records only stat fields in run.toml and does not write a manifest. A later ship into the same run directory replaces the row for the same input name and host. Standalone setup --ship-tree waits for hashing before it returns. drive / go with --ship-tree start hashing after extract and finish it while instantiate, worker launch, and precompile run. outputs is reserved and not written. There is no run.private.toml; public records keep relative paths and aliases only.

terminate! and terminate_run! record how the stop ended. KitRunResult.stop is :within_grace when the process exited after one SIGINT and before SIGTERM, :sigterm or :sigkill when that interrupt_grace window ended and the ladder escalated, and :no_sigint when interrupt_grace was 0. nothing means this call did not run the ladder. The same string is stop in kit.result. When the kit run directory already has this package's run.toml, stop is set there too. terminate_run! finds that directory from kit.run_dir next to kit.pid. A run.toml in the artifact output_dir or log directory is left alone unless the supplied output_dir already holds this package's run.toml; terminate_run! updates that file. schema stays 1. Read it from the return value or from kit_result_from_dir / read_kit_run_toml.

The remote login shell ($SHELL) must be POSIX-compatible (sh, bash, dash, or zsh). ssh host command runs that shell as $SHELL -c, not as a login shell. Distributed's SSHManager starts workers with sh -l -c, and that wrap cannot be changed. fish and csh are unsupported. fish can silently change quoting (a backslash inside single quotes). An SSH name that starts with -, including user@-host, is rejected. A host or remote path that contains a control character (NUL, newline, or any other) is rejected. See julia -m DistSSHRun help setup.

Commands

setup --check fails when the worker Julia patch or prerelease differs from this process. A build suffix alone still passes. --ignore-julia-version turns that skew into a warning. go and drive compare the full VERSION string and the Julia commit. up runs juliaup update even when the channel name already matches.

julia -m DistSSHRun setup --check child:host1
julia -m DistSSHRun go SCRIPT.jl
julia -m DistSSHRun ride parent:2 SCRIPT.jl
julia -m DistSSHRun drive parent:2 SCRIPT.jl
julia -m DistSSHRun hostinfo parent
julia -m DistSSHRun pool

using DistSSHRun
go!("job.jl")

Delete and prune

setup --delete and setup --prune remove a remote tree only when --confirm-path is the path that command prints. -y, --yes, and DISTSSHKIT_YES do not authorize those two modes. Without that flag and without -y, the command still asks you to type delete or prune.

--delete also refuses when a present runs/<run_id>/run.toml under the deploy root is not DistSSHRun-owned, including a symlink runs/ or a symlink directly under runs/. The confirmed path may be a git-work-tree ancestor of that deploy root; the stamp check still uses the deploy root. --prune uses the same ownership rule on .distsshkit candidates. A missing stamp (older trees) still uses structural checks only.

From Julia, setup!(session, :delete; confirm_path) and :prune follow the same rule. session.yes does not delete. One SSH host accepts a String. More than one host needs a Dict whose keys are the configured host aliases. A host with no key is left in place. An unknown key is an error. A mismatch throws ConfirmPathMismatch. Do not copy the target out of that error into a retry. See help setup. teardown is DistSSHQueue, not this package.