DistSSHKit.jl

DistSSHKit is a kit for running the same Julia project locally and over SSH, then collecting the results. It makes SSH-distributed runs easier and more uniform, which helps keep those runs reproducible. It uses Distributed.jl processes, not threads. Supported on macOS, Linux, and WSL2 Ubuntu (not native Windows).

Even small labs and individuals often have a few high-performance machines or workstations. DistSSHKit helps you use that hardware as a small set of compute nodes. A lightweight scheduler, DistSSHKitQueue.jl, is also in progress.

Note

0.3 is not getting major new features for now. The current commands stay put, and ordinary bugs still get fixed. CONTRIBUTING.md · Discussion #26.

What is DistSSHKit?

Two ways to run a script:

Same script on each machine (go) — each host runs your .jl from start to finish. No rewrite needed. Prefer this when every run is already a complete job.
One machine coordinates (drive) — your main Julia process stays in charge and hands pieces of the work to the others (Distributed.jl).

Around that, the kit handles remote project setup, sync, and collecting outputs. Use it from the terminal or from Julia code / notebooks.

How you call it is a separate choice:

Julia API — setup! for remotes, go! / drive! to run, or pipeline! for optional sync → size! → drive! → collect (not setup!)
CLI — julia --project=. -m DistSSHKit go … / drive … (and setup, demo, …)
distsshkit (experimental) — after pkg> app add DistSSHKit, a distsshkit command on the terminal. Same flags as -m, but always the Apps copy, not --project=.. Fine for go / setup / demo; keep drive and size on julia --project=. -m DistSSHKit. When to use it: User Guide.

All of these need Julia 1.12+ (Requirements).

Same host tokens for all of these (local:2, user@host:1). Details: API, User Guide.

Installation

From the Julia REPL, type ] to enter the Pkg REPL mode and run:

pkg> add DistSSHKit

Or, equivalently, via the Pkg API:

julia> import Pkg; Pkg.add("DistSSHKit")

Optional distsshkit command (1.12+, experimental): User Guide.

Also needs ssh, rsync, and git (git deploy only); pkg> add does not install them. Requirements.

Basic terms

Host — the machine that runs the work. Local is local. An SSH target is user@hostname, an IP address, or an SSH config Host alias
Process — one running julia. Each process has its own memory and runs independently at the OS level (this kit launches multiple julia processes, even on a single machine, to run work in parallel — built on Distributed.jl)
Master — the process that coordinates the whole run: it hands out work to workers and collects the results
Worker — a process that receives work from the master and runs it

Example: a local machine plus remotes. Each machine can run several workers (local may run none), and you can add as many remote machines as you like.

Master on the local controller, workers on local and remote machines

There's no limit on the number of remote hosts — more hosts just means more time spent on SSH connections and deployment, so it's best to start with a few and scale up. Each remote host needs:

Passwordless SSH from your local machine
Julia with the same major.minor version as your local machine (setup --check verifies this)

Details: Requirements.

Start at Requirements, then Prepare (remotes) and the bundled Demo.

Later: setup, go, drive, and the rest of the User Guide; or the API to embed from Julia.

Contributing

Bugs and feature requests: Issues. Questions and ideas: Discussions. See CONTRIBUTING.md for how to contribute.