DistSSHKitQueue.jl

DistSSHKitQueue runs jobs one after another on machines that several people share. You can submit a job, check its status, and cancel. DistSSHKit does the run. Supported on macOS, Linux, and WSL2 Ubuntu (not native Windows).

Even small labs and individuals can keep one always-on machine, add SSH hosts, and use them together as a small set of compute nodes. Not on General yet. Julia 1.12+, DistSSHKit 0.4.1+. Placement tokens (parent[:N] / child:NAME[:N]) stay DistSSHKit's — see the kit docs.

What is DistSSHKitQueue?

Day to day you submit from a client (qhost:HOST). If no waiter is up, submit starts one on the queue host. You do not need setup, serve, or enable for a job to run.

How you call it:

CLI — julia --project=. -m DistSSHKitQueue qhost:HOST submit go …
Julia API — submit! / cancel! / serve! on a Queue (API)

qhost:NAME names the queue host (like Kit child:NAME, but not a worker). --hosts / --julia stay on Kit go / drive. Queue-host Julia is --remote-julia / JULIA_DISTRIBUTED_EXE.

Installation

From the Julia REPL, type ] to enter the Pkg REPL mode and run:

pkg> add https://github.com/yamanori99/DistSSHKitQueue.jl#v0.1.0-beta.1

Or, equivalently, via the Pkg API:

julia> import Pkg; Pkg.add(url="https://github.com/yamanori99/DistSSHKitQueue.jl", rev="v0.1.0-beta.1")

Not on General yet, so pin the beta tag by URL. DistSSHKit 0.4.1+ comes from General with it. Do not Pkg.develop Kit for ordinary Queue work.

Also needs ssh, rsync, and git (git deploy only); pkg> add does not install them. Requirements.

Basic terms

Queue host — the always-on machine that holds ~/.distsshkitqueue and runs the waiter. A sleeping laptop is not this box.
Client — a dev machine that submits, lists, watches, or cancels. No cap. It must not become the Kit master.
Waiter — serve on the queue host. It starts DistSSHKit (execute!(…; detached=true)) and waits. Stopping it does not cancel a Kit job that is already running.
Workers — where the script runs. DistSSHKit tokens: parent[:N] on the queue host, child:NAME[:N] on SSH machines.

  clients = dev machines (no cap)          one queue host (always on)
  ───────────────────────────────          ──────────────────────────
  yours / a colleague's / …                waiter   one Kit job at a time
       │                                   table    ~/.distsshkitqueue
       │  julia -m DistSSHKitQueue         add-host / remove-host
       │    qhost:NAME                     serve    now, this terminal
       │    submit | status | list-host    enable   again after reboot
       │    watch | cancel | …
       └────────────────────────────────►  then DistSSHKit go/drive
                                           → workers (Kit tokens)

Queue is not a bigger Kit and does not keep a lab-wide slot ceiling. serve is “run the process”. enable is “register that process with the OS”. They are not two ways to start the same thing.

Start at Requirements, then Prepare (queue host, once) and First job (from a laptop).

Later: submit, status, and the rest of the User Guide; or the API to embed from Julia.

Contributing

Bugs and feature requests: Issues. See CONTRIBUTING.md.

License