Tools and config with carry

A container loses everything added after its image was built. wfnexus does not try to bake every tool a workflow might want into its images. The server and runner images start through carry (carry run --), which restores config files, env defaults and tools from an S3 bucket before the process starts, and commits every change you make with the carry CLI back as a new encrypted version.

Only the agent CLIs (claude, devin, codex) stay baked into the image. Everything else (gh, jq, rg, a .gitconfig, a log level) is carry’s job.

Turn it on

carry is off until CARRY_REPO is set; without it carry run -- simply runs the server or the runner. To turn it on, pass these to the container (the shipped docker-compose.yml already passes them through):

Variable	What it is
CARRY_REPO	the bucket and prefix, e.g. s3://my-bucket/carry
CARRY_S3_ENDPOINT	the S3 endpoint
CARRY_KEY_FILE	the encryption key file for the stored state
CARRY_STATE	which state this machine restores; defaults to the project name on the server and the worker’s name on a runner
CARRY_TOOL_PUBKEY	the public key every tool is verified against before it runs

S3 credentials come from the environment by name (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), never from a file in the repo. wfx doctor prints whether carry is configured on the server.

Publish a tool once

A tool is published to the bucket once, signed, and promoted to a channel:

carry tool keygen                                   # signing key; prints the public key only
carry tool publish jq@1.7.1 --file ./jq-linux-amd64 --platform linux/amd64
carry tool promote jq@1.7.1 --channel stable

Get the server there with wfx

A server is a profile: wfx profile create <name> --install … puts the server where you say, with the same checksum-verified install.sh, docker compose, the published image, or the Helm chart, then registers the profile, switches to it and logs in. A runner is wfx worker install --to …. Both print every command before running it, and --dry-run prints only.

wfx profile create dev  --install local                        # wfx-server into ~/.local/bin
wfx profile create box  --install ssh://ops@box --compose      # server + Postgres with docker compose there
wfx profile create box  --install ssh://ops@box --image        # docker run the GHCR image (loopback port)
wfx profile create here --install local --compose              # the compose stack on this machine
wfx profile create prod --install k8s --version v0.3.0         # helm upgrade --install infra/helm/wfnexus
wfx worker install --to ssh://ops@worker-1                     # a runner; then `wfx workers` for the join line

Skills and MCP servers on the server

wfx skill and wfx mcp work like wfx tool, on the current profile’s server:

wfx skill add pr-reviewer                          # a skill this server ships, by name
wfx skill add ./skills/my-skill                    # a local folder, uploaded
wfx skill add https://github.com/acme/skills.git --ref v2   # a git source, at a ref
wfx skill ls                                       # installed skills and git sources
wfx skill rm my-skill

wfx env set GITHUB_TOKEN                           # the credential goes in the env store…
wfx mcp add github --set command="npx -y @modelcontextprotocol/server-github" --env GITHUB_TOKEN
wfx mcp add docs --set url=https://mcp.example.com/mcp   # a remote server
wfx mcp ls
wfx mcp rm docs

--env NAME stores the name only. The value is read from the env store when a step that uses the server starts, so it never sits in mcp.json, in your shell history, or in a log. A literal --set env.X=… or --set header.X=… that looks like a credential is refused.

Live, no restart. The server re-reads its catalog (skills, MCP servers, providers) when any of its files change: after wfx skill/wfx mcp, after a rollback, or after an edit by hand. A step already running keeps the toolkit it started with; the next step sees the change. WFX_CATALOG_POLL sets how often the server checks, in seconds (default 2, 0 turns it off).

Versioned with carry. After each change the server records all of its skills and MCP servers as one file in its carry state, so the change is one line in wfx config log, and wfx config rollback <n> puts those skills and servers back on disk and reloads them. When carry is off, the change still applies and the command says it was not versioned.

The image path passes configuration by name (-e CARRY_REPO -e AWS_SECRET_ACCESS_KEY …), so a value in your shell reaches the container and never appears on a command line.

Install it, and set config, with wfx

The server runs carry with its own bucket credentials, so wfx needs only your wfx login: no S3 keys and no state key on your laptop. Every verb acts on the current profile (or --context <name>), on the server’s state by default, or a worker’s with --on <worker>.

wfx config status                                   # is carry on for this server
wfx tool add jq@1.7.1                               # install a tool, no rebuild
wfx tool add gh@2.60.0 --on worker-1                # on one worker
wfx tool rm jq                                      # and take it off
wfx tool ls                                         # what is published in the bucket
wfx config put /home/wfx/.gitconfig ./gitconfig     # place a config file
wfx config get /home/wfx/.gitconfig -o ./gitconfig  # read it back
wfx config ls                                       # every file in the state
wfx config env set LOG_LEVEL=debug                  # set an env default
wfx config env ls                                   # names only; a value is never printed
wfx config log                                      # every version
wfx config rollback 3                               # go back to one

Running machines pick up a new version on their next poll. Nothing reaches into a worker: the server writes the bucket, and the worker’s own carry run pulls the change. When carry is not configured on the server, these verbs answer with what to set (CARRY_REPO and friends) instead of failing silently.

Move config between places

wfx sync copies config files and env defaults from one place to another. A place is a context or a local folder (local:DIR, holding files/<absolute path> and a 0600 env.defaults).

wfx sync --from local:./box-config --to prod                 # local → server
wfx sync --from prod --to local:./backup                     # server → local
wfx sync --from staging --to prod --dry-run                  # server → another server, preview
wfx sync --from prod --state worker-1 --to prod --to-state worker-2   # one worker's setup onto another
wfx config put /etc/app.yaml ./app.yaml --to prod            # one file, to another context

Sync is additive: nothing at the destination is deleted (wfx config rm does that). Secrets do not move with it. They stay in the env store. Workflows and skills move with wfx publish / wfx pull.

The same, with the carry CLI

Every wfx verb above is the server running one of these. Use them directly when you hold the bucket credentials yourself:

carry tool add wfnexus jq@1.7.1
carry put  wfnexus:/home/wfx/.gitconfig ./gitconfig
carry env  set wfnexus LOG_LEVEL=debug
carry log  wfnexus
carry rollback wfnexus 3
``` A file deleted inside a container never
touches the bucket; deletes and renames happen only through `carry rm` / `carry mv`.

<Aside type="tip" title="Keys are not carry's job">
  Model and API keys belong in the wfnexus [env store](/wfnexus/concepts/env-and-secrets/)
  (`wfx env set NAME`), where they are sealed and redacted from every run log. Use carry for the
  tools and the non-secret config a machine needs.
</Aside>

## With a server you reach over SSH

carry and the SSH tunnel fit together: carry shapes what is installed on the box, and the box never
exposes its API. Configure the server with carry, keep it on loopback, and reach it with
`wfx profile create … --via ssh://…` as in [Remote access over SSH](/wfnexus/concepts/remote-access/).
Workers do the same from their side: each restores its own `CARRY_STATE` and connects outbound with
`wfx-runner`, so no machine in the pool needs an inbound port.