Skip to content

feat(test-lab): add IPFS devnet - #3453

Open
Alenar wants to merge 31 commits into
mainfrom
djo/ipfs/devnet
Open

feat(test-lab): add IPFS devnet#3453
Alenar wants to merge 31 commits into
mainfrom
djo/ipfs/devnet

Conversation

@Alenar

@Alenar Alenar commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Content

This PR includes a new pure Bash IPFS devnet tool in mithril-test-lab/ipfs-devnet.

It provides a local private Kubo/IPFS swarm that can be initialized with a configurable number of nodes and managed through a single entry point:

./ipfs-devnet.sh --help
Manage a swarm of IPFS Kubo nodes

Usage: ./ipfs-devnet.sh <COMMAND> [OPTIONS]

Commands:
  init                     Download, configure, and create scripts to manage the Kubo swarm
  log                      Tail the logs from all Kubo nodes
  query                    Run an ipfs command against one configured node
  start                    Start the Kubo swarm using the generated start script
  stop                     Stop the Kubo swarm using the generated stop script
  help                     Print this help

Run './ipfs-devnet.sh <COMMAND> --help' for command-specific options.

Environment variables:
  SWARM_DIR                Directory that contains the swarm nodes

The tool supports:

  • downloading a Kubo binary for the current platform;
  • initializing a configurable number of Kubo nodes;
  • configuring a private IPFS swarm with a shared swarm key;
  • disabling external discovery/bootstrap mechanisms to keep the network private;
  • starting and stopping all configured nodes;
  • tailing node logs;
  • querying a selected node with the downloaded ipfs binary;
  • configuring options through CLI arguments and environment variables;
  • generating self-contained helper scripts in the swarm directory;

The supported platforms are aligned with the end-to-end/devnet tooling compatibility:

  • Linux amd64 and arm64;
  • macOS amd64 and arm64 (checked on my own arm64 macbook air);
  • Windows is not supported.

Pre-submit checklist

  • Branch
    • Crates versions are updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • All check jobs of the CI have succeeded
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested
  • Documentation
    • Update README file (if relevant)
    • No new TODOs introduced

Comments

One thing, optional, is missing: a download-webui command (tentative name) that would fetch the webui car file from the ipfs network using the public, and slow, gateway (https://ipfs.io/ipfs/<CID>?format=car&dag-scope=all) store it in a download dir and load it in one of the network node.
Edit: added in the feat(ipfs-devnet): add load-webui command to download and load Kubo Web UI commit.

Issue(s)

Closes #3434

@Alenar Alenar self-assigned this Aug 3, 2026
@Alenar Alenar added the testing 🔁 Something related to tests label Aug 3, 2026
@Alenar
Alenar requested review from jpraynaud and turmelclem August 3, 2026 18:13
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown

Test Results

     5 files  ±0     209 suites  ±0   59m 22s ⏱️ -31s
 3 330 tests ±0   3 330 ✅ ±0  0 💤 ±0  0 ❌ ±0 
11 090 runs  ±0  11 090 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit e789f8b. ± Comparison against base commit eb74d84.

♻️ This comment has been updated with latest results.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new Bash-based ipfs-devnet tool under mithril-test-lab/ to create and manage a local private Kubo/IPFS swarm for development and testing (matching the need from issue #3434).

Changes:

  • Added an ipfs-devnet.sh entrypoint plus command scripts to initialize, start/stop, log, and query a multi-node private Kubo swarm.
  • Added helpers to download a platform-appropriate Kubo binary and generate self-contained management scripts into the swarm directory.
  • Updated Cardano devnet mkfiles scripts to use an absolute SCRIPT_DIRECTORY and read VERSION reliably.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
mithril-test-lab/ipfs-devnet/VERSION Introduces versioning for the new IPFS devnet tool.
mithril-test-lab/ipfs-devnet/README.md Documents installation, commands, and usage examples for the IPFS devnet.
mithril-test-lab/ipfs-devnet/ipfs-devnet.sh Adds the top-level CLI entrypoint and command dispatch.
mithril-test-lab/ipfs-devnet/commands/ipfs-devnet-init.sh Implements init to download Kubo, configure nodes, and generate helper scripts.
mithril-test-lab/ipfs-devnet/commands/ipfs-devnet-exec.sh Implements log/query/start/stop execution by delegating to generated scripts in the swarm dir.
mithril-test-lab/ipfs-devnet/commands/lib/common.sh Adds shared Bash helpers (validation, quoting, directory creation).
mithril-test-lab/ipfs-devnet/commands/mkfiles/kubo-download.sh Adds Kubo download + checksum verification + extraction.
mithril-test-lab/ipfs-devnet/commands/mkfiles/kubo-configure-swarm.sh Adds Kubo node initialization and private swarm configuration/peering.
mithril-test-lab/ipfs-devnet/commands/mkfiles/mkscripts-lifecycle.sh Generates start.sh and stop.sh into the swarm directory.
mithril-test-lab/ipfs-devnet/commands/mkfiles/mkscripts-query.sh Generates log.sh and query.sh into the swarm directory.
mithril-test-lab/cardano-devnet/mkfiles/mkfiles-init.sh Fixes devnet version reading to use ${SCRIPT_DIRECTORY}/VERSION.
mithril-test-lab/cardano-devnet/devnet-mkfiles.sh Switches to an absolute SCRIPT_DIRECTORY and updates sourced paths accordingly.
Suppressed comments (1)

mithril-test-lab/ipfs-devnet/README.md:394

  • The troubleshooting command block includes a stray shell prefix; users copying it will likely get a "command not found" error. The command should be killall ipfs without shell.
```shell
shell killall ipfs
</details>

Comment thread mithril-test-lab/ipfs-devnet/README.md
Comment thread mithril-test-lab/ipfs-devnet/commands/mkfiles/kubo-download.sh

@jpraynaud jpraynaud left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Maybe you can add a .gitignore for the example swarm folder.

Comment thread mithril-test-lab/ipfs-devnet/commands/mkfiles/kubo-configure-swarm.sh Outdated
Alenar added 5 commits August 4, 2026 18:19
…nitialization and configuration

- Extract `init_node` function to handle node initialization and peer ID generation.
- Update `configure_node` to depend on pre-initialized nodes.
… in `kubo-configure-swarm.sh`

Allowing for a number of kubo nodes greater than 9
Alenar added 15 commits August 4, 2026 18:20
- Introduced `mkscripts-lifecycle.sh` to generate start and stop scripts
  for managing Kubo nodes.
- Supports multiple node directories and validates inputs for IPFS
  binary path and node directories.
…ated start script

- Introduced `--log-level` option to control Kubo node logging (debug,
  info, warn, error).
- Added `-h/--help` option to display usage instructions.
- Updated node startup to respect the specified log level.
…node swarms initialization

Similar to its equivalent script in cardano devnet, it download,
configure, and create all needed scripts to start a swarm of kubo nodes.
…ptions in `ipfs-devnet-mkfiles.sh`

- Added support for environment variables such as `DOWNLOAD_DIR`,
  `KUBO_VERSION`, `NUMBER_OF_NODES`, `SWARM_DIR`, and `OVERWRITE`.
- Updated help menu to include new environment variable details.
- Ensured CLI options take precedence over environment variables.
…or command management

- Moved `ipfs-devnet-mkfiles.sh` and related scripts to `commands/`
  directory for better organization.
- Introduced `ipfs-devnet.sh` as an entry point to manage IPFS Kubo node
  swarms with `init` and `help` commands.
…-init.sh`

- Ensure options have corresponding values during parsing.
- Fail on unexpected arguments positional arguments
- Introduced the `log` command to tail logs from all Kubo nodes.
- Added `mkscripts-query.sh` to generate query scripts, including
  log-tail functionality.
- Consolidated `log`, `start`, and `stop` commands into
  `ipfs-devnet-exec.sh`.
- Updated `ipfs-devnet.sh` to route all commands through the unified
  script.
- Removed redundant `log`, `start`, and `stop` individual scripts.
…sage

- Detailed instructions for initializing, managing, and observing a
  local private IPFS network.
- Included command descriptions, options, environment variables, and
  usage examples.
- Introduced the `query` command for running `ipfs` commands against
  specific Kubo nodes.
- Updated README with detailed usage instructions, options, and examples
  for the new `query` feature.
- Enhanced script generation to include `query.sh` for node-specific
  queries.
…nd scripts

- Moved common functions to a new `lib/common.sh` for reusability.
- Replaced redundant error handling and validation logic with reusable
  utilities.
- Refactored `ipfs-devnet` scripts to use shared functions for options,
  directories, and validation.
…mmon utilities

And add missing requirement checks in `kubo-download.sh`
Alenar added 2 commits August 4, 2026 19:49
… Web UI

- Introduced a new `load-webui` command to download the Kubo Web UI as a
  CAR archive and load it into the private IPFS network as it's not
  bundled with the kubo binary.
- Added `ipfs-devnet-load-webui.sh` script to handle the new
  functionality.
- Updated README with detailed command description, options, and
  examples.
- Enforce the private nature of the created networks in commands helps
- add a `.gitignore` for the `./swarm/` folder used in the README
@Alenar
Alenar requested a review from jpraynaud August 4, 2026 17:51
@Alenar
Alenar deployed to testing-2-preview August 4, 2026 18:06 — with GitHub Actions Active
@Alenar
Alenar deployed to testing-preview August 4, 2026 18:06 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing 🔁 Something related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a local IPFS network

3 participants