Manual and CI/CD Testing

netlab uses a GitHub Actions CI/CD pipeline with multiple workflows. The automated tests executed on every push or pull request include:

  • mypy static type checking of all Python code in netsim/ directory

  • ruff linting of all Python code

  • yamllint validation of all YAML files in netsim/ and tests/ directories

  • Transformation tests and error-case tests run with pytest in tests/ directory

  • Integration-test topology validation (netlab create -d none) for fork PRs

You can run the same tests locally in the tests directory. It’s highly recommended that you run them before creating a pull request that modifies Python code. PRs that fail the CI/CD pipeline will not be merged.

Tip

  • The CI/CD tests require additional Python modules. Install them with pip3 install -r requirements-dev.txt

  • The CI/CD tests use PyYAML. You can run them on a system with ruamel.yaml installed, but they might take longer (see bug report and related PR for details). Uninstalling ruamel.yaml might not be a bad idea.

Before Submitting a PR

If your PR includes modifications to Python code:

  • Run the run-tests.sh script in the tests directory to run the automated tests that will be run as part of the CI/CD pipeline (see also Test Runner Scripts)

  • If the tests fail, your code modifications changed the data transformation logic or error messages.

  • If you’re absolutely sure your changes are correct, run the create-transformation-tests.sh and create-error-tests.sh scripts in the tests directory to recreate the expected test results.

  • Check the differences between previous and new expected test results.

  • Add modified test results to your commit.

Tip

Automated CI/CD tests will check your expected test results anyway, and we’ll have a discussion if you submit “suboptimal” content ;)

When modifying documentation:

When modifying device templates or adding new device functionality:

  • Run the relevant integration tests.

  • If possible, add the integration test results to the PR.

Finally:

  • Submit a PR.

  • The PR should be in draft state when you’re still working on it and in ready for review state when you’re done.

  • Ask someone to review your code. GitHub is pretty good at identifying who worked on the code recently; that person might be your best bet.

Automated Tests

The test_transformation.py test harness runs three types of transformation tests:

  • Regular transformations (see below)

  • Error cases – topologies that should generate an error resulting in an aborted transformation attempt. Add tests to this directory only when you need to test error messages in the Python code.

  • Verbose test cases – identical to regular transformations but with more logging. Used only when measuring code coverage (to ensure all logging printouts are triggered)

Test Runner Scripts

The tests/ directory contains several shell scripts to run subsets of the test suite:

Script

Command

Purpose

run-tests.sh

pytest -v -k 'xform_ or error_cases' + mypy + yamllint

Main local pre-PR suite (excludes tests/coverage fixtures). Accepts ci argument for fail-fast mode (set -e).

run-xform.sh

pytest -v -k xform_

Transformation tests only

run-xerr.sh

pytest -v -k error_cases

Error tests only

run-yamllint.sh

yamllint --no-warnings on netsim/ and tests/

Standalone YAML linting

run-typing.sh

mypy --no-incremental -p netsim

Standalone type checking

run-coverage-tests.sh

pytest -v -k 'coverage'

Coverage tests only (see coverage tests)

You can run any of these scripts from the tests directory. For example:

$ cd tests
$ ./run-tests.sh ci        # Full CI/CD suite with fail-fast
$ ./run-xform.sh           # Transformation tests only
$ ./run-coverage-tests.sh  # Coverage tests only

Data Transformation Tests

The regular transformation tests:

  • Take a topology file from tests/topology/input directory

  • Run the transformation code

  • Render the resulting data structure (without address pools or system defaults) in YAML format

  • Compare the results with corresponding file from tests/topology/expected directory

Whenever you’re creating a new test case or modifying an existing one, you HAVE TO create a corresponding expected results file. Please don’t try to create the expected results by hand – the results are compared as strings, not data structures, and it’s just not worth the effort to fix them manually.

To create expected results files, run the create-transformation-tests.sh script in the tests directory. The script assumes your code works flawlessly and that whatever it does is the correct result. That might not be the case, so it’s highly recommended that you execute git diff topology after running the create-transformation-tests.sh script and do a thorough check of the differences.

Transformation Error Tests

The transformation error tests:

  • Take a .yml topology file from the tests/errors directory

  • Run the transformation code that should result in a ‘fatal error’ exit

  • Collect the error messages generated during the data transformation

  • Compare the collected error messages with the corresponding .log file from tests/errors directory

Whenever you’re creating a new error test case or modifying an existing one, you HAVE TO create a corresponding expected error messages log file.

To create the expected error messages files, run the create-error-tests.sh script in the tests directory. The script assumes your code works flawlessly and that any generated error messages are the expected error messages. That might not be the case, so it’s highly recommended that you execute git diff errors after running the create-errors-tests.sh script and do a thorough check of the differences.

Warning

You cannot create a new error test on a system with ruamel.yaml package (details in the bug report and related PR). Uninstall ruamel.yaml before running the create-error-tests.sh script.

Coverage Tests

Coverage tests ensure edge cases and rarely exercised code paths are covered. They are stored separately from the main test fixtures:

  • Coverage transformation tests (tests/coverage/input/ → tests/coverage/expected/): Edge cases not in the main set – dict-key types, empty links, invalid attributes, defaults inheritance, unnumbered edge cases, and more.

  • Coverage error tests (tests/coverage/errors/): Attribute/type validation errors, group errors, node validation, and similar edge cases.

  • Verbose coverage tests: Re-run the main transformation test cases with verbose logging while coverage measurement is active. These only execute under a trace (i.e., when coverage run is active), ensuring all logging printouts are triggered.

Running Coverage Tests

To run coverage tests only (without coverage measurement):

$ cd tests
$ ./run-coverage-tests.sh

To generate a full HTML coverage report:

$ cd tests
$ ./coverage.sh

This uses the coverage package (not pytest-cov) to instrument the code, then generates an HTML report in tests/htmlcov/. The .coverage and htmlcov/ directories are gitignored.

Tip

The coverage report shows which lines of Python code in netsim/ are executed during the test suite. Use it to identify untested code paths that need new test fixtures.

Creating Coverage Test Fixtures

To regenerate the expected coverage test results:

$ cd tests
$ ./create-transformation-tests.sh coverage   # Regenerate coverage transformation expected files
$ ./create-error-tests.sh coverage            # Regenerate coverage error expected files

Warning

As with the main error tests, create-error-tests.sh coverage refuses to run if ruamel.yaml is installed.

Testing Documentation Changes

netlab documentation is built with Sphinx. You can generate and inspect the HTML output locally before submitting a PR.

Prerequisites

From the repository root directory, install the documentation build dependencies:

pip3 install -r docs/requirements.txt

This installs Sphinx, the MyST parser, the Read the Docs theme, and other required packages. The docs/requirements.txt file pins specific versions to ensure consistent CI/CD builds.

Building HTML Documentation

From the docs directory, run:

make html

This invokes sphinx-build and writes the generated HTML files to _build/html/.

netlab documentation uses Sphinx extensions that rely on netsim modules to fetch default settings. The documentation build process might fail if you don’t have the GitHub repository in the Python path. In that case, use:

PYTHONPATH=.. make html

Sphinx will report warnings for broken links, missing references, or markup errors during the build – fix these before submitting a PR.

Tip

Run make clean html to do a full rebuild, avoid stale cached pages, and detect broken links in pages you haven’t modified.

Inspecting the Result

From the docs directory, open the top-level HTML page in your browser:

open _build/html/index.html          # macOS
xdg-open _build/html/index.html      # Linux

Browse the pages you modified.

Integration Tests

Integration tests are run by hand; it’s too much hassle to set up an automated test environment with vendor boxes/containers/license files. The latest results are available at https://tests.netlab.tools/.

The test topologies are stored in the tests/integration directory. If you’re adding new device features or changing device configuration templates, please run the relevant tests before submitting a PR.

Most integration tests include automated validation. The easiest way to use automated validation with a single test is to run the netlab up _test_scenario_ --validate command.

Integration Test Topology Validation

On every pull request, the CI/CD pipeline runs check-integration-tests.sh, which validates all integration test topologies by executing netlab create -o none -d none against every numbered test file in tests/integration/ and tests/platform-integration/. This ensures that the topology transformation code does not break any integration test definitions.

Background: GitHub Actions Workflows

The CI/CD pipeline consists of the following workflows in .github/workflows/:

Workflow

Trigger

Purpose

t-push.yml

Push to any branch (except devc) when **.py or **.yml change

Runs mypy and full pytest suite across Python 3.10, 3.13, and 3.14

t-pull.yml

Pull request touching **.py or **.yml

Runs mypy + pytest (forks only), then integration-test topology validation (check-integration-tests.sh) for all PRs

t-yamllint.yml

Push (non-devc) / PR on **.yml

Runs yamllint --no-warnings on all YAML in netsim/ and tests/

ruff.yml

Push (non-devc) / PR on **.py, ruff.toml, or the workflow file

Runs ruff check --output-format=github . on Python 3.14

integration.yml

GitHub issue comment containing /integration

Runs real integration tests on a self-hosted runner with vendor boxes/containers

pages.yml

Push to master or manual dispatch

Builds Sphinx docs and deploys to GitHub Pages

pypi.yml

Push of release_* tag

Builds and publishes package to PyPI

test-pypi.yml

Completion of “Run CICD Tests” on dev/cicd-fix branches

Builds and publishes to Test PyPI

devcontainer-*.yml

Various

Builds and pushes devcontainer images

What Happens on a Push or PR

  1. t-push.yml (push) / t-pull.yml (PR) runs the full test matrix:

    • Installs requirements.txt and requirements-dev.txt

    • Runs mypy -p netsim (type checking)

    • Runs cd tests && PYTHONPATH="../" pytest (all transformation and error tests)

    • For PRs from forks, also runs check-integration-tests.sh (validates all integration topologies with -d none)

  2. t-yamllint.yml lints all YAML files

  3. ruff.yml lints all Python files

Tip

The push and pull-request workflows run on a matrix of Python 3.10, 3.13, and 3.14. The ruff workflow runs on Python 3.14 only.