Skip to content

CI and Release Automation

This document describes the ETLRelay branch and publishing workflow. Run release checks against the exact code and artifacts intended for publication; results from a previous release do not establish that the current release is valid.

Branch strategy

  • dev is the integration branch for active development.
  • Work-in-progress changes are developed on feature branches and merged into dev through pull requests, or committed directly to dev if working solo and that is the chosen project practice.
  • A release is promoted by opening a pull request from dev into main.
  • main is the production branch. A push to main triggers the PyPI release workflow when the release gate is enabled.
  • The release workflow publishes only when the version in pyproject.toml has not already been published to PyPI. Merging ordinary changes without a version bump therefore does not create a duplicate release.

Before a release PR is merged:

  1. Update project.version in pyproject.toml to the intended version.
  2. Update CHANGELOG.md and confirm release notes describe the actual shipped scope.
  3. Run the established test, lint, format, type, documentation, and packaging checks; ensure CI passes.
  4. Confirm the release version, Git tag, distribution name, and generated documentation version are correct.

A PyPI version cannot be overwritten. Every release that should publish must use a new version.

Coverage requirement

CI runs pytest with line and branch coverage and requires coverage to be strictly greater than 90% (90.1% minimum reported coverage). The build fails if the threshold is not met. The coverage configuration is in pyproject.toml.

pytest-cov must be available in a development dependency group. Ruff and Pyright must also be available in a dependency group installed by CI. If they are only supplied by the local Nix development shell, add them to a CI-only dependency group so GitHub-hosted runners can run the same checks without installing the local Nix shell.

GitHub branch protection

In repository settings, protect main:

  • Require pull requests before merging.
  • Require the CI / quality status check to pass.
  • Require branches to be up to date before merging, if preferred.
  • Block force pushes and deletion.

Protect dev from accidental deletion and force pushes as appropriate. If you require pull requests into dev, also require CI / quality there. Configure the GitHub pypi environment to allow deployments from main only. Keep the production publishing job separate from the build job; only the publishing job receives id-token: write.

CLI release verification (v0.7.0 onward)

Before promoting a CLI release, verify the command boundary from the built package, not only from a repository checkout:

  1. Run uv run mkdocs build --strict to catch broken navigation, missing pages, and documentation warnings.
  2. Run the functional installed-package tests that build and install the wheel in an isolated environment.
  3. Confirm etlrelay --help, etlrelay help run, and etlrelay version work from outside the repository checkout.
  4. Validate a supported local CSV configuration, run it through etlrelay run, and inspect the generated output.
  5. Verify validation does not create output and failures use the documented exit codes without printing sentinel credentials or payload values.
  6. Run the project test suite, Ruff lint/format checks, Pyright, uv build, and distribution metadata validation.

Do not mark the release ready if the strict documentation build or clean-install/installed-CLI checks remain unverified. Record any genuine environment-specific limitation explicitly and distinguish it from a product or documentation failure.

TestPyPI procedure

  1. Merge the release candidate into dev and make sure CI passes.
  2. In GitHub Actions, select Publish to TestPyPI, select the dev branch, and run the workflow manually.
  3. The workflow runs tests, enforces coverage, checks lint/format/types, builds the docs and distributions, then publishes the built distributions to TestPyPI.
  4. A smoke-test job creates a clean environment, installs runtime dependencies from PyPI, installs the ETLRelay package itself from TestPyPI, and checks core imports.
  5. Review the workflow result and package page on TestPyPI before promoting the same version to production.

TestPyPI versions are also immutable. If a version was already uploaded to TestPyPI, use a new pre-release version for another publishing attempt, or inspect the existing uploaded artifact instead of trying to overwrite it.

Configure Trusted Publishing

Create/configure Trusted Publishers separately on TestPyPI and PyPI. These values must match the repository that actually contains the workflows:

Index Workflow file GitHub environment
TestPyPI .github/workflows/publish-testpypi.yml testpypi
PyPI .github/workflows/release-pypi.yml pypi

For each index, configure the repository owner and repository name exactly. Configure the workflow filename and environment name to match the table. If the project has not yet been created on that index, use that index's pending Trusted Publisher flow for the first publication.

Trusted Publishing uses GitHub's OIDC integration, so do not add a long-lived PyPI API token to GitHub secrets. The publish jobs deliberately have only id-token: write; build and test jobs do not receive it.

Historical first-release bootstrap

The following procedure documents the initial 0.1.0 bootstrap and is retained as historical release-process context. It is not the workflow to repeat for v0.6.0.

  1. Set the repository Actions variable PYPI_RELEASE_ENABLED to false (or leave it unset).
  2. Merge the initial release candidate from dev into main, including CI/release workflow files. The production workflow is present but disabled, so the merge does not publish.
  3. Configure the TestPyPI Trusted Publisher and GitHub testpypi environment. Run Publish to TestPyPI with the selected branch set to dev, then wait for its smoke-test job to pass.
  4. Configure the PyPI Trusted Publisher and GitHub pypi environment. Review the successful TestPyPI run and release notes.
  5. Set PYPI_RELEASE_ENABLED to true, then manually run Publish release to PyPI with the selected branch set to main. This publishes the initial 0.1.0 and creates its matching GitHub tag/release. Leave the variable enabled for future releases.

The variable is a one-time bootstrap gate, not a permanent manual release step. After the first verified release, every push to main can publish automatically when pyproject.toml contains a new version. Manual production workflow dispatch is restricted to main.

Production publication behavior

When PYPI_RELEASE_ENABLED is exactly true, every push to main runs .github/workflows/release-pypi.yml. It reads the project name/version and checks whether that exact version exists on PyPI.

  • If it already exists, publication is skipped. This is expected for ordinary main-branch changes that do not bump the version.
  • If it is new, the workflow runs the quality gates, builds the wheel and source distribution, checks the artifacts, publishes them to PyPI, and then creates a matching vX.Y.Z GitHub release/tag.
  • If quality gates or publishing fail, no GitHub release is created.

The CI and release workflows both run checks intentionally: CI blocks unsafe merges, while the release workflow independently verifies the exact code and artifacts it publishes.

Configuration documentation checks

For releases that change declarative configuration, include these documentation checks in the release review:

  • docs/configuration.md must match the implemented schema version, component registrations, validator behavior, and SQL environment preflight.
  • The runnable examples should load and build through ConfigLoader, create_default_component_catalog(), and PipelineBuilder; they must not rely on a CLI if no CLI is implemented.
  • README, getting-started, API reference, configuration architecture, security, changelog, roadmap, and versioned documentation links must consistently reflect the intended release.
  • Run uv run mkdocs build --strict after documentation edits to check navigation, internal links, and generated API references.
  • Run the release workflow's clean-environment smoke test against the built wheel and source distribution; do not rely only on imports from the repository checkout.

Manual release recovery

If publication succeeds but GitHub release creation fails, do not change the published package version. Check whether the vX.Y.Z tag already exists and create the missing GitHub release manually for that same tag/commit. If the PyPI version exists but is wrong, publish a new version; PyPI files cannot be replaced in place.