ETLRelay

ETLRelay is an open-source Python framework for composing lightweight, batch-oriented extract, transform, and load (ETL) pipelines.

Its core execution model is:

Source → Processor(s) → Sink

A source produces batches, processors transform them, and a sink consumes the processed output. File workflows keep storage access independent of format-specific reading and writing.

v1.0.0 — Stable initial release

This documentation describes ETLRelay v1.0.0 and its supported public contract. Install etlrelay==1.0.0. The release retains the verified v0.9.0 Python APIs, local CSV/Parquet and SQLite workflows, processors, YAML schema version 1, and CLI behavior. No code or configuration migration is required from v0.9.0.

Start with Getting Started, the compatibility policy, and release notes. For releases before v0.9.0, use the migration guide.

Current capabilities

ETLRelay includes:

• Arrow-backed Batch data and synchronous pipeline execution.
• The etlrelay CLI with run, validate, version, and help commands.
• Local filesystem access with resolved-path containment validation.
• CSV and Parquet reading and writing.
• Column selection, column renaming, and row filtering processors.
• SQLAlchemy-backed SqlSource and append-only SqlSink.
• YAML configuration schema version 1 with explicit component validation.
• ComponentCatalog and PipelineBuilder for configuration-built pipelines.
• Resource lifecycle management and documented error propagation.
• Unit, integration, functional, and installed-distribution tests.

SQLite is the reference SQL integration-test backend. Other SQLAlchemy dialects require appropriate drivers and application-level verification.

ETLRelay supports CPython 3.12, 3.13, and 3.14. The complete test suite runs on Python 3.12, with installed wheel/source-distribution compatibility checks across the declared Python/platform matrix. See the packaging verification record for verification evidence.

See the v1.0.0 readiness record for verification evidence and release gates.

Documentation

• Getting Started — create and run YAML- or Python-defined pipelines.
• CLI Reference — command usage, validation, execution, and exit codes.
• YAML Configuration — schema version 1, supported properties, preflight behavior, and limitations.
• Core Concepts — batches, processors, schemas, and file formats.
• Security — implemented controls, security boundaries, and deferred risks.
• Logging — operational events, logging configuration, and error reporting.
• API Reference — public Python interfaces and implementations.
• Development — development environment and verification commands.
• Release Automation — CI checks and publication workflow.
• Pipeline Architecture — execution, schema validation, and iterator cleanup.
• Configuration Loading — loading, validation, construction, and execution boundaries.
• Configuration Validation — schema and component validation.
• Component Registry — registrations, factories, and component catalog.
• TabularReader and TabularWriter Contracts — stream ownership and format behavior.
• SQL Source and Sink — SQL parameterization, resources, and transactions.
• Roadmap — released milestones and the remaining v1.0 roadmap.

Scope and limitations

YAML schema version 1 supports file and SQL sources/sinks, local CSV/Parquet workflows, and the built-in select_columns, rename_columns, and filter_rows processors.

The configuration model uses explicit registrations rather than arbitrary dynamic imports or a general-purpose plugin system.

The validate command parses and validates configuration without constructing runtime components, resolving SQL environment variables, or executing the pipeline. Builder preflight occurs during run; neither validation nor preflight guarantees runtime data compatibility, filesystem availability, or database connectivity.

File writes are non-atomic, and SQL sink transactions operate per nonempty batch. ETLRelay does not automatically retry failed operations or roll back earlier persisted output.

S3, HDFS, and other remote-storage integrations remain outside the planned v1.0.0 scope.

Public interfaces follow the compatibility policy. Users remain responsible for evaluating deployment security and operational requirements.