ETLRelay

ETLRelay is an open-source Python framework for composing lightweight, batch-oriented extract, transform, and load (ETL) pipelines.

Its core execution model is:

Source → Processor(s) → Sink

A source produces batches, processors transform them, and a sink consumes the processed output. File workflows keep storage access independent of format-specific reading and writing.

v0.9.0 — Feature-complete beta

This documentation describes ETLRelay v0.9.0 and its frozen v1.0 feature set. Install etlrelay==0.9.0 and review the migration notes when updating Python APIs.

v0.9.0 freezes local CSV/Parquet and SQLite-based SQL workflows, the existing processors, YAML schema version 1, and CLI execution. It completes tabular protocol naming and reviews API contracts, failures, and installation.

Start with Getting Started, Migration, and the release notes.

Current capabilities

ETLRelay includes:

• Arrow-backed Batch data and synchronous pipeline execution.
• The etlrelay CLI with run, validate, version, and help commands.
• Local filesystem access with resolved-path containment validation.
• CSV and Parquet reading and writing.
• Column selection, column renaming, and row filtering processors.
• SQLAlchemy-backed SqlSource and append-only SqlSink.
• YAML configuration schema version 1 with explicit component validation.
• ComponentCatalog and PipelineBuilder for configuration-built pipelines.
• Resource lifecycle management and documented error propagation.
• Unit, integration, functional, and installed-distribution tests.

SQLite is the reference SQL integration-test backend. Other SQLAlchemy dialects require appropriate drivers and application-level verification.

ETLRelay supports CPython 3.12, 3.13, and 3.14. The complete test suite runs on Python 3.12, with installed wheel/source-distribution compatibility checks across the declared Python/platform matrix. See the packaging verification record for verification evidence.

See the beta review and feedback guide for release evidence and reporting criteria.

Documentation

• Getting Started — create and run YAML- or Python-defined pipelines.
• CLI Reference — command usage, validation, execution, and exit codes.
• YAML Configuration — schema version 1, supported properties, preflight behavior, and limitations.
• Core Concepts — batches, processors, schemas, and file formats.
• Security — implemented controls, security boundaries, and deferred risks.
• Logging — operational events, logging configuration, and error reporting.
• API Reference — public Python interfaces and implementations.
• Development — development environment and verification commands.
• Release Automation — CI checks and publication workflow.
• Pipeline Architecture — execution, schema validation, and iterator cleanup.
• Configuration Loading — loading, validation, construction, and execution boundaries.
• Configuration Validation — schema and component validation.
• Component Registry — registrations, factories, and component catalog.
• TabularReader and TabularWriter Contracts — stream ownership and format behavior.
• SQL Source and Sink — SQL parameterization, resources, and transactions.
• Roadmap — released milestones and the remaining v1.0 roadmap.

Scope and limitations

YAML schema version 1 supports file and SQL sources/sinks, local CSV/Parquet workflows, and the built-in select_columns, rename_columns, and filter_rows processors.

The configuration model uses explicit registrations rather than arbitrary dynamic imports or a general-purpose plugin system.

The validate command parses and validates configuration without constructing runtime components, resolving SQL environment variables, or executing the pipeline. Builder preflight occurs during run; neither validation nor preflight guarantees runtime data compatibility, filesystem availability, or database connectivity.

File writes are non-atomic, and SQL sink transactions operate per nonempty batch. ETLRelay does not automatically retry failed operations or roll back earlier persisted output.

S3, HDFS, and other remote-storage integrations remain outside the planned v1.0.0 scope.

ETLRelay is experimental pre-1.0 software. Its public APIs may evolve, and users should assess deployment security and operational requirements before processing sensitive or production data.