API Reference

This reference is generated from the code and its docstrings. Keep public classes and methods documented as their contracts evolve.

Core contracts and data model

etlrelay.core.batch.Batch

Represent a batch of tabular data using a PyArrow Table.

Batch provides a consistent data representation for sources, processors, and sinks throughout a ETLRelay pipeline.

Parameters:

Name	Type	Description	Default
table	Table	The PyArrow Table containing the batch's data.	required

table property

Return the PyArrow Table containing the batch's data.

Returns:

Type	Description
Table	The underlying PyArrow Table.

etlrelay.core.source.Source

Bases: Protocol

Define the interface for components that produce data batches.

Concrete source implementations are responsible for obtaining data from their respective data sources and exposing it as an iterable of Batch objects.

The protocol is runtime-checkable, allowing implementations to be checked for protocol conformance using isinstance().

read()

Return an iterable of batches obtained from the data source.

Concrete implementations may raise implementation-specific exceptions when the underlying data cannot be accessed or read.

Returns:

Type	Description
Iterable[Batch]	An iterable of Batch objects produced by the source.

etlrelay.core.processor.Processor

Bases: Protocol

Define the interface for ETLRelay batch processors.

A processor accepts a Batch, applies a transformation, and returns the resulting Batch. Processors are executed sequentially by the pipeline.

The protocol allows different processor implementations to be used interchangeably without coupling the pipeline to concrete classes.

process(batch)

Transform a batch and return the resulting batch.

Parameters:

Name	Type	Description	Default
batch	Batch	The input Batch to process.	required

Returns:

Type	Description
Batch	The resulting Batch after applying the transformation.

etlrelay.core.sink.Sink

Bases: Protocol

Define the interface for components that consume batches.

Concrete implementations are responsible for delivering batches to their destination, such as a file or database. The protocol is independent of any particular storage backend or output format.

write(batches)

Write an iterable of batches to the configured destination.

Parameters:

Name	Type	Description	Default
batches	Iterable[Batch]	An iterable of batches to consume and write.	required

Raises:

Type	Description
Exception	If writing fails. Concrete implementations may raise more specific exceptions for their respective failure cases.

etlrelay.core.reader.Reader

Bases: Protocol

Protocol for deserializing binary streams into ETLRelay batches.

Implementations handle format-specific parsing independently of the storage system that provides the stream.

read(stream)

Read and deserialize data from a binary stream.

Parameters:

Name	Type	Description	Default
stream	BinaryIO	An open binary stream containing serialized input data.	required

Returns:

Type	Description
Iterable[Batch]	An iterable of batches containing the parsed data.

Raises:

Type	Description
ReaderError	If the input data cannot be read or parsed.

Note

The caller is responsible for managing the stream's lifecycle.

etlrelay.core.writer.Writer

Bases: Protocol

Define the interface for serializing batches to a binary stream.

Implementations are responsible for format-specific serialization. They do not manage filesystem access or the lifecycle of the supplied stream.

write(batches, stream)

Serialize batches and write the resulting data to a stream.

Parameters:

Name	Type	Description	Default
batches	Iterable[Batch]	An iterable of batches to serialize into one output.	required
stream	BinaryIO	A writable binary stream that receives the serialized data. The caller is responsible for closing the stream.	required

Raises:

Type	Description
WriterError	If the batches cannot be serialized.
OSError	If an I/O error occurs while writing to the stream.

Pipeline

etlrelay.pipeline.pipeline.Pipeline

Coordinate batch processing between a source, processors, and a sink.

The pipeline obtains batches from a source, applies each configured processor in sequence, and passes the resulting batches to the sink.

An empty processor sequence allows batches to pass through unchanged. Processing is synchronous, and exceptions from the source, processors, or sink propagate to the caller.

__init__(source, processors, sink)

Initialize the pipeline with its source, processors, and sink.

Parameters:

Name	Type	Description	Default
source	Source	Component that produces batches for processing.	required
processors	Sequence[Processor]	Ordered sequence of processors to apply to each batch. May be empty if no transformations are required.	required
sink	Sink	Component that consumes the processed batches.	required

run()

Execute the pipeline and deliver processed batches to the sink.

Batches are obtained from the source, processed in sequence, and passed to the sink as an iterable. Processing is lazy, avoiding buffering all batches in memory before writing.

Raises:

Type	Description
Exception	Exceptions raised during source iteration, processing, or sink execution propagate to the caller.

Filesystem and security

etlrelay.security.path.PathValidator

Validate filesystem paths against a trusted base directory.

PathValidator resolves both the configured base directory and candidate paths before checking whether the candidate remains within the permitted directory. This prevents path traversal and symlink-based paths from escaping the configured boundary.

Parameters:

Name	Type	Description	Default
base_path	Path | None	The directory that candidate paths must remain within. If None, the current working directory is used.	None

Raises:

Type	Description
PathValidationError	Raised by validate() when a candidate path is outside the base directory or does not exist.

validate(path)

Validate and resolve a filesystem path.

The candidate path is resolved relative to the configured base directory. The resolved path must remain within the base directory and must exist on the filesystem.

Parameters:

Name	Type	Description	Default
path	Path | str	The filesystem path to validate. May be a Path object or a string. Relative paths are resolved relative to the configured base directory.	required

Returns:

Type	Description
Path	The resolved Path for the validated filesystem location.

Raises:

Type	Description
PathValidationError	If the resolved path is outside the configured base directory or does not exist.

etlrelay.storage.filesystem.FileSystem

Bases: Protocol

Define the interface for accessing file resources.

Implementations provide readable and writable binary streams for storage resources. Callers are responsible for closing the returned streams, preferably by using a context manager.

open_read(path)

Open a resource for binary reading.

Parameters:

Name	Type	Description	Default
path	str	The path or resource identifier understood by the filesystem implementation.	required

Returns:

Type	Description
BinaryIO	A readable binary stream.

Raises:

Type	Description
OSError	If the resource cannot be opened for reading.

open_write(path)

Open a resource for binary writing.

Parameters:

Name	Type	Description	Default
path	str	The path or resource identifier understood by the filesystem implementation.	required

Returns:

Type	Description
BinaryIO	A writable binary stream.

Raises:

Type	Description
OSError	If the resource cannot be opened for writing.

etlrelay.storage.local.LocalFileSystem

Provide validated binary I/O for the local filesystem.

LocalFileSystem uses PathValidator to enforce the permitted filesystem boundary before opening files. It returns binary streams so that format-specific readers and writers can operate independently of the underlying storage implementation.

Parameters:

Name	Type	Description	Default
path_validator	PathValidator	Validator used to ensure paths remain within the permitted base directory.	required

open_read(path)

Open a validated local file for binary reading.

The path is validated and must refer to an existing filesystem resource that can be opened for reading.

Parameters:

Name	Type	Description	Default
path	str	Path to the file, relative to the validator's base directory or an absolute path permitted by the validator.	required

Returns:

Type	Description
BinaryIO	A readable binary stream. The caller is responsible for closing
BinaryIO	the stream, preferably by using a context manager.

Raises:

Type	Description
PathValidationError	If the path resolves outside the permitted base directory.
FileSystemError	If the path does not exist.
OSError	If the file cannot be opened or read.

open_write(path)

Open a validated local file for binary writing.

The path is validated before the file is opened. If the destination does not exist, it is created. If it already exists, its contents are truncated. Parent directories are not created automatically.

Parameters:

Name	Type	Description	Default
path	str	Destination path, relative to the validator's base directory or an absolute path permitted by the validator.	required

Returns:

Type	Description
BinaryIO	A writable binary stream. The caller is responsible for closing
BinaryIO	the stream, preferably by using a context manager.

Raises:

Type	Description
PathValidationError	If the path resolves outside the permitted base directory.
OSError	If the destination cannot be opened for writing, such as when its parent directory does not exist or access is denied.

File source and sink

etlrelay.sources.file.FileSource

Read data from a file and expose it as an iterable of batches.

Attributes:

Name	Type	Description
path	str	The path to the file to read.
storage	FileSystem	The filesystem used to open the file.
reader	Reader	The reader used to parse the file contents into a batch.

__init__(path, storage, reader)

Initialize the file source.

Parameters:

Name	Type	Description	Default
path	str	The path to the file to read.	required
storage	FileSystem	The filesystem used to access the file.	required
reader	Reader	The reader used to parse the file contents.	required

read()

Read the file and return its data as an iterable of batches.

Opens the file through the configured filesystem, parses its contents using the configured reader, and returns the resulting batch in an iterable.

Returns:

Type	Description
Iterable[Batch]	An iterable containing the batch parsed from the file.

etlrelay.sinks.file.FileSink

Write batches to a file resource using a storage backend and writer.

FileSink obtains a writable binary stream from the supplied FileSystem and passes the batches to the supplied Writer for serialization. It is independent of the underlying storage implementation and file format.

Parameters:

Name	Type	Description	Default
path	str	Path identifying the output resource.	required
storage	FileSystem	Storage implementation used to open the output stream.	required
writer	Writer	Writer used to serialize the batches.	required

__init__(path, storage, writer)

Initialize the file sink with its destination and dependencies.

Parameters:

Name	Type	Description	Default
path	str	Path identifying the output resource.	required
storage	FileSystem	Storage implementation used to access the resource.	required
writer	Writer	Writer used to serialize batches.	required

write(batches)

Serialize batches and write them to the configured destination.

Opens the destination through the storage implementation and passes the resulting binary stream and batches to the writer. The stream is closed when writing completes or an exception occurs.

Parameters:

Name	Type	Description	Default
batches	Iterable[Batch]	Iterable of batches to serialize.	required

Raises:

Type	Description
OSError	If the storage implementation cannot open or write to the destination.
Exception	Any exception raised by the writer is propagated.

CSV reader and writer

etlrelay.readers.csv.CsvReader

Read CSV data from a binary stream and yield batches.

read(stream)

Read CSV data from a binary stream and yield batches.

Parameters:

Name	Type	Description	Default
stream	BinaryIO	A binary stream containing CSV data.	required

Yields:

Type	Description
Iterable[Batch]	Batches containing parsed CSV data, one per record batch
Iterable[Batch]	produced by PyArrow's streaming CSV reader.

Raises:

Type	Description
ArrowInvalid	If the CSV data cannot be parsed.
OSError	If an error occurs while reading from the stream.

etlrelay.writers.csv.CsvWriter

Serialize iterable ETLRelay batches as CSV data.

write(batches, stream)

Serialize batches to CSV in the supplied binary stream.

Writes the CSV header only once, using the first batch's schema. Subsequent batches contribute rows without additional headers.

Parameters:

Name	Type	Description	Default
batches	Iterable[Batch]	Iterable of batches containing tabular data.	required
stream	BinaryIO	Writable binary stream that receives the CSV data.	required

Raises:

Type	Description
OSError	If an error occurs while writing to the stream.