go-streams

go-streams

go-streams provides a lightweight and efficient stream processing framework for Go. Its concise DSL allows for easy definition of declarative data pipelines using composable sources, flows, and sinks.

Wiki
In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

Overview

The core module has no external dependencies and provides three key components for constructing stream processing pipelines:

• Source: The entry point of a pipeline, emitting data into the stream. (One open output)
• Flow: A processing unit, transforming data as it moves through the pipeline. (One open input, one open output)
• Sink: The termination point of a pipeline, consuming processed data and often acting as a subscriber. (One open input)

Flows

The flow package provides a collection of Flow implementations for common stream processing operations. These building blocks can be used to transform and manipulate data within pipelines.

• Map: Transforms each element in the stream.