This project is a template for creating input connectors for Bytewax.
To create a new connector, click "Use this template" and create a new
repository. Start creating your connector by customizing
pyproject.toml to include any dependencies your connector requires, along
with any development dependencies.
For a completed example that uses this template, see https://github.com/bytewax/bytewax-kafka.
This template includes an optional set of development dependencies which can be
installed with pip install .[dev]. If you get an error using zsh you may need to quote the path like this - pip install '.[dev]'
This template also contains a configuration for using
pre-commit hooks. You can install the pre-commit
hooks after running pip install .[dev] with pre-commit install.
This project comes with an example test directory that uses
pytest.
Included in this project are GitHub actions that will build and publish your package to PyPI.
Publishing to PyPI using the GitHub Action configured here uses their Trusted Publishing framework. See the docs for configuring your project on PyPI.
This action is triggered when creating a GitHub release from a tagged version.
This project contains a GitHub action for building and publishing documentation to GitHub Pages using Sphinx.
To view the documentation locally, run make html from the docs/ folder.
This action is triggered when creating a GitHub release from a tagged version.
Included in this project is a skeleton implementation of PartitionedInput and DynamicInput.
For more information about how to create custom connectors, see our API Docs and the blog post that details building a custom connector for Bytewax.
It is important to yield control from your input connector to Bytewax when processing input. Bytewax's runtime is based on a cooperative multitasking model. Unless your code yields execution to Bytewax, no progress will be made in processing items in your dataflow, and important work like storing state for recovery will not take place.