v2.1.0

• Fix last file erased when restarting from a cursor
• Bumped substreams to v1.7.3
• Enable gzip compression on substreams data on the wire

v2.0.1

Substreams Progress Messages

Important

This client only support progress messages sent from a to a server with substreams version >=v1.1.12

• Bumped substreams-sink to v0.3.1 and substreams to v1.1.12 to support the new progress message format. Progression now relates to stages instead of modules. You can get stage information using the substreams info command starting at version v1.1.12.

Changed Prometheus Metrics

• substreams_sink_progress_message removed in favor of substreams_sink_progress_message_total_processed_blocks
• substreams_sink_progress_message_last_end_block removed in favor of substreams_sink_progress_message_last_block (per stage)

Added Prometheus Metrics

• added substreams_sink_progress_message_last_contiguous_block (per stage)
• added substreams_sink_progress_message_running_jobs(per stage)

v2.0.0

Highlights

This release drops support for Substreams RPC protocol sf.substreams.v1 and switch to Substreams RPC protocol sf.substreams.rpc.v2. As a end user, right now the transition is seamless. All StreamingFast endpoints have been updated to to support the legacy Substreams RPC protocol sf.substreams.v1 as well as the newer Substreams RPC protocol sf.substreams.rpc.v2.

Support for legacy Substreams RPC protocol sf.substreams.v1 is expected to end by June 6 2023. What this means is that you will need to update to at least this release if you are running substreams-sink-files in production. Otherwise, after this date, your current binary will stop working and will return errors that sf.substreams.v1.Blocks is not supported on the endpoint.

From a database and operator standpoint, this binary is fully backward compatible with your current schema. Updating to this binary will continue to sink just like if you used a prior release.

Retryable Errors

The errors coming from sink files handler are not retried anymore and will stop the binary immediately.

Added

• Added --infinite-retry to never exit on error and retry indefinitely instead.

• Added --development-mode to run in development mode.

  Warning You should use that flag for testing purposes, development mode drastically reduce performance you get from the server.

• Added --final-blocks-only to only deal with final (irreversible) blocks.

v0.2.0

Changelog

The Protobuf definitions are now under sf.substreams.sink.files.v1. Update to use the latest .spkg attached to this issue, re-generate your code and ensure your mod.rs in your project points to the right generated file sf.substreams.sink.files.v1.rs.

Commits

• c1fb775 Added sink-files tutorial example code
• 5b1d2d8 Updated substreams.sink.files.v1 to be sf.substreams.sink.files.v1
• 004b89a Updated tutorial

v0.1.0

Changelog

Initial release of Substreams Sink Files tool. Documentation to come later.

Commits

• a59ec3d Added master as branch on which images are built for
• b8ccf3b Added buffered writer
• b978d39 Added buffered writter
• 3c12d66 Added build
• fe8367c Added entities mapping to run
• 3a4bfc1 Added in memory writer
• c91f365 Added lines encoder support
• 2ce96b7 Added metered writer
• 77335de Added more logs
• e9a0389 Added new mode that bridge best of both world of a a BufferedWriter
• 04bbf8d Added parallel file upload
• 1d05b28 Added proto query language support to extract "entities" out of message
• 8ef2ace Added readme
• c65df5d Added release elements
• 4bd1c88 Added state store abstractions
• 3ead367 Brought back Proto definitions in project, use buf and replace Lines to use repeated string
• 9b3e837 Cleaned up bundler and tester
• 5049191 Ensure we create full directory structure for state file prior writing it
• 72a84c1 Fix bundler
• c533b50 Fixed run command example and logger registration
• d71f3d5 Init commit
• a34b2ab Made working file relative to working dir, update devel script to write allin localdata, fix -c not working
• 8362527 Merge branch 'line'
• cc76c8b Removed leftover print statement
• a531bc0 Updated to latest substreams-sink
• 9bcfb97 Various improvements to get even more speed
• a18a521 WIP: adding local file copying
• 424ae5d add local file working
• 7f4ca56 added bundler count
• 37ea63d added bundler shutter
• f707232 added more stats
• 1ca2cf7 added new line between every block
• 5f1cf42 added noop support
• 04f87f7 added time elapsed measure
• bfb2364 better stats logging
• 1ddc5d2 bump substreams-sink
• c1bdc56 bumped sink lib
• c3c3093 cleaned up github action
• 494859f cleaned up sinker and example
• 0c1fe90 fix bug in dstore writer
• 9c35b13 fix bugs
• 09ca634 fix proto-message
• 4a8ddfe fix typo
• 227d0f3 improvement measure and benchmarking
• 5398aec sink and pq extracted to their own repositories
• d85cd26 skip test
• e086ffd wip