You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently state is managed transparently by the user by storing it in the cache and automatically using it if present.
The only control that's granted to the user is to ignore the current state.
There are some cases where this doesn't work properly:
No way / no desire to retain the cache (e.g. because no local files can be kept, like in a Github action)
When using streaming via get_records
In these cases it would be nice to still allow a user to do incremental syncs by managing the state themselves. This means there is a method to get the current state as an opaque string, which can be stored whatever and be passed in again:
state_str=cache.dump_state()
# persist this string somewhere# ...# in another runsource.read(cache, state=state_str)
For get_records:
my_stream=source.get_records("my_stream")
forrinmy_stream:
# do something# can also be called if the iterator is not exhausted yet - it keeps track of the latest state messagesstate_str=my_stream.dump_state()
# in another runmy_incremental_stream=source.get_records("my_stream", state=state_str)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Currently state is managed transparently by the user by storing it in the cache and automatically using it if present.
The only control that's granted to the user is to ignore the current state.
There are some cases where this doesn't work properly:
get_records
In these cases it would be nice to still allow a user to do incremental syncs by managing the state themselves. This means there is a method to get the current state as an opaque string, which can be stored whatever and be passed in again:
For get_records:
Beta Was this translation helpful? Give feedback.
All reactions