You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, data flow from Oracle to ES runs every hour, and some "empty" records appear in ES now and then.
I made sure that it runs in a proper environment and that the SQL query is fixed (see PR #110), yet they keep on appearing in the new data.
I suspect it can happen due to the fact that in the original database some data can appear with a delay, while the timestamp value for them is defined not by the database, but on previous steps.
Suggestion
When started with None "final timestamp" parameter, Connector tries to pull data "from X until Right Now".
I think what we need is to add new config parameter: timestamps.delay with default value, say, 1m (or better 5m), and teach Stage 009 to pull data "from X until Right Now minus delay".
The text was updated successfully, but these errors were encountered:
Motivation
Currently, data flow from Oracle to ES runs every hour, and some "empty" records appear in ES now and then.
I made sure that it runs in a proper environment and that the SQL query is fixed (see PR #110), yet they keep on appearing in the new data.
I suspect it can happen due to the fact that in the original database some data can appear with a delay, while the
timestamp
value for them is defined not by the database, but on previous steps.Suggestion
When started with
None
"final timestamp" parameter, Connector tries to pull data "from X until Right Now".I think what we need is to add new config parameter:
timestamps.delay
with default value, say,1m
(or better5m
), and teach Stage 009 to pull data "from X until Right Now minus delay".The text was updated successfully, but these errors were encountered: