Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data4es: missed values (Stage 009: delay) #114

Open
mgolosova opened this issue Feb 19, 2018 · 0 comments
Open

data4es: missed values (Stage 009: delay) #114

mgolosova opened this issue Feb 19, 2018 · 0 comments

Comments

@mgolosova
Copy link
Collaborator

mgolosova commented Feb 19, 2018

Motivation

Currently, data flow from Oracle to ES runs every hour, and some "empty" records appear in ES now and then.
I made sure that it runs in a proper environment and that the SQL query is fixed (see PR #110), yet they keep on appearing in the new data.
I suspect it can happen due to the fact that in the original database some data can appear with a delay, while the timestamp value for them is defined not by the database, but on previous steps.

Suggestion

When started with None "final timestamp" parameter, Connector tries to pull data "from X until Right Now".
I think what we need is to add new config parameter: timestamps.delay with default value, say, 1m (or better 5m), and teach Stage 009 to pull data "from X until Right Now minus delay".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant