manifest syle connector #46349
Unanswered
guykatz
asked this question in
Connector Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi all
my source data sits in muiltiple remote http web servers.
each server has a manifest file (predefined name) in a well known address I can connect to.
inside this manifest there are many lines, each line is a http address of a source data file I need to transfer to the datawarehouse.
for example:
in remote web server 1.1.1.1 I have a manifest style file accesible via 1.1.1.1/manifest.mf
the content of this file can be:
i do not control this manifest file creation. lines are added and removed from it periodicly and i need to get the new ones only and trasfer them.
I assume that even though this type of source is not rare, its still rare enough not to exist as a connector in airbyte?
in case it does not exist, I assume i need to build a custom connector?
thanks for the help./
Beta Was this translation helpful? Give feedback.
All reactions