Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move pipeline specifiation to strategies #73

Open
sirnicolaz opened this issue Aug 17, 2022 · 1 comment
Open

Move pipeline specifiation to strategies #73

sirnicolaz opened this issue Aug 17, 2022 · 1 comment

Comments

@sirnicolaz
Copy link

sirnicolaz commented Aug 17, 2022

Problem statement

Currently, the definition of the dependencies between different steps of the pipeline is done inside the core package at crawl_path.mjs.

This leads to the issue that contributors that want to write strategies will necessarily have to modify both the strategies package and the core package, whilst, given my understanding of the architecture, core should just represent the crawling boilerplate and orchestration component, unaware of the specifics of the implemented strategies.

Mitigation proposal

Move crawl_path.mjs inside strategies.

Technical credit

The more pipelines will be integrated, the more the complexity of their dependency will increase. Already at this point there is a non-trivial graph defining how the soundxyz + zora pipelines should work: it involves a common father step involving the web3subgraph crawling, which branches out to the crawling of the two platforms, to then merge again in the musicosaccumulator.

Further down the road there might be other platforms that require to be integrated in this same pipeline and with additional in-between transformation steps.

Having a way to clearly define the dependency graph on the strategy repository would help future contributor understanding how their new strategy should be integrated, either by adding it on an existing graph or by creating a whole independent one (such as the get-xkcd crawler).

@TimDaub
Copy link
Collaborator

TimDaub commented Aug 17, 2022

@sirnicolaz pls define this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants