Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

We should run a mini crawl frequently #67

Open
TimDaub opened this issue Jul 11, 2022 · 0 comments
Open

We should run a mini crawl frequently #67

TimDaub opened this issue Jul 11, 2022 · 0 comments

Comments

@TimDaub
Copy link
Collaborator

TimDaub commented Jul 11, 2022

  • neume-network/data runs a full crawl in roughly 2-3hrs every day
  • But to get e.g. feedback whether some strategies are properly integrated with each other can hence take a very long time. E.g. if we ship a change in strategy X, and downstream strategies break because of it, it may take up to one day for this to surface in a crash of the neume-network/data crawl
  • However, we don't have to crawl the entire chain and all platforms to get the information whether e.g. all strategies are properly integrated with each other. Instead, we could e.g. set up web3subgraph or any other starting strategy to e.g. only return a single id e.g. [{"id":"0x01ab7d30525e4f3010af27a003180463a6c811a6/1","createdAtBlockNumber":"14323199","platform":"sound"} and then do the crawl for just that id. (I've just taken the first value of: https://raw.githubusercontent.com/neume-network/data/main/results/web3subgraph-extraction)
  • If we found a set of start ids that resulted in all strategies being invoked at least once, we could probably finish an entire mini crawl in a matter of seconds or minutes: and we'd have a fast response/feedback for making coding mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant