Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: remove scrape method + cli cmd #40

Closed
2 tasks
newsroomdev opened this issue Jun 10, 2024 · 0 comments · Fixed by #82
Closed
2 tasks

refactor: remove scrape method + cli cmd #40

newsroomdev opened this issue Jun 10, 2024 · 0 comments · Fixed by #82

Comments

@newsroomdev
Copy link
Member

newsroomdev commented Jun 10, 2024 •
edited
Loading

Description

The scrape method downloads a lot of data for users. @zstumgoren advises removing the method but running the scrape in the cloud. We can split this process but maintain the scrape_meta methods in this codebase. Everyone can use the metadata of the file paths and URLs to create their own scalable scrapers

TODO
@newsroomdev newsroomdev changed the title refactor: remove scrape method refactor: remove scrape method + cli cmd Jun 10, 2024
@newsroomdev newsroomdev linked a pull request Aug 14, 2024 that will close this issue
refactor: deprecate scrape method #82
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

refactor: deprecate scrape method biglocalnews/clean-scraper
1 participant
@newsroomdev