Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make pipeline less verbose #253

Open
KarenJewell opened this issue Aug 19, 2023 · 1 comment
Open

Make pipeline less verbose #253

KarenJewell opened this issue Aug 19, 2023 · 1 comment
Labels
data engineering Things related to data: scraping, cleaning, labelling, transformation

Comments

@KarenJewell
Copy link
Member

Is your feature request related to a problem? Please describe.
Current pipeline is extremely verbose as all the scrapers are doing their own thing. It becomes a bit too much to read when debugging.

Describe the solution you'd like
Options:

  1. Switch of all individual print statements
  2. standardise messages at main() level
  3. add parameter to switch on/off verbosity

Describe alternatives you've considered
rip it all out and rebuild as modules.

Additional context
None.

@KarenJewell KarenJewell added data engineering Things related to data: scraping, cleaning, labelling, transformation back end labels Aug 19, 2023
@JackGilmore
Copy link
Member

I did introduce Loguru when I worked on main.py last but didn't really get round to applying it to any other scripts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data engineering Things related to data: scraping, cleaning, labelling, transformation
Projects
Development

No branches or pull requests

2 participants