Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load.py SQL insertion performance enhancement #30

Open
shawngu39 opened this issue Jan 1, 2023 · 0 comments
Open

Load.py SQL insertion performance enhancement #30

shawngu39 opened this issue Jan 1, 2023 · 0 comments

Comments

@shawngu39
Copy link

First thank you so much for creating this amazing repo!

While using the Load.py, I noticed that the data is generally inserted row-by-row, which could be really slow for some of the bigger tables (although they are not that big after all). I am using a low-tier AWS RDS instance with MySQL engine, and it is taking hours to insert certain tables, such as invNames, etc.. I put together some local quick modifications to switch to the bulk insertion from sqlalchemy lib and it speeds up things tremendously, from hours down to a few seconds for the invNames table as an example. Just want to bring this up and see how interested people are with this enhancement. If people believe this is a very useful enhancement, I can clean up my temp walkaround solution and turn it into a more prod quality change. Please let me know. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant