Use SQLite for data storage #63
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will allow Towtruck to use SQLite as a relational database for data storage instead of JSON files.
danlivings-dxw
force-pushed
the
feature/use-sqlite-db
branch
from
2ad9ca1
to
9c9855d
Compare
danlivings-dxw
force-pushed
the
feature/use-sqlite-db
branch
8 times, most recently
from
f905ed7
to
81fe697
Compare
This introduces the `TowtruckDatabase` class, which serves as a wrapper around the `Database` class in `better-sqlite3` and provides utility methods to setup the database schema and interact with it in particular ways. The key-value approach serves as an intermediate step between the current flat file database and a proper relational database - it allows for improved querying of the data while maintaining flexibility of the shape of the data.
This uses the SQLite database to store both dependency lifetimes and repo information using the key-value table in a way that introduces minimal changes to the structure of the persisted data, which aids the migration both for seeding and for retrieving and presenting the data.
danlivings-dxw
force-pushed
the
feature/use-sqlite-db
branch
from
81fe697
to
cd2ea2e
Compare
The NODE_VERSION variable was not being correctly set, so the `setup-node` action was defaulting to the latest LTS version (20.17.0). This was causing `better-sqlite3` problems as it relies on native code which does not target that version.
danlivings-dxw
force-pushed
the
feature/use-sqlite-db
branch
4 times, most recently
from
975f99f
to
bf6e33c
Compare
This will make it easier to update individual pieces of data in the future, such as via a webhook.
danlivings-dxw
force-pushed
the
feature/use-sqlite-db
branch
from
bf6e33c
to
95df3c1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently Towtruck uses two JSON files for data storage,
data/repos.jsonanddata/lifetimes.json. While this is good enough for now, as Towtruck expands in functionality it will require information that is both more comprehensive and more up to date. Using JSON files as a database means that the entire file must read from or written to disk, which will introduce performance concerns as the file size and number of disk accesses increases.This PR introduces SQLite as the data storage mechanism, which as a relational database provides several advantages over storing data in a flat file. However, as Towtruck is still in early development, the data model is still considered to be in flux, so the full advantages of having a properly normalised set of tables is not taken here. Instead, there is a single table called
towtruck_datawhich uses a modified key-value structure. This allows the transition to gain some advantages of using a relational database, such as being able to request data only for a single repository, while retaining the flexibility of data shape originally provided by using JSON files. By retaining some of this flexibility, the refactoring to use this new data storage is relatively minor.