Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add seed data #27

Merged
merged 9 commits into from
Sep 12, 2024
Merged

Add seed data #27

merged 9 commits into from
Sep 12, 2024

Conversation

richpjames
Copy link
Contributor

@richpjames richpjames commented Sep 11, 2024

Previously we were hitting the Github API to fetch the repos every time the server received a request.

This next iteration adds a seed step where the repo data is fetched and saved to JSON per #21. Every request the server receives then reads from this data and renders it to this page. We also add some more data to the table UI. It is a naive approach with lots of room for improvement but I think it's enough for now.

Future improvements

  • We can iterate on the seed step to enrich the data with (EG) the number of open PRs from the Github the API and the endoflife.date/api API.
  • Pagination
  • We could listen to webhooks and run this seed job to populate the JSON as an inefficient way to keep the data up to date, but we may decided to add that functionality later.
  • We can now sort on any property in the JSON, not just the the ones supported by the Github API

NB to run the app locally you will now either need to run script/seed or script/boostrap -seed.

image

@richpjames richpjames force-pushed the add-repos-data branch 4 times, most recently from 1399faa to 82ec11e Compare September 11, 2024 13:30
@richpjames richpjames changed the title Add repos data Add seed data Sep 11, 2024
@richpjames richpjames marked this pull request as ready for review September 11, 2024 13:45
@richpjames richpjames force-pushed the add-repos-data branch 2 times, most recently from 5bbac31 to 9091893 Compare September 11, 2024 14:16
Copy link
Contributor

@danlivings-dxw danlivings-dxw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implements issue #26

Looks good to me, I just have a minor comment.

I think we might want to look into running the seed script on a schedule when this is deployed to avoid having to do it manually, but shouldn't be a blocker for a first iteration.

index.njk Outdated Show resolved Hide resolved
@danlivings-dxw danlivings-dxw linked an issue Sep 11, 2024 that may be closed by this pull request
@richpjames richpjames force-pushed the add-repos-data branch 3 times, most recently from 737e602 to 5211e27 Compare September 11, 2024 15:59
rich added 9 commits September 12, 2024 09:45
This function won't be used to map repos from the Github API to the UI
anymore but from our stored repos to the UI repos.
This means we can just convert the date from ISO8601 to a JS Date
object. We retain the total number of repos as this may come in handy
later.
This is called pre-mapping so we use the casing from Github's API
This script pulls in all repos for the org where Towtruck is installed.
It pulls the relevant properties from the repo data and pushes this info
to an array. We ignore any archived repos which reduces the number of
repos by >50%.
We then write this repo data to a JSON file to be used later.
I couldn't work out a way to unit test the function that didn't result
in tests that were harder to read than the function itself. The mocking
in Node's built in test runner is still experimental so we may need to
introduce a standalone test runner (Vitest, Jest) to be able to mock
properly.
We want to be able to display the name of the organisation too
This simply reads from the data added by the function in the last
commit.
Previously we were fetching all the repos on each request to the server
by iterating through each installation of the app and every repo of that
installation. Now we just read from the seeded JSON instead.
The app will no longer work unless the seed script is run.
We don't always want to seed this data as we hit the prod GH API.
For now the seeding can be run by passing a flag to the script/bootstrap
command.
We have some new data it is useful to show
@richpjames richpjames merged commit 57e546d into main Sep 12, 2024
2 checks passed
@richpjames richpjames deleted the add-repos-data branch September 12, 2024 08:46
richpjames pushed a commit that referenced this pull request Sep 12, 2024
I broke this with #27.
I switched from passing repos as a second parameter to passing it as
part of the object in the first parameter in the consuming function but
not in the function signature itself.
This format follows the signature the Octokit builtin functions use so I
think it makes sense to align our approach with theirs.
richpjames pushed a commit that referenced this pull request Sep 12, 2024
I broke this with #27.
I switched from passing repos as a second parameter to passing it as
part of the object in the first parameter in the consuming function but
not in the function signature itself.
This format follows the signature the Octokit builtin functions use so I
think it makes sense to align our approach with theirs.
richpjames pushed a commit that referenced this pull request Sep 12, 2024
I broke this with #27.
I switched from passing repos as a second parameter to passing it as
part of the object in the first parameter in the consuming function but
not in the function signature itself.
This format follows the signature the Octokit builtin functions use so I
think it makes sense to align our approach with theirs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Store and retrieve repository information in JSON files
2 participants