Terminology used in the codebases

[zackkrida]

Some thoughts on various jargon used in the codebase.

Openledger

The term 'openledger', is confusingly the name the database in the catalog AND in the API. This name comes from the original CC Search prototype at Creative Commons¹. This term isn't something that is commonly used or something that provides clarity on the role of the database. It's essentially an external name for internal databases.

I propose switching to another name that makes the role of the databases in the system clearer. There's a part of me that wants to call the dbs upstream and downstream for the catalog and API, but I'm curious for other ideas. They could also just as easily be catalog and api, and so on.

Ingestion

The term 'ingestion' seems to be super overloaded, used to describe:

• Grabbing images from providers
• Copying the database from the upstream and downstream databases

These are both critical processes, and it would be nice to have clarity when discussing one or the other.

Footnotes

1. https://github.com/cc-archive/open-ledger ↩

[zackkrida]

CC @WordPress/openverse

[AetherUnbound]

Openledger - I think I would prefer catalog and api over upstream/downstream, personally. It better codifies the purpose of each database without having to know about the other.

Ingestion - I think this one is fine! Ingestion feels like the right term for both of these 🤷‍♀️ I think it will help when we fold the latter into the catalog repo. We could call the upstream -> downstream the "data refresh" though, that seems like it describes what's happening well.

[zackkrida]

That all makes sense. Regarding ingestion, my goal is to make it so if I say 'Darn, ingestion failed last night' folks don't need to wonder what it's referring to. Maybe a word before 'ingestion' for each could fix that...

[AetherUnbound]

In that case I think "data refresh" or even "API data refresh" would be good!

[dhruvkb]

Agree with @AetherUnbound about the use of catalog and api instead of upstream/downstream. To me the latter feel "relative" (as in upstream to what?) instead of the more direct nature of the former proper nouns.

[krysal]

I like catalog and api names, I think we could also go with api-upstream and api-downstream, or opendledger-upstream and opendledger-downstream for the API repo. I would prefer any of these a bit over Ingestion, this name suggests to me more that we are talking about a process rather than a database.

[obulat]

I like catalog and api names for databases. I have a small question regarding that: in development, the api's upstream database is different from the catalog database. Does production use the catalog's main (openledger? upstream?) database as the upstream database where it loads the data from? Or are there two upstreams: one catalog's main db, which is copied to another upstream in the api and then used to get the data into api database?

[zackkrida]

Good question, there's only 1 upstream DB. "the api's upstream database is different from the catalog database." is only true in development.

[sarayourfriend]

The problem with upstream/downstream is that they're relative terms. api-upstream I would think points to the downstream of some other service (in this case the catalog maybe?)

My understanding of the Openverse architecture is too small to share a solution, but I think upstream and downstream are pretty confusing as a new comer who hasn't quite grokked the entire infrastructure yet.

If api-upstream is the catalog's DB (#107 (reply in thread)) then we can just call it something related to the catalog rather than something that describes its relationship to another piece of the puzzle. Something like "the catalog DB lives upstream of the API" or something something like that.

Again, I really don't have a strong enough understanding of the architecture to help too much but I can share what seems confusing to me right now as a new comer to the project.

[zackkrida]

Based on the discussion I'm thinking @AetherUnbound's idea of the DB names mirroring their respective repositories, and not using names relative to the project or the relationship between the repositories is a good idea.

So...here's how Openverse works:

Image metadata from providers is gathered into the catalog database, in a process called ingestion. Then, data is copied from the catalog database to the api database in a process called the data refresh.

[dhruvkb]

Does this mean that the ingestion_server inside WordPress/openverse-api repo is actually a data_refresh_server?

[zackkrida]

That would be more accurate with the new names, yes. But also, I've has some limited discussion with @AetherUnbound and @rbadillap around the idea that the ingestion_server could probably just be a DAG and doesn't need to exist as a standalone service. Historically, I think it only was in terms of philosophical purity RE: keeping the catalog and api decoupled.

[dhruvkb]

That's very true. Would love to see it changed into DAG(s) or spun off into it's own repo. It's in a very strange place right now sitting alongside the API.