Fetch non-food product data from Open Product Facts & Open Beauty Facts

[raphodn]

Story

Currently we consider that a new barcode is necessarily a food product, and we fetch its data from Open Food Facts.

What if a user scans a book ? or a boardgame ? or a deodorant ?

Solution

Sync with the OBF & OPF databases.
How to manage duplicates ? Or products that switch from one db to another ? Or are deleted ?

Extra info

	Link	Number of products	csv size
OFF	https://world.openfoodfacts.org	3M+	855 MB
OBF	https://world.openbeautyfacts.org	38k	78 MB
OPFF	https://world.openpetfoodfacts.org	10k	22 MB
OPF	https://world.openproductsfacts.org	17k	39 MB
Obsolete products (OFF)	see issue #281	24k

[raphodn]

Status : not a priority, decided to concentrate our efforts on food products, while the OPF & OBF databases are revamped in 2024

Nevertheless, users are regularly adding non-food prices. It could be a good starting point to nudge users to add new products in the corresponding database 😉

[raphodn]

Opened an issue in the Python package to figure out how to fetch easily the dumps of the other databases

openfoodfacts/openfoodfacts-python#220

[raphodn]

@raphael0202 is it possible to get dumps from OBF & co ?

[teolemon]

https://world.openbeautyfacts.org/data
https://world.openpetfoodfacts.org/data
https://world.openproductsfacts.org/data

They should have similar URLS, but we probably don't have fancy new stuff like JSONL

[teolemon]

https://static.openbeautyfacts.org/data/openbeautyfacts-products.jsonl.gz
https://static.openpetfoodfacts.org/data/openpetfoodfacts-products.jsonl.gz
https://static.openproductsfacts.org/data/openproductsfacts-products.jsonl.gz

[raphodn]

Ok cool !

Maybe i'll try to improve the openfoodfacts-python package first ? It seems limited to only fetching the OFF data dumps.

I'll see if I manage to get something working.

Issue opened : openfoodfacts/openfoodfacts-python#220

[raphodn]

Ok cool ! Maybe i'll try to improve the openfoodfacts-python package first ? It seems limited to only fetching the OFF data dumps.

PR opened ! here: openfoodfacts/openfoodfacts-python#223