Price proof: link a date, location & currency

[monsieurtanuki]

          @monsieurtanuki in the next few days i will focus on integrating these new Proof fields.

• the UI will change a bit, we'll be asking first the proof data, and then allow adding prices (slightly more proof-oriented)
• if a user selects an existing proof with these info, the form will be pre-filled (and the fields set as readonly)
• new info will be displayed under each proof, and possibility to order & filter as well
• futur step will be to send these proofs to Proof: send to Google Cloud Vision, and store the matching JSON output open-prices#320 and get extra data (and one day pre-fill these info automatically)

Originally posted by @raphodn in openfoodfacts/open-prices-frontend#634 (comment)

[monsieurtanuki]

@raphodn I've just tried to update a proof, and I got the following error message. Any idea what's wrong with my call?

{
  "detail":[
    {
      "type":"extra_forbidden",
      "loc":["body","app_name"],
      "msg":"Extra inputs are not permitted",
      "input":"off-dart integration tests",
      "url":"https://errors.pydantic.dev/2.6/v/extra_forbidden"
    }
  ]
}

[raphodn]

I've just tried to update a proof, and I got the following error message. Any idea what's wrong with my call?

From what i see the server expects a strict list of items in the body.

• Do you always pass app_name in the body ? vs just in the url params instead ? (it should be in the url params)
• The fields expected for the proof update endpoint are currently : type, currency, date (you don't necessarily have to pass them all btw) https://github.com/openfoodfacts/open-prices/blob/67f0d0b58ef639cc65fd8b028c695cf831937ee6/app/schemas.py#L225

[monsieurtanuki]

Thank you @raphodn for your answer, now it's solved!

[raphodn]

What's still missing is to be able to update the proof's location. I need some thoughts on this first...

[monsieurtanuki]

What's still missing is to be able to update the proof's location. I need some thoughts on this first...

I don't see the problem at all, actually. It's just some other fields.
You mean UX-wise?

[raphodn]

Also in terms of fields. We currently ask for a location_osm_id and location_osm_type, and then link it to an actual location object (and set the location_id.

Not sure if this is maintainable in the long run. I'll see :)

[monsieurtanuki]

Anyway you already have to deal with incoherences, e.g.:

• proof currency and prices currencies
• setting a locationOSMId but no locationOSMType (or one that doesn't match)
• for locations, you can say that the priority is locationId (if passed as an argument), and locationOSM* then if relevant

[raphodn]

True, but the incoherences would only come from non-web-or-mobile users (that use the API directly).
And for them (and everyone else) we could start enforcing data quality rules : openfoodfacts/open-prices#319

[monsieurtanuki]

Indeed.

Btw I've just tried to create a price with a value for price_per (never tried before), and the value doesn't seem to be set (it is null).
It works with updatePrice, though.
Could you have a look at it?

[raphodn]

What value did you try for price_per ? the 2 valid options are UNIT or KILOGRAM. https://github.com/openfoodfacts/open-prices/blob/6d3c84ddf10116a98ad6db146a3c51eee60ad89f/app/enums.py#L26
And there's also a validator that sets this value to null if the price has a product (as its only for non-barcode prices). https://github.com/openfoodfacts/open-prices/blob/6d3c84ddf10116a98ad6db146a3c51eee60ad89f/app/schemas.py#L417

[monsieurtanuki]

@raphodn Understood now that price_per is only for non-barcodes, thanks!
Then, there's a little bug in your update checks, as I managed to update a price_per for a barcode product.

[raphodn]

Oh yeah I see the problem, we only do the validation on create, not update 🤦

[raphodn]

ok @monsieurtanuki I did some quick changes in the backend here : openfoodfacts/open-prices#342 & openfoodfacts/open-prices#343

• rearchitectured the price & proof schemas
• have a basic validation check on price update (but it doesn't solve the price_per case you mentioned yet)
• it now returns an error if extra fields are passed ⚠️

Doing some fixes in the frontend to reflect this : openfoodfacts/open-prices-frontend#668

I merged so its available in staging, i'll wait for your go to deploy in prod (to be sure that it doesn't break your current price/proof addition in the mobile app)

[monsieurtanuki]

@raphodn I've just re-run our 17 off-dart tests about prices, and they all pass.
All the "write" tests work on TEST .net env.
As long as you've added optional parameters for existing methods, that should be fine.

[raphodn]

Ok awesome, I'll deploy then, so that if there is any issue I can fix it tonight or tomorrow 🙏

[monsieurtanuki]

@raphodn LGTM: I've just added a receipt today with Smoothie (without changing anything in the code)
https://prices.openfoodfacts.org/app/users/monsieurtanuki

[raphodn]

On the "location update" subject I just created an issue for a new endpoint to create a location, will work on it in the coming hours/days : openfoodfacts/open-prices#346

[monsieurtanuki]

@raphodn I think you should systematically update the proof location/currency/date from the first price, if empty.
I've just had at look at today's proof, and the "missing fields" effect doesn't look good.

[raphodn]

Well the idea is that the location, date & currency of a proof should be mandatory.
I didn't implement it that way (yet) because there are proofs without prices (so i wasn't able to init them with this data).

In the web app I made the changes so that the users have to provide these 3 info if you want to add a price afterwards (and there's indeed a bug/limitation if you choose an existing proof without these info).

Your screenshot is a proof uploaded from mobile i presume ?

Do you plan on implementing these new fields at the proof level ? It would be the best workflow don't you think ?

[raphodn]

These new fields are optional because some proofs (like GDPR) may have multiple locations & dates. But the idea is that they should be mandatory for PRICE_TAG & RECEIPT proofs..

Will implement these validation rules ASAP

[monsieurtanuki]

Your screenshot is a proof uploaded from mobile i presume ?

The screenshot was taken on a mobile, from the web app.
The related data came from Smoothie, as the test I ran yesterday.

Do you plan on implementing these new fields at the proof level ?

First the code in dart has to be reviewed, approved and merged, then it will be easy to integrate in Smoothie/ flutter.

It would be the best workflow don't you think ?

Don't trust anyone who uses the API, by default.
Like our discussion about you accepting images as .bin because the mime header was not set. The solution there was to add a mandatory mime type parameter, if I remember well.

These new fields are optional because some proofs (like GDPR) may have multiple locations & dates. But the idea is that they should be mandatory for PRICE_TAG & RECEIPT proofs..

Then I would recommend considering GDPR so different from receipt/price tags than they deserve a different API entry point.

Will implement these validation rules ASAP

Please don't! Smoothie wouldn't work anymore.
Or do it as api/v2.

[raphodn]

Please don't! Smoothie wouldn't work anymore.

Then how should we proceed ? This is an evolution that was validated collectively in the weekly Wednesday meeting 10 days ago. And we pushed back the mobile launch (discussion on Slack) because we considered that these changes (having a proof location, date & currency) was necassary to increase data quality and simplifying user input. cc @teolemon

I'm going to do the PRs nevertheless, but won't merge them.
edit : PR opened here : openfoodfacts/open-prices#349

Fyi i checked the database

7481 proofs
461 without date
1 without date and souce is 'Smoothie - OpenFoodFacts'

[monsieurtanuki]

I wasn't aware of those decisions.
I assumed that an api/v2 version would do the trick: letting "old" users work without errors with v1 (though deprecated) and letting up-to-date users setting better quality data.
If we're supposed to be in such an early stage with limited number of apps (e.g. only web app and Smoothie) that we can reject new proofs if fields are missing because of "old" code, why not, fair enough.
In the same spirit, we shouldn't populate currency for product prices as it is redundant (if we're lucky) or incoherent.

[raphodn]

It's early stage, there's no known API users apart from the web and now the mobile, so it will be under the same api/v1 :)

[monsieurtanuki]

@raphodn Fair enough, but the current Smoothie users (beta only?) won't be able to add prices while the code isn't changed / merged for Smoothie.
And for the record the code in dart hasn't even been reviewed yet.

[raphodn]

As long as the price have a location/date/currency (which is the case in the mobile's current version) then we'll be able to retro-actively populate the linked proof location/date/currency fields (with a quick script).

[monsieurtanuki]

As long as the price have a location/date/currency (which is the case in the mobile's current version)

Perhaps it would be a good idea to keep those parameters anyway at the price level (what would a price of 1 mean?) and to use them to check the consistency with the proof (and reject inconsistent proof and price parameters).

[raphodn]

Yes for now there's no plan to remove the price location/date/currency fields 😇