Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sensitive counts not visible for recorder on website, also not in admin download #691

Open
chrisvanswaay opened this issue Jun 11, 2024 · 24 comments
Open

sensitive counts not visible for recorder on website, also not in admin download #691

chrisvanswaay opened this issue Jun 11, 2024 · 24 comments
Assignees
@johnvanbreda

Comments

@chrisvanswaay
Copy link
Collaborator

@kazlauskis @DavidRoy
Last Sunday (9 June) Irma Wynhoff did five counts, and marked one of them as sensitive. She uploaded all of them, and this is what the situation is now:

  • She sees five counts as uploaded in the app on her phone: one starting 10:38h, one 11.26h, one 12:41h, one 13:12h and one starting 14:05h
  • the one from 13:12h is marked as sensitive by her.
  • This one cannot be seen when visiting the website for others (which is fine).
  • On the website after logging in, she only sees four counts on that day.
  • A personal download by her also only shows data for the four other counts, not the one starting 13:12h.
  • My download as admin (or however we call it) only show four counts that day, the one from 13:12h is not in my download files. That means I cannot use this count for anything.

I don't think this is a righ way of dealing with sensive records. Sensitive records should be:

  • visible for the recorder in the app (this works).
  • visible for the recorder on the website when searching their own records (this is not the case)
  • in a personal download from the recorder (which is not the case).
  • in a download made by an admin (who has to work with the data).

This is also important for my colleages at Vlinderstichting, as we have marked some visits as sensitive, but now it means we cannot get to them anymore. That makes 15min counts useless for our professional work.
I haven't tested how this works for the api.

If you agree to my interpretation of sensitive counts, could this please be corrected. And if not, then we might discuss how sensitive records should be treated.
@kazlauskis
Copy link
Member

@johnvanbreda probably some reports need tweaking. We set privacy_precision=0 from #468

@chrisvanswaay
Copy link
Collaborator Author

@johnvanbreda I hope this issue can be solved (I hope it is just a simple change of a few queries) as we need the sensitive data too. Not sure how much it has been used, but most likely sensitive sites contain rare and threatened species, just what we need the 15 min counts for.

@johnvanbreda
Copy link
Collaborator

I've changed the required configuration so that the private samples should now be visible. For the occurrences, they are ready, for the samples it may take another hour to update. Also note that the samples I could find for Irma were indexed in France so won't appear on your scheme download.

Please close if OK.

@chrisvanswaay
Copy link
Collaborator Author

@johnvanbreda Thanks. I have access to all 15 min counts via download. I think we are almost there.

  • I can see all observations of that visit in the occurrences download. As there is a rare species involved, I could search for that.
  • However at the moment I cannot find the sensitive sample in the download of the samples. The Visit Sample ID==26100171
  • Irma also does not see this visit in her https://butterfly-monitoring.net/mydata/samples

@chrisvanswaay
Copy link
Collaborator Author

@johnvanbreda We also discovered that sensitive species are not included in personal downloads by recorders (so a recorder does a 15 min count, sees a species marked as sensitive, and this does not show up in the download).
@DavidRoy This influences our professional work. Let me know if this cannot be fixed soon, because then we have to find another way (or back to paper). We started relying on the app.

@Irmawyn
Copy link

Irmawyn commented Jun 26, 2024 via email

@johnvanbreda
Copy link
Collaborator

Am working on this now...

@Irmawyn
Copy link

Irmawyn commented Jul 1, 2024 via email

@johnvanbreda
Copy link
Collaborator

FYI there is a need to populate the private samples into the Elasticsearch to make them available for reporting - this is running now but will take a few hours to complete.

@Irmawyn
Copy link

Irmawyn commented Jul 1, 2024 via email

@Irmawyn
Copy link

Irmawyn commented Jul 3, 2024 via email

@johnvanbreda
Copy link
Collaborator

Hi @Irmawyn - please can you let me know which page you are expecting to see them on the website? Do you mean https://butterfly-monitoring.net/mydata/samples or somewhere else?

@Irmawyn
Copy link

Irmawyn commented Jul 5, 2024 via email

@johnvanbreda
Copy link
Collaborator

Hi @Irmawyn - thanks for confirming. Can you try again please - I think it should be OK now?

@Irmawyn
Copy link

Irmawyn commented Jul 18, 2024 via email

@DavidRoy
Copy link
Collaborator

Thanks for confirming

@chrisvanswaay
Copy link
Collaborator Author

@johnvanbreda @DavidRoy
I have to reopen this one, just discovered that my counts of a sensitive species (Phengaris teleius) on my field day yesterday do NOT show up in the api download that we use internally. This applies to $user = 'NLD';
Could the sensitive species for NL be added to this api call?

@chrisvanswaay chrisvanswaay reopened this Jul 31, 2024
@DavidRoy
Copy link
Collaborator

@johnvanbreda can you pick this one up?

@johnvanbreda
Copy link
Collaborator

@DavidRoy will do.

@johnvanbreda
Copy link
Collaborator

Hi @chrisvanswaay, I've just created a test download page which points to your API endpoint (user NLD) and downloaded the records since the 28th. The data file I received did contain an example record I was looking for (37477231) - are you certain that you didn't receive this record? Is it possible to try again?

@chrisvanswaay
Copy link
Collaborator Author

Indeed I can see this one on the webpage, but I'm sure all the P teleius were left out of the API download yesterday. I will have a look and come back.

@DavidRoy
Copy link
Collaborator

DavidRoy commented Aug 1, 2024

@johnvanbreda Is there a caching delay with sensitive species such as this?

@johnvanbreda
Copy link
Collaborator

@DavidRoy there is normally a short delay while data are transferred to Elasticsearch - allowing for all the various indexing that goes on, typically 30 minutes would cover everything assuming no server backlog at the time. The sensitive records are done on a separate feed to other records so there is a possibility of one set arriving before the other, but both should be within this timeframe. I'm not aware of any backlogs over the last few days.

@chrisvanswaay
Copy link
Collaborator Author

@johnvanbreda @DavidRoy I fear this will have to wait till Monday when my colleague is back, at the moment I can't seem to get proper data, but this might also be my fault. Sorry for the delay.

@CrisSevilleja CrisSevilleja mentioned this issue Sep 2, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@johnvanbreda johnvanbreda

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@johnvanbreda @DavidRoy @kazlauskis @chrisvanswaay @Irmawyn