event: Automate adding registered users to iNaturalist event projects

[synrg]

This is a proposal to automate the chore of adding registered members to the user rules for projects associated with a Discord server (e.g. yearlisting and bioblitz event projects). This proposal is not about self-registration to the bot - see #140 which is a separate proposal about that.

Up until this point, in Dronefly we have called these projects user projects because they use user rules to manage membership. From here on, I refer to them as event projects, as that accurately describes the "what" instead of the "how".

What's the problem? Why can't people just join the project on the web?

While iNaturalist projects conventionally are self-joinable merely by clicking "Join this project" on the project page on the iNat website, this does not work for us. The issue is that any iNat user can do that, and we want to restrict membership to be only for members of our Discord communities. Therefore, ever since the creation of the iNaturalist Discord server, we have used iNat project user rules to define membership of our event projects: one yearlisting project per year, and shorter term bioblitz event projects throughout each year. The admins of each project have had to add members manually. Users cannot do this themselves.

While bioblitz event registration is a chore that we're managing fairly painlessly, nowhere is this work more painful and labour-intensive than the handling of the yearlisting projects, and it gets to be more of a burden each year as our community grows. Starting with the https://www.inaturalist.org/projects/discord-inaturalist-yearlisting-2019 yearlisting project for 2019, and continuing through present day, every year from late December through the early months of the next year, volunteers on the iNat Discord mod team have manually added all registered members who have indicated they still want to be a part of the next year's project. It has reached the point where it is no longer reasonable to do it this way. We need to automate the chore.

Commands to manage event project membership

The most straightforward way to implement this, I think, is to provide an event command group to manage event project membership. Who gets to use this command is a separate issue: we can initially make the command group mods-only to make it easier for the mod team to handle project join requests. Later on we can develop the feature to require even less mod intervention by allowing users to use the commands themselves, or by pressing buttons on menus that invoke those commands.

Additionally, commands in the [p]inat command group will perform server admin functions to define event projects, etc.

Set iNat credentials via Red's Shared API Keys module: [p]set api

Refer to Red's documentation at: https://docs.discord.red/en/latest/framework_apikeys.html

The bot admin would set the inat_events API key as a distinct key, separate from any others we might want to create for other purposes. Obviously, the password in this example is not the real password:

Example: [p]set api inat_events login dronefly password notlikethis

Because the credentials are stored on the bot owner's own machine, we will make this an owner-only command. Other server communities that want to use this feature should run their own instance of Dronefly so they retain control over how their credentials are used.

Set an event project: [p]inat set event <project_abbrev> <project_id> [creds] [role]

This is just the [p]inat set user_project <project_id> <emoji> command renamed and with parameters modified. In this command, we formally drop support for emojis and replace them with plain text abbreviations. (We stopped using emojis to stand for event projects a long time ago because it doesn't scale well to represent the large and growing number of event projects we've created to date.) Note that the order of order of the id# and abbreviation (was emoji) have flipped to make it consistent with other commands, e.g. the registered user command ,project add <abbrev> <project_id>.

The optional [creds] is the credentials abbreviation set in [p]set api (e.g. inat_events). It is a bot-owner-only option which enables that iNat admin to make changes to the project. Note: the inat login must be added as an admin to the iNat project (do this on iNat web; don't forget you need to join that user to the project before they can be added as an admin). If [creds] is not specified, then updates to the event project membership cannot be made for the defined project.

The optional [role] is a role for the server that is allowed to issue authenticated requests via the bot to manage event membership. It is a bot-owner-only option that requires the [creds] to also be specified. This allows the bot owner to delegate event project management to role-holders (e.g. @Moderator) for the server.

If the project is already defined, this command may be executed again to supply or change the optional parameters. Each new project abbrev

Set event projects for [p]me display: [p]inat set me <project_abbrevs>

In the current implementation, all event projects are included in the [p]me display, but it is impractical both in terms of computational cost and space in the display to put all of a user's event stats in that display. Therefore, we need a separate command to manage which events are shown in [p]me. The command would accept a space-delimited list of projects to include in the display.

Example: [p]inat set me ever 2022 would set the projects with abbrevs ever and 2022 as the projects to include in the [p]me display.

Joining an event project: [p]event join <project_abbrev> <user>

This command will have two forms, only the second of which needs to be implemented at this time.

1. self-registration: [p]event join <project_abbrev> would be harder to do, as we need additional checks to ensure they don't register themselves to two opposing teams (as in a bioblitz event) or join after project joins are closed for the project
2. registration by a mod: [p]event join <project_abbrev> <discord_user>

In either case, only users registered to the bot for the server are allowed to join. This means a mod has previously issued a [p]user add command for that user.

The command issues an API request using the event project credentials set via [p]set api and defined in [p]inat define event. to add the user rule for the project.

Leaving an event project: [p]event leave <project_abbrev> <user>

As per [p]event join but removes the user.

Listing an event project's user rules: [p]event list <project_abbrevs>

As per the current [p]user list command, but lists only members joined to one or more space-delimited event projects indicated by <project_abbrevs>.

[synrg]

Parts of this issue are not easy at all to implement due to missing API endpoints to update the project rules. Therefore, over the past month, I have worked on all of the parts of the issue that don't involve having the bot actually update the project.

When creating a new event, instead of creds, this has been replaced with main, a flag that indicates whether or not an event project is a main project for the server. This supersedes the spec for [p]inat set me.

It's unlikely we'll provide [p]event join and [p]event leave since the bot's inability to actually do the registrations means we will continue to manage event project membership in only a semi-automated event, by self-assignable roles.

To that end, the role parameter is not going to be used to indicate the event manager role. Instead, it is now used as the role the user self-assigns to gain admittance to the event project. This role can now be used as a filter when a mod does a [p]user list, e.g. [p]user list cetaceans to list all users with the cetaceans event role.

A new teams parameter is used to link a team event with other projects associated with the same event.

Putting this all together, this is how it all works:

Any discrepancies (like a user with two team roles, a user in both team projects, or not in either, or a user in a project but without any role) can be easily spotted from the [p]user list cetaceans output.

[synrg]

The event members list should also include project members who are not known to the bot on this server. This has happened a couple of times and is hard to figure out when it does because it requires an exhaustive review of the project member list (and/or the reactions on a role reaction menu that assigns the event roles) to figure out who is missing.

Reasons for the discrepancy might include:

• a project admin added the wrong person
• a former server member joined the project, then left the server

The ,user list event output should cover this case:

• currently one pass is made through known-on-this-server members to see if they are role holders and/or team project members
• add to this list of member/inat id pairs any not-known-on-this server members found in the project being listed
• make those discrepancies visually distinct in the listing, e.g. 👻 (unicode "ghost") to indicate their absence

[synrg]

ee22e68 implements the idea in the previous comment

[synrg]

The last kind of discrepancy is if people reacted to the menu who aren't known to the bot. That is most likely because they haven't shared their profile yet and had a mod add them.

Rather than write more code in ,user list to try to do this, in c18f955 and 31659e5 I did a more general change to ,inat inspect to list counts of all reactions to a message, then under each, which known users reacted, then the keyword unknown followed by a list of unknown users, if any. This list can be very long, so it is paginated (the 2nd commit).

It was kind of rushed for the in-progress iNaturalist Discord bioblitz, and therefore messy. Needs some fixing up later.

[synrg]

I changed my mind and am reopening this issue to implement the complete automation of project updates rather than only semi-automating it with the new event project support. My reasons are:

1. Every time we manually edit one of our projects, it throws an error at the end. I reported over a year ago, but it still remains unfixed: https://forum.inaturalist.org/t/project-edit-fails-to-return-to-project-page-after-save/22649 . This is a real pain. It slows the job down and makes it unpleasant.
2. There is actually an API endpoint we could use, the same one the web uses, PUT /v1/projects/#. It's just unpublished. This is something I discussed with pisum on the iNat forums a while ago and then forgot about: https://forum.inaturalist.org/t/collection-project-user-filter-management-webapp-with-pkce-oauth-flow/19232
3. There's another, more recent bug I reported. This completely breaks a project with "must be observed by user" rules if one of them is an undefined user (e.g. a deleted/suspended account, perhaps? i haven't identified the specific user that's breaking it yet). In the comments on this report, pisum reminded me again about using PUT /v1/projects/#, perhaps just to remove the one user to fix the project: https://forum.inaturalist.org/t/edit-project-breaks-entirely-for-project-with-undefined-user/30070

Well, the first two reasons are good enough reasons to reopen, but the third is the last straw that I think will put this back in my queue to do soon. If we're going to use PUT to fix up the 2020 yearlisting project, I might as well think seriously about writing the code around PUT /v1/projects/# to do it properly from the bot, rather than just hacking together something (e.g. via some throwaway script - using curl or something).

[synrg]

Even though this is mostly implemented now, (i.e. we now use the PUT endpoint so mods can add each user as they sign up) and this has been working fairly well for almost a year, it is still incomplete for these reasons:

1. adding to a very large project breaks (fixed in Send only changed items when updating project observation rules pyinat/pyinaturalist#460 and will be added to Dronefly once pyinaturalist 0.18 is released)
2. while discrepancies for members of the server which have the role and aren't in the project, or are in the project but don't have the role are detected and reported now, the reactions to the menu aren't checked at all. this consumes a lot of my time when it happens to manually review and fix.

Today, a variant of 2 that I hadn't thought of before occurred. Often those discrepancies are because the user isn't a member to begin with, and they reacted to the menu. But a second case was triggered today with 3 users: they all reacted during a few-minute-long power outage, so Polistes was not connected to the network at that time, didn't detect their reactions, and didn't assign them the role. I only found out later in the afternoon when I had all but completely forgotten the brief outage earlier in the day. It took an hour of frustrating manual searching through the entire list of 43 reactions, bringing up the ,user output for each one, and comparing it to the 40 users added to the project. Upon discovering they were all valid members (i.e. added to the bot in the server, and present in the main project) and yet still weren't a part of the discrepancy report, I just gave up and added them all manually to the current blitz project, ants ... lo and behold, after adding them, they did show up in the report! This is because they were in the project, and yet did not have the role. That's when it clicked with my memory of the power outage, that was the only way that could've happened.

Given the large amount of time it takes to manually review and correct mismatches between reactions to the role menu vs. being in the project vs. having the role, doing that reconciliation needs to be prioritized before this issue can be closed.