[Bug] Import Viewer Database drops users if they are included in a batch with banned/deleted users

[Marstead]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When importing users to Firebot using the standard import tool in the Viewer Database, some percentage of viewer records are dropped.

Expected Behavior

Banned & deleted users should be (and are) dropped. However, legitimate Twitch users are also dropped if they are included in the same batch as a bad entry.

Steps To Reproduce

1. Import a viewer .xlsx file containing viewers who are banned or deleted on Twitch.
2. Review the logs to see viewers which have been culled
3. Check for viewers which are legitimate (not banned, not deleted) but still dropped during the import.

I've attached a sample xlsx of my own viewer database where you can see some of these viewers being dropped.

Points.xlsx

Further info from @Taronin, a dev helping me with a temporary user import solution:

I ran into an issue with 400 errors from the Twitch 'request users' API that were the result of including a blank string in the list of usernames. This silently resulted in missing data for every other user sent in the same 100-user request as the bad entry, in terms of the top level API call for ~40k users from my code.

There may be other possible malformed username input values that would similarly break a whole API request, which the JS binding seems to treat as "none of the users in this batch exist"

Version

5.63.2

What operating system are you using Firebot on?

Windows 10

Relevant log output

No response

Contact Details (Optional)

Marstead

[Marstead]

Related to this issue are two feature requests, #2788 and #2789 . The current Firebot import tool seems insufficient for the import use case of streamers moving from another bot to Firebot as, in addition to dropping viewer records in error, it also can only import two field values (Viewer Name & Watch Time), and not loyalty points, join date, last seen, chat messages, or other data.

@Taronin developed a third-party script to act as a temporary hack for DeepBot users migrating to Firebot:

https://github.com/codemage/firebot-deepbot-importer/releases/tag/v0.0.2

This script could be retuned to accept input from other bots as well. But it's still just a temporary hack, a more robust import tool native to Firebot would work much better. The tool rewrite likely required in #2789 would resolve this issue as well as #2788 .