Support canary events

[ChlodAlejandro]

Filter out canary (heartbeat) events by default by checking if meta.domain == "canary", per Data Engineering and Event Platform Team at T266798. Canary events can be re-enabled by passing enableCanary when creating the stream.

Will be released as 2.1.0 when finished.

To-do:

• Add option to re-enable
• Document canary event behavior
• Utilize canary events as part of tests
• Not-so-possible since canary events are only fired once every hour; much longer than the CI timeout.

[ChlodAlejandro]

Given that canary events will start appearing by next month, it might also be worth backporting this to 2.0.x and 1.0.x tags so that no bumps to package.json would be required for anyone on those versions. I'll think about this more tomorrow.

[ottomata]

Given that canary events will start appearing by next month

FWIW, canary events are already present in many of the available streams. The ones that will have them newly enabled are the ones listed in the ticket: mostly ones that were created before 2020.

[ChlodAlejandro]

Most of the streams covered by this library are part of the list (only exception being mediawiki.revision-tags-change), so it's more or less the same for this specific library. This is probably still something worth backporting to older versions. (currently running tests to see if this is something worth backporting)

[ChlodAlejandro]

Based on what I understood from wikitech:Data Engineering/Systems/Hadoop Event Ingestion Lifecycle#Canary Events, the meta.stream field remains intact and unchanged by ProduceCanaryEvents. The following lines are what fires the events for any client using this library:

// Emit event.
this.emit( data.meta.stream, data, event );
// Emit event to aliases of event stream.
if ( WikimediaEventStreamAliasesKey[ data.meta.stream ] ) {
    for ( const alias of WikimediaEventStreamAliasesKey[ data.meta.stream ] ) {
	this.emit( alias, data, event );
    }
}

which means if meta.stream remains unchanged, canary events will still be received by dependents. TL;DR: The way the code is written right now won't protect current users of this library from receiving canary events when they are enabled. So a backport (even one that only drops canary events and doesn't include an option to enable them) might help in getting this to users who use an older version of this library. This does mean everyone will be required to update when the events become enabled, but I've set the old versions as deprecated on npm anyway, so that was already a given from long ago.

Will now be merging this in a bit!