Generate event monitoring view for all apps with events ping #5799
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
scholtzan
reviewed
Jun 18, 2024
Comment on lines
+63
to
+65
| if bq_dataset_name in self._get_prod_datasets_with_event(): | ||
| pings.add("events") | ||
|
|
There was a problem hiding this comment.
Suggested change
| if bq_dataset_name in self._get_prod_datasets_with_event(): | |
| pings.add("events") |
The materialized views only need to monitor tables that have event-type probes as defined in probe-info. Some apps don't seem to have such probes in the events tables, so we shouldn't add them here since not monitoring them reduces cost a bit.
There was a problem hiding this comment.
Ok so do we only care about custom events defined in the app and not in the libraries? In that case, #5798 would be at least partially correct and a bunch of materialized views would need to be deleted to get what's deployed consistent with the generated sql. I didn't think it made sense to suddenly delete them.
If we do care about events in libraries then I think they all will get at least glean.restarted in the events ping because of glean-core. If it's more complicated than that then I can leave this for now since nothing is broken at the moment.
There was a problem hiding this comment.
Yeah, good point. I guess adding events_v1 to the monitoring shouldn't cause any issues of stuff breaking.
Comment on lines
+44
to
+49
| return [ | ||
| s.bq_dataset_family | ||
| for s in get_stable_table_schemas() | ||
| if s.schema_id == "moz://mozilla.org/schemas/glean/ping/1" | ||
| and s.bq_table == "events_v1" | ||
| ] |
There was a problem hiding this comment.
This was a bit of a hacky way to get all relevant datasets, since (almost?) all of them do have event pings.
scholtzan
approved these changes
Jun 18, 2024
Integration report for "Make events order deterministic for diffs"
|
|
I filed #5804 to track deployed materialized views not getting updated. If I'm looking at it correctly, merging this shouldn't actually change any of the deployed views until that's fixed. I also just sorted the events lists that get passed into the templates so the diffs should be more stable |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #5797
This recreates the materialized views that were removed from the generated sql in #5576. Looking at the generated sql diff,
monitoring/event_monitoring_live/view.sqlwasn't changed so I think that means the nothing new gets added or removed as expected (other thangleanjs_docs_derived.event_monitoring_live_v1).@scholtzan One thing I'm not sure about is that this adds
events_v1to the union in some of the materialized views. Did we intentionally want to exclude those from the views for apps not listed in the bqetl config?Checklist for reviewer:
<username>:<branch>of the fork as parameter. The parameter will also show upin the logs of the
manual-trigger-required-for-forkCI task together with more detailed instructions.For modifications to schemas in restricted namespaces (see
CODEOWNERS):┆Issue is synchronized with this Jira Task