Crash Monitoring

Important

• The main goal here is not to file an issue for every single distinct crash report, but to find regressions of new problems that need to be addressed.
• Once you're familiar with the process this should not take more than 10 mins in the morning.
• Quick links that you should check everyday Sentry Query and Socorro Query

Things to note before starting:

• Since we are focused on Java crashes, Sentry currently is the better choice.
• Ignore level:info issues (blue labels in Sentry). These issues are informational only.

What to do when monitoring crashes:

• Look at Sentry Fenix-nightly overview. Go though trending issues. Sentry Dashboard.
• Look at Sentry custom search Sentry Query.
• Sign up for https://groups.google.com/a/mozilla.org/g/stability to get a daily email on the stability of Fenix.

How to determine a crash requires actions:

• Crashes that have level either Fatal or Error.
• Crashes that are occurring on latest Fenix and A-C versions.
• Crashes that are spiking.
• Crashes that are new.
• Crashes that happen repeatedly and often.

When a crash report requires actions:

• Is this a crash due to a recent change? If so, contact the developer.
  • The histogram on the right side can help determine this along with checking the Firefox-Beta and Firefox Sentry products.
• Triage the crash to determine if the issue is real and requires a Github issue to track it.
  • When filing an issue add a link to it as a comment in the Sentry crash for the products (nightly, beta, release) where the crash appears.
• If a Github issue is required to track the crash, put the issue in the Android Team Backlog Staging Board.
• Notify the relevant teams on Slack/Matrix that there's a new crash in Nightly that needs urgent attention, e.g. #synced-client-integrations for all things involving application services (A-S), #nimbus-rust-sdk for Nimbus, and GeckoView on Matrix.

What can you do to help when not monitoring crashes

• If you recently landed a new module / change that is significant, contact the crash monitor so they are aware of it.

Crash monitoring with Socorro

• Look at Top Crashers for Fenix Nightly for reports on Nightly builds.
  • This will return zero results if GV build ID is greater than 3 days old. Change to the 7 day view and ask #releaseduty-mobile in Slack about the GV upgrade task being broken.
• Ignore [INFO] issues.
• Use Sentry, Github and Bugzilla to determine if the crash has already been reported.
• If the crash is new and the volume is high, then consider raising an issue.
• If the crash is a Native crash, file a bug using the crash-stats Bugzilla tab from a crash ID.
• If the crash is a Java crash, then consider opening an issue on Github.