Blog: How we built mobile replay

contents/blog/mobile-session-replay.md

@@ -0,0 +1,156 @@
+---
+title: How we built mobile replay (and why it took so long)
+date: 2024-10-08
+author:
+ - ian-vanagas
+ - manoel-aranda-neto
+rootpage: /blog
+featuredImage: >- 
+  https://res.cloudinary.com/dmukukwp6/image/upload/posthog.com/contents/images/blog/posthog-engineering-blog.png
+featuredImageType: full
+tags:
+ - Guides
+---
+
+Web session replay has been a core part of PostHog since our first hackathon way back in 2020, but we only released our first betas for mobile apps in April of this year.
+
+Why the long wait? 
+
+1. We had much more demand from web-based B2B customers during our early-stage phase.
+
+2. We went wide instead of deep. We shipped a whole suite of products, including feature flags, experiments, surveys and (more recently) our own data warehouse.
+
+3. Building performant and functional replay for mobile apps is way harder than web.
+
+Happily, after several months of hard work from the replay team, we have betas for [iOS](/docs/session-replay/ios), [Android](/docs/session-replay/android), and [React Native](/docs/session-replay/react-native), with [Flutter](https://github.com/PostHog/posthog-flutter/issues/69) coming soon. 
+
+This post covers how it works and some the technical challenges we overcame it make it.
+
+## What's so difficult about mobile session replay?
+
+Developers complain about [the lack of good mobile replay options](https://medium.com/goodones/15-years-later-there-is-still-no-good-session-replay-for-ios-f8d335999737). The existing options were too expensive, ruined performance, lacked privacy controls, or locked behind [annoying salespeople](/founders/negotiate-software-better). These are all antithetical to how we do things at PostHog.
+
+Why was it this way for so long?
+
+### 1. Multiple platforms
+
+The industry's big secret about web session replay is that it largely relies on a single open-source library to work: [rrweb](https://github.com/rrweb-io/rrweb). It includes tools for recording web interactions and state changes, structuring session data, and playback.
+
+Unfortunately, rrweb for mobile doesn't exist. To build mobile session replay, you simply need to do a lot of work. Instead of a single JavaScript SDK, mobile requires multiple (like iOS, Android, React Native, and Flutter) and all the building and testing that goes with it.
+
+There are even breaking differences within platforms. For example, Jetpack Compose uses a compositional model for UI, which is different from Android's traditional view-based model. This means you need to develop separate ways of doing replays when using it. iOS has a similar problem with SwiftUI versus UIKit.
+
+### 2. Performance
+
+Phones are much less powerful than desktops, and not all phones are a top-of-the-line iPhone. Because of this, we need to be much more sensitive about performance. 
+
+If you've ever tried to record your screen on an older phone, you would know its impact on performance. Apps take longer to load, animations become choppy, reactivity degrades, and your phone heats up.
+
+Developers don't know what devices their users are on, so they need to be especially careful about performance. Anything that degrades user experience is not an option for many developers.
+
+### 3. Privacy
+
+To protect users privacy, developers often mask and exclude sensitive information. The tricky part of this is identifying what information is sensitive.
+
+Web apps rely on the DOM to do this. The DOM provides a hierarchical tree structure that represents and uniquely identifies elements. The web also has standardized elements, like `<input type="password">`. Combining these makes it easier to identify, mask, and exclude sensitive elements on the web.
+
+Mobile doesn't have standardized structures or elements. Accessibility identifiers are also inconsistently implemented. This means identifying, masking, and excluding content is a lot trickier.
+
+### 4. Testing
+
+We're big fans of [dogfooding](/product-engineers/dogfooding) at PostHog. This helps us [test in production](/product-engineers/testing-in-production), find issues, and fix them before they affect users. We can often ship a feature early because we can test it ourselves.
+
+Unfortunately, like most analytics tools, PostHog is built for the desktop and we don't have a mobile app. We can't dogfood mobile replay and need to spend more time testing in development to make up for it.
+
+This means relying more on demo and open source apps as well as our users and their feedback. If we were not careful, we would ship more bugs and build something that doesn't work well for larger, production-quality apps.
+
+## How we solved these problems
+
+### 1. Multiple platforms
+
+The solution to this problem is plastered on motivation posters everywhere: **do the work** (to develop for each platform).
+
+The one thing that made this easier was reusing our existing session replay infrastructure. This included the data structure, the way to store replays, as well as a complete product for playback and analysis.
+
+![Replay](https://res.cloudinary.com/dmukukwp6/image/upload/replay_03a8c56981.png)
+
+The first platform-specific work we did was rewriting the SDKs in Kotlin and Swift, removing code we didn't use, improving tests, automating deployments, and making sure they worked with the latest platforms.
+
+Once updated, we could get on with the work developing platform-specific proof-of-concepts.
+
+### 2. Performance
+
+Not slowing down people's apps was core to our mission with session replay. 
+
+To do this, we decided to focus on capturing wireframes instead of screenshots. Although we built both in the end, wireframe mode is much less performance-intensive, meaning it is the mode users prefer.
+
+![Screenshot vs Wireframe](https://res.cloudinary.com/dmukukwp6/image/upload/wireframe_78ce94bd4b.png)
+
+Building wireframe mode meant starting from capturing and rendering the smallest amount of data and building up to more complex components. In our [Android proof-of-concept](https://github.com/PostHog/posthog-android/pull/69), we started by using `Curtains` to capture the view hierarchy, listen for changes, and track touch events. We then transformed this data to render it as an HTML wireframe (as rrweb expects).
+
+![PoC](https://res.cloudinary.com/dmukukwp6/image/upload/mobile_b85c032c93.png)
+
+Once this was working, we worked to capture standard Android components like radios, checkboxes, Calendar, [Toggle](https://github.com/PostHog/posthog/pull/19279), and RatingBar. Many of them required custom transformation and components to render properly. For example, radio buttons didn't group, padding wasn't applied, and positioning was wrong.
+
+Beyond this, fitting the data into a service meant for the web caused many challenges. For example, click events weren't showing even though the types and data were the same as the web data. After investigation, we found that rrweb expects touch events to be associated with specific elements. Setting the ID to the `body` element was enough to fix it.
+
+Once we got consistent and useful results from what we built, we recruited our first test users and iterated from there. Later, we followed a similar process for [iOS](https://github.com/PostHog/posthog-ios/pull/115).
+
+Beyond our focus on wireframe mode, the biggest performance improvements came from getting mobile replay into the hands of our users. We found and solved multiple performance issues thanks to their feedback.
+
+### 3. Privacy
+
+As for privacy, we built the ability to mask all text inputs and images as well as redact certain views by adding `ph-no-capture` tags like this:
+
+<MultiLanguage>
+
+```xml
+<ImageView
+    android:id="@+id/imvProfilePhoto"
+    android:layout_width="200dp"
+    android:layout_height="200dp"
+    android:tag="ph-no-capture"
+/>
+```
+
+```swift
+let imvProfilePhoto = UIImageView(frame: CGRect(x: 50, y: 50, width: 100, height: 100))
+imvProfilePhoto.accessibilityIdentifier = "ph-no-capture"
+```
+
+</MultiLanguage>
+
+We added this functionality on both wireframe and screenshot mode. As nice as it would have been to have automatic masking like we do on the web, the ability to have masking at all enables privacy-sensitive teams to actually use replay. 
+
+There are more challenges to solve, specifically around [SwiftUI](https://github.com/PostHog/posthog-ios/issues/162) and Jetpack Compose, since the way they transpile code causes the representation to not be a 1:1 match and properties are lost.
+
+### 4. Testing
+
+We solved the testing problem in 3 key ways:
+
+1. **Building testing tools:** <TeamMember name="Paul D'Ambra" /> and <TeamMember name="Manoel Aranda Neto"/> wanted to keep the rrweb schema for mobile replay to ensure the fewest changes possible to our API and player. To do this, they wrote a [validation and testing tool](https://github.com/PostHog/mobile-replay-data-transformer) to rapidly test the transformations before deploying it in our main app.
+
+2. **Using open source apps:** The team found a [list of open source apps](https://github.com/pcqpcq/open-source-android-apps) they could test with. This helped find issues, support more edge cases, and build test coverage.
+
+3. **Testing in production:** Luckily, mobile replay had massive demand. We could rely on our users to test in production and give us feedback. We recruited users from the public issues for mobile replay, sales conversations, and existing mobile SDK users.
+
+## Making mobile session replay available for everyone
+
+In many ways, mobile has been neglected by the analytics industry. Tools like session replay have either not existed, been locked behind enterprise plans, or been too expensive for most developers.
+
+<ProductScreenshot
+  imageLight="https://res.cloudinary.com/dmukukwp6/image/upload/v1725526016/posthog.com/contents/Screenshot_2024-09-05_at_9.46.17_AM.png"
+  imageDark="https://res.cloudinary.com/dmukukwp6/image/upload/v1725526016/posthog.com/contents/Screenshot_2024-09-05_at_9.46.32_AM.png" 
+  alt="iOS session replays in PostHog" 
+  classes="rounded"
+/>
+
+We want to change this. Mobile replay is free while in beta, and once it's out of beta, we'll follow [our pricing principles](/handbook/engineering/feature-pricing), making it as affordable as possible.
+
+This enables us to help more developers have the tools they need to build successful products.
+
+## Further reading
+
+- [The 80/20 of early-stage startup analytics](/founders/early-stage-analytics)
+- [We decided to make session replay cheaper](/blog/session-replay-pricing)
+- How to set up [Android](/tutorials/android-session-replay), [iOS](/tutorials/ios-session-replay), and [React Native](/docs/session-replay/react-native) session replay

contents/blog/posthog-vs-hotjar.mdx

@@ -93,7 +93,7 @@ Both Hotjar and PostHog are good options if you're looking for a [Microsoft Clar
   <ComparisonRow column1={true} column2={true} feature="Download replays" description="Save replay files offline for storage" />
 </ComparisonTable>
 
-> **Note:** Our mobile session replay is free while it's in development, and we're actively working on session replay support for React Native and Flutter, too. See our [mobile replay docs](/docs/session-replay/mobile) for more info on development and how replay works on mobile apps.
+> **Note:** Our [mobile session replay](/blog/mobile-session-replay) is free while it's in development, and we're actively working on session replay support for React Native and Flutter, too. See our [mobile replay docs](/docs/session-replay/mobile) for more info on development and how replay works on mobile apps.
 
 ### Heatmaps
 

contents/handbook/growth/sales/new-sales.md

@@ -218,7 +218,7 @@ Oh no! It's ok - the most important thing here is that we learn. You should capt
 - Product/feature gap:
   - Threshold-based alerting
   - Role-based access
-  - Mobile session replay
+  - [Mobile session replay](/blog/mobile-session-replay)
   - Experimentation
   - Error monitoring
   - (Create a new category if something else comes up, don’t just have an ‘other’ bucket)

contents/newsletter/beyond-the-10x-engineer.md

@@ -113,7 +113,7 @@ Our teams have a lot of autonomy. They decide on their roadmap, goals, and imple
 - Choosing the right metrics to [measure the success of new features](https://newsletter.posthog.com/p/how-to-build-new-features-users-love).
 - Understanding the infrastructure requirements and scalability.
 
-To guide the development of mobile replay, [Manoel](/community/profiles/1166) researched competitors and feedback about them. He and [Paul](/community/profiles/69) also developed a spec for mobile replay that included details about the client lifecycle, API request, data structure, and compatibility with [rrweb](https://github.com/rrweb-io/rrweb). This research and planning helps them build a feature that makes business sense while also being scalable and maintainable.
+To guide the [development of mobile replay](/blog/mobile-session-replay), [Manoel](/community/profiles/1166) researched competitors and feedback about them. He and [Paul](/community/profiles/69) also developed a spec for mobile replay that included details about the client lifecycle, API request, data structure, and compatibility with [rrweb](https://github.com/rrweb-io/rrweb). This research and planning helps them build a feature that makes business sense while also being scalable and maintainable.
 
 The ability to see the bigger picture enables us to make better long-term decisions. If we’re too tunnel-visioned, we can miss opportunities or build things that break in the long run.
 

src/data/authors.json

@@ -271,6 +271,14 @@
         "link_type": "linkedin",
         "link_url": "https://www.linkedin.com/in/igorkotua/"
     },
+    {
+        "handle": "manoel-aranda-neto",
+        "name": "Manoel Aranda Neto",
+        "role": "Mobile Engineer",
+        "image": "../../static/images/authors/manoel.png",
+        "link_type": "twitter",
+        "link_url": "https://twitter.com/marandaneto"
+    },
     {
         "handle": "mathew-pregasen",
         "name": "Mathew Pregasen",