[HOLD for payment 2024-04-09] Live Markdown Preview

[thienlnam]

Proposal

https://expensify.slack.com/archives/CC7NECV4L/p1675122898707719

Design Doc: https://docs.google.com/document/d/1rcmXjgwvsM3MtdQnkJlMs6IS-i8SHJd0Vs7B_U0K3UU/edit

Proposal: Implement a Live Markdown Preview feature in NewDot
Context: Markdown is a text formatting syntax that we use in NewDot to add rich text formatting to messages. It allows users to add basic formatting such as bold, italic, bullet points, headings, links, etc. to their messages without having to use HTML or other complex formatting tools.

Problem:
As NewDot becomes more widely used for group communication, it’s essential to offer ways to effectively format and structure messages to improve productivity and avoid miscommunication similar to how we do resyncs in WN.
The problem is that without a visual representation of the final message, it can be difficult for users to confirm that their message is correctly formatted and conveys the intended information. Inconsistent or improperly formatted messages can hinder effective communication and decrease productivity, especially for users who are not familiar with markdown.

Solution:
To enhance user experience and improve communication, add a live markdown preview feature in the chat application. This will allow users to preview their message format in real-time and make any necessary adjustments before sending the message, resulting in clearer and more easily understood communications.
The live markdown preview feature is already present in Slack, and gives you the assurance that the message you are crafting, will show up exactly like you expect once sent.
The implementation will be done in collaboration with Callstack, utilizing the Design Doc process, and will be executed as a concurrent project. The live markdown preview feature is a front-end focused project and will not impact any internal employees or ongoing newsletters.

Tasks

• Post Proposal (full Problem/Solution statement) in #expensify-open-source
• Wait at least one full business day, and until the post has a majority (2/3) of positive reactions (👍)
• Paste Proposal in the space above with a link to the Slack thread
• Email [email protected] and paste in the Proposal
• Fill out the High-level overview of the problem, Timeline, and Terminology sections of the Design Doc
• Email [email protected] (continue the same email chain as before) with the link to your Design Doc
• Host a pre-design meeting (example) in #expensify-open-source to discuss any necessary details in public before filling out the High-level of proposed solution section.
• Fill out the High-level of proposed solution section
• Email [email protected] again with links to the doc and pre-design conversation in Slack
• Add the DesignDocReview label to get the High-level of proposed solution section reviewed
• Respond to any questions or concerns and bring up blockers in Slack to get a consensus if necessary
• Confirm that the doc has the minimum necessary number of reviews before proceeding
• Host another pre-design meeting in #expensify-open-source to ask for engineering feedback on the technical solution.
• Fill out the Detailed implementation of the solution and related sections.
• Re-add the DesignDocReview label to this issue
• Respond to any questions or concerns and bring up blockers in Slack to get consensus if necessary
• Confirm that the doc has the minimum necessary number of reviews before proceeding
• Email [email protected] one last time to let them know the Design Doc is moving into the implementation phase
• Implement the changes
• Send out a follow up email to [email protected] once everything has been implemented and do a Project Wrap-Up retrospective that provides:
  • Summary of what we accomplished with this project
  • What went well?
  • What could we have done better?
  • What did we learn?

[melvin-bot]

Triggered auto assignment to @bfitzexpensify (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details.

[thienlnam]

Creating a new issue to track this as we're taking a different direction with this and the old issue is long.

Currently - we're waiting for next month and having Tomasz Zawadzki from SWM take this up. I need to update the design doc with the new direction we've decided on

[roryabraham]

Excited to see this one play out 🙂

[thienlnam]

Tomasz is back from OOO and is looking at the design doc. Asked for an update here

[tomekzaw]

Just wanted to confirm that we're investigating the feasibility of implementing this feature in React Native. At this point things look promising. I will keep you updated.

[thienlnam]

@tomekzaw How are things looking? Do you think we could also get some regular updates here as well so we're all kept up with progress on this?

[tomekzaw]

@thienlnam Things are still looking promising, at least on iOS. Sorry for lack of updates last week, I had very little time to work on this due to lots of duties in other projects but at least I've managed to complete them so I can focus on this topic.

Yesterday I was trying to dive into paragraph styling (e.g. for blockquotes or codeblocks) but I encountered some obstacles like retaining correct cursor position when typing. Because I'm building the PoC on top of React Native's TextInput component, it also suffers from the problems with asynchronous updates like a regular TextInput. Right now I'm trying to focus on the core of the feature (i.e. applying formatting to Markdown syntax) in an uncontrolled component and once it's done I will investigate other issues.

At this point I'm pretty confident this feature can be done on iOS with some amount of native code. Now I would like to prepare a similar PoC on Android in order to validate the approach on this platform as well.

There are some more questions and remarks I've gathered while working on this feature:

• [Q1] I'm still thinking if we should use third-party dependency for rich text editing. Yesterday I managed to setup an example app with react-native-aztec which works quite well although it seems to be designed for a different use-case (i.e. formatting text like in Microsoft Word or Slack). Apart from the section "Using Aztec Editor with ExpensiMark" in the "Live Markdown Preview" design doc, do you have some more information on why this solution was not chosen? I tried to reach @matkoson on Expensify Slack yesterday and I'm waiting for a response.

• [Q2] I'm thinking about embedding ExpensiMark library and calling it directly from native code on the UI thread. This way we could re-use existing code which is already well-tested. However, it looks like it can only parse Markdown into HTML but removing the Markdown syntax, i.e. the output for "Hello, *world*!" would be "Hello, <b>world</b>!" and not "Hello, *<b>world</b>*!" (notice the asterisks being still there, but also not properly formatted). Technically, we could convert a Markdown string into a HTML-formatted Markdown string and then convert it once again into NSAttributedString using initWithHTML method but I'm worried there will be some differences between actual and expected output of this whole pipeline. At this point I see two possible solutions: 1. Extend ExpensiMark library so it can parse Markdown string into JS data structures that we can easily convert to NSAttributedString, 2. Write the parser in Objective-C or Swift for better performance and just re-use test cases from ExpensiMark library. What do you think?

• [Q3] How should we format escape characters like the backslash in * \* *? Should it be the same color as asterisks? Here's how Discord does it: Also, I expect there will be some questions on how to format some edge-cases, who is the best person to contact here?

• [Q4] It looks like iOS doesn't support margin/padding in NSAttributedString so we won't be able to style inline code 100% accordingly to the provided designs. Even the Slack iOS app suffers from this limitation, compare the screenshots: (desktop, i.e. web on Electron) vs. (iOS app). I guess we could hack it by adding some spaces before and after the code but I'm afraid this would break pretty quickly and cause many issues with selection and line breaks (at this point, we assume that there are no "fake" characters in the native TextInput, so if we just read the current text and remove all formatting, we get the input Markdown string). Is it okay to sacrifice the padding of inline code on iOS, at least at this stage? FYI, NSAttributedString supports background color (which looks like text highlight) but does not support border and border radius which is also inconvenient. However, we have an idea how to overcome this limitation (i.e. measuring the position and size of those substrings and rendering additional UILabel on top of them) and we think this is also what Slack does. Another alternative is to implement a custom text renderer, but this sounds way more difficult. For us, it would make more sense to implement this feature iteratively, i.e. start with monospace font and simple background using NSAttributedString, then add border according to the designs using UILabel, then think about padding. What do you think about it?

[thienlnam]

Thanks for the update!

I'm still thinking if we should use third-party dependency for rich text editing. Yesterday I managed to setup an example app with react-native-aztec which works quite well although it seems to be designed for a different use-case (i.e. formatting text like in Microsoft Word or Slack). Apart from the section "Using Aztec Editor with ExpensiMark" in the "Live Markdown Preview" design doc, do you have some more information on why this solution was not chosen?

What problems have they already solved that we can benefit from? Is it mainly the issues that stem from "the problems with asynchronous updates" with the native textInput?

We actually initially went this direction (PR) but the PR was held up and we didn't get around to continuing with this. We can continue down that route, but I think it's worthwhile to explore what our current issues that make it difficult to do this natively.

At this point I see two possible solutions: 1. Extend ExpensiMark library so it can parse Markdown string into JS data structures that we can easily convert to NSAttributedString, 2. Write the parser in Objective-C or Swift for better performance and just re-use test cases from ExpensiMark library. What do you think?

Interesting, I agree we might end up with some unintended complexity if we end up converting it twice to add back in the syntax. I think the first solution, but worried that it will make the library harder to update in the future. I'm not entirely sure what this would look like though. The Objective-C parser is also pretty interesting and would definitely be a lot more performant. I think in the end it comes down to how much of a performance difference between the solutions and also maintainability. Currently I'm leaning towards the first solution as then we don't have to maintain another parser in native code but curious for your thoughts here

[Q3] How should we format escape characters like the backslash in * * *? Should it be the same color as asterisks?

I think the discord format looks good, essentially make it look like the same as the other text within the markdown syntax. As for other questions, feel free to ask first in the expensify-swm channel if there are any questions I already have an answer for but for other things that may just be a matter of preference we might just need to start a conversation in expensify-open-source

Is it okay to sacrifice the padding of inline code on iOS, at least at this stage?

Yeah this definitely doesn't look like a blocker for any means, and figuring out a workaround appears to be more headache than it is worth. I agree with your decision, let's just build this feature and then iterate on the spacing if we need to but that's not a problem at this point

[roryabraham]

1. Extend ExpensiMark library so it can parse Markdown string into JS data structures that we can easily convert to NSAttributedString, 2. Write the parser in Objective-C or Swift for better performance and just re-use test cases from ExpensiMark library. What do you think?

Definitely I don't think we'd want to use Objective-C or Swift – if we want to implement a native markdown parser I would strongly encourage that we build it with C/C++ so that it can be used across platforms.

Furthermore, there's an existing md4c library that's very fast, has a wasm wrapper for use on web, and is compliant with the CommonMark standard. I'd be open to using that, but we have additional requirements:

• Need to parse HTML back into markdown
• Need a flag to parse markdown into HTML with markdown syntax preserved

I don't think we have such features in md4c, so that means we'd probably have to essentially rebuild Expensimark from scratch in C++. Overall that sounds like a lot of work when it's unclear whether Expensimark is a performance bottleneck for us, so I think the first option of just extending the JS implementation of Expensimark to meet our needs is the way to go for the V1 👍🏼

[roryabraham]

How should we format escape characters like the backslash in * * *? Should it be the same color as asterisks?

The way discord does it looks good 👍🏼

I expect there will be some questions on how to format some edge-cases, who is the best person to contact here?

My advice is that you should just make a decision that you think is best and post it in slack. That way if anyone feels passionately they can have the chance to object, but you're not blocked on us.

[roryabraham]

Even the Slack iOS app suffers from this limitation, compare the screenshots: web (desktop, i.e. web on Electron) vs. iOS (iOS app)

They look exactly the same to me 😅 let's not hack with extra spaces

Edit: Oh, I see it now. I'd say let's not worry about the padding of inline code blocks on iOS as it's a platform limitation that doesn't really affect usability. Your iterative approach for addressing the platform limitations of NSAttributedString sounds like a good approach 👍🏼

[roryabraham]

question: does Aztec implement its own text renderer?

[tomekzaw]

What problems have they [react-native-aztec] already solved that we can benefit from? Is it mainly the issues that stem from "the problems with asynchronous updates" with the native textInput?

I think they already solved some problems with native text formatting and selection but not sure if they also suffer from asynchronous updates, I would need to check that.

Topic: Extend ExpensiMark vs. implement native Markdown parser

Thanks @thienlnam and @roryabraham for your feedback on this topic.

I'd be open to using that [md4c], but we have additional requirements:

• Need to parse HTML back into markdown
• Need a flag to parse markdown into HTML with markdown syntax preserved

Could you please explain where these requirements come from? If we don't use react-native-aztec, do we need the HTML-formatted string? Currently, the iOS PoC that I'm working on does not need to use HTML, it parses Markdown directly to NSAttributedString.

I think the first option of just extending the JS implementation of Expensimark to meet our needs is the way to go for the V1

I strongly agree with this approach, for now let's just use something that gives a correct result (maybe even for some subset of supported syntax, e.g. bold/italic/links), then measure its performance, see if it's a bottleneck and plan next steps.

question: does Aztec implement its own text renderer?

I don't know yet but I can check it. Currently, I'm leaning towards not using react-native-aztec due to its complexity and the fact that it's another third-party dependency, and also needs to use HTML as the intermediate format. I think at some point we may face issues with text selection but they will be definitely easier to debug on a smaller codebase.

[thienlnam]

Could you please explain where these requirements come from? If we don't use react-native-aztec, do we need the HTML-formatted string? Currently, the iOS PoC that I'm working on does not need to use HTML, it parses Markdown directly to NSAttributedString.

Parsing the the HTML back into markdown is when we get to editing an existing message. Rory will have a better answer but I believe the second requirement is for rendering the html in the listview

I strongly agree with this approach, for now let's just use something that gives a correct result (maybe even for some subset of supported syntax, e.g. bold/italic/links), then measure its performance, see if it's a bottleneck and plan next steps.

Yup sounds great, let's do that!

[thienlnam]

For updating this issue - here's the latest demo from @tomekzaw https://expensify.slack.com/archives/C04878MDF34/p1698167840008629

[tomekzaw]

Update: Today I've implemented all formatting options on iOS (bold, italic, strikethrough, mention, link, email, inline code, codeblock, quote, heading). It feels quite robust and even the PoC implementation seems to be performant. There are some minor issues that need to be addressed (e.g. trailing whitespace or nested styles) but nothing critical.

Parsing the the HTML back into markdown is when we get to editing an existing message.

How are messages stored in the database? I thought they are stored in Markdown and converted to HTML on the client side so that we can display them using react-native-render-html. If they are stored in HTML, we can still convert them to Markdown (once) and use the obtained Markdown string as the initial value for the TextInput.

Definitely I don't think we'd want to use Objective-C or Swift – if we want to implement a native markdown parser I would strongly encourage that we build it with C/C++ so that it can be used across platforms.

I've just realized that we still need some Obj-C logic to build NSAttributedString and we'll also need similar logic on Android to build Spannable. Additionally, we would like to re-use ExpensiMark library which is implemented in JavaScript. Here's the way how the PoC works currently, I believe we can stick to this approach:

Input (Markdown): Hello, *world*!

Step 1. Convert Markdown to HTML using already existing ExpensiMark library. Output: Hello, <strong>world</strong>! Note that currently the output does not contain * characters! We can either extend the library so we add them now (e.g. as <span class="syntax">*</span>) but it also makes sense to restore them at some later stage (when parsing <strong>, we know there must have been a * before it).

Step 2. Parse HTML to common intermediate tree format. For instance, this could be JSON. This step is relatively easy since we can just find < and > characters thanks to the fact that all other occurrences in the text were replaced with < and >. Also, it's written in JavaScript so it will work across all platforms. Note that this indeed needs to be a tree to support nested styles, e.g. strikethrough inside bold inside blockquote. Output:

{
  "type": "regular",
  "children": [
    "Hello, ",
    { "type": "syntax", "children": [ "*" ] },
    { "type": "bold", "children": [ "world" ] },
    { "type": "syntax", "children": [ "*" ] },
    "!"
  ]
}

Step 3. Convert tree in common intermediate format to platform-specific tree (i.e. NSAttributedString on iOS, Spannable on Android). This needs to be implemented in the language of the platform (Objective-C/Swift, Java/Kotlin).

This pipeline is already implemented in the PoC and seems to work fine. The presence of the intermediate format also makes it easy to inspect, debug and test. Also, it minimizes the amount of code that needs to be written separately for each platform.

However, there's one problem with using ExpensiMark library for Markdown to HTML conversion in its current form (i.e. with removing Markdown syntax). Consider the following inputs:

• Input 1: https://example.com
• Input 2: [https://example.com](https://example.com)

Both of them give the following output: <a href="https://example.com">https://example.com</a>. When we parse HTML, we don't know which Markdown syntax we should restore. We could assume that if URL and label is the same, we skip []( and ), though. At this point it makes sense to leave the Markdown syntax in the output HTML string (as <span class="syntax"> or even <syntax>) but this effectively doubles the number of tests that we need to maintain. There's a similar problem with blockquotes as both >Hello and > Hello give the same HTML output.

Another problem is related to whitespace handling and especially trailing spaces and newlines which are removed from the HTML output, for instance "> Hello \n" becomes <blockquote>Hello</blockquote> (note no trailing space and no trailing <br />). If we are inside a quote and type a space (or hit Enter), it will be immediately removed because the HTML string does not contain it (and thus neither NSAttributedString will). So we would need an option to disable some ExpensiMark rules.

Some more questions:

[Q1] All spaces in codeblocks are represented as   to prevent them from collapsing into a single space. I've fixed it by replacing all occurrences of " " with " " in the output JSON tree. Note that if the code actually contains text " ", the & will be replaced with & so it becomes  , so this replacement is indeed safe to perform. The question is, can we guarantee that there will be no HTML tags inside <pre> in the output HTML? Or, alternatively, is it possible to format some substrings inside a codeblock (e.g. bold or links)?

[Q2] ExpensiMark library wasn't designed to handle Live Markdown Preview, so the following input:

will be formatted this way:

Do we want to prevent "```" (triple backticks) from being formatted as inline code containing the middle backtick?

Sorry for long comment again and looking forward to your answers and remarks!

[thienlnam]

How are messages stored in the database? I thought they are stored in Markdown and converted to HTML on the client side so that we can display them using react-native-render-html. If they are stored in HTML, we can still convert them to Markdown (once) and use the obtained Markdown string as the initial value for the TextInput.

They're stored as HTML in the database actually, so yeah agree that we can convert them into markdown and then use the obtained markdown for the TextInput

At this point it makes sense to leave the Markdown syntax in the output HTML string (as or even ) but this effectively doubles the number of tests that we need to maintain.

You bring up a pretty interesting point about the Markdown to HTML conversion with links. I agree it would be helpful if we had the existing markdown syntax in the output string but I'm not entirely sure we want to add that now - I think that those current edge cases seem fine. Like if we always restore markdown as a URL / add an extra space with blockquotes and it only happens when we edit a comment then I'd say that seems fine

Another problem is related to whitespace handling and especially trailing spaces and newlines which are removed from the HTML output,

👍

[Q1] All spaces in codeblocks are represented as to prevent them from collapsing into a single space. I've fixed it by replacing all occurrences of " " with " " in the output JSON tree. Note that if the code actually contains text " ", the & will be replaced with & so it becomes  , so this replacement is indeed safe to perform. The question is, can we guarantee that there will be no HTML tags inside

 in the output HTML? Or, alternatively, is it possible to format some substrings inside a codeblock (e.g. bold or links)?

IIRC we decided that we would prevent any nested markdown within code blocks. So that is true - and I don't think we would want to support any formatting within a codeblock as normal markdown doesn't support it either

Do we want to prevent "```" (triple backticks) from being formatted as inline code containing the middle backtick?

That could work, realistically I don't think anyone would be intentionally formatting a middle backtick in a code block. Though could we add or adjust a rule that just waits until it finds a resulting triple backtick? In any case, if that would overcomplicate things we can just go with your solution

Thanks for the questions and updates! It's looking great so far

[thienlnam]

Latest update here

[melvin-bot]

@tomekzaw, @thienlnam Eep! 4 days overdue now. Issues have feelings too...

[thienlnam]

https://expensify.slack.com/archives/C06BDSWLDPB/p1709306810896539?thread_ts=1708724096.745109&cid=C06BDSWLDPB

[melvin-bot]

@tomekzaw, @thienlnam Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[melvin-bot]

@tomekzaw, @thienlnam 6 days overdue. This is scarier than being forced to listen to Vogon poetry!

[thienlnam]

We're close to re-enabling on web! Latest is here https://expensify.slack.com/archives/C06BDSWLDPB/p1708724096745109

[melvin-bot]

⚠️ Looks like this issue was linked to a Deploy Blocker here

If you are the assigned CME please investigate whether the linked PR caused a regression and leave a comment with the results.

If a regression has occurred and you are the assigned CM follow the instructions here.

If this regression could have been avoided please consider also proposing a recommendation to the PR checklist so that we can avoid it in the future.

[melvin-bot]

⚠️ Looks like this issue was linked to a Deploy Blocker here

If you are the assigned CME please investigate whether the linked PR caused a regression and leave a comment with the results.

If a regression has occurred and you are the assigned CM follow the instructions here.

If this regression could have been avoided please consider also proposing a recommendation to the PR checklist so that we can avoid it in the future.

[melvin-bot]

⚠️ Looks like this issue was linked to a Deploy Blocker here

If you are the assigned CME please investigate whether the linked PR caused a regression and leave a comment with the results.

If a regression has occurred and you are the assigned CM follow the instructions here.

If this regression could have been avoided please consider also proposing a recommendation to the PR checklist so that we can avoid it in the future.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.58-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Enable Live Markdown Preview on web #38152

If no regressions arise, payment will be issued on 2024-04-09. 🎊

For reference, here are some details about the assignees on this issue:

• @tomekzaw does not require payment (Contractor)

[melvin-bot]

BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@tomekzaw] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
• [] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.

[melvin-bot]

Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.

[thienlnam]

We've been live! Now we're handling clean up in #36071