Emails sent from the site are not rendered nicely any more

[adrianbanks]

This is a regression caused by #330.

Before the change, the HTML emails were formatted nicely, and had a text-specific version inside them (different content and differently formatted so as to be suitable for reading in a text client):

After the changes to email the submissions to submitters, the emails now don't look nice, and the text and HTML versions appear to have the same content:

There is also a duplicate bit of text about how to contact us (reported separately in #335).

[adrianbanks]

This is more serious than it first seems. In doing some more testing around this, the link can sometimes get truncated in the linked portion of the email, meaning that the user can never click on it and change their password:

When they click it, they get to the page to reset their password, but it never works because the token is incorrect.

To make it work, they would have to notice and then copy and paste the text portion.

[adrianbanks]

This is getting worse. In the above example, the token rendered in the email is xjYydWSjyIohr6eNSkzw2. In the hyperlink, it is xjYydWSjy. In the plain text portion of the email (and in the database) it is xjYydWSjy_Iohr6eNS_kzw2. Only by specifying the latter one of these will the user be able to change their password correctly, but they have to work very hard to be able to even find the correct token. All others will fail, but don't make it obvious that it has.

[alastairs]

Does the SendGrid API allow for sending text emails only? A workaround until we can get it working properly would be to comment out the line that includes the HTML version so only the text version of emails is sent.

[adrianbanks]

I'm not convinced this is down to the email being in HTML, but something to do with how SendGrid formats them.

In the previous reset emails (before the recent refactorings), the link to reset (on the button) was of the form https://u688375.ct.sendgrid.net/wf/click?upn=1HDJdoZVPwX9q8uTLy4EOQWZ4JgXGHAl1OhqhgZbmrf0oyBWW6ahFDvsXT5ZW5VsY0cyBwM1v32QtHVFNhHXXkSbRZ-2FZBgfTWp1G4FYZnh4DkEqmT8H4uYew1EpYhkxG_mJ2vvicTuK2lzdbeKkJ1BXY2M6gwtGGnImiVkg16ToZcSlwT71-2F7aVRwCZvWXhbqDKDEpslvBh8L8ujkurewD5LUvOoQTsYB6pOTiCB56S9tAJgxliYpr9ZdSQklJBOuOvBgX0PnXAD3IRTiDpBO-2FaItr-2FNLuJDRs2tgO4dR2Pw9uQjeRst-2Bp6ZwMdu2kvTYvICJPmc9NsBv28wILuIQjzPrl7MjNrtLXEU8zLc9ZjQ-3D, which when clicked did a redirect to the correct url, including the correct token.

In the new emails, no such reformatting occurs, resulting in a broken link.

For the time being, I've disabled sending of HTML emails and pushed that to production (it does make the submitted session emails look really pants, but that's better than preventing people from being able to log in). The recent email changes will need looking at again to fix it though.

[alastairs]

Agreed it needs looking at again.

The problem here isn't SendGrid, it's the Markdown library. The new emailing code works as follows:

1. A mail template is defined as Markdown text
2. Any necessary substitutions in the template are performed
3. The Markdown is rendered to HTML
4. The resulting HTML fragment for the message body is inserted into the HTML document
5. The results from steps 2. and 4. are passed to SendGrid as the text and HTML versions of the email respectively.

The issue is that the password reset link is substituted before the template is passed for Markdown rendering, so the URL is run through the Markdown parser too. Pairs of underscores in Markdown of course mean "make this text italic", and there seems to be a precedence issue in the Markdown parser causing underscores in the URL to be replaced with HTML.

The fix might be as simple as upgrading to a newer version of MarkdownSharp (or whatever it's called).

[adrianbanks]

Ah yes, that would explain why the link appears as partially italicised. We should probably only run the session abstract through the markdown parser, not the whole email, as that is the only bit we want converting.

[adrianbanks]

I've undone the PR that introduced this bug, to ensure that the password reset emails are working properly. The changes that originally went in will need looking at again to not introduce the issue.