Skip to content

Latest commit

 

History

History
248 lines (173 loc) · 9.4 KB

ADDING_ESPS.md

File metadata and controls

248 lines (173 loc) · 9.4 KB
Raw

Adding a new ESP

Some developer notes on adding support for a new ESP.

Please refer to the comments in Anymail's code---most of the extension points are (reasonably) well documented, and will indicate what you need to implement and what Anymail provides for you.

This document adds general background and covers some design decisions that aren't necessarily obvious from the code.

Getting started

  • Don't want to do all of this? That's OK! A partial PR is better than no PR. And opening a work-in-progress PR early is a really good idea.
  • Don't want to do any of this? Use GitHub issues to request support for other ESPs in Anymail.
  • It's often easiest to copy and modify the existing code for an ESP with a similar API. There are some hints in each section below what might be "similar".

Which ESPs?

Anymail is best suited to transactional ESP APIs. The Django core mail package it builds on isn't a good match for most bulk mail APIs. (If you can't specify an individual recipient email address and at least some of the message content, it's probably not a transactional API.)

Similarly, Anymail is best suited to ESPs that offer some value-added features beyond simply sending email. If you'd get exactly the same results by pointing Django's built-in SMTP EmailBackend at the ESP's SMTP endpoint, there's really no need to add it to Anymail.

We strongly prefer ESPs where we'll be able to run live integration tests regularly. That requires the ESP have a free tier (testing is extremely low volume), a sandbox API, or that they offer developer accounts for open source projects like Anymail.

Boilerplate

You should add entries for your ESP in:

  • pyproject.toml:
    • in the [project] metadata section under description and keywords
    • in the [project.optional-dependencies] section
  • integration-test.yml in the test matrix
  • tox.ini in the [testenv] section under setenv
    • if your ESP requires any extra dependencies, also update the tox.ini [testenv] extras and the "partial installation" at the bottom of [tox] envlist
  • README.rst in the list of ESPs

EmailBackend and payload

Anymail abstracts a lot of common functionality into its base classes; your code should be able to focus on the ESP-specific parts.

You'll subclass a backend and a payload for your ESP implementation:

  • Backend (subclass AnymailBaseBackend or AnymailRequestsBackend): implements Django's email API, orchestrates the overall sending process for multiple messages.

  • Payload (subclass BasePayload or RequestsPayload) implements conversion of a single Django EmailMessage to parameters for the ESP API.

Whether you start from the base or requests version depends on whether you'll be using an ESP client library or calling their HTTP API directly.

Client lib or HTTP API?

Which to pick? It's a bit of a judgement call:

  • Often, ESP Python client libraries don't seem to be actively maintained. Definitely avoid those.

  • Some client libraries are just thin wrappers around Requests (or urllib). There's little value in using those, and you'll lose some optimizations built into AnymailRequestsBackend.

  • Surprisingly often, client libraries (unintentionally) impose limitations that are more restrictive than than (or conflict with) the underlying ESP API. You should report those bugs against the library. (Or if they were already reported a long time ago, see the first point above.)

  • Some ESP APIs have really complex (or obscure) payload formats, or authorization schemes that are non-trivial to implement in Requests. If the client library handles this for you, it's a better choice.

When in doubt, it's usually fine to use AnymailRequestsBackend and write directly to the HTTP API.

Requests backend (using HTTP API)

Good staring points for similar ESP APIs:

  • JSON payload: Postmark
  • Form-encoded payload: Mailgun

Different API endpoints for (e.g.,) template vs. regular send? Implement get_api_endpoint() in your Payload.

Need to encode JSON in the payload? Use self.serialize_json() (it has some extra error handling).

Need to parse JSON in the API response? Use self.deserialize_json_response() (same reason).

Base backend (using client lib)

Good starting points: Test backend; SparkPost

If the client lib supports the notion of a reusable API "connection" (or session), you should override open() and close() to provide API state caching. See the notes in the base implementation. (The RequestsBackend implements this using Requests sessions.)

Payloads

Look for the "Abstract implementation" comment in base.BasePayload. Your payload should consider implementing everything below there.

Email addresses

All payload methods dealing with email addresses (recipients, from, etc.) are passed anymail.utils.EmailAddress objects, so you don't have to parse them.

email.display_name is the name, email.addr_spec is the email, and str(email) is both fully formatted, with a properly-quoted display name.

For recipients, you can implement whichever of these Payload methods is most convenient for the ESP API:

  • set_to(emails), set_cc(emails), and set_bcc(emails)
  • set_recipients(type, emails)
  • add_recipient(type, email)

Attachments

The payload set_attachments()/add_attachment() methods are passed anymail.utils.Attachment objects, which are normalized so you don't have to handle the variety of formats Django allows. All have name, content (as a bytestream) and mimetype properties.

Use att.inline to determine if the attachment is meant to be inline. (Don't just check for content-id.) If so, att.content_id is the Content-ID with angle brackets, and att.cid is without angle brackets.

Use att.base64content if your ESP wants base64-encoded data.

AnymailUnsupportedFeature and validating parameters

Should your payload use self.unsupported_feature()? The rule of thumb is:

  • If it cannot be accurately communicated to the ESP API, that's unsupported. E.g., the user provided multiple tags but the ESP's "Tag" parameter only accepts a (single) string value.

  • Anymail avoids enforcing ESP policies (because these tend to change over time, and we don't want to update our code). So if it can be accurately communicated to the ESP API, that's not unsupported---even if the ESP docs say it's not allowed. E.g., the user provided 10 tags, the ESP's "Tags" parameter accepts a list, but is documented maximum 3 tags. Anymail should pass the list of 10 tags, and let the ESP error if it chooses.

Similarly, Anymail doesn't enforce allowed attachment types, maximum attachment size, maximum number of recipients, etc. That's the ESP's responsibility.

One exception: if the ESP mis-handles certain input (e.g., drops the message but returns "success"; mangles email display names), and seems unlikely to fix the problem, we'll typically add a warning or workaround to Anymail. (As well as reporting the problem to the ESP.)

Batch send and splitting to

One of the more complicated Payload functions is handling multiple to addresses properly.

If the user has set merge_data, a separate message should get sent to each to address, and recipients should not see the full To list. If merge_data is not set, a single message should be sent with all addresses in the To header.

Most backends handle this in the Payload's serialize_data method, by restructuring the payload if merge_data is not None.

Tests

Every backend needs mock tests, that use a mocked API to verify the ESP is being called correctly. It's often easiest to copy and modify the backend tests for an ESP with a similar API.

Ideally, every backend should also have live integration tests, because sometimes the docs don't quite match the real world. (And because ESPs have been known to change APIs without notice.) Anymail's CI runs the live integration tests at least weekly.

Webhooks

ESP webhook documentation is almost always vague on at least some aspects of the webhook event data, and example payloads in their docs are often outdated (and/or manually constructed and inaccurate).

Runscope (or similar) is an extremely useful tool for collecting actual webhook payloads.

Tracking webhooks

Good starting points:

  • JSON event payload: SendGrid, Postmark
  • Form data event payload: Mailgun

(more to come)

Inbound webhooks

Raw MIME vs. ESP-parsed fields? If you're given both, the raw MIME is usually easier to work with.

(more to come)

Project goals

Anymail aims to:

  • Normalize common transactional ESP functionality (to simplify switching between ESPs)

  • But allow access to the full ESP feature set, through esp_extra (so Anymail doesn't force users into least-common-denominator functionality, or prevent use of newly-released ESP features)

  • Present a Pythonic, Djangotic API, and play well with Django and other Django reusable apps

  • Maintain compatibility with all currently supported Django versions---and even unsupported minor versions in between (so Anymail isn't the package that forces you to upgrade Django---or that prevents you from upgrading when you're ready)

Many of these goals incorporate lessons learned from Anymail's predecessor Djrill project. And they mean that django-anymail is biased toward implementing relatively thin wrappers over ESP transactional sending, tracking, and receiving APIs. Anything that would add Django models to Anymail is probably out of scope. (But could be a great companion package.)