PolicyStat's Dev Blog

Using Django-mailer With Django-ses for Amazon SES Goodness

Amazon’s new Simple Email Service is pretty awesome.

Previously, there were basically two options:

  1. Pay out the wazoo per message
  2. Hassle with setting up a mail server and continuously wrangle with the email world to avoid spam filters

spam

With SES, you avoid the hassle of deliverability and you get 2k messages per day for free with 1 penny per hundred messages over that.

At PolicyStat, we use django-mailer to queue up emails for sending so that we don’t need to make an SMTP connection inside the request/response cycle. It also gives us nice logging and do-not-send abilities and we wanted to keep those. We also really wanted to use SES.

Luckily, as is usually the case with the Python and Django community, in the ~2 weeks between the SES announcement and the time we wanted to implement, the wonderful community around Boto had already grown SES support and a gentlemen named Harry Marr out of the UK had created a Django app for SES Email called django-SES.

The integration was painless for us thanks to Django 1.2’s email backend support. The problem we ran in to was that django-mailer was specifically designed for use with SMTP-based email sending. In the send_all() function used for sending all of the emails in your queue, there was code like:

except (socket_error, smtplib.SMTPSenderRefused, smtplib.SMTPRecipientsRefused, smtplib.SMTPAuthenticationError), err:
                mark_as_deferred(message, err)
                deferred += 1

This worked great for SMTP sending. If a message failed for any of the expected reasons, that message was deferred for later (in case of a temporary problem) and the function kept chugging through the rest of the queue. With django-ses and boto though, you don’t get SMTP errors. You get things that look like:

400 Bad Request
<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
  <Error>
    <Type>Sender</Type>
    <Code>MessageRejected</Code>
    <Message>Address blacklisted.</Message>
  </Error>
  <RequestId>eb0e8eda-48c2-11e0-8b2e-91b9805ad73d</RequestId>
</ErrorResponse>

When this happens, the send_all function fails out, your message isn’t deferred and all emails after this one are effectively blocked. That’s bad.

Our fix, after experimenting with the bad idea of wrapping Boto exceptions in SMTP exceptions, was to simply use a bare except: statement for send_all(). Now SES errors don’t block our queue and we’re back to being happy. We’re using our django-ses compatible django-mailer fork in production right now.

Next step will be to find a way to automatically handle “Address blacklisted” messages, but that’s for another day. I’m just happy to stop our ghetto system of getting a pagerduty notification at 3am and manually rotating our SMTP user in production as we hit our quota. Thank you SES, boto, django-ses and django-mailer for taking back my sleep time.

Comments