An Event-Driven Email List Architecture Ideal for Rapid Time-to-Market and Feature Growth

StockZoomAi’s most important user interface is email

StockZoomAi’s stock reports, stock movement alerts, and other stock tips and updates are delivered via email.

Even when a feature is not entirely delivered via email, email is still often the launchpad.

We need a system design for our emails that is quick to rollout and quick to iterate on.

Below is an explicit set of requirements

Requirement #1: Users must be able to easily join the lists they care about and leave the lists they don’t

We don’t want customers to get overwhelmed with irrelevant emails.

Therefore, we need a simple architecture for building LOTS of email lists, but also making it very easy for customers to join or leave those lists.

Keep in mind that these email lists are NOT marketing lists

These emails are ONLY going to customers.

Therefore, we’re not worried about spam filters, etc.

For one thing, StockZoomAI has its own email domain, so most emails will be internal to that domain.

But also, even for email addresses to other domains, we expect customers to add our email address to their safe senders list.

Requirement #2: We want rapid time-to-market for new features

Time-to-market is everything. So we need an architecture that makes it easy to add new email lists.

Requirement #3: We want minimal maintenance overhead and operational cost

At such an early stage, we don’t want to be slowed managing hardware and platform resources.

We also don’t want high monthly costs at low volumes. (e.g. We want to avoid fixed monthly costs.)

Requirement #4: We want easy development across the languages our microservices are written in

Our microservices are built in Python, Node.js, and C#; therefore, we want our architecture to use SDKs that are easy to work with from a variety of languages.

Given the above requirements, we need an email list architecture that provides the following

1. The ability to rapidly roll-out new lists for new microservices
2. Minimal code required to trigger and send emails to a specific list
3. Reliability in the method for triggering and sending emails from our microservices
4. An easy to understand architecture for debugging and complexity growth
5. Good ergonomics across different languages
6. Low cost to manage and operate

The design

#1: Our microservices raise events by writing to a message queue

Putting email logic in our microservices would needlessly complicate them

Suddenly:

1. Each microservice would have to know what email service we’re using. What happens if the email service changes over time?
2. Each microservice would have to know what email lists their computations apply to, and send the appropriate emails for every list, etc.

By decoupling raising the event and sending of the email, we drastically reduce churn in our microservice code.

Now we can refactor and re-design email as many times as needed with hopefully minimal impact on our microservices.

For our message queue, we use AWS SQS

The cloud can get expensive, but in this case, our scale is small, so we save money and time using a fully managed message queue, rather than trying to host a 3rd party message queue solution ourselves.

This satisfies our goals of low fixed monthly costs to operate.

Using a message queue also helps us satisfy some of our reliability goals

SQS will hold an event until the event is processed and the consumer indicates that the event has been handled.

This also gives us flexibility as the system grows.

Spinning up and talking to a new queue has low dev and operations overhead.

Decoupling email and writing to a message queue keeps each microservice simple

Writing a message to a message queue is a simple operation that helps satisfy our desire for minimal code required and our desire for good ergonomics for the developer.

For example, below are the six lines of Python added to the DCF_ENGINE microservice to raise an event that will later trigger the sending of an email.

sqs_queue_url = ...
msg_body = ...
...
sqs_client = boto3.client('sqs')
  queue_response = sqs_client.send_message(QueueUrl=sqs_queue_url, MessageBody=msg_body)
  print(f"SQS Queue Response: {queue_response}")
  print(f"SQS Message ID: {queue_response['MessageId']}")

#2: Email send is handled by a lambda triggered by the message queue write

Many AWS services can fire off a lambda automatically when various events occur.

This is perfect for the case of sending email(s) when an event is written to SQS.

Each SQS event queue has its own simple Lambda that fires to send the appropriate email(s) for that specific queue.

#3: A NoSQL database stores our email lists

An email list is just an array of contacts with metadata

In its simplest form, an email list is really just a collection of objects, where each object must get us to an email address along with other relevant data.

This leaves us with a few options to store email lists.

Options rejected for storing email lists

We need options that are very low overhead to roll-out and maintain.

1. An RDBMS like SQL Server, PostgreSQL, etc. (Way too complicated for this use case. Too much overhead. — See notes about RDBMSs below)
2. Local file storage (Not durable. Much of our system is serverless to keep operational cost low. Requiring local file storage would drastically reduce our architecture flexibility.)
3. Object storage like S3. Makes list edits tedious and brittle. Are these file operations? How do we solve concurrency issues? Introduces overhead managing URLs. (e.g. where should each component look to find this data?)
4. Self-managed NoSQL database. For such a small amount of data, hosting ourselves is too much operational overhead.

Fully-managed NoSQL database; our solution for storing email lists

We use one collection per email list. Each record in the collection contains the email address and all data relevant to the email list.

So, we can get the subscribers to dcf_engine_results, with a simple query like:

subscribers = nosql_client.collection('dcf_engine_results').all()

This design has the following benefits:

1. At our scale, read/write costs should be close to zero.
2. Every client talks to a single known nosql endpoint (configuration overhead is low)
3. Adding a new email list is as simple as adding a new collection
4. Using the nosql client, a new email list can be consumed from any of our systems with two or three lines of code and no new configuration

Yes, duplicating email addresses makes email address changes more difficult

For now, this is a new product. Therefore, for the foreseeable future, this should not be a major problem.

If the system grows to where this is an issue we have very good problems.

Migrating to a slight better solution for email address changes is.

1. Add a Customer collection.
2. Swap out the email addresses in each mailing list with the CustomerId of the Customer in the Customers collection.

But, to speed time-to-market, we can ignore this concern for now.

Why an RDBMS is overkill for our use case

We could use an RDBMS to model an email list as a pointer to an email address that has metadata that is global to all of our system.

Maybe each mailing list could be an association table, etc. Or a single mailing lists table and a single mailing_list_to_customer association table.

But, this is already showing the operational downsides of using an RDBMS.

There is simply too much schema work for such a simple problem. We’re already adding multiple tables and foreign keys…

And the normalization of the schema bleeds into complicating querying data from the database.

By far our most common coding task will be to get data from our database. And complicating querying complicates every code base that needs to query data.

The schema design overhead of using an RDBMS is overkill for our scale

De-normalization of our data is perfectly fine at this scale.

An RDBMS provides far more functionality than we are likely to need. And using one comes with the extra overhead of managing a database, defining tables, defining schemas, and on and on.

Whereas a fully managed NoSQL system like AWS DynamoDb or Google Firestore is simple. They are also very cheap at this scale.

(Note: Our system uses resources from both AWS and GCP, so both DynamoDB and Firestore could work here.)

#5: Each Lambda loads the appropriate email list from a managed NoSQL store

When an event hits SQS, SQS launches the Lambda associated with the target queue.

The Lambda then loads all records in the appropriate mailing list NoSQL collection.

For example, see the below pseudo code:

recipients = nosql_client.collection('this-mailing-list').all()

Now, the lambda can iterate through recipients and send the appropriate email.

#6: Sending the email itself can use a number of different email providers

This part isn’t important. We can use a number of different email providers.

In our case, because we use Google to manage our email domain and client inboxes, we will send email via Google’s SMTP servers.

But, this could change in the future to a number of any providers.

Alternatives rejected: A dedicated email microservice

We could have built a dedicated email microservice that reads from AWS SQS message queues and has logic for sending emails to mailing lists.

This approach has drawbacks.

Trying to handle every mailing list through a single service is going to make that service increasingly complex.

For example:

1. Where do we save email templates for each email list?
2. Every time we add a new SQS, we need to add branching to support this new queue. Or, we have to generalize the service to handle an arbitrary number of queues.
3. How do we ensure high uptime of the service? It requires dedicated resources. Then it requires redundancy.
4. And on and on…

Hopefully, the above is sufficient to see that this email service would quickly become difficult to reason about, a major single point of failure, and add lots of code to the system.

Overall, a dedicated email service would become a major distraction.

Splitting a single email microservice into multiple email microservices is even worse

We have the same problems with uptime, redundancy and managing the resources to provide both.

But, now we’re just multiplying these problems.

For our use case, such an approach doesn’t make sense.

An image of the final architecture

StockZoomAI Email List Architecture, A Low-Cost Event Driven Architecture That is Ideal for Rapid Time-to-Market, Feature Growth, and List Expansion!

For this architecture, we bring together just the right set of tools to provide: flexibility, simplicity, reliability, and low cost of operation!

1. AWS SQS to raise events
2. AWS Lambdas to respond to raised AWS SQS events, and to handle sending the emails
3. Fully managed noSql database for storing mailing lists (e.g. AWS DynamoDB or Google Firestore)

These choices SPEED time-to-market, while keeping the system scalable, low cost at our scale, and make it very easy to iterate and grow!

If you enjoyed this article or this article helped you, please hit the Clap buttons a few times to help me write more articles like this! Also, hit follow to get future articles like this one.