How Email Works: A Deep Dive for Developers and IT Pros

Email is the backbone of modern Internet communication. As a developer or IT professional, you likely interact with email systems on a daily basis, whether it‘s configuring mail servers, parsing email data, or securing your organization‘s email infrastructure.

In this deep dive, we‘ll explore the technical intricacies of how email works under the hood, from the core protocols to the evolving email threat landscape. By the end, you‘ll have a solid foundation to build and secure email systems fit for the digital age.

The Anatomy of an Email System

Let‘s start with an overview of the key components that make up a typical email system:

• Mail User Agents (MUAs): Email clients like Outlook, Apple Mail, or Gmail that allow end-users to read, compose, and manage their email.

• Mail Transfer Agents (MTAs): Server-side programs like Sendmail, Postfix, or Exchange that are responsible for routing and delivering email between mail servers using the SMTP protocol. Examples include Microsoft Exchange, Sendmail, and Postfix.

• Mail Delivery Agents (MDAs): Server-side programs like Dovecot or Cyrus that receive email from an MTA and deliver it to the relevant user mailbox. MDAs also provide access to stored messages via protocols like IMAP or POP3.

• DNS (Domain Name System): The Internet‘s "phonebook" that maps human-readable domain names to IP addresses. DNS is crucial for email, as it‘s used to locate the mail servers responsible for handling email for a given domain.

Peeling Back the Layers: Core Email Protocols

Now that we‘ve seen the 10,000-foot view, let‘s zoom in on the key protocols that power email communication.

SMTP: The Workhorse of Email Transmission

The Simple Mail Transfer Protocol (SMTP) is the lingua franca of email. As defined in RFC 5321, SMTP specifies how MTAs communicate to send and relay messages.

Here‘s a simplified example of an SMTP conversation between a sender‘s MTA (client.com) and a recipient‘s MTA (server.com):

S: 220 server.com ESMTP 
C: HELO client.com
S: 250 Hello client.com
C: MAIL FROM:<[email protected]>
S: 250 OK
C: RCPT TO:<[email protected]>
S: 250 OK
C: DATA
S: 354 Send message content; end with <CRLF>.<CRLF>
C: From: "John Doe" <[email protected]>
C: To: "Jane Doe" <[email protected]>
C: Subject: Hello world
C: 
C: This is a test message.
C: .
S: 250 OK: queued as 12345
C: QUIT
S: 221 Bye

As you can see, SMTP consists of a series of text-based commands (e.g. MAIL FROM, RCPT TO) and responses (e.g. 250 OK). The actual message content is sent after the DATA command, terminated by a line with a single period (.).

By default, SMTP uses TCP port 25. However, servers usually also support secure SMTP over TLS on port 465 or 587 to encrypt email traffic.

The Battle of the Mailbox Protocols: POP3 vs IMAP

Once an email reaches the recipient‘s mail server, how does it get to their inbox? That‘s where mailbox protocols like POP3 and IMAP come in.

POP3 (Post Office Protocol version 3) is a simple protocol that allows MUAs to download new messages from a mailbox and (usually) delete them from the server. POP3 is defined in RFC 1939 and runs on port 110 (or 995 for POP3S over TLS).

In contrast, IMAP (Internet Message Access Protocol) provides a richer set of features for managing mailboxes remotely. With IMAP, messages stay on the server by default, and clients can synchronize message state across devices. IMAP is defined in RFC 3501 and uses port 143 (or 993 for IMAPS).

Here‘s a comparison of key differences between POP3 and IMAP:

Feature	POP3	IMAP
Connection model	Download-and-delete	Remote mailbox access
Folder support	No	Yes
Message state syncing	No	Yes
Search functionality	Limited	Extensive
Append new messages	No	Yes

While POP3 is still used for legacy reasons, IMAP is generally preferred for its flexibility and robustness.

The Email Threat Landscape: Data and Trends

So far we‘ve focused on the technical building blocks of email. But equally important is understanding the evolving email threat landscape.

Consider these eye-opening statistics:

• Email usage continues to grow rapidly. The number of global email users is projected to reach 4.6 billion by 2025, up from 4.0 billion in 2021 (Statista).

• Over 300 billion emails are sent and received each day (Statista).

• However, over 55% of all emails are estimated to be spam (Kaspersky Lab).

• Phishing remains the top attack vector, featuring in 36% of breaches (Verizon DBIR 2021).

• Business Email Compromise (BEC) attacks cost organizations an average of $5.01 million per incident (IBM Cost of a Data Breach Report 2021).

As these numbers illustrate, securing email is a major challenge for organizations of all sizes. Fortunately, a multi-layered approach can significantly reduce email risk.

Securing Email: A Multi-Layered Approach

Given the diversity of email-based threats, there‘s no silver bullet for email security. Instead, a combination of technical controls and user education is needed.

At a high level, email security encompasses:

1. Confidentiality: protecting email contents from unauthorized access
2. Integrity: ensuring email contents haven‘t been tampered with
3. Authenticity: verifying the identity of the sender
4. Availability: maintaining the accessibility of email systems

Let‘s examine some of the key protocols and standards for achieving these goals.

Encryption: Protecting Email Confidentiality with TLS

The first line of defense is email encryption. The primary standard for email encryption is Transport Layer Security (TLS), which can secure SMTP, POP3, and IMAP connections.

TLS provides encryption, authentication, and integrity protection for email traffic. When a TLS connection is established, the mail server presents an X.509 certificate to prove its identity. The client verifies this certificate before proceeding.

However, several challenges with email encryption remain:

• Opportunistic TLS: By default, most servers use "opportunistic TLS", meaning they still allow unencrypted connections. Enforcing strict TLS requires additional DNS records (MTA-STS) or server configurations.

• Difficulty of end-to-end encryption: While TLS protects email in transit, it doesn‘t provide end-to-end encryption. Once an email reaches the recipient‘s server, it‘s decrypted and stored in plaintext. True end-to-end encryption for email remains challenging, as it requires both the sender and recipient to use compatible encryption software like PGP.

Authentication: Proving Sender Identity with SPF, DKIM, DMARC

The next pillar of email security is authentication – ensuring emails are actually from who they claim to be from. The main email authentication standards are:

Sender Policy Framework (SPF): Allows domain owners to list which IP addresses are allowed to send email on their behalf. Recipient servers can check the SPF record in the sender‘s DNS to verify the email‘s source. Example SPF record:

example.com. IN TXT "v=spf1 ip4:192.0.2.0/24 ip6:2001:db8::/32 -all"

DomainKeys Identified Mail (DKIM): Allows senders to cryptographically sign specific headers in an email. Recipients can verify this signature using the sender‘s public key published in their DNS. Example of a DKIM signature header:

DKIM-Signature: v=1; a=rsa-sha256; d=example.com; s=default;
        c=relaxed/relaxed; q=dns/txt; l=1234; t=1117574938; x=1118006938;
        h=from:to:subject:date:keywords:keywords;
        bh=MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTI=;
        b=dzdVyOfAKCdLXdJOc9G2q8LoXSlEniSbav+yuU4zGeeruD00lszZ
                VoG4ZHRNiYzR

Domain-based Message Authentication, Reporting, and Conformance (DMARC): Allows domain owners to publish a policy specifying what to do with emails that fail SPF or DKIM checks. DMARC also enables reporting back to senders about failed messages. Example DMARC record:

_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=mailto:[email protected]"

Together, SPF, DKIM, and DMARC form a powerful framework for email authentication. However, their effectiveness depends on wide-scale adoption and proper implementation.

Emerging Standards and the Future of Email Security

Despite the progress made with protocols like SPF and DMARC, email security remains an uphill battle. Phishing and BEC attacks are still on the rise, and many organizations have yet to implement email authentication.

However, promising developments are on the horizon:

• ARC (Authenticated Received Chain): a proposed standard for preserving email authentication results across intermediary servers (like mailing lists). ARC aims to address DMARC compatibility issues that can cause legitimate messages to fail authentication.

• BIMI (Brand Indicators for Message Identification): a new standard that allows companies to display verified brand logos next to authenticated messages in the inbox. BIMI provides a visual incentive for senders to adopt DMARC.

• MTA-STS (Mail Transfer Agent Strict Transport Security): a mechanism enabling domains to declare they only send and receive secure SMTP connections, preventing downgrade attacks.

• Proposed RFCs for securing message content: Several draft RFCs propose methods for end-to-end encryption and integrity protection of message contents, such as S/MIME and SENDER. While promising, these have yet to see wide adoption.

Beyond technical standards, the fight for email security also hinges on user education and AI/ML-powered threat detection. As an IT professional, you can champion security best practices like:

• Enabling strict SPF, DKIM, and DMARC policies
• Enforcing TLS for all inbound and outbound email
• Adopting an email signing certificate (S/MIME or PGP)
• Training users to spot and report phishing
• Implementing BIMI for brand protection
• Leveraging AI/ML to detect and block advanced email threats

Conclusion: Securing Email, One Byte at a Time

Email is a complex, ever-evolving ecosystem. As a developer or IT pro, understanding the ins and outs of email protocols and security standards is key to building and maintaining secure email systems.

While email threats continue to proliferate, the future of email security is not without hope. By staying on top of emerging protocols, advocating for security best practices, and continuing to innovate, we can work towards a more trustworthy email landscape byte by byte.

Here are some helpful resources to learn more:

• How to Build DMARC Records
• Configure SPF and DKIM with Postfix
• The BIMI Group: Implementing BIMI
• M3AAWG: The Best Practices for Email Security

As the famous programmer Grace Hopper once said, "The most damaging phrase in the language is: ‘It‘s always been done that way.‘" Let‘s challenge the status quo and build a more secure email ecosystem, together.