A Postfix deployment postmortem: Debugging short-circuiting of mapping lookups

A Postfix deployment postmortem:Debugging short-circuiting of mapping lookups - HedgeDoc

<center> # A Postfix deployment postmortem: Debugging short-circuiting of mapping lookups *Originally published 2020-02-28 on [docs.sweeting.me](https://docs.sweeting.me/s/blog).* *Also published as an [answer on ServerFault](https://serverfault.com/questions/948362/postfix-multiple-smtpd-sender-login-maps/1005041#1005041), and in the [Mailu Issue Tracker #1096](https://github.com/Mailu/Mailu/issues/1096).* <img src="https://docs.monadical.com/uploads/upload_6e59cbf30deeba744cdce2c6024b6a95.png" style="width: 35%"/> <img src="https://docs.monadical.com/uploads/upload_9ca60c0fc523e381ddd262d3625ab373.png" alt="Diagram of SMTP over the internet" style="width: 60%"/> How we tracked down an email sending bug caused by Postfix's counterintuitive `type:table` lookup behavior, and how we fixed it with Postfix 3.0's new `unionmap` feature. --- </center> <style> h1, h2, h3, h4, h5 { margin-top: 4px !important; } </style> [TOC] --- ## How it all went wrong > One week ago we had a mailserver that was running perfectly fine... > or so we thought.... Prompted by [error reports from other people](https://github.com/Mailu/Mailu/issues/1096#issuecomment-583147927) using our config with [Mailu](https://mailu.io/) (a Docker mailserver), we went down a rabbit hole of investigation before discovering a critical issue with our setup. ![](https://docs.monadical.com/uploads/upload_c5c0ec36e6122041770a252729a36c3a.png) Several months ago we had deployed a Mailu-based Postfix mailserver using the `smtpd_sender_login_maps` option to control which users could send email from which addresses. Our intent was to allow a single admin user to send email from any address, but restrict normal users to their own address and their aliases. It was working fine for our purposes: relaying mail for other servers in our cluster using the admin user account, and handling inbox duties for a few (mostly inactive) human users. > Our mail traffic: 👾99% bots, 👩‍💻1% humans => leads to very little human feedback Our config allowing the admin user to masquerade as anyone was working fine, but unbeknownst to us, our normal non-admin users were actually **completely unable to send** email! Any non-admin attempts to send messages were met with this perplexing error: :::danger `postfix/smtpd[16645]: NOQUEUE: reject: RCPT from mailu_webmail_1.mailu_default[192.168.203.11]: 553 5.7.1 <test@zalad.io>: Sender address rejected: not owned by user test@zalad.io; from=<test@zalad.io> to=<redacted@example.com> proto=ESMTP helo=<mail.zervice.io> ` ::: The root cause ended up being a surprisingly simple Postfix config assumption that led to unexpected consequenses. The rest of this post dives into the counter-intuitive way that Postfix determines lookup table result precedence, and how you can use `unionmap` / SQL to control it. > Tip: EveryCloud's free [Mail Flow Monitor](https://www.everycloud.com/free-mail-flow-monitor) is a great little monitoring/alerting solution, it helped ensure this issue didn't affect *any* of our production mailserver users. <center><img src="https://docs.monadical.com/uploads/upload_f4316c22add4e63a7dbd5f7921ba995d.png"> This is how SMTP is used to send an email from a client to its recipient.</center> ## Background Imagine a simplified setup where we have Postfix and MySQL running on a mailserver. There are 3 users stored in the MySQL DB who can authenticate via SMTP with Postfix: - `alice@example.com`: can only send from `alice@example.com`,`alice.smith@example.com` - `bob@example.com`: can only send from `bob@example.com`,`bob.jones@example.com` - `admin@example.com`: should be able to send from *any* `.*@example.com` address Regular users can only send from their main address and their aliases, but we want to allow the admin to be able to masquerade as *any* sender without restrictions. <center><img src="https://docs.monadical.com/uploads/upload_630aba0208fb9289013db5627cdd8ed5.png"> The admin could add any any sending address they wanted in their mail client.</center> > In real life this type of setup is common whenever a user needs to be able to send from many addresses `.*@example.com` using only one SMTP login. > > e.g. if the addresses are dynamically-generated or belong to other users The normal approach to achieve fine-grained control over sender address validation is to point [`smtpd_sender_login_maps`](http://www.postfix.org/postconf.5.html#smtpd_sender_login_maps) to a SQL query or PCRE file that defines FROM addr -> SMTP users. ## Using `smtpd_sender_login_maps` The purpose of using `smtpd_sender_login_maps` is to validate whether the currently authenticated SMTP user is allowed to send an outbound email based on its FROM address. It's a powerful feature to restrict the addresses a client impersonate based on their SMTP auth. ### The Config Spec :::warning > [**`smtpd_sender_login_maps`** (default: empty)](http://www.postfix.org/postconf.5.html#smtpd_sender_login_maps) > > Optional lookup table with the SASL login names that own the sender > (MAIL FROM) addresses. > Specify zero or more [`type:name`](http://www.postfix.org/DATABASE_README.html) lookup tables, separated by whitespace or comma. *Tables will be searched in the specified order until a match is found.* http://www.postfix.org/postconf.5.html#smtpd_sender_login_maps ::: ### Use Cases As you can imagine, being able to granularly filter allowed senders has many use cases: - **Prevent a user from logging in and sending a message as someone else** *> Bob can send as `bob@example.com`, but can't send as `alice@example.com`* - **Allow multiple users to send from a single shared address neither of them own** *> Bob and Alice can both send as `marketing@example.com`* - **Allow a single user to send from multiple addresses they own** *> Alice can send as both `alice@example.com` and `alice.smith@example.com`* - **Allow an admin to send messages from many addresses (even owned by others)** *> Tracy, the IT admin can send as anyone `.*@example.com`* - **And more...** (remember, lookups can be via TCP socket, SQL query, regex, etc.) --- ## Example Scenario We'll investigate this use-case of `smtpd_sender_login_maps` in particular: > **Allow an admin to send messages from many addresses** (even owned by others) We start by defining `smtpd_sender_login_maps` in `main.cf` to lookup the message FROM address in two lookup tables to determine whether the currently authenticated SMTP user is allowed to send from that address. <center> <img src="https://docs.monadical.com/uploads/upload_1cc8424a8b49e4f2da28424a63bd1200.png" style="padding-right: 60px; width: 350px"/> </center> Each lookup table used by `smtpd_sender_login_maps` is a mapping of: > `msg FROM addr -> SMTP users allowed to send as that address` For `alice@example.com` to be able to log in and send from `alice.smith@example.com`, one of the lookups must return `alice@example.com` in the allowed SMTP user list: > `alice.smith@example.com -> alice@example.com,admin@example.com` ### Main Postfix Config :::warning #### `/etc/postfix/main.cf` [`type:postconf`](http://www.postfix.org/postconf.5.html): ```ini ... smtpd_sender_login_maps = mysql:/etc/postfix/sender_logins.cf, pcre:/etc/postfix/sender_overrides.cf ``` MySQL is checked first, then pcre is checked only if the mysql lookup returned 0 results. ::: Then, we create the two mappings referenced above, one using SQL, the other using regex. ### Sender Login Mapping Definitions :::warning #### `/etc/postfix/sender_logins.cf` [`type:mysql_table`](http://www.postfix.org/mysql_table.5.html): ```ini hosts = 127.0.0.1 user = postfix password = yourDatabasePasswordHere dbname = mail query = SELECT email FROM users WHERE email='%s' ``` This query checks the DB and returns the allowed SMTP users for a given FROM addr. > `alice@example.com -----> alice@example.com` > `bob.jones@example.com -> bob@example.com` ::: :::warning #### `/etc/postfix/sender_overrides.cf` [`type:pcre_table`](http://www.postfix.org/pcre_table.5.html): ```ini /.*@example.com/ admin@example.com ``` This regex matches *all* `@example.com` FROM addrs and returns the admin SMTP user. > `.*@example.com ---------> admin@example.com` ::: ## Testing it out Great, so far we've configured our server to check two lookup tables to determine whether an outbound message is allowed to be sent by the logged in SMTP user. **Now let's take a look at two scenarios and see exactly what Postfix does in each case.** :::success #### Scenario 1: `alice@example.com` tries to send from `alice@example.com` 1. `alice@example.com` logs in via SMTP to send a msg from `alice@example.com` 2. `alice@example.com` lookup in `smtpd_sender_login_maps` returns `alice@example.com` 3. Sending succeeds, `alice@example.com` == SMTP authed user `alice@example.com` ✅ This works fine, the email is sent because the address matches the sender as expected. ::: :::danger #### Scenario 2: `admin@example.com` tries to send from `alice@example.com` 1. `admin@example.com` logs in via SMTP to send a msg from `alice@example.com` 2. `alice@example.com` lookup in `smtpd_sender_login_maps` returns `alice@example.com` 3. Sending fails, `alice@example.com` != SMTP authed user `admin@example.com` ``` postfix/smtpd[16645]: NOQUEUE: reject: RCPT from webmail.mailserver[192.168.1.5]: 553 5.7.1 <alice@example.com>: Sender address rejected: not owned by user admin@example.com; from=<alice@example.com> to=<bob@example.com> proto=ESMTP helo=<mail.example.com> ``` ❌ This does not work, the email is rejected because the SMTP authed user `admin@example.com` doesn't match the first lookup result `alice@example.com`. ::: :::danger #### Scenario 3: What if we switch the order of the lookup tables so `pcre` comes first? **`/etc/postfix/main.cf`:** ```ini smtpd_sender_login_maps = pcre:..., # does moving the pcre mapping above mysql work? mysql:... ``` **Flipping the order of the mappings so that the pcre file is checked before mysql won't fix the problem.** It even makes it worse because the catchall `.*@example.com` will overshadow all the real users in MySQL and prevent any user other than `admin@example.com` from sending email. 1. `alice@example.com` logs in via SMTP to send a msg from `alice@example.com` 2. `alice@example.com` lookup in `smtpd_sender_login_maps` returns `admin@example.com` 3. Sending fails, `admin@example.com` != SMTP authed user `alice@example.com` When it looks up `alice@example.com` in the pcre file, it returns `admin@example.com` as the only allowed user and fails before it ever checks the mysql database. ::: --- <center> Hmm... <img src="https://docs.monadical.com/uploads/upload_97148b3c67f496c293f79eb648dd0d0e.gif" style="width: 70%; border-radius: 8px; box-shadow: 4px 4px 4px rgba(0,0,0,0.02)"/> Clearly there's something preventing us from using both mappings simultaneously... </center> --- ## The problem ### The behavior the user expects 1. Postfix looks up the FROM addr in the first `sender_login_maps` db 2. it finds a matching entry for the FROM addr 3. the returned SMTP user != logged in user, **so we try the next mapping db** 4. it finds a matching entry for the FROM addr & the SMTP user == logged in user > ✅ Postfix sends the message succesfully Because the second mapping matched the logged in user. ### What actually happens 1. Postfix looks up the FROM addr in the first `sender_login_maps` db 2. it finds a matching entry for the FROM addr, **the lookup process stops** 3. the returned SMTP user != logged in user > ❌ Postfix rejects the message. Lookup stoped after the first mapping matched, but matched SMTP user != the logged in user. ### Explanation The issue is that Postfix doesn't check both mappings and merge the results, instead it stops the lookup process the moment it encounters the *first* matching lookup returning any SMTP user. If the returned SMTP user doesn't match the currently authed user, it won't proceed to lookup the address in the next database, it'll just immediately `DENY`. The same lookup conflict can happen with any two mappings that share keys, e.g. two mysql databases `mysql:...,mysql:...`, not necessarily just a `pcre` with `.*` in it. Any exact match or wildcard match like `@example.com someuser@example.com` in the first mapping will take precedence and prevent the second mapping from being queried at all. --- ## The solutions ### Solution A. Make the `smtpd_sender_login_maps` disjoint <img src="https://media.giphy.com/media/SEv6jiT1OhPRC/giphy.gif" style="float: right; width: 38%; margin-left: 10px;"/> If the mappings don't contain any overlap in keys, then the order doesn't matter, and any lookup that doesn't match the first db will proceed to check the subsequent ones as expected. Mappings earlier in the list cannot have any "catchall" or wildcard keys or they will match everything and overshadow results from later mappings. ### Solution B. Join multiple lookup results in SQL with `UNION` If you're using the `mysql` table type for your `smtpd_sender_login_maps` mappings, then you can control the SQL query run when doing an address lookup and you may be able to join multiple mappings at the SQL level. Assuming all your mappings are accessible in the same MySQL database, you can concatenante the results of multiple address lookups at the SQL level using a `UNION` statement like so: `main.cf`: smtpd_sender_login_maps = mysql:/etc/postfix/sender_logins.cf `sender_logins.cf`: hosts = 127.0.0.1 user = postfix password = yourDatabasePasswordHere dbname = mail query = SELECT email FROM users WHERE email='%s' UNION SELECT destination FROM aliases WHERE source='%s' UNION SELECT email FROM users WHERE wildcard_sending=1 In this example we would set `admin@example.com` to have `wildcard_sending=1` in SQL, and then it would be returned with every lookup result along with the normal user and alias matches, e.g. alice.smith@example.com -> alice@example.com,admin@example.com bob@example.com -> bob@example.com,admin@example.com ### Solution C. Use `unionmap` to combine multiple mappings If you're using Postfix 3.0 or above, you might be able to try using the new `unionmap` feature, which performs a lookup to all the mappings at once and concatenates the results together. smtpd_sender_login_maps = unionmap:{ mysql:/etc/postfix/sender_logins.cf, pcre:/etc/postfix/sender_overrides.cf } With this setup, the mysql results will be concatenated with the pcre lookup results, e.g. alice.smith@example.com -> alice@example.com,admin@example.com bob@example.com -> bob@example.com,admin@example.com See http://www.postfix.org/DATABASE_README.html#types **unionmap** for more info. <center> <img src="https://media.giphy.com/media/YmjleYhDTUiYw/giphy.gif~c200" style="width: 200px"/> </center> --- :::info <center> This post talked about a specific Postfix config option `smtpd_sender_login_maps`, but **the behavior described above applies to any Postfix config option that accepts multiple [`type:table`](http://www.postfix.org/DATABASE_README.html) parameters** and looks them up in order `"until a match is found"`, e.g.: </center> - `alias_maps`, `canonical_maps`, `transport_maps`, `local_recipient_maps` - `virtual_alias_maps`, `virtual_uid_maps`, `virtual_mailbox_maps` - `authorized_flush_users`, `authorized_mailq_users`, `authorized_submit_users` - ... and many more in http://www.postfix.org/postconf.5.html <center> They all have something like this in the config parameter documentation: </center> > Specify zero or more "type:table" lookup table names, separated by comma or whitespace. The tables are queried in the specified order until a match is found. The first table match wins. <center> **So watch out for `"until a match is found"` when setting up your own config!** </center> ::: --- ## Further Reading - https://www.mail-archive.com/postfix-devel@postfix.org/msg00677.html (I highly recommend reading this entire mailing list thread for full context) - http://www.postfix.org/postconf.5.html#smtpd_sender_login_maps - http://www.postfix.org/SMTPD_ACCESS_README.html#lists - http://www.postfix.org/DATABASE_README.html#types - http://www.postfix.org/access.5.html - https://unix.stackexchange.com/questions/294300/postfix-prevent-users-from-changing-the-real-e-mail-address - https://groups.google.com/forum/#!topic/mailing.postfix.users/X7Wj_nSEyKI - https://workaround.org/ispmail/jessie/relaying-smtp-authentication - https://serverfault.com/questions/948362/postfix-multiple-smtpd-sender-login-maps/1005041#1005041 - https://github.com/Mailu/Mailu/issues/1096 - https://www.everycloud.com/free-mail-flow-monitor <img src="https://docs.monadical.com/uploads/upload_d05b3d09f8e9d75f457bbcd80451665e.png" style="width: 100%"/> --- <center> <a href="https://monadical.com"><img src="https://monadical.com/static/logo-blue.png" style="height: 60px"><h4>Monadical Inc. | Full-Stack Software Consultancy</h4></a> Need help with your mail server? <a href="https://monadical.com/index.html#consulting">Hire us!</a> </center>

Nick Sweeting

Monadical Co-Founder @theSquashSH

Published on 2020-02-28