Hosting an e-mail service definitely feels like something that can be done without significant effort. (It is so common!) Unfortunately, there are many caveats if we want to host a reasonable e-mail service that can send and receive e-mails from popular e-mail service providers like Gmail. I spent a good-old three days of trying to Google up everything and now hopefully you don't have to do it again :)
E-mail architecture is very complicated and consists of many different protocols and jargons. This section covers the basics of everything so we know what we are building.
At the beginning of the world, there is a protocol called SMTP (Simple Mail Transfer Protocol). SMTP allows us to input a message and deliver it to the other side. Postfix is an open-source implementation of such protocol:
Everything is good. However, it very quickly turns out that no one wants to deal with a bunch of unsorted e-mail files. Therefore, people developed the POP3/IMAP protocol to allow a user to sort all the e-mails into Unread/Read, Inbox/Junk, etc. POP3 is the older/outdated protocol and IMAP is the newer/fancier one. Dovecot is an open-source implementation of both:
The system turned out to be very successful, and everyone is sending e-mails. Actually it became too successful, and people are starting to receive a lot of spam e-mails! To remedy this issue, spam filters are introduced to detect spam emails; signatures (DKIM) and a lot of DNS-related measures (PTR, SPF, DMARC) are adopted to defend against impersonation. There is a hot debate on which spam filter is the best, but we are not going to indulge in it and just use SpamAssassin, an open-source spam detector, coupled with Dovecot Sieve to filter the mails according to the detection results:
OK, that roughly sums up the thing we are building! It is indeed very complicated, but we will go through it step by step.
Prerequisite: PTR Record and Port Availablity
To successfully set up a real e-mail server that can communicate with Gmail we will need two things beyond simply setting up the VPS: one is a PTR record, and the other is that our VPS provider cannot block us from sending e-mails.
PTR Record
A PTR record is like a reverse-DNS: you tell DNS servers what domain name you are using for your IP address, and when people query for your IP address, they can find your domain name. This mechanism is used to prove your identity and make sure that outgoing mails are not sent from frivolous domain names. Unfortunately for us, PTR records can only be set by our VPS provider since they have control of the IP address, so we must contact our VPS provider or do a search to find out.
Port Availablity
Most VPS providers limit (or ban) the connection to other servers' port 25 to limit their servers' potential as an e-mail spammer. Unfortunately we need the port to properly send e-mails.
One way to test if our server has access to port 25 is to do a telnet test:
$ telnet smtp.gmail.com 25
Trying 173.194.209.109...
Connected to smtp.gmail.com.
Escape character is '^]'.
220 smtp.gmail.com ESMTP m17-20020ae9e711000000b006aedb35d8a1sm9336266qka.74 - gsmtp
We also need to set up what service Postfix should provide (submission, SMTPS and give received mail to Dovecot). This is done via editing /etc/postfix/master.cf. It should look something like this:
Click here:
## Postfix master process configuration file. For details on the format# of the file, see the master(5) manual page (command: "man 5 master" or# on-line: http://www.postfix.org/master.5.html).## Do not forget to execute "postfix reload" after editing this file.## ==========================================================================# service type private unpriv chroot wakeup maxproc command + args# (yes) (yes) (no) (never) (100)# ==========================================================================
smtp inet n - y - - smtpd
#smtp inet n - y - 1 postscreen#smtpd pass - - y - - smtpd#dnsblog unix - - y - 0 dnsblog#tlsproxy unix - - y - 0 tlsproxy# Choose one: enable submission for loopback clients only, or for any client.#127.0.0.1:submission inet n - y - - smtpd
submission inet n - y - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_tls_auth_only=yes
-o smtpd_reject_unlisted_recipient=no
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
# Choose one: enable smtps for loopback clients only, or for any client.#127.0.0.1:smtps inet n - y - - smtpd
smtps inet n - y - - smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_reject_unlisted_recipient=no
-o smtpd_recipient_restrictions=
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING
#628 inet n - y - - qmqpd
pickup unix n - y601 pickup
cleanup unix n - y - 0 cleanup
qmgr unix n - n 3001 qmgr
#qmgr unix n - n 300 1 oqmgr
tlsmgr unix - - y1000? 1 tlsmgr
rewrite unix - - y - - trivial-rewrite
bounce unix - - y - 0 bounce
defer unix - - y - 0 bounce
trace unix - - y - 0 bounce
verify unix - - y - 1 verify
flush unix n - y1000? 0 flush
proxymap unix - - n - - proxymap
proxywrite unix - - n - 1 proxymap
smtp unix - - y - - smtp
relay unix - - y - - smtp
-o syslog_name=postfix/$service_name
# -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq unix n - y - - showq
error unix - - y - - error
retry unix - - y - - error
discard unix - - y - - discard
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - y - - lmtp
anvil unix - - y - 1 anvil
scache unix - - y - 1 scache
postlog unix-dgram n - n - 1 postlogd
## ====================================================================# Interfaces to non-Postfix software. Be sure to examine the manual# pages of the non-Postfix software to find out what options it wants.## Many of the following services use the Postfix pipe(8) delivery# agent. See the pipe(8) man page for information about ${recipient}# and other message envelope options.# ====================================================================## maildrop. See the Postfix MAILDROP_README file for details.# Also specify in main.cf: maildrop_destination_recipient_limit=1#
maildrop unix - n n - - pipe
flags=DRXhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
## ====================================================================## Recent Cyrus versions can use the existing "lmtp" master.cf entry.## Specify in cyrus.conf:# lmtp cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4## Specify in main.cf one or more of the following:# mailbox_transport = lmtp:inet:localhost# virtual_transport = lmtp:inet:localhost## ====================================================================## Cyrus 2.1.5 (Amos Gouaux)# Also specify in main.cf: cyrus_destination_recipient_limit=1##cyrus unix - n n - - pipe# flags=DRX user=cyrus argv=/cyrus/bin/deliver -e -r ${sender} -m ${extension} ${user}## ====================================================================# Old example of delivery via Cyrus.##old-cyrus unix - n n - - pipe# flags=R user=cyrus argv=/cyrus/bin/deliver -e -m ${extension} ${user}## ====================================================================## See the Postfix UUCP_README file for configuration details.#
uucp unix - n n - - pipe
flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
## Other external delivery methods.#
ifmail unix - n n - - pipe
flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp unix - n n - - pipe
flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix - n n - 2pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FRX user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py ${nexthop} ${user}
dovecot unix - n n - - pipe
flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver
-f ${sender} -d ${recipient}
To conclude, we uncommented the part regarding submission and SMTPS, and added Dovecot to the end of the file.
The main configuration directory of Dovecot is /etc/dovecot/conf.d. We are going to create a file 99-main.conf in the folder and put in our own configuration:
First of all, we need to decide what e-mail address real Linux users on the VPS should use. This is done in /etc/aliases:
# See man 5 aliases for format
postmaster: webmaster@zhtluo.com
root: webmaster@zhtluo.com
www-data: webmaster@zhtluo.com
We should then update the aliases database with:
newaliases
Then, we should set up virtual users who are not Linux users but can send and receive e-mails. Namely we need to create these four files as outlined in main.cf:
The fourth file is /etc/postfix/canonical-maps, which allows us to map addresses so that legacy systems with a fixed e-mail address can still function. I leave it empty.
So we should create this file. Generate a password with:
# doveadm pw -s SSHA512
Enter new password:
Retype new password:
{SSHA512}EPqCQ0Gz0+BBRtCL643gIt4G+mlyxGwma6VFziDBAZrJDCN4fSN25vH9cM3kUKRqMDvPP/3yvLSjIDPRo8+C1yV5Jos=
There are a lot DNS records to set up. We will need to provide a DKIM key for the record and sign our outgoing message with the key. We will use OpenDKIM for the purpose (substitute the last part with your domain name):
apt install opendkim opendkim-tools
mkdir /etc/opendkim
chown opendkim:opendkim /etc/opendkim
cd /etc/opendkim
opendkim-genkey -r -h sha256 -d zhtluo.com -s mail
Create a file named /etc/opendkim/KeyTable to set up the keys:
Create a file named /etc/opendkim/SigningTable to decide what keys to use for what domain:
*@zhtluo.com zhtluo.com
Create a file named /etc/opendkim/TrustedHosts to allow localhost to sign:
127.0.0.1
Give every file to OpenDKIM so that it works:
chown -R opendkim:opendkim /etc/opendkim
Finally, edit the configuration file at /etc/opendkim.conf:
Click here:
# This is a basic configuration for signing and verifying. It can easily be# adapted to suit a basic installation. See opendkim.conf(5) and# /usr/share/doc/opendkim/examples/opendkim.conf.sample for complete# documentation of available configuration parameters.
Syslog yes
SyslogSuccess yes#LogWhy no# Common signing and verification parameters. In Debian, the "From" header is# oversigned, because it is often the identity key used by reputation systems# and thus somewhat security sensitive.
Canonicalization relaxed/simple
#Mode sv#SubDomains no
OversignHeaders From
# Signing domain, selector, and key (required). For example, perform signing# for domain "example.com" with selector "2020" (2020._domainkey.example.com),# using the private key stored in /etc/dkimkeys/example.private. More granular# setup options can be found in /usr/share/doc/opendkim/README.opendkim.#Domain example.com#Selector 2020#KeyFile /etc/dkimkeys/example.private# In Debian, opendkim runs as user "opendkim". A umask of 007 is required when# using a local socket with MTAs that access the socket as a non-privileged# user (for example, Postfix). You may need to add user "postfix" to group# "opendkim" in that case.
UserID opendkim
UMask 007
# Socket for the MTA connection (required). If the MTA is inside a chroot jail,# it must be ensured that the socket is accessible. In Debian, Postfix runs in# a chroot in /var/spool/postfix, therefore a Unix socket would have to be# configured as shown on the last line below.
Socket local:/run/opendkim/opendkim.sock
#Socket inet:8891@localhost#Socket inet:8891#Socket local:/var/spool/postfix/opendkim/opendkim.sock
PidFile /run/opendkim/opendkim.pid
# Hosts for which to sign rather than verify, default is 127.0.0.1. See the# OPERATION section of opendkim(8) for more information.#InternalHosts 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12# The trust anchor enables DNSSEC. In Debian, the trust anchor file is provided# by the package dns-root-data.
TrustAnchorFile /usr/share/dns/root.key
#Nameservers 127.0.0.1# Our configuration
Canonicalization relaxed/relaxed
ExternalIgnoreList refile:/etc/opendkim/TrustedHosts
InternalHosts refile:/etc/opendkim/TrustedHosts
KeyTable refile:/etc/opendkim/KeyTable
SigningTable refile:/etc/opendkim/SigningTable
LogWhy Yes
PidFile /var/run/opendkim/opendkim.pid
Socket local:/var/spool/postfix/opendkim/opendkim.sock
SyslogSuccess Yes
TemporaryDirectory /var/tmp
UserID opendkim:opendkim
The first line tells the DNS server what mail server we are using; the second line is the SPF record and the third line is the DMARC record. The fourth line is our DKIM public key: we need to copy the content in /etc/opendkim/mail.txt to the record. All these records are needed for external servers to recognize us as a legit mail server.
Edit the configuration file in /etc/default/spamassassin, mostly let it use the user we just created:
Click here:
# /etc/default/spamassassin# Duncan Findlay# WARNING: please read README.spamd before using.# There may be security risks.# Prior to version 3.4.2-1, spamd could be enabled by setting# ENABLED=1 in this file. This is no longer supported. Instead, please# use the update-rc.d command, invoked for example as "update-rc.d# spamassassin enable", to enable the spamd service.# Options# See man spamd for possible options. The -d option is automatically added.# SpamAssassin uses a preforking model, so be careful! You need to# make sure --max-children is not set to anything higher than 5,# unless you know what you're doing.OPTIONS="-x --max-children 5 --helper-home-dir=/var/lib/spamassassin -u debian-spamd -g debian-spamd --siteconfigpath=/etc/spamassassin --socketpath=/var/spool/postfix/spamassassin/spamd.sock --socketowner=debian-spamd --socketgroup=debian-spamd --socketmode=0660"# Pid file# Where should spamd write its PID to file? If you use the -u or# --username option above, this needs to be writable by that user.# Otherwise, the init script will not be able to shut spamd down.PIDFILE="/var/run/spamd.pid"# Set nice level of spamd#NICE="--nicelevel 15"# Cronjob# Set to anything but 0 to enable the cron job to automatically update# spamassassin's rules on a nightly basisCRON=1
Edit the configuration file in /etc/default/spamass-milter to specify the socket:
Click here:
# spamass-milt startup defaults# OPTIONS are passed directly to spamass-milter.# man spamass-milter for details# Non-standard configuration notes:# See README.Debian if you use the -x option with sendmail# You should not pass the -d option in OPTIONS; use SOCKET for that.# Default, use the spamass-milter user as the default user, ignore# messages from localhostOPTIONS="-u spamass-milter -i 127.0.0.1 -m -I -- --socket=/var/spool/postfix/spamassassin/spamd.sock"# Reject emails with spamassassin scores > 15.#OPTIONS="${OPTIONS} -r 15"# Do not modify Subject:, Content-Type: or body.#OPTIONS="${OPTIONS} -m"####################################### If /usr/sbin/postfix is executable, the following are set by# default. You can override them by uncommenting and changing them# here.####################################### SOCKET="/var/spool/postfix/spamass/spamass.sock"# SOCKETOWNER="postfix:postfix"# SOCKETMODE="0660"######################################
Edit the configuration file in /etc/spamassassin/local.cf and add a couple lines at the top to use Pyzor and Razor:
Click here:
# This is the right place to customize your installation of SpamAssassin.## See 'perldoc Mail::SpamAssassin::Conf' for details of what can be# tweaked.## Only a small subset of options are listed below############################################################################# Force global Bayesian databases instead of per-user
bayes_path /var/lib/spamassassin/.spamassassin/bayes
bayes_file_mode 0777# Set Pyzor & Razor config file paths
razor_config /var/lib/spamassassin/.razor/razor-agent.conf
pyzor_options --homedir /var/lib/spamassassin/.pyzor
# A 'contact address' users should contact for more info. (replaces# _CONTACTADDRESS_ in the report template)# report_contact youremailaddress@domain.tld# Add *****SPAM***** to the Subject header of spam e-mails## rewrite_header Subject *****SPAM*****# Save spam messages as a message/rfc822 MIME attachment instead of# modifying the original message (0: off, 2: use text/plain instead)## report_safe 1# Set which networks or hosts are considered 'trusted' by your mail# server (i.e. not spammers)## trusted_networks 212.17.35.# Set file-locking method (flock is not safe over NFS, but is faster)## lock_method flock# Set the threshold at which a message is considered spam (default: 5.0)## required_score 5.0# Use Bayesian classifier (default: 1)## use_bayes 1# Bayesian classifier auto-learning (default: 1)## bayes_auto_learn 1# Set headers which may provide inappropriate cues to the Bayesian# classifier## bayes_ignore_header X-Bogosity# bayes_ignore_header X-Spam-Flag# bayes_ignore_header X-Spam-Status# Whether to decode non- UTF-8 and non-ASCII textual parts and recode# them to UTF-8 before the text is given over to rules processing.## normalize_charset 1# Textual body scan limit (default: 50000)## Amount of data per email text/* mimepart, that will be run through body# rules. This enables safer and faster scanning of large messages,# perhaps having very large textual attachments. There should be no need# to change this well tested default.## body_part_scan_size 50000# Textual rawbody data scan limit (default: 500000)## Amount of data per email text/* mimepart, that will be run through# rawbody rules.## rawbody_part_scan_size 500000# Some shortcircuiting, if the plugin is enabled#
ifplugin Mail::SpamAssassin::Plugin::Shortcircuit## default: strongly-whitelisted mails are *really* whitelisted now, if the# shortcircuiting plugin is active, causing early exit to save CPU load.# Uncomment to turn this on## SpamAssassin tries hard not to launch DNS queries before priority -100. # If you want to shortcircuit without launching unneeded queries, make# sure such rule priority is below -100. These examples are already:## shortcircuit USER_IN_WHITELIST on# shortcircuit USER_IN_DEF_WHITELIST on# shortcircuit USER_IN_ALL_SPAM_TO on# shortcircuit SUBJECT_IN_WHITELIST on# the opposite; blacklisted mails can also save CPU## shortcircuit USER_IN_BLACKLIST on# shortcircuit USER_IN_BLACKLIST_TO on# shortcircuit SUBJECT_IN_BLACKLIST on# if you have taken the time to correctly specify your "trusted_networks",# this is another good way to save CPU## shortcircuit ALL_TRUSTED on# and a well-trained bayes DB can save running rules, too## shortcircuit BAYES_99 spam# shortcircuit BAYES_00 hamendif# Mail::SpamAssassin::Plugin::Shortcircuit
Finally, restart every service and we should be done!
service postfix restart
service dovecot restart
A couple things to test:
Send a message to Gmail from this server.
Send a message from Gmail to this server.
Send a GTube message from Gmail to this server and check that the spam filter works.
And now we have finished building an e-mail server! If you run into any problem you are more than welcome to shoot me an e-mail at the address at the bottom of the page.
Contact Information
Lawson Computer Science Building
305 N. University Street
West Lafayette, IN 47907-2107
United States of America
Institutional: luo401 [at] purdue [dot] edu
Personal: zhtluo [at] gmail [dot] com