Building a High-Volume Newsletter Server

Steven Champeon
2000-06-15
Reprinted from WebTechniques

We've come a long way from the early days of the Internet, when many "mailing lists" were simply multiuser aliases maintained by the postmaster of a UNIX server. In those days, it was common for such "list" aliases to have a "-L" suffix, so sys admins and users could easily tell the difference between user accounts and multiuser lists. Subscription was a matter of emailing the sys admin and asking to be added to the alias. All mail sent to the list alias was simply resent, or "exploded," to all the users on the alias.

As the lists grew, and as users and administrators demanded more complex and powerful features, it became clear that the world needed something better. Developers introduced software packages that became known as mailing list managers (MLMs), including the Perl-based Majordomo and the popular and powerful Listserv. As you might expect, given that Listserv predates the Web by over eight years, these early MLMs were managed via commands sent to specific email addresses. These programs were more than just a different way to reroute mail from one person to several. They offered advanced features such as digest mode (with which a list member could elect to receive all messages sent within a given time period at once, instead of individually), the ability to block certain people from sending mail to the list, and a wealth of other configuration options, including custom headers, footers, and filtering capabilities. Listserv is the more powerful of the two programs, placing more control in the hands of the user. Listserv supports a dizzying range of options, but Majordomo is much smaller and simpler.

Both Majordomo and Listserv are remotely administered and let users perform certain administrative tasks themselves, simply by sending commands to the list software via specially configured aliases. For example, if you wanted to subscribe to a list named "coffee-lovers@example.com," you might send the command "subscribe coffee-loversme@mydomain.com" to the address coffee-lovers-request@ example.com. If you wanted to retrieve a set of instructions, you might send the command "help" tomajordomo@example.com. Before long, there was a wide variety of commands supported by the different packages, causing great confusion and irritation when those familiar with a given set of commands issued them to a completely different list manager that didn't support them. It's still fairly common to see SIGNOFF THISLIST, a Listserv command, inadvertently sent to a Majordomo list, for example.

With the advent and subsequent popularity of the Web, many list admins shifted to Web-based interfaces, in a desperate attempt to make the now-cryptic plethora of commands accessible to the average user, and to prevent the sort of confusion to which even experienced users found themselves prone. These interfaces ranged from home-grown CGI scripts that simply handled subscribe and unsubscribe requests, to full-blown GUIs such as MajorCool, which acts as a Web front end for Majordomo, to software such as Lyris or GNU Mailman, written from the ground up to enable GUI-driven administration and user-level preference editing. The intended nature and audience of your list will help you decide which features you need, and therefore which software you need.

Questions and Considerations

Before setting up an email server, consider the needs and skill level of your audience. Are your users email-savvy? Are they prone to sending giant HTML-formatted messages with vCards and other attachments? Do you want them to be able to subscribe or unsubscribe on their own via a Web interface? Will they be allowed to post to the list, or will it simply be a one-way newsletter list? Is there a chance that the list might start out as a one-way and change or expand into full-blown discussions later? How many lists will you run, and of what type? What sort of control do you want or need over the traffic on your list? Will there be only one person posting, or will you need to specify the email addresses of several people who are allowed to post? How often will this change? Do you need digests, where the traffic from a discussion list is posted in one long message every day or whenever you choose to configure it this way?

Next, ask yourself who will manage the list-a seasoned administrator, or a relative newbie? By management, I mean not only the initial administrative details such as setup and configuration, but also the mundane, day-to-day tasks such as unsubscribing users whose mail is bouncing, adding new members, and the like. Some packages automatically unsubscribe bouncers, provide Web-based (and therefore remote) administrative capabilities, and so forth. Others expect you to have command-line access to the server for certain advanced administrative tasks. Still others are extremely configurable and flexible, and are distributed as source code so a programmer can extend and customize them if the features provided still don't cover your needs. If the source code is something you'll be interested in, pay close attention to the language in which the list software is written. Trust me-it's not much fun to try to hack a Python-based package if the majority of your programming experience has been with Perl, C, and JavaScript.

What is your budget? Don't just think purchase price, either-consider the total cost of ownership. A freeware package may be fine for a programmer, but could pose some challenges for a nontechnical person. The day-to-day administrative tasks take time as well, so choose the software best suited to your planned time investment. Take into account whether you'll need support, and whether such support is going to cost you many times more than the original package costs over the lifetime of use.

A few other considerations come to mind as well. How large will the list membership be? What sort of growth will it experience? If you're only going to be sending occasional short messages to a few dozen of your favorite customers, or to a small (and easily estimable) group of friends, you can get by with the basic installation options of many packages. If your list will serve a few thousand or even tens of thousands, however, you'll need to tweak the installation. Don't worry, I'll walk you through a few examples of how to do so later.

The final question you have to answer before you choose a mailing list manager is which server OS and hardware platform you prefer to run. Some say that this is the first question you should ask, but I hate to limit myself based on OS bigotry. To be honest, all of the software packages we discuss here can be happily served from old, slow, clunky servers that have outlived their usefulness. I've run high-volume, high-traffic mailing lists from an old 166-MHz Pentium with 128MB of RAM that also served as a Web hosting farm, FTP server, DNS server, and mail server for three dozen hosting clients. Of course, this was on a Linux box, so your mileage may vary. Treat any recommended system requirements as what they are: recommendations. If the server you plan to use for these lists is underutilized, or if you only expect to use it for the lists, you really don't need much hardware or memory. On the other hand, if you plan to run a large, enterprise-class list manager such as Lyris to serve lists for an entire college campus or large-scale organization, feel free to ignore this advice.

Choosing a List Manager

My preference leans toward code you can fix or extend yourself. Perhaps because I've been using Majordomo since 1994, or perhaps because I'm a Perl hacker, I tend to favor Majordomo and sendmail over the newer, Web-based, GUI-driven list managers. I've also used several other list managers, and have taken to using Mailman when the list owner wants a Web-based GUI, usually because the list owners need remote administration, or because the list users aren't very technical.

I'll look at Majordomo version 1.9.4, a freeware, UNIX-based mailing list manager. And in the follow-up article , I'll show you how to set up Mailman 1.1. The installations assume you'll be running them with sendmail (8.9.3) as your mail-transfer agent on a Red Hat Linux server. If you want or need to run Windows NT or its more recent incarnations, there's always Lyris, which also runs under UNIX (Linux and Solaris) and which I recommend. If you prefer to use Macintosh computers, consider LetterRip Pro from Fog City Software. If you're a Java freak, I would recommend ezmlm, particularly if your mail-transfer agent of choice is qmail under Linux.

Installing Majordomo

You'll need root permissions to install Majordomo. After installation, you can configure everything with basic user-level access. Because Majordomo is written in Perl, you'll also need to make sure that the Perl interpreter is installed on your system. The Majordomo FAQ suggests that you need version 5.005_02 or later, so check that with the command perl -v. See " Online Resources" for download sites.

Save the distribution into a temporary directory (I usually use /usr/local/src for this purpose). Unpack it, using the command: tar xvzf majordomo-1.94.x.tar.gz (or whatever you find appropriate). Move into the distribution directory, and take a look at the README and INSTALL documents, which contain more detailed and up-to-date instructions than I have time to include here. You'll need to choose a user ID and group under which to run Majordomo and its associated programs. I recommend selecting a dedicated user, such as "majordomo," and the "daemon" group, although the truly paranoid among you will prefer to create a dedicated group as well. Just remember that if you use a dedicated group ID, you'll need to add that group ID to sendmail's list of trusted users (either in the sendmail.cf using the "T" line, for example, "Tmajordomo," or in whatever external file you've configured as your trusted users file, usually sendmail.ct). If you don't, the messages you send will contain an ugly "X-Authentication-Warning" header, saying that you didn't bother to take this step. Not a big deal, but if you have any mailing list administrators on your lists, you'll get pestered into doing it anyway.

Next, choose a directory in which to install Majordomo-I usually use /usr/local/majordomo for this purpose. Figure out where your C compiler, Perl binary, and system manual pages live, and edit the Makefile to reflect those locations. You'll also need the numeric UID and GID of the user and group under which Majordomo will run. Edit the Makefile to reflect that information, as well. If you have extreme paranoia about ordinary users being able to run Majordomo programs, you may also wish to modify the permissions for all installed files as further documented in both the INSTALL document and Makefile.

Next, create a new majordomo.cf file, by copying the sample.cf file to majordomo.cf-this will be the main configuration file for all Majordomo mailing lists run from this server. You should keep that in mind as you choose a value for the $whoami_owner variable. The user specified there will receive all problem reports and status messages in case of problems. This should probably be set to the administrator, rather than to the email address of whoever will run your first list. The full list of values you should modify is shown in Table 1.

When you've configured everything to the point where you're ready to build, run make wrapper, make install, and make install-wrapper. This will compile and install the Majordomo program and wrappers into the directory you specified. The wrapper program is a small C program that allows Majordomo to run as the user and group you've installed it under, for additional security. (Sendmail has, in the past, had some rather egregious security holes.) At the time Majordomo was written, Perl didn't provide such security on its own (you had to use a special version of Perl called taintperl), though it does now in version 5. So the wrapper is mostly historical, though it still performs its function admirably.

Next, configure sendmail to use a special majordomo aliases file, either by adding a "OA" line to sendmail.cf, like so:

OA/usr/local/majordomo/majordomo.aliases

or, if you're using sendmail's M4 macro feature to build your sendmail.cf file, add the majordomo.aliases file there:

define(`ALIAS_FILE',`/etc/mail/aliases, 
/usr/local/majordomo/majordomo.aliases')

and rebuild your sendmail.cf.

In the majordomo.aliases file, add the aliases for Majordomo (see Example 1). Note that you should be replaced by the actual user ID of the Majordomo administrator. When you've created and edited your aliases file, be sure to run the command newaliases to regenerate the sendmail aliases databases.

When you're finished, move into the Majordomo home directory and run ./ wrapper config-test to ensure that you've installed everything properly. Once you've successfully installed the software and confirmed that the installation is configured properly, it's time to create a list.

Configuring a Basic List

All Majordomo list configuration can be done via email, as befits its origins-Majordomo was created before the introduction of the Web. To make a new list, you can take one of several paths. The first, and simplest, is to create an empty file with the name of the list (for example, "example-list") in the lists subdirectory. Then, send the single line command lists to majordomo@example.com (substituting your own domain for example.com, naturally). This, as is the case with all Majordomo commands, should be in the body of the email message, not the Subject: line. You should receive a message from Majordomo itemizing all the lists that are currently available. To configure the new list, simply issue the command config example-list example-list.admin (again in the body of an email message tomajordomo@example.com.) All new lists have the default password listname.admin, where listname is the name of your new list. You will receive a reply from Majordomo containing the entire default configuration for your new list. I won't go into the entire list of options available, as they're fairly thoroughly documented, but I will discuss a few important values you'll want to change.

The first things I change whenever I create a new list are detailed in " Majordomo List Configuration Options ". They have to do with security, keeping signal high and noise low (for discussion lists), ensuring that list members know how to unsubscribe, and personalizing the messages sent out from the list. When you're finished, simply send the new list configuration to the addressmajordomo@example.com, preceded by the command

newconfig listname list.password

on the first line of the message. Copy and paste the new configuration into the body of a new message-it's important that the text isn't prefixed by quote characters and the like.

Once the list has been created and configured, you need to add a few aliases to the majordomo.aliases file, so sendmail knows how to deal with mail sent from and to the list. These are discussed in Example 2 . Substitute the list owner's userid for "you", and the name of your list for "example-list." Run "newaliases" to make sure the aliases took effect.

There! That's all you need to do; your new list is now ready to go. People can subscribe to the list by sending an otherwise blank message containing the text subscribe example-list tomajordomo@example.com.

Virtual Hosting Support

Some installations, particularly those in a virtual hosting environment, will need to deal with multiple domains. I've found that Majordomo is easy to reconfigure to support virtual hosting. Simply follow the instructions in the previous section, with a few minor differences.

First of all, you'll need to create a different configuration file for each domain-I usually just use domainname.cf for each (changing "domainname" accordingly). Make sure to alter the $whereami variable in the new configuration files to reflect the name of the domain in question. I also create a subdirectory for each domain in the $homedir/lists directory, and modify $listdir to suit (in other words, set it to $homedir/lists/domainname). Voila! Majordomo now understands multiple domains.

Secondly, because sendmail handles virtual hosted mail slightly differently from mail for a single domain, you'll need to set up aliases for each list in the majordomo.aliases file, then you'll have to map public aliases in the virtusertable to those "internal" aliases, so that mail sent tolistname@virtuallyhosteddomain.com is redirected to the alias you defined for that list in the majordomo.aliases file. You'll need to specify the Majordomo configuration file for each domain on the command alias, using the -C switch to wrapper. This provides a simple but powerful level of indirection and makes it possible to host lists for many domains with a single Majordomo installation. For an example majordomo.aliases entry, see Example 3 , and for an example virtusertable. aliases entry, seeExample 4 . As before, replace you with the address of the list owner, and example-list with the name of your list.

To test each installation, simply run ./wrapper config-testdomain1.com.cf for the configuration files you've created for each domain.

Advanced Server with bulk_mailer

After a few weeks or months with either of the above installations-or sooner, if you host extremely high-traffic or large-member lists-you may find that performance isn't up to snuff. What happens in most cases, in my experience, is that someone on the list has an unresponsive mail server, which has the unfortunate side effect of slowing down the processing of the queue.

Some background information may be in order here.

Sendmail processes mail by breaking the message up into a few different files in what is known as the "queue"; one contains the actual message body, one contains message headers, the addressee list (in a form that sendmail can understand), and a transcript file that's used when a message bounces or is undeliverable. If a given address near the top of the addressee list fails, sendmail may wait and try again after a configurable time period. When this happens frequently with a long list of addresses, the mail is delayed accordingly. Sending one copy of a message to multiple recipients at the same mail server is the only way sendmail knows to optimize queue processing. To work around this limitation of sendmail's queuing algorithm, Keith Moore developed bulk_mailer, a small C program.

Compiling and installing bulk_mailer is pretty easy; just download and unpack the source, then run ./configure and make. Then, install the bulk_mailer binary in /usr/local/bin or wherever you store optional system binaries. To immediately put the performance improvements into place, change the majordomo.aliases entry for example-list-out, as shown in Example 5 . The format is |/path/to/bulk_mailer sender-address member-list-file.

You should bear in mind that, because bulk_mailer enables large-scale bulk email, it has been abused by unscrupulous folk to send unsolicited commercial email (UCE, in anti-spam lobby argot) and many people have elected to filter out all mail with a bulk_mailer header. I worked around this problem by editing the header bulk_mailer inserts into every message, replacing it with a custom header. You may wish to do this as well, so I've included a sample patch against the bulk_mailer source. It changes the program identifier in the Received: header to avoid filtering, and adds a header that allows recipients to follow up to the appropriate abuse@ address, should the program be misused. To apply the patch, create two copies of the distribution, one in a folder named bulk_mailer1.12.new. Save the patch to a file named bulk_mailer.patch in the folder named bulk_mailer-1.12, and run patch < bulk_mailer. patch. This assumes you have the popular patch(1) program. Be sure to update the abuse address and identifier to reflect the appropriate domain and address. Now, you should see an amazing improvement in the efficiency of your list.

I've given you the tools to install and use mailing list managers in multiple environments. Be sure to check the online follow up to this article where I explain how to set up and use Mailman.

Steve explains how to set up and use Mailman in the follow up to this article, " Mailman Made Easy ".

Resources

bulk_mailer
ftp://cs.utk.edu/pub/moore/bulk_mailer

ezmlm
http://cr.yp.to/ezmlm.html

LetterRip Pro
www.fogcity.com

Listserv
www.lsoft.com

Lyris
www.lyris.com

Mailman
www.list.org

Mailman FAQ
www.list.org/faq.html

MajorCool
ftp://ftp.informatik.uni-hamburg.de/pub/soft/mail/majorcool/majorcool-1.32.tar.gz

Majordomo
www.greatcircle.com/majordomo 
ftp://ftp.greatcircle.com/pub/majordomo

Majordomo FAQ
www.greatcircle.com/majordomo/FAQ.html

MLM FAQ
ftp://ftp.uu.net/usenet/news.answers/mail/list-admin/software-faq

qmail
www.qmail.org

sendmail
www.sendmail.net 
www.sendmail.org

Table 1

Variables to modify in the majordomo.cf configuration file.

variable modification
$whereami Set to the domain name of the server.
$whoami This should probably be "Majordomo\@$whereami";
$whoami_owner The email address of the administrator of this computer.
$homedir Set to "/usr/local/majordomo" or wherever you've decided to install.
$listdir Set to "$homedir/lists"
$log Set to "$homedir/Log" or as preferred (you may want to use /var/log/majordomo, instead)
$sendmail_command Set to the path and name of your sendmail installation (usually /usr/lib/sendmail or /usr/sbin/sendmail)
$mailer Set to "$sendmail_command -oi -oee -f\$sender"
$bounce_mailer Set to "$sendmail_command -oi -oee -f\$sender -t"

Example 1

	# note that each entry should be on its own line
# also note that each alias name is followed by a colon			
	majordomo: "|/usr/local/majordomo/wrapper majordomo"
owner-majordomo: you,
majordomo-owner: you

Back to "Building a High-Volume Newsletter Server" article >>

Example 2

	example-list:         "|/usr/local/majordomo/wrapper resend -l 
                         example-list example-list-out"
example-list-out:      :include:/usr/local/majordomo/
                        lists/example-list
owner-example-list:    you,
example-list-owner:    you
example-list-request:  "|/usr/local/majordomo/
                        wrapper request-answer example-list"
example-list-approval: you

Example 3

	majordomo-domain1:     "|/usr/local/majordomo/wrapper majordomo -C /       
                         usr/local/majordomo/domain1.com.cf"
example-list:          "|/usr/local/majordomo/wrapper resend -C 
                        domain1.com.cf -l example-list 
                        example-list-out"
example-list-out:      :include:/usr/local/majordomo/lists/
                        domain1.com/example-list
example-list-request:  "|/usr/local/majordomo/wrapper majordomo -C 
                        domain1.com.cf -l example-list"
owner-example-list:    you,
example-list-approval: you,

Example 4

	# the format is:
# alias@domain                    local_alias
majordomo@domain1.com              majordomo-domain1
example-list@domain1.com           example-list
example-list-request@domain1.com    example-list-request
owner-example-list@domain1.com      owner-example-list
example-list-approval@domain1.com   example-list-approval

Example 5

	# majordomo.aliases entry for example-list
majordomo-domain1:     "|/usr/local/majordomo/wrapper majordomo -C /       
                         usr/local/majordomo/domain1.com.cf"
example-list:          "|/usr/local/majordomo/wrapper resend -C 
                        domain1.com.cf -l example-list 
                        example-list-out"
example-list-out:      "|/usr/local/bin/bulk_mailer 
                        owner-example-list@domain1.com/usr/
                        local/majordomo/lists/domain1.com/
                        example-list"
example-list-request:   "|/usr/local/majordomo/wrapper majordomo -C 
                        domain1.com.cf -l example-list"
owner-example-list:    you,
example-list-approval: you,

Sample Patch

	*** bulk_mailer.c       Fri Feb  5 18:29:53 1999
--- ../bulk_mailer-1.12.new/bulk_mailer.c       Mon Mar 20 19:23:01 2000
***************
*** 957,964 ****
      /*
       * add any prefix headers we want.
       */
!     fprintf (out, "Received: by %s (bulk_mailer v1.%d); %s\n",
!            local_domain, PATCHLEVEL, arpadate (&t));
  
      /*
       * copy header, checking for various fields
--- 957,964 ----
      /*
       * add any prefix headers we want.
       */
!     fprintf (out, "Received: by %s (example.com exploder); %s\n",
!            local_domain, arpadate (&t));
  
      /*
       * copy header, checking for various fields
***************
*** 1150,1155 ****
--- 1150,1157 ----
        fprintf (out, "List-Subscribe: %s\n", list_subscribe_hdr);
      if (list_unsubscribe_hdr)
        fprintf (out, "List-Unsubscribe: %s\n", list_unsubscribe_hdr);
+ 
+     fprintf(out, "X-Abuse-Contact: if this header is found in UCE/UBE, please contact <abuse@example.com>\n");
  
      fprintf (out, "\n");