Populate Mailman Lists from Django Projects

Django projects can end up with complex sets of Users, Groups, and multiple profile types representing different types of people. For example a school site might have Students, Parents, Teachers, Staff, and Alumni. The mailing lists for that school will live completely outside of the Django project, but there’s a good chance you’d like to be able to populate list membership from your membership database rather than maintaining lists by hand. And you’d like to be able to use any combination of criteria to populate your lists (Group membership,  profile types, join date, privileges, etc.)

Since Mailman is installed on so many web hosts by default, there’s a good chance you’re using it, and have lots of overlapping and non-overlapping groups subscribed to various lists. I recently went through the process of integrating a bunch of Mailman lists with the membership representing a school intranet built in Django and thought I’d document it for anyone going through a similar process.

Once everything is set up, you’ll never need to use the Mailman interface to manage lists again – everything will be automated and self-maintaining.

Manual Overrides / Prerequisites

• The simple queries are easy. But in the real world you end up in situations where, say, the Maintenance person needs to be able to write to the Teachers list but not receive mail from it. Or maybe teachers need to be able to write to the Board, but not be privy to the Board’s list traffic.
• Some people may fit a particular group query but not want to receive mail from lists for those groups (because maybe their spouse handles that stuff).
• Some people don’t match a particular query group but need to be made into regular list members anyway.  For example you might want to have the principal be on all mailing lists, even though he’s not a parent of any child enrolled in a class.

The key to making some of this happen is a feature of Mailman called “nomail” mode, which allows certain subscriber addresses to send to a list but not receive mail from it. This recipe interfaces with Mailman from the command line, using the add_members command. But here’s the rub – the official Mailman distribution’s add_members command is missing the option to add a person to a list in nomail mode (grrrr). However, the version of add_members that comes with OS X Server does include this option. I have no idea why Mailman hasn’t seen fit to integrate Apple’s changes into the official distro, since it’s insanely useful.  But since add_members is just a Python script, you can replace the version of add_members that comes with the official distro with Apple’s version and it’ll work fine on any server platform.

Because the Mailman command-line scripts must be run as root, you must be the administrator of the server this recipe runs on.

To handle all the exceptions and special cases your organization probably has, you’re going to need two tweaks to your apps/models. First, on your Profile model(s), create a new field for people who don’t want to be on the lists at all:

no_lists = models.BooleanField(default=False,
     help_text="When checked, this parent will NOT be subscribed 
     to the mailing lists they normally would be.")

You can now check this box in any user profile for people who show up in group queries but don’t wish to receive mail from any of the organization’s lists.

You’re also going to need an additional model to store the exceptions – additional people to add to lists, either in yesmail or nomail mode:

class ListExtra(models.Model):
    """
    Extra email addresses to be added to mailing lists.
    """
    list = models.SlugField()
    addresses = models.TextField(blank=True,
         help_text='Add addresses here, one per line.')

    def __unicode__(self):
        return u'%s' % (self.list)

Once the model has been created and you’ve run syncdb, create a record for each list you want to track, e.g. “teachers” for the list “teachers@ourschool.org” .  Do not enter real teachers here – they’ll come from ORM queries on the regular User table.  Instead, this is where you’ll add non-teachers  who should be regular members of the teachers@ mailing list anyway. Add such people’s addresses to the field one-per-line.

If you have the need to let certain people write to the teachers list but NOT receive mail from it, create an additional record called “teachers-nomail”.

How It Works

This recipe is really two scripts – one in Python and the other bash – that I end up running from a single shell alias as needed. The two scripts I use can be downloaded at the end of this post and are well commented.

The Python script (listgen.py) interacts with the Django ORM, extracting list memberships out into text files. The bash script (listgen.sh) iterates through that set of text files and, for each, quickly unsubscribes the entire membership and then re-subscribes them from the text files generated by the python script. On my server, the whole process unsubs and resubs 14 lists covering 125 people in  5 seconds flat, so there’s no real danger of anyone freaking out because they were temporarily unsubscribed.  And  listgen.sh makes sure that subs and unsubs occur without notifying users of list membership changes, so they don’t get scary messages.

listgen.py sets up the Django environment for the given project, defines a function for processing query results into text files, then iterates through a list of mailing list names (without the domain name). For each, it runs a Django ORM query. Then it excludes from that queryset anyone who has opted out of all lists.

Next, it looks for a corresponding ListExtras record and extracts the “extra” addresses it contains. Finally, it looks for a corresponding “nomail” ListExtra record (e.g. teachers-nomail). It then removes any duplicate members and passes the final query data to the file processing function and writes text files to a directory on the server.

listgen.sh iterates through an array of all mailing lists and, for each, unsbuscribes all members (without sending unsub messages). It then locates the text file corresponding to the current list and subscribes all regular members in yesmail mode. Finally, it looks for a corresponding listname-nomail.txt file and subscribes the addresses it contains in nomail mode.

Deployment

The scripts I’m using are linked below.

First, check to see whether the version of add_members on your server supports the -e flag (for nomail mode). If it does not, rename the add_members script on your server to -old and put the replacement add_members script in its place. Run “locate add_members” if not sure where to put it. If it complains when you try to run it, make sure its first line matches the python path shown on other scripts in the same dir.

Put listgen.py in a “scripts” directory in your Django project.

Put listgen.sh somewhere where root can get to it.

You’ll need to modify the list of mailing lists in each script, as well as the most important bit – the Django ORM queries in listgen.py. This will probably take quite a bit of testing and experimentation – make sure you’re working with test data until everything is humming – users will get righteously pissed if you break their lists.

You’ll also need to make sure the two scripts agree on the filesystem location for the output  (for the generated files that listgen.py creates and listgen.sh reads).  By default this will be a dir at myproject/scripts/listgen. You’ll need to edit the python path vars at the top of listgen.py.

During testing you’ll probably want to just run listgen.py and not listgen.sh — then you can study the text file output without actually altering the lists. When you’re confident listgen.py is doing the right thing, add the listgen.sh step.

Once it’s all working, create an alias for root that runs both scripts, something like:

alias schoollist='python /path/to/scripts/listgen.py; \
/bin/sh /root/scripts/for_customers/school/listgen.sh'

You could set this up to run as a cron job, but there’s a good reason not to: Mailman keeps track of bad addresses with its bounce processing feature, which counts the number of times a message has bounced from a bad address and then unsubscribes that user and alerts the list administrator. If you run listgen.sh every night, every member will always be “new” as far as Mailman is concerned and bounce thresholds will never be reached.  You’ll never be informed when/if you have bad addresses in your Django system. To address this, I wrote a Django signal handler that sends me email whenever a User object has been updated. I then run the schoollist command whenever I know something has changed (after the system stabilizes this shouldn’t happen very often).

The Scripts

add_members (copied from OS X Server’s Mailman distro, but works on any platform)

listgen.py (put in a scripts dir inside your Django project)

listgen.sh (put somewhere root can access easily)