• About
scot hacker's scripts and utils

Populate Mailman Lists from Django Projects

September 7, 2009 1:19 am / 12 Comments / shacker

Django projects can end up with complex sets of Users, Groups, and multiple profile types representing different types of people. For example a school site might have Students, Parents, Teachers, Staff, and Alumni. The mailing lists for that school will live completely outside of the Django project, but there’s a good chance you’d like to be able to populate list membership from your membership database rather than maintaining lists by hand. And you’d like to be able to use any combination of criteria to populate your lists (Group membership,  profile types, join date, privileges, etc.)

Since Mailman is installed on so many web hosts by default, there’s a good chance you’re using it, and have lots of overlapping and non-overlapping groups subscribed to various lists. I recently went through the process of integrating a bunch of Mailman lists with the membership representing a school intranet built in Django and thought I’d document it for anyone going through a similar process.

Once everything is set up, you’ll never need to use the Mailman interface to manage lists again – everything will be automated and self-maintaining.

Manual Overrides / Prerequisites

  • The simple queries are easy. But in the real world you end up in situations where, say, the Maintenance person needs to be able to write to the Teachers list but not receive mail from it. Or maybe teachers need to be able to write to the Board, but not be privy to the Board’s list traffic.
  • Some people may fit a particular group query but not want to receive mail from lists for those groups (because maybe their spouse handles that stuff).
  • Some people don’t match a particular query group but need to be made into regular list members anyway.  For example you might want to have the principal be on all mailing lists, even though he’s not a parent of any child enrolled in a class.

The key to making some of this happen is a feature of Mailman called “nomail” mode, which allows certain subscriber addresses to send to a list but not receive mail from it. This recipe interfaces with Mailman from the command line, using the add_members command. But here’s the rub – the official Mailman distribution’s add_members command is missing the option to add a person to a list in nomail mode (grrrr). However, the version of add_members that comes with OS X Server does include this option. I have no idea why Mailman hasn’t seen fit to integrate Apple’s changes into the official distro, since it’s insanely useful.  But since add_members is just a Python script, you can replace the version of add_members that comes with the official distro with Apple’s version and it’ll work fine on any server platform.

Because the Mailman command-line scripts must be run as root, you must be the administrator of the server this recipe runs on.

To handle all the exceptions and special cases your organization probably has, you’re going to need two tweaks to your apps/models. First, on your Profile model(s), create a new field for people who don’t want to be on the lists at all:

no_lists = models.BooleanField(default=False,
     help_text="When checked, this parent will NOT be subscribed 
     to the mailing lists they normally would be.")

You can now check this box in any user profile for people who show up in group queries but don’t wish to receive mail from any of the organization’s lists.

You’re also going to need an additional model to store the exceptions – additional people to add to lists, either in yesmail or nomail mode:

class ListExtra(models.Model):
    """
    Extra email addresses to be added to mailing lists.
    """
    list = models.SlugField()
    addresses = models.TextField(blank=True,
         help_text='Add addresses here, one per line.')

    def __unicode__(self):
        return u'%s' % (self.list)

extras Once the model has been created and you’ve run syncdb, create a record for each list you want to track, e.g. “teachers” for the list “teachers@ourschool.org” .  Do not enter real teachers here – they’ll come from ORM queries on the regular User table.  Instead, this is where you’ll add non-teachers  who should be regular members of the teachers@ mailing list anyway. Add such people’s addresses to the field one-per-line.

If you have the need to let certain people write to the teachers list but NOT receive mail from it, create an additional record called “teachers-nomail”.

How It Works

This recipe is really two scripts – one in Python and the other bash – that I end up running from a single shell alias as needed. The two scripts I use can be downloaded at the end of this post and are well commented.

The Python script (listgen.py) interacts with the Django ORM, extracting list memberships out into text files. The bash script (listgen.sh) iterates through that set of text files and, for each, quickly unsubscribes the entire membership and then re-subscribes them from the text files generated by the python script. On my server, the whole process unsubs and resubs 14 lists covering 125 people in  5 seconds flat, so there’s no real danger of anyone freaking out because they were temporarily unsubscribed.  And  listgen.sh makes sure that subs and unsubs occur without notifying users of list membership changes, so they don’t get scary messages.

listgen.py sets up the Django environment for the given project, defines a function for processing query results into text files, then iterates through a list of mailing list names (without the domain name). For each, it runs a Django ORM query. Then it excludes from that queryset anyone who has opted out of all lists.

Next, it looks for a corresponding ListExtras record and extracts the “extra” addresses it contains. Finally, it looks for a corresponding “nomail” ListExtra record (e.g. teachers-nomail). It then removes any duplicate members and passes the final query data to the file processing function and writes text files to a directory on the server.

listgen.sh iterates through an array of all mailing lists and, for each, unsbuscribes all members (without sending unsub messages). It then locates the text file corresponding to the current list and subscribes all regular members in yesmail mode. Finally, it looks for a corresponding listname-nomail.txt file and subscribes the addresses it contains in nomail mode.

Deployment

The scripts I’m using are linked below.

First, check to see whether the version of add_members on your server supports the -e flag (for nomail mode). If it does not, rename the add_members script on your server to -old and put the replacement add_members script in its place. Run “locate add_members” if not sure where to put it. If it complains when you try to run it, make sure its first line matches the python path shown on other scripts in the same dir.

Put listgen.py in a “scripts” directory in your Django project.

Put listgen.sh somewhere where root can get to it.

You’ll need to modify the list of mailing lists in each script, as well as the most important bit – the Django ORM queries in listgen.py. This will probably take quite a bit of testing and experimentation – make sure you’re working with test data until everything is humming – users will get righteously pissed if you break their lists.

You’ll also need to make sure the two scripts agree on the filesystem location for the output  (for the generated files that listgen.py creates and listgen.sh reads).  By default this will be a dir at myproject/scripts/listgen. You’ll need to edit the python path vars at the top of listgen.py.

During testing you’ll probably want to just run listgen.py and not listgen.sh — then you can study the text file output without actually altering the lists. When you’re confident listgen.py is doing the right thing, add the listgen.sh step.

Once it’s all working, create an alias for root that runs both scripts, something like:

alias schoollist='python /path/to/scripts/listgen.py; \
/bin/sh /root/scripts/for_customers/school/listgen.sh'

You could set this up to run as a cron job, but there’s a good reason not to: Mailman keeps track of bad addresses with its bounce processing feature, which counts the number of times a message has bounced from a bad address and then unsubscribes that user and alerts the list administrator. If you run listgen.sh every night, every member will always be “new” as far as Mailman is concerned and bounce thresholds will never be reached.  You’ll never be informed when/if you have bad addresses in your Django system. To address this, I wrote a Django signal handler that sends me email whenever a User object has been updated. I then run the schoollist command whenever I know something has changed (after the system stabilizes this shouldn’t happen very often).

The Scripts

add_members (copied from OS X Server’s Mailman distro, but works on any platform)

listgen.py (put in a scripts dir inside your Django project)

listgen.sh (put somewhere root can access easily)

Posted in: Django, Python

12 Thoughts on “Populate Mailman Lists from Django Projects”

  1. Tim on September 8, 2009 at 5:19 pm said:

    Hello, I am looking to do something that I assume is pretty simple. All I want to do is send a message, either directly from actionscript or from PHP (from the “Submit Address” form) to Mailman containing an email address and have the address automatically added to our existing mailing list. I’m very new to Python, and the add_members script above has basically been my tutorial, so in the short term, is there an easy way to achieve this with the scripts as they are written originally? I’ve been searching forums for hours but haven’t had any luck.

    Thanks for your help

  2. shacker on September 8, 2009 at 6:28 pm said:

    Tim, the reason that’s going to be difficult is that the command line scripts that come with Mailman must be run by root, but your web apps will never run as root. Your only alternative is to have your form submit addresses as lines to a text file on the server, then use a cronjob run by root to process that text file once per day.

  3. Tim on September 11, 2009 at 1:50 pm said:

    Thanks for the quick response. Fortuitously, I’ve already written the script for the text file operation, I’ll read up on the process you mentioned to make it happen.

    Thanks for the help, I’ll let you know how it goes!

  4. Bernd on September 18, 2009 at 3:10 am said:

    I don’t have access to the add_members script because I’m using webfaction and the mailman installation runs on another server.

    So I wrote a small django-app instead which uses the webinterface.
    https://launchpad.net/django-mailman

  5. shacker on September 18, 2009 at 9:23 am said:

    Bernd – That sounds really useful. Will there be any documentation or examples coming for it? I can’t tell from looking at the source code how one might go about deploying it.

  6. Bernd on September 19, 2009 at 3:51 am said:

    @shacker:
    I cannot promise anythink, but I will put it on my todo list. I wrote it a while ago and have to look into it again 😉

  7. Bernd on September 19, 2009 at 6:16 am said:

    @shacker:
    you can now find a README file in the source code. You should also take a look at the INSTALL file.

    You can ask question or file bugs for my project on launchpad directly

  8. shacker on September 22, 2009 at 12:02 am said:

    Thanks Bernd! The world is always grateful for good documentation.

  9. Olive on November 18, 2010 at 4:00 am said:

    Thank you very much for the script. Its really useful too for the migration from majordomo to mailman. I have found something interesting which you can set in mailman. To allow members of another list to post to your list, eventhough they are not members, you can add @ to the “List of non-member addresses whose postings should be automatically accepted” setting. Then all members of @listname can also post to the list but not receive the emails. Thanks again for a most useful script !

  10. shacker on November 18, 2010 at 10:18 am said:

    Olive, glad you find it useful! That’s a brilliant tip about allowing @listname to post but not receive. I had no idea! That could really simplify some of my work. Thanks much!

  11. Mike C. on February 5, 2015 at 6:56 pm said:

    Any chance you still have the Django User Signal code you mentioned?

    “I wrote a Django signal handler that sends me email whenever a User object has been updated.”

  12. shacker on February 5, 2015 at 7:54 pm said:

    Mike, no, but they’re pretty easy to write. If you want to detect when a User object has been saved, include this somewhere in your codebase:

    signals.post_save.connect(foobar, sender=User)

    where foobar refers to the name of a function with a signature something like this:

    def foobar(sender, instance, signal, created, **kwargs):
    …
    # send your mail here

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Post Navigation

← Previous Post
Next Post →

Categories

  • AMP
  • BeOS
  • cPanel
  • Django
  • Geo
  • Mac
  • Movable Type
  • Music
  • Performance
  • Python
  • QuickTime Streaming Server
  • Spam
  • Twitter
  • WordPress
© Copyright 2023 - scot hacker's scripts and utils
Infinity Theme by DesignCoral / WordPress