/admin/spam system

part of the ArsDigita Community System by Philip Greenspun

The Big Picture

This is a module to be used only by the publisher. Moreover, it has very little user interface of its own: URLs underneath /admin/spam are typically only invoked from within /admin/users. The data model is in /doc/sql/spam.sql. However, there is a facility for posting scheduled, periodic alerts or mailings, which has an interface under /admin/spam.

Under the Hood

Sending spam from the browser

The /admin/spam page has a link to send a plain spam message to a class of users, or to send a combined plain and HTML message. In both cases you will be given a form which asks for a subject line, a target class of users, and a date to send the message. The from-address will probably be overridden with a machine-generated email address in order to make the automated bounce handling scripts work.

Internal API for other modules to post scheduled spam messages to a user or group of users

If another module in the ACS wishes to post a spam message, they should use the following interface function:
# insert a new spam message into the spam queue, returns the spam_id of the new message
spam_post_new_spam_message {{
          -db 0 
          -template_p "f"
          -from_address ""
          -title ""
          -body_plain ""
          -body_html ""
          -body_aol ""
          -target_users_description ""
          -target_users_query ""
          -send_date "sysdate"
          -creation_user ""
          }}
The target_users_description should be an English description of who the designated target users are, for administrative documentation purposes.

The target_users_query should be a SQL query which returns the following columns for each user you want to send email to:

Column NameMeaning
user_iduser_id
emailuser's email address
email_type
[optional]
text/plain, text/html, or text/aol-html.
If you don't return this column in your query, the email type default to plain text.
If the message body is going to be evaluated as a Tcl template, include these columns as well:
first_namesUser's first name
last_nameUser's last name

Important

Make sure you do an order_by user_id, so that the query returns a list of users in increasing order by their user_id. This allows the spam system to checkpoint itself and resume sending where it left off in case the server is restarted in the middle of a long send job. See examples below:

Example SQL Queries for spam messages

It is usually a good idea to query over the users_spammable view, which will exclude users who have expressed a desire not to receive email alerts from the systems.

A simple plain text spam to members of a specific user_group:
select u.user_id, email, 'text/plain' as email_type
from users_spammable u, user_group_map ugm 
where ugm.group_id = 12345
and ugm.user_id = u.user_id
order by u.user_id

A spam to members of a specific user_group, which uses either the plain-text or html alternative content for the message, depending on the users email-type preference:

select u.user_id, email, email_type
from users_spammable u, user_group_map ugm, users_preferences up
where ugm.group_id = 12345
and ugm.user_id = u.user_id
and up.user_id = u.user_id
order by u.user_id

General Tcl Substitution in Message Body and Subject Line

The spam entry forms all have a checkbox labeled Template?.

If checked, then Tcl evaluator will be run at the time the message is actually sent on message subject and body, substituting variables or procedure calls wherever \$ and \[\] chars are found. This is specially useful for creating automatically generated templates, such as a daily report which runs a tcl procedure to query the database or create some other summary message.

Note: if you have the Template? option selected, make sure you use backslash to quote any $ or '[]' characters in your message, if you do not want them to undergo evaluation by the Tcl parser.

The following variables are guaranteed to be set in the environment, for use by your template.

Guessing user's email preferences

In the site .ini file, you can set up a list of patterns to be used to guess the email type preference of new users, based on their email address.

The param is an association-list, with SQL patterns on the left, and pseudo-mime types on the right. Supported types right now are text/html, text/html, and text/aol-html.

EmailTypes={%@hotmail.com text/html} {%@aol.com text/aol-html}

Manually Sending Spam From a File on the Server

You can schedule a spam which gets its content from files in the file system, using the "Post spam from file" link on the admin spam page. It will look n the "Drop Zone" directory (described below) for files with the specified names, and send the spam using them as the content to the target user class of your choice.

Spam Which Is Sent Automatically: Newsletter Groups

The spam daemon can send out daily (or any period you want) email to groups of users in designated user classes. There is a configurable list of "daily spam" files, accessible from the /admin/spam page via the "View/modify automatic daily spam file settings " link.

Every time you create a new user group, a new user_class will be created to go along with it. You may want to designate a specific group type to be the "Newsletter" group type, and then you can make a user self-service subscriptions page, which simply adds or removes the user from the groups of type Newsletter. You can then configure the Daily Spam Locations admin page to look for content files which correspond to the different Newsletter groups. Here is an example of what the admin page might look like for sending three types of newsletters; Daily, Weekly, and Special Updates.

Daily Spam File Locations

Your Workspace : Admin Home : Spam : List Daily Spam Files

Spam files to look for in drop-zone directory "/web/greentravel-dev/spam/".

To delete an entry, just enter an empty string for the filename and subject, and press the Modify button.
User Class Subject Filename
Add new daily spam
User Class Subject Filename

You can enter the following information for an automatic spam daily message:

User Class
(pulldown menu)
Subject
Note that you can include the current date in the subject line of the spam, by including the string "%%DATE%%" in the subject.
File Prefix
The filename prefix where you will deposit the new content for periodic mailings.

Drop Zone Directory

The files should be placed in the "Drop Zone" directory specified by the .ini parameter DailySpamDirectory:

Example:

DailySpamDirectory=/home/johnny/spam
For each spam defined in the Daily Spam list, the system will look for the following files:
file_prefix-MM-DD-YYYY
file_prefix
Note: Be sure to always use two-digits for Month and Day fields, i.e., 03-06-1999. Don't forget the leading zeros.

If a file whose name matches with the specified prefix and the current day's date is found, the contents are queued up to be sent as spam to the designated target user class.

The spam system will only send a dated spam once. It keeps a history of all spams sent, and will be robust across server restarts. If the server is restarted in the middle of sending a spam, the spam daemon will resume sending where it left off in the list of users.

Be very careful with filenames that have no date suffix!

If you use a filename with no date suffix, the spam will be sent once a day from the file. This behavior is designed to support a special case spam for new users, where the user class contains a magic query like
select user_id from users where trunc(registration_date)=trunc(sysdate-1) 
which is carefully constructed to always select a mutually exclusive set of users each day, and never repeat the same user twice.

HTML and AOL content types

Some users will have designated preferred MIME types for email via the users_preferences table. Currently we support HTML and AOL types in addition to plain text. If you create auxiliary files with the names
file_prefix-html-MM-DD-YYYY
file_prefix-aol-MM-DD-YYYY
Then content from these files will be sent preferentially to users who have their email types preferences set in the database.

Setting the .ini Parameters

Here is a summary of the .ini params you will need
 
[ns/server/yourserver/acs/spam]

; Pairs of {email_addr_pattern pseudo-mime-type}
EmailTypes={%@hotmail.com text/html} {%@aol.com text/aol-html}
DailySpamDirectory=/web/yourserver/spam
SpamRobotFromAddress=email-robot@yourdomain.com

BulkMail

By default the spam system uses the built-in AOLserver ns_sendmail routine. This is adequate for low volume mailings, but if you need to send mail to more than about 500 users, then it has serious limitations. A high-performance module called bulkmail which has a multi-threaded mail client which can talk to multiple mail servers concurrently, and can generate the QMAIL style VERP return addresseses to make automatic bounce handling possible. This will soon be available as part of the core ACS distribution, and the spam module will have an option to use this as the mail transport rather than sendmail.

For more info on the configuration of bulkmail and qmail, see bulkmail and qmail configuration

Utilities

There are some useful utilities defined by the spam and email handling modules:

Other module writers who want to post spam, Please Read This:

API for posting spam messages

If you want to write code that posts spam messages to the spam_history table directly, then you will have to obey some conventions. The most important of these is the structure of the query which returns the list of target users for the spam message.

This is the spam_history.user_class_query column, which should be a well formed SQL query which when run returns the following columns for each target user: user_id, email, email_type.

It is suggested that the query be done over the users_spammable table, in order to not annoy users who have expressed their preference not to get email notifications.

Example user_class_query SQL:

select user_id, email, email_type from users_spammable
where (email like 'john%') and (last_visit < (sysdate - 20))
Note that the email_type field comes from the users_preferences table, not the users table. The users_spammable view internally does a join of users and users_preferences, and selects the email_type column for you, but if you bypass the users_spammable table, you will need to supply email_type in your query.
philg@mit.edu

hqm@arsdigita.com