Email2Moin

Background

Prior to selecting StatusNet as a messaging platform for the CDRC, I considered a forum integrated into MoinMoin, with email-based access. This would provide a means for people with low bandwidth and sporadic Internet connections to receive updates on documents and comments added to the system, while also having the opportunity to reply with comments of their own. In this respect I envisage a system that is diachronic (like a mailing list or news feed) but also allows for collaborative editing and revisioned content, and that can be accessed via offline email clients. We chose not to implement this method mainly because it is conceptually a challenge for non-technical users, who may be more comfortable with discussion that is clearly delineated from--but that can still reference--other types of content, such as wiki pages, images, laws, or archival materials. And ultimately, email delivered to mobile phones uses up more bandwidth due to bulky headers than the same set of messages combined into a single web page, compressed by Opera Mini or a comparable proxy server. Nonetheless an email-based cms/forum can be an extremely powerful and portable solution.

The nice thing about email is that it is a good transport mechanism for interacting with various storage systems. It can be composed, stored, and accessed offline. Users have a variety of options for clients and providers, both web-based and device-based (Desktop, mobile, etc.).

A MoinMoin-based setup has several advantages over a basic mailing list:

Users can actually contribute (and update) revisioned documents, though substantive revisions need to be done through the web or xmlrpc interface (the email interface supports only appending text). This differs from mailing lists (updated documents have to be re-posted) and web forums (individual posts can be edited, but these are not presented as standalone pages).
We can add attachments (documents, media files) to a given folder via the email interface (as opposed to individual messages).
Users can browse and write content using the web interface, rather than their email client exclusively.
We can activate hyperlinks that make the web archive browsable.
The archive can easily be replicated, edited, and synchronized; it is not static but a dynamic document set. The documents and related discussion can be pulled into an offline instance.

Comment storage format

One question is whether comments should be stored individually or appended to a single page, which is the default setup in MoinMoin and MediaWiki (pre-Liquid Threads). The advantages to individual storage:

comments can be filtered and listed using pagename arguments
we can change display format and order (e.g., chronological or reverse chronological, or most recent 10, etc.)
linking to individual comments is easy
we can set the number of comments per page, without having to archive threads
individual comments can be edited, revisioned, deleted, and even renamed
comments can be copied/linked/embedded to other discussions. This is useful if we want to fork a discussion, or create a summary page that embeds "featured" comments (similar to the New York Times discussion feature).
comments can be protected by ACL (i.e., so that contributors can't edit one another's comments)
for a long discussion, readers can start at a paged view rather than having to scroll down
it is easy to format the forum automatically to show author details, etc.

The advantages to having everything on a single page:

Easy to create threads
We can archive the entire discussion as a single page, which is nice and logical (like meeting minutes, rather than a series of discrete interventions)
We can add attachments to the page, etc.
The default setup works (though we lose the formatting available from the macro; this would need to be arranged somehow with css)
Do we have an "append" macro for the web and localedit interfaces? We will end up with a revision metadata line for each comment, which we don't necessarily want.
We can insert summaries etc. within the discussion

Our choice here partly depends on how we view this discussion--as a series of independent interventions, or as a complete discussion.

If we want to view this as a series of independent interventions, we might think of using a naming scheme where each author has their own comment page. But then chronological reading becomes difficult.

Forum action and macro

See MoinMoinCustomizations for details and code. We are using subpages for individual comments, in the format Pagename/C{timestamp}{UserName}:

MoinForum/TopicName/C1100457894EricThrift
MoinForum/TopicName/C1100566879ХангайсайханыНаранцацрал
MoinForum/TopicName/C1100572409AnonymousUser

Configuring email replies

The first challenge here is to figure out how to allow people to contribute to a discussion thread by responding to an email. Options:

The "Reply-To" header gives an incoming address for forum comments (e.g., forum@example.org), which generates a new subpage in the format [Subject][Sender][Time].
Replies are to the moderator's email, who forwards them to the wiki.
Text is simply appended to the topic page. (This can get messy with quoted text, unless we automatically remove it.) This is actually the default behaviour for inbound mails.

In our solution, the user can do one of two things:

Reply directly to the email, which creates a sub-page containing their comment; or
Forward and modify the text of the email, which creates a new version of the page.

When sending a comment, we get the parent page name and append a subpage using the author's name and timestamp. (Change the "date" variable in mailimport.py to a UNIX timestamp.)

 To: <wiki@example.org>
 Subject: Re: "PageName/" (UserName)

or for the page SandBox/C1321188838UserName

 Subject: Re: "SandBox/" (UserName)

The slash activates mail_import_subpage_template (C$date$from). Make sure that this slash is present in notification messages. We need to modify mail_import_pagename_regex if we are going to get rid of the wiki link brackets and use quotation marks instead. The regex can potentially require CamelCase pagenames, though this will break attempted replies to non-standard pagenames.

mail_import_subpage_template = u"C$date$from"

Signature for outgoing mail

We need to include "subscribe/unsubscribe" links, as a legal requirement in some jurisdictions. If someone is posting a reply we can remove quoted text (i.e., lines beginning with a hash sign).

-- 
# This message has been sent automatically by the MandalWeb system 
# because you are subscribed to this page. To unsubscribe or change 
# your subscription preferences, visit http://mandal.ca/web.cgi/?
# action=userprefs&sub=notification.
#  
# Changes are indicated above using the "+" symbol to show added lines 
# of text, and the "-" symbol to show removed lines of text.
# 
# To post a follow-up comment by email, simply reply to this message, 
# leaving the subject intact. Please do not use special formatting in 
# your message (bold, underline, italic, etc.). Wiki formatting is 
# allowed; see http://mandal.ca/web.cgi/HelpOnWikiFormatting for 
# details.
# 
# The main page for this topic is located at "http://mandal.ca/web.cgi/
# PageName". To view existing discussion, visit "http://mandal.ca/
# web.cgi/PageName?action=supplementation".

Bookmarks