HTTP Access to Email Stores

Email access via the Web is ubiquitous. The most common situation seen in deployment is for a mail server administrator to setup IMAP, POP and Web access to the same mail store, giving users choice. The Web-based access seen today is referred to in this document as "Web mail UI" to distinguish from HTTP mail access as defined in this document. Note that HTTP use in Internet mail already goes beyond Web mail UIs and can be quite sophisticated. See "Existing Work". IMAP, POP and Web mail UIs already co-exist, and this proposal makes no attempt to replace any of these, but instead to augment them. Different situations require different tools. HTTP access will enable new use cases for both users and administrators, and foster new clients and helper applications. Adoption may be found far from traditional rich email applications: in status bar applications, client utilities, administrative utilities, server-to-server communication or even mashups. See "Applicability". The functionality specified here is quite limited and cannot, by itself, replace POP or IMAP even if that were the intent. The approach here is to combine what is easiest and most natural to do in HTTP with a quick analysis of use cases and take the intersection. The features found in that intersection: A persistent HTTP URL used to download a single message/rfc822 object HTTP URLs and response formats to download full or partial mailbox content listings Persistent URLs and response formats to discover what mailboxes exist Use of DELETE to instruct the server to remove the email from normal visibility. Use of POST, together with a mailbox URL, to add an email to an existing mailbox. The choice of how to provide mailbox content listings in HTTP responses is a difficult decision. While some deployed systems already use WebDAV PROPFIND responses for message listings, we believe that a few conventions for using Atom will be more easily implemented and used by clients and servers that do not already have this functionality. Atom offers more control to the server over how, and even what entries to present in a feed, and can support search folders as well as traditional mailboxes. Simple operations are very simple indeed with Atom: a GET request leading to a single document in a well-known XML format. This document's current mappings even makes it possible for a unmodified newsreader to browse mailboxes. Support for Atom does not preclude more sophisticated support for WebDAV PROPFIND, REPORT or for HTTP-based search operations. Indeed, if this proposal sees any adoption, further work will likely be needed depending on which use cases crop up. Skip ahead to section 2 and the rest of this specification for further technical details on these allegedly easily accomplished features.

Servers MUST support: HTTP , including REQUIRED support for strong ETags Atom Feed documents , generated to show listings of messages Downloading messages in message/822 format Atom Feed history , to break up listings of messages TLS , REQUIRED unless all messages and mailboxes are publicly readable. Clients MUST support: HTTP 1.1 , including chunked transfer-encoding and all other required HTTP 1.1 features. Clients MUST support TLS in order to be able to communicate with secure servers. Client support for other features depends on the usage. For example, a status monitoring client might only need to know a bit of summary information about new or unread messages, and would not need to support message/822. Some examples of optional client features and consequences: If the client does not support the Atom Feed format, it will be unable to parse message listings. It may still be able to access individual messages but finding the URLs to these messages would have to be done in some other way. If the client does not support message/822, it will be unable to read messages on all servers, but may still be able to see message listings, and read messages if other formats are available. HTTP Cookies MAY be used to maintain authentication status if the transport is secure. If the client does not support feed history , it can only see some of the most recent messages in each mailbox.

[Note: This section should be deleted if this document is adopted by a WG or published as an RFC. Information in this section may become inaccurate over time and is useful mainly to motivate starting this work in the form proposed.] Web access to email, as commonly experienced in 2008, involves presentation information (HTML, CSS, images and links) as well as mail content. Some Web mail UIs do separate presentation from content under the surface. For example, some servers support an AJAX-like programming pattern so that the Web browser can be instructed to download message listings or message bodies separately from downloading templates, images or scripts. With this architecture, different server processes or even different servers can handle the tasks of serving presentation logic, presentation formats and mail store contents. Some commercial mail stores already have HTTP interfaces that are predictable to the point of being machine-readable. Sometimes this interface is called a third-party programming interface, an interface that allows third-party or add-on applications to access the mail store and provide some auxiliary service that the site administrators wish to offer users. Microsoft did some early work on HTTP access to email. The Outlook Express client application uses a fully HTTP-based interface to do all mail functions on Exchange mail stores as well as Hotmail. At the time, Hotmail could or did not support IMAP. The Microsoft API uses PROPFIND to list mailbox contents (this mechanism predated Atom by several years). The Yahoo Mail Web Service uses SOAP or JSON-RPC. This adds a layer of complexity even to operations such as downloading an email body, which is retrieved inside a SOAP envelope. In an indication that email providers see value in enabling mashups, Yahoo announced this API at Yahoo Hack Day "so programmers can develop extensions and new interfaces for Yahoo Mail". Google's mail API is undocumented. However, Google does have the GData API, which extends Atom for access to many different types of user content other than email, and GData is similarly available for third-party developers. SquirrelMail is a Web mail UI that can work with a number of IMAP backends. This is interesting, and related, because it is a generic gateway from IMAP to an HTTP interface, albeit a human-readable one. A machine-readable HTTP interface would be even more natural than IMAP as an interface for a front-end like SquirrelMail.

The focus in this specification is on read-only access to the mailbox contents and message contents stored in a mail store. This specification does not cover sending email or handling email delivery status, managing spam or other filters on the server, creating mailboxes, moving messages or even marking them as read. These limitations clearly rule out many complex applications. Still, there are many situations where this work may be applicable, and further work can build on this base. Associated clients and utilities: Users may install applications or utilities which are not full-fledged email clients, but can benefit from access to mailbox listings and mail messages. Some examples: A user might install a Challenge/Response spam-limiting system as a client tool, rather than use a server-hosted system. The Challenge/Response utility can scan the Inbox periodically for mail from new senders. It can use SMTP with the user's normal credentials to send the Challenge message, along with some Turing test such as an image to identify or describe. The same Inbox scans allow it to find responses from validating senders. The main problem that this specification does not help with is how the Challenge/Response utility can then deal with the message that is now considered valid -- this specification does not explain how to use HTTP to change the message flags or move it to another mailbox. A calendar client might poll the Inbox for messages containing calendar contents or calendar attachments, and automatically add these as "tentative" meetings. A productivity client (e.g. Chandler) may scan special folders that the user sets up to contain emails describing tasks or task information. A custom spam detection engine, tuned or trained by the user, may be able to poll the user's Inbox and delete spam before the user gets to it. This can help in situations when the user is required to use an otherwise-adequate mail client/server with an insufficiently accurate spam filter. While there are many spam solutions within the traditional email framework, we (as protocol designers) have no reason to artificially limit the means by which users attempt to manage spam in their own mailboxes. The next category is third-party server applications and server 'mashups', where the user's existing email store and its functionality is augmented by a separate service. The major differentiator within this category is whether the augmenting service is setup and trusted by email administrators or not. There already exist mail to Twitter gateways. These could be easier to use with (authorized) access to the user's own email account. This kind of service is currently adopted by individual users, without assistance from email administrators. Third-party applications to archive email, and apply email retention policies, are sometimes offered by different vendors than the email server itself. This is an example of a service setup and trusted by the email administrator. The final category is new services: places where mail standards have not yet penetrated. We see silos of personal messages in social interaction sites, career networking sites, volunteer coordination sites and even knitting sites. Perhaps some of these sites can offer a standard API as long as it's simple enough and under sufficient server control to be low-cost and low-risk for the site management. The good news is that the functionality required by such sites can be very small: list email, read email, and delete (which may or may not be implemented moving to trash.) Marking messages as 'read' may also be a requirement, and sending new messages may involve a POST to a well-known mailbox. The benefit to users of even partial support by these sites may simply mean that with a local client able to poll and access site messages, the user would be better able to keep abreast of conversations and maintain participation. Mailing list postings are delivered via SMTP, but the access to the public mailing list archive is as-yet unstandardized. Most mailing list software comes with HTTP access to archives, but these vary widely and don't allow easy "mashup" access. A machine-readable HTTP interface to public mailing list archives would allow users to catch up on mailing list history and choose between Web UIs, email-style client UIs and newsreader style client UIs.

Analysis of use cases for helper applications and mashups (anything other than a full, rich mail reader) shows that marking messages as "read" may be the most common requirement so far unmet by this specification. Using IMAP to mark a message as "read" would involve a number of operations: establishing the session and logging in if that's not already done, selecting the correct folder, selecting the correct message (e.g. in a SEARCH command), and storing the correct "Seen" flag. Note: Other message flag operations involve the same steps, but do not seem to be as commonly required in the use cases. One might consider removing the "Seen" flag to be necessary to undo user errors in marking as "read". The next most common use case requirement is that of submitting messages. While SMTP is a reasonable simple approach for many use cases, it is not universally acceptable. For example, the social networking sites that offer site-specific messaging and mailboxes may have significant trouble operating a fully standards-compliant SMTP server in order to provide mail sending to their site members. In RESTful HTTP email interfaces, message submission has been modeled as the action of putting a message into an outbox, possibly a write-once "Sent" folder. This operation does not fit as nicely into HTTP's methods as one would like. The third most common unmet use case is marking a message as spam.

The message/rfc822 content type is registered and discussed in and , but the format itself is defined in . registers the MIME types message/rfc822 and message/partial. Servers supporting this specification MUST offer messages in the message/rfc822 format, and MUST use this format if it is mentioned in an Accept header in a GET request for the message. Servers MUST also serve the message as a Atom Entry document that can be used in clients such as existing newsreaders that do not accept message/rfc822. The Atom Entry document is handy as a default link to the message, which can be used to find other links like the message/rfc822 version. Servers MAY offer other formats which can be obtained through content negotiation (HTTP headers) or by offering alternate URLs in message metadata. In addition to returning the entire message/rfc822 document, with headers as well as text body and other attachments, servers MUST support direct GET access to attachments. This is described further in the section on entries in mailbox feed documents.

The request specifies that the user agent prefers message/rfc822.

Request

Response Date: Wed, 24 Sep 2008 10:01:54 -0400 From: Participant MIME-Version: 1.0 To: ietf@example.com Subject: Re: Call for review of proposed IESG Statement on Examples References: <48D584879F297C944CE2D568@JCK-ACR.example.com> In-Reply-To: <002C61D09CE7B135B04EFCA@p3.example.com> List-Id: IETF-Discussion Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" +1 ]]>

The same request without specifying message/822 format, results in an Atom entry instead. Re: Last Call for proposed IESG Statement on Examples 48DA4852.8080707@example.net 2008-08-24T10:01:54-04:00

+1 ]]>

A list of messages, which may be the contents of a mailbox, is represented as an Atom document with the 'feed' root element. Within 'feed', there are separate 'entry' elements for each mail. In addition, a number of elements directly inside 'feed' describe the feed itself, in this context the mailbox or list of messages. In Atom, the server can choose how many entries to put in a feed document.

With this basic structure explained, here is an example:

INBOX 123456789321 Short notes to help you set up your presentation 000e0cd47d9@example.org 2008-10-18T05:08:13-07:00

Dear Jim, Here a few short notes, based on our experiences so far, to help you set up you ...

Dear Jim, Here a few short notes, based on our experiences so far, to help you set up your presentation: * Please do not move, resize, close, or open pods in the rooms. The rooms have been setup for the conference. * If you haven't had a chance to, please watch the speaker orientation video, it will answer a number of common questions. alvin@example.org alvin@example.org Re: Intro: Web Hooks 4304feb30810181219s322f27bcs570b8c80826a7da1@example.com 2008-10-18T12:19:45-07:00

Joe, Let's setup a time to talk about the role. we don't have many slides believe it or ...

Joe, Let's setup a time to talk about the role. we don't have many slides believe it or not :-). Red Red Herring red@example.com ]]>

If the listing of messages maps directly to the contents of a single mailbox, the feed title MUST be the mailbox name. In IMAP this is limited to UTF7 and will always be a valid Atom title. Example: "FORK" If the listing of messages is from a search or filter, the feed title should contain a descriptive name. Example: "Mail filtered by Sender containing 'leslie'"

Required by Atom to be permanent and universally unique. Since IMAP allows users to RENAME mailboxes, it would be best if the id were based off something more permanent than the mailbox name. In gateway use cases where the gateway can't tell the difference between a renamed mailbox and a new mailbox, the gateway MUST treat the mailbox as new. Effectively this creates a new feed (or more than one if the mailbox has inferior hierarchical names) and one or more old feeds are coincidentally removed.

When the listing of messages is from a personal mailbox, the 'author' element SHOULD identify the owner of the mailbox. This helps distinguish from public mail archives. When the mailbox does have an owner or account represented in an 'author' element, the 'email' element MUST contain the full email address. In some cases, the name will contain the email address again (it is used for display), but it MUST be present.

Example author element: Jeffrey Lebowski dude@example.com ]]>

Example for mailbox of messages sent to support@example.com: support@example.com support@example.com ]]>

The feed itself contains several 'link' elements, with different 'rel' attribute values. rel=self: The URL to this feed. rel=alternate: MAY provide an IMAP mailbox URL if this feed represents a mailbox listing.. Other URLs and multiple values are also appropriate. rel=service: The URL to the document that describes server features and other mailboxes (see ). The following link relations are from Feed paging , see : rel=current: The URL to the most recent entries in this feed rel=next: the URL to the next section of entries in this feed rel=prev: the URL to the previous section of entries in this feed

Within the feed, a number of 'entry' elements each contain further elements. The minimal legal Atom entry consists of a 'title', a 'link', 'id', 'updated' and 'summary'.

This MUST contain the core of the message-id used in IMAP and SMTP, and generated by the originator of the message. The 'message-id' syntax in has surrounding whitespace and angle-brackets, which MUST be omitted. Example: <id>d691a2530810071141y757@mail.example.net</id>

This MUST contain the subject of the message. It MUST be empty if the subject of the message is empty.

For email received and unchanged, this contains the date received. Otherwise, it should represent the last time the item was changed. Given that some email systems allow users to modify received messages, the inside of the message needs to be examined to get an accurate date received.

Contains the sent date.

This SHOULD be present and contain a piece of the body of the message. 100 characters or less is RECOMMENDED.

The link elements provide both a link to a machine-readable version of the complete email, to support rich and smart applications, and a human-readable version, so that this interface can be reasonably browsed in extant newsreader software. Newsreaders display the rel=alternate link as the main link to the entry itself. Attachments and other messages related by threading should also be represented with link elements. The 'self' link is used for the link to the message in its default human-readable format. At least one 'alternate' link provides access to the message/rfc822 format. Other formats can be provided in additional 'alternate' links. Attachments are linked as enclosures. Maps directly to messages in the "References" header. Maps directly to messages in the "In-Reply-To" header. Defined in The 'related' and 'in-reply-to' messages might be available on the same server store, but they might not be. If the replied-to message is available in the same store, the URL MUST be an HTTP URL to the message entry (which gives links to the message/rfc822 format and other metadata). Otherwise, the URL MAY be a 'mid' URL, see .

Example Servers MUST support Feed History Paged Feeds (Section 3.). This gives servers the control over how many messages to send to clients in one feed document. Clients make further requests if more of the feed history is desired, though in many use cases, only the most recent feed page is needed. Paged Feeds do not quite allow for synchronization or ensuring that all mails have been seen. Archived feeds would, but are inappropriate for mailboxes because messages are frequently removed from mailboxes (and even if tombstones were used, an inbox might quickly fill up with thousands of tombstones and only a few undeleted messages). A TODO for this draft is to consider whether a simple extension to paged feeds could provide the message-ID for the last message on the previous page, and the first message on the next page. By matching these up, clients could see that all messages had been seen. Clients can usefully display a screenful of entries at a time. This varies highly depending on the display size, screen resolution and even font size chosen. Twenty is an absolute minimum number, whereas a forty to fifty entry display is a more common capacity. Servers are RECOMMENDED to put at least fifty entries in a page, unless fewer than fifty entries are available. Servers are RECOMMENDED to apply paging if there are more than a hundred entries in a full logical feed.

Links directly to the attachments on the message MUST be shown in the entry as 'link' elements with rel=enclosure.

Clients need to be able to find out which mailboxes exist. The server creates a feed of feeds, a document that lists all mailboxes in a flattened namespace, along with other related feeds. For example, a feed for all unread messages might behave as a "saved search", accessible to even very simple clients because it acts like a mailbox feed. This kind of feed, listing both mailboxes and other feeds, is not suited to clients making changes, reorganizing email or authoring email. The Atom Service document format is more suited to that purpose, because along with listing workspaces and collections, it indicates what kinds of resources may be created in those workspaces and collections. However, this approach is simpler and does not preclude more careful design of a mapping to Atom Service documents in the future.

progrium@example.com 2008-10-18T05:08:13-07:00 John Doe progrium@example.com urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af7 INBOX 123456789321 2008-10-18T05:08:13-07:00 personal 123456789323 2008-10-08T04:17:02-07:00 All unread mail 123456789322 2008-10-18T05:08:13-07:00 ]]>

Jeff Lindsay helped with initial modeling, early review and a very early prototype IMAP-HTTPMail proxy.

No IANA Considerations arise in this document.

The biggest concern in offering HTTP access to mail stores is probably dealing with authentication differences between HTTP and IMAP/POP. HTTP authentication mechanisms are not as powerful as SASL. Solving this problem is both an implementation and an administrative concern. Common practice demonstrates that this concern can be met in implementations and deployments, at least to the satisfaction of administrators and users. For example, the same password is commonly supplied in an HTML login form, and sent to the server over TLS, that the user types into an IMAP client interface for use in a SASL authentication interaction. Naturally this approach weakens the security of that password, and habituates the user to typing it into Web interfaces. This specification does not worsen that common situation. Privacy concerns dictate use of TLS in most cases. Clients MUST support TLS, so that servers are free to secure private email in the confidence that this will interoperate.

&RFC2026; &RFC2045; &RFC2046; &RFC2109; &RFC2392; &RFC2616; &RFC4287; &RFC4346; &RFC4685; &RFC5005; &RFC5023; &RFC5322;