I assume that we're going to use Sun's javamail APIs. I also assume we're going to be able to use very little of their actual implementation, unless we get to copy their code and hack it to death. We have ambitious performance goals, and also support of things like intertwingle and RDF will probably require changing most of the implementations.

I assume that we'll end up supporting RDF. I assume we'll have RDF objects corresponding to folders, messages, and the important message headers. I assume we will NOT have RDF objects corresponding to MIME parts, although I know there are people who think this real important.

The diagram

Any document like this needs a diagram. Here's mine. (Click on it to see a larger, more readable version.)

This shows the basic relationship between most of the important packages within grendel:

The grendel.storage package implements javamail APIs for four different kinds of message stores: POP3, Berkeley mail folders, IMAP, and NNTP.
Underneath the message stores, the grendel.storage.intertwingle package maintains the intertwingle database describing what messages are in any of the stores. This database is used to support RDF functionality, and to allow efficient searching and filtering.
The grendel.filters package presents javamail Folder objects that represent the results of applying a filter on some set of the existing folders. Depending on the filter, this will work either directly on the existing Folder objects or by querying the intertwingle database.
The grendel.view package takes a Folder object (generally obtained from either the grendel.storage or grendel.filters package) and sorts or threads the messages. It directly supports most operations that will be needed from the UI.
The grendel.ui object sits above everything, and implements our user interface.
Third parties may write their own applications or UIs, that call into any of the above packages. They may also implement their own javamail Folder objects, and our entire system should work with those objects.

There are many important pieces of Grendel that do not show up in this diagram. This includes:

The grendel.addressbook package, which implements the user's address book and speaks to LDAP servers.
The grendel.composition package, which presents the user interface for composing new messages. This will interact with the grendel.addressbook and grendel.storagepackages, as well as using Alchemy to implement message editing.
The grendel.mime package, which implements MIME parsers and implements Magellan content handlers for MIME messages and other messaging-related MIME types.

Issues of global scope (affecting all packages)

Reusable

customizable

Scriptability

Yeah, we want it. I wish I understood better what we have to do to get it. I'm hoping just publish a list of classes and methods somehow?

Package grendel.storage

This package is where we implement most (all?) of the javamail API. Private to this package are any details about particular protocol implementations. All the differences between NNTP, IMAP, POP, etc. are packaged away in here and hidden. This also implements the RCFdata interfaces; that is, this is the stuff that supports RDF queries.

One problem that comes from using the javamail API throughout is that it makes it difficult to innovate and optimize. For example, javamail does not provide an API to get the number of undeleted messages in a folder. This is a number we want to display in our UIs. We can't just add a new API to our subclasses of Folder, because we want to work with third-party implementations of Folder as well. And while it's possible to determine this number using the existing APIs, it is a very slow and expensive process to do so.

To solve all this, we will define a FolderExtra interface. FolderExtra has all the extra APIs we need, like getUndeletedMessageCount(). Our implementations of Folder will also implement the FolderExtra class.

When we have code (e.g., our UI code) that is given a Folder object and wants to call a method on FolderExtra, it gets a FolderExtra object by calling FolderExtraFactory.Get(). FolderExtraFactory determines if the given Folder directly implements FolderExtra. If so, it returns that object. If not, it creates a new object which implements all the FolderExtra methods in terms of methods available on Folder.

This all makes things a bit convoluted for callers. But it means we can use our fast code for our own Folders, and be assured that things will work (though slower) on Folders provided by other parties.

Similarily, there is a MessageExtra interface and a MessageExtraFactory. Other Extra interfaces will be invented as necessary.

It would be nice if we could just use the pk.core.Supports interface for this kind of stuff. However, this strategy only makes sense if the base Folder object implements Supports. Since Supports is one of our own interfaces, and Folder comes from javasoft, I can't see this happening.

The grendel.storage package is where the database problem lives. Do we use summary files, like 3.0? Do we use some general database? Here is the current thinking: we will use something like 3.0 summary files. A background thread will attempt to maintain all changes in a general-purpose database (see the grendel.storage.intertwingle package). This database will only be used for fancy functions (cool searches and filters). If the database blows up, you'll still be able to read your mail; if the database works well, you'll be able to do cool stuff too. If we find that the database is just too expensive or too hard to maintain, then we'll still have a product even after we rip it out.

Package grendel.mime

This is where we put in our MIME parsing code. Maybe this is just an implementation of javamail APIs; maybe it is our own version, since we have suspicions that javamail's won't be good enough, and their APIs won't fit the problems we're trying to solve.

The message/rfc822 parser also goes here, maybe. And the Magellan content handlers, that know how to display various messaging-related MIME types in a Magellan window also go here.

Package grendel.filters

These are classes that take Folder objects and use them to create other Folder objects. The idea is to end up with a Folder object that acts just like any other javamail Folder object, but the messages in it can actually come from any number of real Folders, and are chosen via some constraint.

Package grendel.view

This is what gathers together the messages in a Folder, calculates a sorting/threading order on them, and allows for a UI to present them.

Much of the UI work is in the ``thread list'' -- a scrollable list of messages, possibly organized in a threaded tree, which the user can scroll through and select and drag and manipulate. Most of the smarts of the ``thread list'' are really implemented here in the grendel.view package.

Interfaces to the view package

There are three interfaces defined: the view, messages within the view, and observers of the view.

MessageSetView

public interface MessageSetView {
  /** Gets the root of the tree of
      messages that are being viewed. */
  public ViewedMessage getMessageRoot();
  public void setSort(SortOrder);
  public SortOrder getSort();
  public void setIsThreaded(boolean b);
  public boolean isThreaded();

  public int getMessageCount();
  public int getUnreadMessageCount();
  public void setFlags(Message msgs[],
		       Flags flag,
		       boolean value);
  public boolean copyMessages(Message msgs[],
			      Folder folder)

  public void addObserver(MessageSetViewObserver obs);
  public void removeObserver(MessageSetViewObserver obs);
}

When a view is created (via the appropriate factory), the filtering rules that determine which messages are part of the view are specified. To change the filtering rules, a new view must be created.

Once the messages are determined, however, the ordering and threading can be controlled. (Need to figure out exactly what the SortOrder class looks like.)

The number of messages, and number of unread messages, can be obtained through the getMessageCount and getUnreadMessageCount calls. The setFlags and copyMessages calls do the same things that the routines javamail's javax.mail.Folder class does, but lets the user of the view ignore the fact that the messages in the view may in fact be from different Folder's.

Any time the set of messages changes, or the order is changed, registered observers will get notified. (The set of messages can change when new messages appear or disappear from the underlying message store.)

ViewedMessage

This represents one message in a view. It is just a pointer into the storage representation of a Message, as well as pointers to other messages in the list or tree for this view.

public interface ViewedMessage {
  /** Gets the view that this
      message is a part of. */
  public MessageSetView getView();

  /** Gets the message itself.  You
      need to go through this to find out
      stuff about the message (like its
      subject, author, etc.). */
  public Message getMessage();

  /** Returns the parent of this
      message.  (This is always null
      unless threading is turned on in
      the view.) */
  public ViewedMessage getParent();

  /** Returns the first child of this
      message.  (This is always null unless
      threading is turned on in the view.) */
  public ViewedMessage getChild();

  /** Returns the next message.  This
      is the next message that has the
      same parent as this message.*/
  public ViewedMessage getNext();
}

Note that there is no random-access method of jumping around in the view. Do we need one?

MessageSetViewObserver

The grendel.view package does not implement this class; rather, users of the package will provide implementations.

This is probably too simplistic. We should probably imitate the javamail observer classes instead.

public interface MessageSetViewObserver {
  /** Some messages changed.  Each 
      enumeration parameter might be null,
      or it might be an enumeration of
      ViewedMessage objects.

      @param inserted   new messages that have
                        appeared

      @param deleted    old messages that are no
                        longer considered part
                        of the MessageSetView

      @param changed    Messages that have had
                        their internals tweaked
                        in some way
   */
  public void messagesChanged(Enumeration inserted,
			      Enumeration deleted,
			      Enumeration changed);
};

Factories in the view package

There will be factories to generate the different kinds of views. One important view is the FolderView, which just shows every message in one storage folder:

public class FolderViewFactory {
  static public MessageSetView Make(Folder f);
}

There will be another factory that take filter specifications and maybe a set of Folders and create a view that show messages from within those folders that match that filter. This needs to be hashed out more when filters are hashed out more.

Package grendel.ui

This is enormous. Much of the work is to create the outline view of a set of messages, although much of that work is split off into the grendel.view package.

Package grendel.composition

This is the UI to compose messages. It uses Alchemy as the editor; alchemy needs to support a plaintext mode for us, as well as full HTML editing.

It's tempting to think of compositions as totally isolated from the rest of Grendel, but some important ties exist. Compositions get their headers initialized from other messages when you do a Reply or Forward. Quoting needs to work to suck in the text of an original message. When you send a message, you often want to annotate the original message that it was replied-to or forwarded. When you send a message, you want to append a copy of that message into the ``Sent'' folder. You need to save and restore messages into a ``Drafts'' folder. We also need to interact with an ``Outbox'' folder for offline compositions.

Package grendel.addressbook

This talks to LDAP, and provides a nice API to all things dealing with addressbook.

Package grendel.storage.intertwingle

There are lots of spiffy ideas in intertwingle but the one I'm focusing on here is just to keep a global database optimized for interesting searches across all folders. It would be nice if we had a simple, unified database story that solved this and all our other concerns. However, if we don't, another idea is to create a separate database that gets updated in the background.

Pieces needing a package
(i.e., pieces that need more thought)

In the current picture, all of these things would get lumped into grendel.ui. But they aren't very directly UI-oriented, and we would like to split them out into their own places.

Incorporation filters

An incorporation filter is a filter with an associated action that changes something about the message, or otherwise takes a concrete action. Actions include moving the message to a folder, or deleting it, or forwarding it to another user. The action only happens when the incorporation filter finds a message that matches its criteria. Incorporation filters get their name because they are generally invoked on new messages that are delivered to the user. However, it is also useful for the user to be able to manually invoke an incorporation filter on a selected set of messages.

Display filters

A display filter is a filter that changes some displayable attribute of a message. The filters described in grendel.filters are a simple kind of display filter: they control whether a message is to be displayed or not. But we would also like make filters that control other display attributes: what color to highlight this message with, or tweak a ``score'' associated with the message that would affect its sorting.

Biff

Biff is the lingo for ``check for new mail.'' Unlike Communicator 4.0, we would like new messages to automatically get fetched, without the user having to do a manual ``Get New Mail'' operation. This will then need to run the user's incorporation filters on the new messages, and decide how to alert the user about the new messages.

grendel architecture (a 10,000-foot view) by Terry Weissman 6-Nov-1997

Assumptions