Date:	Sat, 16 May 1998 10:20:37 -0700
From:	Jamie Zawinski <jwz@mozilla.org>
Newsgroups:	netscape.public.mozilla.mail-news
Subject:	mail filtering and kill files
Message-ID:	<355DCAE5.3F586E3D@netscape.com>

So I believe there are two kinds of filters that are useful: those that control the routing (initial physical location) of newly-arrived messages; and those that control presentation of sets of messages (kill files being the simple case of, ``don't present these messages.'')

Here's part of a thread some of us had about this quite some time ago. This conversation wasn't in the context of Netscape Messenger, so some of the terminology below might not quite fit, but this should explain what I think about the two types of filters.

We have been using the term "filter" to mean two entirely different things. Many people don't realize it.

I want to make it clear that they are different things, so that we can be talk about each of them. I want to give each of them a unique name.

The two kinds of filters differ on when they're executed, and what they do:

• Filters that happen at some time (get new mail, user manually fires over a range of messages) that affect stored message properties. These filters do things to the message, like
  • Move messages to another folder
  • Delete message
  • Change message priority
  • Automatically generate reply or forward

• Filters that compute some dynamic property on the message. These filters are usually executed when computing a view to display. They don't change anything at all about the message itself. Some example of dynamic properties computed might be
  • Should we include this message in the current view at all?
  • Determine "score" to base sorting on
  • What color should we use to display the thread line for this message

Ok, what is the difference between automatically moving messages or generating a reply or forwarding without user intervention from an agent?

It's just a matter of your perception. I don't think of an agent as something that only wakes up when I do a particular operatoin (getting new mail). Of course, getting new mail itself is probably going to (optionally) start happening automatically, and I guess that makes the whole thing more agent-like.

I don't think it's just perception. One type of filter actually modifies the message (or the place where the message is stored.) The other type controls how a set of messages are displayed, or displayed relative to each other.

Put another way, one alters intrinsic properties, and the other alters (actually, generates) extrinsic, or derived, properties.

I like the terms ``Incorporation Filters'' and ``Display Filters''.

Examples of Incorporation Filters:

• if From contains "@netscape.com", move to folder "work".
• if Received contains "205.199.212.", move to folder "spammers".
• if To contains "all@netscape.com", move to Trash.

Examples of Display Filters:

• if From contains "loser@some.host", omit it from the list.
• if From contains "winner@some.host", increase score so that it moves closer to the top of the displayed list.
• if Subject contains "foo", decrease its score.
• if size 100k, decrease its score a lot.
• if priority = 3, display it in red.

Incorporation Filters would be associated with a particular maildrop: with a particular POP3 server, or with an IMAP inbox (as distinguished from what we call the Inbox folder today.) They would be associated with the ``Get Mail'' action (even in cases where that action is an implicit background operation.)

Display Filters would be associated with a particular view, or with a particular folder (or with all views/folders.)

I think these are disjoint sets; I don't think it makes sense for folder rules to have ``move to'' operations, and I don't think it makes sense for maildrop rules to have ``omit'' or ``score'' operations.

(I believe Eudora's two kinds of filters, ``automatic'' and ``manual'', are both Incorporation Filters; they both alter the messages, the difference being whether the alteration occurs as a result of Get Mail or whether it is triggered by explicit user action.)

I like the terms "Incorporation Filters" and "Display Filters".

Jamie is right-on about all of this, and I like his names, except for:

• ``Incorporation Filters'' can do more than just move to a folder. For example, they can tweak the message (change its priority, for example), and they can cause e-mail to be sent (though we can have a separate big discussion about why this is a really scary idea).

• ``Incorporation Filters'' need to be able to be run manually, too. (This is a big 4.0 feature request; I think it will probably be done in 5.0.) I guess the name is still workable, but this is a bit confusing.

``Incorporation Filters'' can do more than just move to a folder.

Perhaps, but even so, they're still run at incorporation-time rather than view-time.

For example, they can tweak the message (change its priority, for example),

First, is there another ``for example'' in there? Because I think that ``priority'' is the only example, and further, I think (now) that changing the intrinsic priority header is the wrong approach.

Second, the reason that people want to change the priority header is that they want absolute control over the priority of their messages: just because one of my correspondents says a message is important doesn't mean that I think it's important, and my mailbox viewer is in my service, not theirs.

So, my take is, the way to handle this is to let the priority header in the message itself (in the disk file) be whatever the sender set it to. But, express my preference for believing, ignoring, or otherwise altering the priority at display time rather than incorporation time.

The end result to the user is the same; but by eliminating the need for incorporation filters to do things other than pick destinations, we simplify the distinction between the two.

It's easy to explain that score is the property that controls the ordering of messages in the list, and Priority is a header of the message, just like Subject, that may be used to influence the score. I think that special-casing Priority in some way would be... complicated.

``Incorporation Filters'' can do more than just move to a folder. [...] they can cause e-mail to be sent (though we can have a separate big discussion about why this is a really scary idea).

One can still look at this as a ``destination.'' Rather than the destination being a folder, the destination is a command of some kind.

``Incorporation Filters'' need to be able to be run manually, too. (This is a big 4.0 feature request; I think it will probably be done in 5.0.) I guess the name is still workable, but this is a bit confusing.

This is the Eudora thing. Maybe the name doesn't quite fit, but I think the concept fits fine... And the name is probably close enough.

I think of this ``manual filters'' thing as a way of debugging the filters. Is it something else?

I think of this ``manual filters'' thing as a way of debugging the filters. Is it something else?

Well, the debugging is an important case.

After I get tired of having Xena mail clutter my Inbox, and I make an incorporation filter to cause all Xena mail to get shuffled off into its own Xena folder, the very next thing I want to do is run it over my entire Inbox, to clean all the junk that has already accumulated.

I could imagine a filter that says ``if this message is more than a week old put it in my `archive' folder.'' And then I manually run this filter every so often to clean up. (Or I run it automatically from my calendar; whee!)

Those are all good things. But here's another idea...

What if, instead of this idea of ``running filters manually'', we had the idea of transforming filters to and from search results?

So, I want to add a new filter for my Xena mail. First, I do a cross-folder search for ``Return-Path contains xena'' and see what comes up.

Then, bonk the ``turn this search result into an incorporation filter'' button. (Or drag the icon over to the inc-filter-drop-pocket or something.)

Then (here's the manual filter part), I select all of the messages and drag them into the ``xena'' folder. (Assuming that drag means ``move'' and not ``copy''.)

And it can go the other way, too -- I can take an inc-filter and say ``show me all messages that would be affected by this filter.'' Which really means, ``take the contents of this inc-filter and compose a search-term out of it.''

I don't necessarily think that the idea of ``run a filter manually'' is a bad thing, but I wonder if building it up, as a composite command, from searching and refiling, might be more appropriate or easier to understand.

In Eudora, people have filters marked as ``manual only''. In Grendel, these might merely be ``saved queries'' on which one can easily perform a ``refile'' command (among any number of other commands.)

(Note: when I talk about ``search results'' I mean the concept of doing a search and seeing a ``virtual folder'' that contains the messages which matched the search. A ``search term'' would be the thing from which one was able to construct search results.)