TapestryProtoDoc.tioga
Doug Terry, July 5, 1990 11:01:52 am PDT
Theimer, April 24, 1990 11:38 am PDT
Brian Oki, April 4, 1991 4:54 pm PST
Tapestry Prototype
CEDAR 7.0 — FOR INTERNAL XEROX USE ONLY
Tapestry Prototype
preliminary support for mail filtering and annotation
Doug Terry
© Copyright 1989 Xerox Corporation. All rights reserved.
Abstract: The Tapestry protoype provides preliminary support for mail filtering and annotation. The exact features are constantly evolving...
Created by: Doug Terry
Maintained by: Doug Terry <Terry.pa or Terry:PARC:Xerox>
Keywords: mail, filtering, annotation, databases, Tapestry, BlackCherry
XEROX  Xerox Corporation
   Palo Alto Research Center
   3333 Coyote Hill Road
   Palo Alto, California 94304

For Internal Xerox Use Only
Tapestry Filtering in BlackCherry
Introduction
The Tapestry prototype implements filtering agents that can be used to automatically annotate messages. Currently, these agents can be used in conjunction with the BlackCherry mail reader to filter and prioritize incoming mail. If you are interested in becoming an "alpha" user of these facilities, please read on...
Caveat: These facilities are still very rudimentary. Alpha users are solicited at this time to help debug and develop the ideas. Do not be surprised if you find bugs or severe limitations. Any and all suggestions for improvement will be greatly appreciated, but may not be instantaneously implemented.
Reassurance: Using the Tapestry filtering facilties with BlackCherry will never cause you to lose messages, even if Tapestry has horendous bugs that lock up your screen and do other nasty things. This is because messages are handed to the filtering agent only after they are written to BlackCherry's log and flushed to disk. If you trust BlackCherry, you can trust TapInBlackCherry as well.
Becoming an alpha user
To get started, perform the following steps:
1. (DCedar step only) Bringover TapestryProto.df into a convenient directory, e.g.
Pushv Commands
Bringover -p /CedarChest7.0/Top/TapestryProto.df
Pop
2. Create a LoganBerry database to store your filters and one to store your annotations. For instance, you might want to call these "TapFiltersLB.df" and "TapAnnotationsLB.df". The simplest way to to this is to use the LBCreate command, e.g.
LBCreate TapFiltersLB.df FID User Name
LBCreate TapAnnotationsLB.df AID MsgID User FID Level
Alternatively, you can copy these from the /Tapestry/Top/ directory and copy empty log files from /CedarChest7.0/LoganBerry/, e.g.
PSAdd Tapestry [Eich-NFS]<vert>Tapestry> [Eich-NFS]<vert>Tapestry>
Copy TapFiltersLB.df ← /Tapestry/Top/TapFiltersLB.df
Copy TapFilters.lblog ← /CedarChest7.0/LoganBerry/Empty.lblog
Copy TapAnnotationsLB.df ← /Tapestry/Top/TapAnnotationsLB.df
Copy TapAnnotations.lblog ← /CedarChest7.0/LoganBerry/Empty.lblog
In either case, you should then edit the DF files so that the files go to an appropriate place when you Smodel them. Remember to smodel these databases periodically or you may regret it if you have a local disk crash (this mostly applies to DCedar users).
Once you have the schema and log files created, you need to build the indices, e.g.
LBBuildIndices TapFiltersLB.df
LBBuildIndices TapAnnotationsLB.df
3. Add entries to your user profile that specify the database names and your user name, e.g.
Tapestry.FilterDB: ///Users/Terry.pa/Tapestry/TapFiltersLB.df
The LoganBerry database containing filters used by Tapestry.
Tapestry.AnnotationDB: ///Users/Terry.pa/Tapestry/TapAnnotationsLB.df
The LoganBerry database containing annotations added by Tapestry.
Tapestry.UserName: Terry.pa
The current user's name.
Tapestry.DocFreqName: ///Users/Terry.pa/Tapestry/DFs.tapestry
The file containing frequencies of words in documents for Tapestry.
4. Add some filters, i.e. queries and associated annotations, to your filter database. This can be done using the TapAddFilter command described below. Or you can use the LoganBerryBrowser to update the database. Or, if you are a real LoganBerry wizard, you can edit the log file.
It is okay to start with empty databases. It is also okay to start with a non-empty BlackCherry log. In this case, none of the existing messages will have associated annotations; that's no big deal.
Note that your databases can reside on LoganBerry servers if you like. However, in this case, your step #2 will be slightly more complicated.
Running Tapestry with BlackCherry
Instead of invoking BlackCherry you invoke
Tapestry
It's that simple! The install file for TapInBlackCherry (or .require file in PCedar) installs BlackCherry, LoganBerry, and the Tapestry prototype code. The "Tapestry" command invokes BlackCherry with its command-line arguments passed through. Thus, for example, invoking Tapestry -d will cause invocation of BlackCherry -d.
What's new?
The behavior of BlackCherry with TapInBlackCherry installed differs from the standard version of BlackCherry. These differences are described below. For the most part, they augment rather than change BlackCherry's functionality. In particular, all menu operations and mouse clicks work the same as they always did. See BlackCherryDoc.tioga for a description of the basic BlackCherry operations.
The message set displayer
Each message has an associated "interest level". An interest level is a number between 0 and 100 (actually it could be any number). The interest level of a message is intended to denote the user's level of interest in the message. The default interest level for messages is 50.
Interest levels are attached to messages by the filtering agent. An interest level is an annotation of type "Level" with an integer value, e.g. Level: 75. If a message has several annotations of type "Level", then the maximum value is taken as the message's interest level. An interest level is simply one of many possible types of annotations. It is the only one that BlackCherry looks at.
When TapInBlackCherry has been installed, the BlackCherry message displayer looks almost identical to a vanilla BlackCherry displayer. There are two main differences. First, each message's "table of contents" is preceeded by the message's interest level. For instance, a line of the displayer may contain "70 23 Feb 90 Mark Weise... parcpad, ubicomp". The "70" is the message's interest level. Second, the messages are listed in interest-level order. That is, the highest level messages are at the top and the lowest level messages are at the bottom.
As new messages are retrieved, they are added to the bottom of the displayer. Those messages within a group of messages that are retrieved at the same time are displayed in interest-level order. That is, if 5 new messages are retrieved, those 5 messages will be sorted in order, but they will not be interspersed with previously retrieved messages. (I'm not sure if this is ideal behavior.)
If you would like to see all of the messages displayed strictly in interest-level order, then you can destroy the displayer and retype "BlackCherry" to a CommandTool. (Eventually, there may be a menu operation for doing this and other useful things.)
New menu entries
When you install TapInBlackCherry, a new pop-up menu is added to BlackCherry's main menu. This menu item, labeled "Filters" has several sub-menu operations:
InterestLevel?: Prints the set of annotations for the currently selected message in BlackCherry's typescript. This enables you to figure out why the messages has the given interest level.
DropConv: Creates a new filter such that any future messages with the same subject as the current message (except for possibly a leading "re: ") are given a low interest level, settable by the user. Clicking this button results in a pop-up menu labeled Interest, which lists values from 5 to 95 inclusive, in increments of 5, along with the system "default" and the "original" value if there is one. Clicking outside the menu or waiting indefinitely aborts the entire operation. The system default is "50." In selecting the interest level to "drop" to, the value chosen must be less than the current value; anything greater than or equal to aborts the operation.
BoostConv: Creates a new filter such that any future messages with the same subject as the current message (except for possibly a leading "re: ") are given a higher interest level, settable by the user. Clicking this button results in a pop-up menu labeled Interest, which lists values from 5 to 95 inclusive, in increments of 5, along with the system "default" and the "original" value if there is one. Clicking outside the menu or waiting indefinitely aborts the entire operation. The system default is "50." In selecting the interest level to "boost" to, the value chosen must be greater than to the current value; anything less than or equal to aborts the operation.
BoostSim: Creates a new filter such that any future messages that are "similar" to the current message and whose similarity metric exceeds a user-defined threshold value "match" successfully. Clicking on the Filters BoostSim buttons results in a pop-up menu, the first is labeled "Threshold" and the second is labeled "Interest". "Threshold" menu lists values from 5 to 95 inclusive, in increments of 5, along with the system "default" and the "original" value if there is one. Threshold means that if the computed similarity metric exceeds the threshold value, the user is indeed interested; otherwise, he isn't. "Interest" means that messages that exceed the threshold value can be annotated with an interest level set in the second pop-up menu. The user must pick values for both threshold and interest. It is impossible to pick a threshold and not an interest level, or vice versa. Clicking outside the menu or timing out aborts the entire operation. In selecting the threshold value to "boost" to, the value chosen must be greater than to the current value; anything less than or equal to aborts the operation. The default value for threshold and interest is "50."
DropSim: Creates a new filter that drops the similarity threshold value to a value less than the current value, if there is one. The user can also set the interest level. The default value for threshold and interest is "50."
These are clearly not a complete set of desired menu items. In fact, you might not like them in their current form. The intention is to get started somewhere. If you have suggestions, and especially if you want to help implement new operations or a better user interface, great!
Usage hints
I generally use BlackCherry to read my mail and delete all of the messages that I don't want to keep for any length of time. Periodically, say once a week, I transfer my BlackCherry mail into Walnut for long-term storage. When I first start BlackCherry, I use the "-n" flag so that it does not immediately retrieve new mail, e.g. "BlackCherry -n". After BlackCherry starts up, I then explicitly click on "NewMail". This way, I can see all of my new mail at the bottom of the message set displayer.
CommandTool Commands
Overview
Currently, only a few commands have been implemented. These commands are described below. There should exist more commands and these existing commands could be better. Do you have suggestions?.
TapAddFilter
syntax
TapAddFilter name {field(matcher): pattern}* -> {tag: value}*
description
Adds a new filter to the filter database. The arguments are as follows:
name is an identifier intended as a short description of the filter.
{field(matcher): pattern}* is a query specification, i.e. a LoganQuery attribute-pattern specification as described in LoganBerryToolsDoc.tioga.
-> separates the query specification from the annotation
{tag: value}* is an annotation, i.e. a list of LoganBerry attributes, that is associated with the given query.
examples
% TapAddFilter msgsFromDoug from: *Terry* -> Level: 99
% TapAddFilter onDistributedSystems text(wildcard): "*distributed* system*" -> MsgSet: DistrSys Level: 80
% TapAddFilter dontWantToSeeAnotherMsgOnBathroomFixtures to: HomeImprovement^ subject(re): "(Re':| )*Bathroom Fixtures" -> Level: 0
TapDeleteFilter
syntax
TapDeleteFilter filterName
description
Deletes a filter from the filter database, if the filter exists. The argument is as follows:
filterName is an identifier that briefly describes the filter
example
% TapDeleteFilter msgsFromIsaac
% TapDeleteFilter msgs
Can't delete filter: 'msgs.' NonExistentFilter - Cannot delete non-existent filter
TapListFilter
syntax
TapListFilter {filterName | }
description
Lists filters in the filter database. The arguments are as follows:
filterName is an identifier that briefly describes the filter
no argument means list all filters, if they exist.
examples
% TapListFilter msgsToBOki
Filter name: msgsToBOki
User: BOki
Query: to(re): "(#*Brian Masao Oki#*|#*Brian Oki#*|#*BOki#*)"
Annotation:
Level: 90
% TapListFilter msgs
Can't list filter: NoSuchFilter - Filter doesn't exist
Implementation Details
Overview
The Tapestry implementation is divided into several basic modules. It uses LoganBerry databases to store vital information. These modules and databases are discussed in this section.
Messages and message queues (TapMsgQueue.mesa)
Messages are lists of fields where each field has a type (tag) and a value (contents). For instance, a message header is parsed into a list of fields. The message body is often a single field with type $Text; however, structured messages may be parsed into several fields. Messages are defined to be compatible with LoganBerry entries. TapMsgQueue.mesa provides operations for converting between messages and LoganBerry entries. The operations defined in LoganBerryEntry.mesa can be used for extracting the contents of a message field and for manipulating messages in various ways. TapFilter.mesa provides an operation for parsing a textual message contents into a structured message.
Message queues, as defined in TapMsgQueue.mesa, are the basic facilities for passing around messages in Tapestry. Messages can be read from a queue and written to a queue. Queue's can also be laziliy evaluated, i.e. the contents are generated as requested. While the basic operations on a message queue remain the same, they can be implemented in different ways by different "classes". Message queues are modeled after LoganQuery's complex cursors.
Filters (TapFilter.mesa)
Filters are associated with particular users and stored in a LoganBerry database, e.g. TapFiltersLB.df. Filters are queries in the form of LoganQuery attribute patterns and their associated annotations. Each entry in the filter database is a tuple of the form <$FID: user$filtername, $User: user, $Name: filtername, $Query: query, annotation>. An "annotation" is an arbitrary list of type: value pairs. Several entries may exist in the database for each user.
Annotations (TapFilter.mesa)
Annotations are associated with particular messages and stored in a LoganBerry database, e.g. TapAnnotationsLB.df. Each entry is a tuple of the form <$AID: msgID, $FID: filterID, annotation, ..., $FID: filterID, annotation>. Any number of annotations may exist for each message. Only a single entry may exist in the database for each msgID.
Filtering agents (TapFilter.mesa)
Agents process a queue of messages by running them through a set of filters. If a message "passes" a particular filter, then the annotation associated with that filter is attached to the message. Once an agent is created, it exists until it is terminated. When an agent discovers that there are no more messages to be processed, it goes to sleep, that is. into the idle state. An agent may periodically wakeup and look for more messages or it may be explicitly awakened. Operations for creating, wakening, monitoring, and terminating filtering agents are defined in TapFilter.mesa.
Input sources (TapInput.mesa)
Not yet implemented.
Storage of message (TapStorage.mesa)
Not yet implemented.