1. Introduction
WallTapestry is the Walnut mail reader that has been adapted to use LoganBerry as its database manager and has been ported to run in PCedar on SUN workstations. The basic Walnut functionality has been augmented by message appraising, a term that includes filtering, prioritizing, sorting, and other means for automatically organizing messages. For historic reasons, "appraising" is sometimes called "filtering" in the documentation and code. If you have not used Walnut previously, see WalnutDoc.tioga for information on how to use the basic Walnut facilities. This document provides information on how to use the new message appraising capabilities. It also explains how to get started using WallTapestry.
WallTapestry is different than the Tapestry clipping service. Tapestry's clipping service selects messages from a stream of NetNews articles and other information sources by running user-specified filter queries over the messages. The messages matching a user's filter queries are then sent to the user via electronic mail. WallTapestry may be used to read messages sent by the Tapestry clipping service along with personal mail and messages that arrive via distribution lists. WallTapestry's appraisers, used to prioritize messages, are independent of Tapestry's filters, used to select messages. The two work together to provide personalized message delivery and organization.
Warning: WallTapestry does not have the benefit of transactions (something that Walnut relied on). The major result of this is that inopportune machine crashes may leave your database in an inconsistent state. You will likely have to scavenge after any such crash. But, since Walnut keeps the truth in an append-only log file, it is unlikely that you will lose any real data (unless, of course, you encounter an NFS or PFS or other system bug). I am still thinking about how to make WallTapestry more robust. Again, any contributions of ideas or programming time would be appreciated.
2. Becoming a WallTapestry User
If you are just starting out as a WallTapestry user, there are several steps that you need to take. These steps supercede the "Becoming a User" section of WalnutDoc.tioga. assume that you are a past Walnut user, i.e. that you have a Walnut database that you used with DCedar. You should follow these steps carefully:
Step 1. Get your old Walnut files or create new ones
If you are a past Walnut user, i.e. you have a Walnut database that you used with DCedar, you need to copy all of your Walnut files to either an NFS server or to your workstation's local disk. Unfortunately, your Walnut files were most likely on an Alpine server that no longer exists. You may be able to get them from some archive, or just pretend to be a new user. The files that you need are:
Walnut.LogX
Walnut.LogY
Walnut.msgSetButtonsFile
Walnut.NewMailLog
Walnut.ReadArchiveLog
Walnut.root
Actually, you really only need to copy Walnut.root and either Walnut.LogX or Walnut.LogY, whichever is non-empty. The other files are needed but can be created as empty files (they are probably all empty anyway). You do not need the Walnut.segment file!
If you are a new user, create empty files with the above mentioned names. These should all reside in the same directory (preferably one that is backed up). The only file that cannot be empty to start with is the Walnut.root file. Copy WalnutTemplate.root from the release directory as a starting point.
Make sure that the files are writeable. The Cedar copy command seems to create UNIX files with only read permission. You may need to use the UNIX "chmod" command to add write permission to the files.
Step 2. Create a LoganBerry database
You need to create the database that Walnut uses. First, change to the directory in which you want the database to reside. LoganBerry can only run in a "ux" directory. Next, you build a database with either a full or minimal schema. If you want to use Wallaby, the message browser, then you should have a full schema, otherwise you are better off with a minimal schema. (Wallaby works with a minimal schema but it is real slow.)
To create a database with the minimal schema, type the following:
LBCreate WalnutLB.df Key MsgSet ShowIs
LBBuildindices WalnutLB.df
This will give you an empty LoganBerry database in the current directory. The database will have a single log file and three indices (the minimum set).
To create a database with the full schema, type the following:
LBCreate WalnutLB.df Key Date Subject Sender MsgSet ShowIs
LBBuildindices WalnutLB.df
This will give you an empty LoganBerry database with a single log file and six indices.
By the way, LoganBerry schema files are real DF files. This means that you can smodel them to some other place. If you want to smodel a database that you create using LBCreate, you will probably need to edit the DF file to change the directory statements.
Step 3. Edit your Walnut.root file
You need to edit your Walnut.root file so that it points at the LoganBerry database that you just created. Be careful not to introduce Tioga formatting to the file while you are editing it. The root file contains a bunch of entries separated by blank lines. Find the line in the file that says "Database". Change the line below this one so that it contains the full pathname of the database that you created in step 2. For instance, your Walnut.root should contain something like:
Database
/tilde/terry/Cedar/Databases/WalnutLB.df
If you started with an old Walnut root file, find the entries that start with "SegmentID" and "SegmentIndex". Remove these two entries from the root file. They are no longer needed and, in fact, are no longer allowed.
Lastly, if you want to be able to retrieve new mail using Walnut, change the "MailFor" entry. It currently should contain your Grapevine rname; change it to your XNS name (I believe that just having your last name will work for most people, though I'm not sure. I'll look into this.). If you do not care about reading new mail, you can leave the "MailFor" entry as it is; do not remove it.
Everything else in the file should remain the same.
Step 4. Update your user profile
The set of basic user profile options for Walnut has not changed, though new ones have been added (see below). So, you may want to copy the ones that you used in DCedar or just leave them out completely and use the defaults. It is a good idea to at least have the
Walnut.WalnutRootFile profile option set. This should point to the full pathname of your walnut root file (the one you copied in step 1 and edited in step 3). Your user profile should contain something like:
Walnut.WalnutRootFile: /tilde/terry/Cedar/Databases/Walnut.root
Step 5. Create an appraiser and annotation database
To use message appraising, you need to create a LoganBerry database to store your filters (aka appraisers) and one to store your annotations (aka appraisals). For instance, you might want to call these "TapAppraisersLB.df" and "TapAnnotationsLB.df". The simplest way to to this is to use the LBCreate command to create the schema and log files and then use the LBBuildIndices to build the indices, e.g.
LBCreate TapAppraisersLB.df FID User Name
LBCreate TapAnnotationsLB.df AID MsgID User FID Level
LBBuildIndices TapAppraisersLB.df
LBBuildIndices TapAnnotationsLB.df
The full names of these two LoganBerry databases should be specified in your user profile entries: WallTapestry.FilterDB and WallTapestry.AnnotationDB. Your user profile might contain something like (see below for a full description of the new user profile entries):
WallTapestry.AnnotateNewMail: TRUE
WallTapestry.DisplayPriority: TRUE
WallTapestry.FilterDB: /tilde/terry/Cedar/TapestryDatabases/TerryAppraisersLB.df
WallTapestry.AnnotationDB: /tilde/terry/Cedar/TapestryDatabases/TerryAnnotationsLB.df
WallTapestry.SortByPriority: Active, Deleted
WallTapestry.SortByDate: *
Step 6. Run WallTapestry
The command names have not changed. That is, you still type "Walnut" to start the mail reading tool. Be sure to do a "XNSLogin" before starting WallTapestry if you want to retrieve new mail. Type "Walnut -n" if you don't want it to retrieve new mail. The Walnut command can take an argument, namely the file name of a root file.
Initially, you will need to scavenge your walnut database. You can do this by typing "WalnutScavenge" either before or after Walnut is running. Scavenging your database can take awhile. (It took 2.5 hours to scavenge my 19 MByte log file.) I believe that if your machine crashes while scavenging, you will need to start over.
See WalnutDoc.tioga for details on how to use Walnut.
3. Scavenging, Expunging, and Compacting
Scavenging is the operation of taking a Walnut log and parsing it to build the database. This is basically the same operation as was always in Walnut. There is still a WalnutScavenge command, but it only works if you are starting with an empty database. If you want to scavenge an existing database, type "DoWalnutScavenge .". This is a script that deletes the correct files and such before scavenging. Be sure to type the "." as an argument to DoWalnutScavenge (this says to look in your user profile for the root file name) or else provide an explicit root file name.
Expunging is the operation of reclaiming the space for messages that have been deleted. Recall that deleting a message in Walnut simply means moving to the Deleted message set. You can always display or undelete deleted messages until the next Expunge operation. In WallTapestry, an expunge operation is still available in the main menu. This operation rewrites the Walnut log so that the deleted messages are no longer present. While this reclaims log space, it does not reclaim database space. In fact, the LoganBerry database log gets larger since it records the fact that each message has been deleted. Thus, to really reclaim space you should use the new "DoWalnutCompact" script.
Compacting is the operation of rewriting the Walnut log so that it contains the absolute minimum amount of information necessary to describe your Walnut world, and then rebuilding the database based on this new log. To do this, simply type "DoWalnutCompact .". This is a good alternative to the Expunge operation. The DoWalnutOnlyCompact operation creates the minimum Walnut log, but does not automatically scavenge it.
If you want to break up your Walnut database into smaller databases that are partitioned according to date ranges, use the "DoWalnutIntervalCompact" script. This takes four parameters: the name of your Walnut root file, the name of the root file for your new partial database, a date indicating the earliest messages that should be included in the new database, and a date indicating the latest messages that should be included in the new database. For example, you might type "DoWalnutIntervalCompact Walnut.root WalnutJan88ToDec91.root 1/1/88 12/31/91". You will need to create the root file for the new database by hand. You can do this by copying your existing root file and other Walnut files (creating new names for these files in the process). Be sure to edit the root file so that it includes the new names for the Walnut files and LoganBerry database.
In summary, here are the scripts that you should use:
DoWalnutScavenge <rootFileName> - Scavenges the given Walnut database. Use this instead of WalnutScavenge (for the time being).
DoWalnutCompact <rootFileName> - Compacts the given Walnut database, and then scavenges it. Use this instead of Expunge.
DoWalnutOnlyCompact <rootFileName> - Compacts the given Walnut database, but does not scavenge it.
DoWalnutIntervalCompact <oldRootFileName> <newRootFileName> (<earliest>) (<latest>) - Extracts a range of messages from the old database and writes them to the new one. The earliest date defaults to long, long ago; the latest date defaults to now.
Thanks go to Dan Swinehart for creating these scripts. Eventually, their functionality will be incorporated directly into the Walnut commands.
4. New Functionality in WallTapestry
Overview
WallTapestry allows users to specify queries that get run against all new messages. Messages that match a particular query get assigned particular annotations, such as priority levels. The user can specify whether or not message set displayers should display a message's priority and which message sets are to be sorted by priority order. A new menu in each message set displayer permits users to change the ordering of messages in the displayer. A new menu in the Walnut window allows users to easily add new appraiser queries, list appraisers, and delete existing appraisers.
Displaying messages by priority (and other orders)
Each message has an associated "priority" or "interest" level. An interest level is a number between 0 and 100 (actually it could be any number). The interest level of a message is intended to denote the user's level of interest in the message. The default interest level for messages is 50 (though this can be changed in a user's profile).
Interest levels are attached to messages by appraisers when the messages arrive. An interest level is an annotation of type "Level" with an integer value, e.g. Level: 75. If a message has several annotations of type "Level", then the maximum value is taken as the message's interest level. An interest level is simply one of many possible types of annotations. It is the only one that WallTapestry currently looks at.
In the WallTapestry message set displayer, each message's "table of contents" is preceded by the message's interest level. For instance, a line of the displayer may contain "70> 23 Feb 90 Mark Weise... parcpad, ubicomp". The "70" is the message's interest level. Depending on the value of various user profile entries, the messages may be listed in interest-level order. That is, the highest level messages are at the top and the lowest level messages are at the bottom.
Each message set displayer has a notion of its current sort order. By default, messages in a displayer are listed in date order (the previous Walnut behavior). The user profile entries, WallTapestry.SortByDate and WallTapestry.SortByPriority, permit users to change the default ordering for various named message sets. Moreover, the "SortBy" menu button on each displayer, allows the user to change the current sort order. Clicking on this button gives a pop-up menu with the following sorting options: Date, Priority, Unread, Unread then Priority, Subject, Sender. Selecting a new order causes the displayer to be resorted by the selected order. This order also applies to new mail, as described below. Shift-selecting "SortBy" forces the displayer to be resorted even if the selected sort order is the same as the current sort order. This is useful for sorting new mail with the old. Changing the sort order for a message set does not change its default sort order. That is, each time the message set displayer is destroyed and then recreated, the default sort order is used.
As new messages are retrieved, they are added to the bottom of the Active displayer. Those messages within a group of messages that are retrieved at the same time are sorted according to the current sort order for the Active message set displayer. That is, if 5 new messages are retrieved, those 5 messages will be sorted in order, but they will not be interspersed with previously retrieved messages. (I'm not sure if this is ideal behavior.)
At any time, interest levels can be attached to old messages by reappraising them. The message set displayer's "MsgOps" menu contains two new operations for this purpose:
Reappraise Selected Msg(s) - resends the selected messages to the appraising agent so they can be reannotated.
Reappraise MsgSet - resends the complete message set to the appraising agent.
Both of these operations cause the message set displayer to be refreshed so that the correct priority level is displayed for the reannotated messages and they are displayed in the correct sort order.
Adding and manipulating appraisers
Appraisers can be added, listed, modified, and deleted using a variety of tools. Simple appraisers can be created and managed using the "Appraisers" entry in the main WallTapestry menu. Clicking on this entry produces a pop-up menu with the following operations:
Add Appraiser - takes the currently selected text and adds an appraiser where the selection serves as the query. That is, the selection should be of the form "field: value". An appraiser is added with the query "field(wildcard): *value*" and the name "field:value" where all blanks in value are replaced by underscores. A pop-up menu allows the user to select the desired interest (priority) level. For example, a user might select a header field of a currently displayed message, such as "from: terry", and click "Add Appraiser" to annotate future messages that contain this header.
List Appraisers - takes the currently selected text and lists (in the main Walnut window) the names of all appraisers whose query includes that text.
Show Appraiser - takes the currently selected text as an appraiser name and returns the complete set of information about that appraiser.
Delete Appraiser - takes the currently selected text as an appraiser name and deletes the appraiser.
Re-read Appraiser Database - causes WallTapestry to re-read the current set of appraisers and restart the appraising agent. This should be done if the appraiser database is modified outside of Walnut, e.g. using "TapAddFilter" or "TapDeleteFilter".
These operations are intended as a simple starting point for using the WallTapestry appraising capabilities. They are not the ultimate user interface. They do not even provide access to the full set of appraising and annotation capabilities.
Appraiser queries can be more complicated than those that can be added using "Add Appraiser". These queries can be arbitrary LoganQuery queries (such as those given as arguments to the LBQuery command). In particular, these queries can use various types of pattern matching and can refer to multiple fields of a message (see "Boolean Queries" in LoganQueryDoc.tioga for more details). Moreover, appraisers can add arbitrary annotations, i.e. more than just the interest level.
To add more complex appraisers, one can use the TapAddFilter command as described in TapestryProtoDoc.tioga. Any header field(s) of the message can be referenced in an appraiser query; these should be written in all lower case. For example:
% TapAddFilter filterName (to: *me* OR cc: *me*) AND NOT from: myboss -> Level: 90
Remember to click "Re-read Appraiser Database" after doing this or the new appraiser(s) will not be used until you next re-start WallTapestry. One can also add, delete, modify, or list appraisers using the LoganBerry browser directly on the appraiser database.
Suggestions for improved appraiser management tools are solicited. This is an ongoing area for investigation.
Explaining appraiser results
Debugging appraisers (i.e. understanding why certain appraisers matched certain messages) can be difficult. To aid in this process, an entry has been added to the message set displayer's "MsgOps" menu:
Explain Msg's Priority - prints the appraisers that matched the selected messages along with the priority level assigned by these appraisers.
This operation is also available on the message displayer's "MsgOps" menu along with a couple of highlight operations:
Explain Msg's Priority - prints the appraisers that matched the displayed message along with the priority level assigned by these appraisers.
Highlight Msg's Priority - highlights those portions of a message that contributed to its priority. In particular, it finds all of the appraiser queries that annotated the message and highlights the fields and words that they mention.
Highlight Keywords - takes the current selection as a collection of words that are highlighted wherever they occur in the displayed message.
The current implementation of "Highlight Msg's Priority" is quite simple. It highlights fields of the message that are mentioned in the query and words that occur in the query patterns. For example, if the query is "to: Joe AND from: Sam" then the following words will be highlighted wherever they occur in the message: "to:", "Joe", "from:", "Sam". This simple scheme words okay for most queries. It does not work for complex queries, i.e. queries involving regular expressions or soundex. It also may show portions of a message that did not contribute to the query, e.g. if "Joe" occurs in both the to: and cc: fields.
Providing feedback on read messages
If you use the Tapestry clipping service, you can help out other Tapestry users by indicating Tapestry messages that you find particularly interesting or particularly uninteresting. To make this easy, two new buttons are available in the main menu at the top of WallTapestry's message displayer:
LikeIt - indicates that you like the currently displayed message, i.e. that you recommend the message to others. Holding down the shift key while clicking this button means that you "really like it", while holding down the control key means that you "mildly like it". Clicking with the right mouse button, adds an "anonymous" recommendation for the message.
HateIt - indicates that you dislike the currently displayed message, i.e. that it is not worth reading. Holding down the shift key while clicking this button means that you "really hate it", while holding down the control key means that you "mildly hate it".
These buttons record your feedback in the Tapestry database. Normally, Tapestry records not only the fact that someone likes or dislikes a message, but also the name of the person making the recommendation. Clicking either "LikeIt" or "HateIt" with the right mouse button, rather than the left or middle button, records the feedback anonymously.
This user feedback can be accessed by Tapestry users' filter queries. For instance, a Tapestry user can have filter queries like "show me all messages that were recommended by at least one person" or "show me all Dave Barry articles that were recommended by Joe".
Currently these buttons have no effect for a message that does not reside in the Tapestry database, i.e. one that was not delivered by the Tapestry clipping service. One can imagine scenarios in which your appraisers make use of this feedback to adjust their behavior, i.e. they employ a learning algorithm; this is not currently done. Even so, you can be a good citizen by recommending messages to others.
Pruning message headers
When displaying a message, WallTapestry will prune uninteresting header fields of the message if so desired. For example, you can specify that you only want to see the "Subject" and "From" fields. In this case, WallTapestry will not display fields like "Received" or "Message-ID" or "Return-Path". WallTapestry keeps track of whether or not header pruning is desired. To turn header pruning on and off, the "MsgOps" menu in a message displayer contains the following entry:
Toggle Header Pruning - toggles whether or not full or pruned message headers are displayed. If full message headers are currently being displayed then headers will now be pruned, and vice versa. The selection of full or pruned headers applies to all displayed messages in all instances of WallTapestry running in the Cedar world.
By default, full headers are displayed. This can be changed by clicking on the menu entry described above or by setting the Walnut.PruneHeaders user profile entry to TRUE. By default the following header fields are considered interesting: Subject, From, To, Cc, and Date; all others are considered uninteresting and are elided if header pruning is turned on. To change the set of interesting header fields, add a Walnut.UnprunedHeaders entry to your user profile (see below).
Caveat: The headers for messages that include Tioga formatting are always shown in their entirety. In other words, header pruning currently only works for non-Tioga formatted messages, i.e. for messages that are received from non-Cedar users. This probably includes most of your messages, and undoubtedly includes all of your messages with lots of extraneous header items.
Selecting multiple messages
WallTapestry allows several messages to be selected within a message set at the same time. Operations that used to operate on the current selected message, now operate on the set of selected messages. For instance, you can select a range of messages and click on the "Delete" menu entry to delete the whole set or click "MoveTo" to move them or click on the "Display" menu entry to see a "mail movie."
To use these new selection features, set "Walnut.FancyNewMessageSelect:
TRUE" in your user profile. This slightly changes the meaning of certain mouse clicks used to select messages within a message set. It enables multiple message In particular, the following mouse clicks cause the following actions (all other mouse clicks work as before, i.e. as described in WalnutDoc.tioga):
Left => select the buttoned message (deselecting any previous selected messages).
Right => extend the selection set to include all messages between the buttoned message and previously selected message(s).
Shift-Right => deselect the buttoned message.
Cntl-Shift-Right => add the buttoned message to the set of selected messages.
(FancyNewMessageSelect also changes the way "come from" works, but I don't know how; see Dan Swinehart for details.)
User profile options
If you want to use WallTapestry the same way that you used Walnut in the DCedar world, then you can ignore this whole section and need not add any new entries to your user profile. If you want to try out the new functionality, then you will want to set some or all of these new user profile options:
WallTapestry.AnnotateNewMail: FALSE
-- If TRUE and WallTapestry.FilterDB is non-nil, new mail is annotated by comparing it to a set of appraisers
WallTapestry.DisplayPriority: FALSE
-- If TRUE, a message's priority is displayed in its TOC
WallTapestry.DefaultPriority: 50
-- a message's priority if not annotated
WallTapestry.FilterDB: a loganberry database name
-- Specifies the name of a LoganBerry database that contains a user's appraiser queries. These queries are run over new mail.
WallTapestry.AnnotationDB: a loganberry database name
-- Specifies the name of a LoganBerry database for storing message annotations. Only used if WallTapestry.AnnotateNewMail = TRUE or WallTapestry.DisplayPriority = TRUE.
WallTapestry.SortByDate: a list of folder names
-- Lists the message sets that are to be sorted by date when displayed. A "*" means that all message sets should be sorted by date unless otherwise specified (this is the default).
WallTapestry.SortByPriority: a list of folder names
-- Lists the message sets that are to be sorted by priority when displayed. A "*" means that all message sets should be sorted by priority unless otherwise specified.
Walnut.
FancyNewMessageSelect: FALSE
-- If TRUE, enables Swinehart's new multiple message selection.
Walnut.PruneHeaders: FALSE
-- If TRUE, only the UnprunedHeaders of a message are displayed.
Walnut.UnprunedHeaders: subject from to cc date
-- The headers of a messages that are always displayed, even if PruneHeaders is TRUE.
See WalnutDefault.profile for a list of other Walnut profile options.
Usage possibilities
There are two ways that you might use this new functionality:
A. WallTapestry users can read and prioritize their mail directly using WallTapestry. In this case, WallTapestry.FilterDB and WallTapestry.AnnotationDB should both specify LoganBerry databases. WallTapestry.AnnotateNewMail and WallTapestry.DisplayPriority should be TRUE.
B. TapInBlackCherry users can continue to read new mail and have it prioritized using TapInBlackCherry. Mail can then be transferred into WallTapestry for long-term storage. In this case, WallTapestry.FilterDB can be empty. WallTapestry.AnnotationDB can specify the same database as Tapestry.AnnotationDB so that WallTapestry and TapInBlackCherry share the annotation database. When messages are transferred to WallTapestry, they automatically retain their annotations and WallTapestry uses these annotations. (This currently does not work because the annotation database keys are slightly different.)
5. Known Bugs/Problems
The following is a list of known bugs, suspected problems, and unfinished business. If you encounter any bugs or problems that are not on this list, please send <Doug Terry> a message.
Limitations in basic Walnut functionality:
* The Wallaby browser is way too slow. Fixing this requires a better LoganBerry schema, which is hard.
* Walnut should have a mode where it is willing to accept mail from BlackCherry, but will not retrieve new mail from servers.
* WalnutScavenge does not empty the LoganBerry database before starting. Use DoWalnutScavenge instead.
* Should convert from using FS to PFS.
* There is code sprinkled throughout the source files that is commented out. Eventually it should either be made to work or should be removed.
* After one edits the Walnut.root file, Walnut requires the user to re-scavenge the database. Perhaps, the user should be given the option of avoiding the scavenge if they know that it is not necessary, i.e. if the changes to the root file were inconsequential.
* WalnutRescue, NewWalnutUser, and other miscellaneous tools have not been ported or upgraded.
* WalnutScavenge should work in a viewer-less world, i.e. it should not create a Walnut viewer but rather should simply report its progress in the commander viewer.
* The LoganBerry database could be much more compact in size. Currently, it is very verbose. Moreover, some of the stored info is probably not needed.
* Wallaby searches should be able to proceed in parallel with reading mail. (Requested by Weiser, 11 Jan 93)
Limitations in new WalnutTapestry functionality:
* The Do* scripts, e.g. DoWalnutScavenge, do not cope with errors very well. If some step fails, the script plows along anyway, producing unknown results. Eventually, these scripts should be incorporated into the associated Walnut commands and the error-recovery improved.
* Appraisers that are added (or modified or deleted) through means other than Walnut, will not be noticed or used until Walnut is restarted.
* When a message or message set is reappraised, through "Reappraise Selected Msg(s)" or "Reappraise MsgSet", WallTapestry fails to discard existing annotations for the message(s). Thus, the messages may end up with duplicate annotations in the database. More importantly, reducing the priority level for an appraiser does little good since the old (higher) priority still annotates the message.
* The annotation database is not expunged along with Walnut log. That is, annotations for deleted messages are never discarded so the database grows indefinitely.
* The annotation database used by TapInBlackCherry uses a different format for its primary key than the annotation database used by WallTapestry. Thus, the two mail reading programs cannot currently share an annotation database.
* There should be a means for explicitly assigning priorities to messages.