WalnutInterfacesDoc.tioga
Jim Donahue, July 11, 1985
Donahue, July 20, 1985 9:42:12 am PDT
Walnut Interfaces
CEDAR 6.0 — FOR INTERNAL XEROX USE ONLY
Walnut Interfaces
Programmer's Access to a Walnut Database
Jim Donahue, Willie-Sue Orr
© Copyright 1985 Xerox Corporation. All rights reserved.
Abstract: The details of the various programmer's interfaces to Walnut. The document WalnutDoc.tioga gives information about the user's interface that Walnut provides.
Created by: Donahue
Maintained by: Donahue, Orr <Donahue.pa>
Keywords: Walnut, databases, electronic mail
XEROX   Xerox Corporation
    Palo Alto Research Center
    3333 Coyote Hill Road
    Palo Alto, California 94304

For Internal Xerox Use Only
1. Introduction
We begin with a very brief description of a Walnut database. It contains two entity types: message and message set.
A message entity corresponds to a message retrieved from the Grapevine mail transport system. Like all database entities it has a name, consisting of the sender's RName concatenated with a unique message ID provided by Grapevine. A message also has several immutable properties: its sender, its subject, and so on. Its unread property is a BOOL whose value is TRUE when the message is first stored in the database, and is set to FALSE when the message is first displayed.
A message entity is also a member of one or more message sets. A message set entity is named by a text string containing no embedded blanks. There are two distinguished message sets: Active and Deleted. A newly-retrieved message is made a member of Active. A message that is removed from all other message sets is added to Deleted. Using the Active and Deleted message sets in this way ensures that each message belongs to at least one message set.
Walnut provides three programmer's interfaces, so that other programs can easily browse or modify Walnut databases. These interfaces are:
1. WalnutWindow, for access at the user interface level of Walnut,
2. WalnutRegistry, for information about events that change the state of a Walnut database, and
3. WalnutOps, the low-level client interface which provides access to all of the details of a Walnut database.
Below, we will present each of them and discuss some of the finer details of using the interfaces.
2. Walnut Window
2.1 The Interface
DIRECTORY
Menus USING [MenuProc],
Rope USING [ROPE],
ViewerClasses USING [Viewer];
WalnutWindow: CEDAR DEFINITIONS =
BEGIN
Viewer: TYPE = ViewerClasses.Viewer;
ROPE: TYPE = Rope.ROPE;
Msg and MsgSet viewer procedures
GetMsgName: PROC[v: Viewer] RETURNS[mName: ROPE];
If v is a Viewer for a Walnut Msg, then return its name else NIL
GetMsgSetName: PROC[v: Viewer] RETURNS[msName: ROPE];
If v is a Viewer for a Walnut MsgSet, then return its name else NIL
AddToMsgMenu: PROC[label: ROPE, proc: Menus.MenuProc, clientData: REF ANY ← NIL, onQueue: BOOL ← FALSE, doReset: BOOL ← FALSE];
adds menu button, synchronised using walnutQueue if onQueue is TRUE. If restAfterFinish, then Walnut will do an internal reset after completion of the menu procedure
RemoveFromMsgMenu: PROC[name: ROPE];
removes the named button from the msg displayer menu, if such a button exists; no errors
Walnut control viewer procedures
SelectMsgSetsFromMSNames: PROC[msNames: LIST OF ROPE] RETURNS[notFound: LIST OF ROPE];
Selects the message set buttons in the Walnut Control viewer which corresponds to the list of msNames given. Returns a list of any not found
CurrentVersion: PROC[msName: ROPE] RETURNS[version: INT];
Give the current "expected version" of the message set (-1 means "don't care").
Displaying Msgs and MsgSets
DisplayMsg: PROC[msg: ROPE, oldV: Viewer ← NIL, shift: BOOL ← FALSE] RETURNS[v: Viewer];
displays Msg name (if it exists) in Viewer oldV; if oldV is NIL it creates a viewer
DisplayMsgSet: PROC[msgSet: ROPE, shift: BOOL ← FALSE, repaint: BOOL ← TRUE] RETURNS[v: Viewer];
returns NIL if msgSet is not a MsgSet. This operation also inserts the viewer into the state recorded by the Walnut control window, so that operations on the messages in the message set are guaranteed to use a consistent version of the set. If the message set is already displayed, then it will be repainted if repaint is TRUE
Walnut control operations
StartUp: PROC[rootFile: ROPE, scavengeFirst: BOOL ← FALSE];
Open up a WalnutWindow on the database named in the root file
Shutdown: PROC;
Close down the WalnutWindow
QueueCall: PROC[proc: PROC RETURNS[doReset: BOOL]] RETURNS[outCome: OutCome];
places proc on Walnut's procedure queue; If the procedure returns doReset = TRUE, then Walnut will do a reset of its internal state after the call completes. outCome lets the caller know what happened with the call.
OutCome: TYPE = {ok, flushed, notRunning};
GetNewMail: PROC;
Expunge: PROC;
Scavenge: PROC[rootFile: ROPE];
WriteArchiveFile: PROC[fileName: ROPE, msgSetList: LIST OF ROPE];
If the message set list is NIL, then the entire database is dumped
ReadArchiveFile: PROC[fileName, msgSet: ROPE, useCategoriesInFile: BOOL ← FALSE];
IF msgSet is NIL and useCategoriesInFile is FALSE, then messages go into Active; IF msgSet is NIL and useCategoriesInFile is TRUE, then use the ones specified in the file
IF msgSet is non-NIL THEN put all messages into it
END.
2.2 Commentary
WalnutWindow provides a programmer's interface to the Walnut user interface data structures and operations: one can change the menu used for message viewers, display the contents of messages and message sets, interrogate the Walnut control window, and perform the collection of operations that the Walnut user interface makes available. The Walnut control operations all used Walnut's internal MBQueue; when they are called, they queue a procedure on the Walnut queue (to synchronize with other user operations) and then wait until they are called.
The QueueCall procedure allows a WalnutWindow client to perform a "user-level transaction." The procedure is called when it is pulled from the Walnut queue; this prevents other user level operations from beginning until the procedure completes. If the procedure changes the state of the Walnut database (by calling WalnutOps operations), it must return doReset = TRUE so that the state of the WalnutWindow view of the database will be recomputed (this will cause repainting of Walnut viewers only if what is being displayed is now inconsistent with the contents of the database).
3. WalnutRegistry
3.1 The Interface
DIRECTORY
MBQueue USING[Queue],
Rope USING [ROPE];
WalnutRegistry: CEDAR DEFINITIONS =
BEGIN
WalnutState: TYPE = {unknown, active, stopped};
unknown means that Walnut has not notified the registry yet
Event: TYPE = {started, stopped, mailRead, expungeComplete};
EventProc: TYPE = PROC[event: Event, clientData: REF ANY];
MsgEvent: TYPE = {firstRead, deleted, unDeleted};
MsgProc: TYPE = PROC[msgName: Rope.ROPE, event: MsgEvent, clientData: REF ANY];
MsgGroup: TYPE = ARRAY[0..MsgGroupSize) OF Rope.ROPE;
MsgGroupSize: CARDINAL = 20;
MsgGroupEvent: TYPE = {added, destroyed};
MsgGroupProc: TYPE = PROC[
msgGroup: REF MsgGroup, event: MsgGroupEvent, clientData: REF ANY];
MsgSetEvent: TYPE = {created, destroyed};
MsgSetProc: TYPE = PROC[msgSetName: Rope.ROPE, event: MsgSetEvent, clientData: REF ANY];
MoveProc: TYPE = PROC[
msgName: Rope.ROPE, fromMsgSet, toMsgSet: Rope.ROPE, clientData: REF ANY];
ProcSet: TYPE =
RECORD[eventProc: EventProc ← NIL,
eventProcData: REF ANY ← NIL,
msgProc: MsgProc ← NIL,
msgProcData: REF ANY ← NIL,
msgGroupProc: MsgGroupProc ← NIL,
msgGroupData: REF ANY ← NIL,
msgSetProc: MsgSetProc ← NIL,
msgSetData: REF ANY ← NIL,
moveProc: MoveProc ← NIL,
moveProcData: REF ANY ← NIL];
Registration: TYPE = REF;
Register: PROC [procSet: ProcSet, queue: MBQueue.Queue] RETURNS[registration: Registration];
Register a set of procedures to be invoked when the appropriate database changes occur. Return a registration that can be used to name the set
InvalidRegistration: ERROR;
Each of the procedures below will raise this error if given a registration that does not currently exist in the table of registered procedures
UnRegister: PROC [registration: Registration];
Remove the procedures named by the registration
GetProcs: PROC[registration: Registration] RETURNS[procSet: ProcSet, queue: MBQueue.Queue];
Return the procedures named by the registration
CurrentWalnutState: PROC RETURNS[walnutState: WalnutState];
a quick and safe way to find out if Walnut has been loaded and if it is active or not
END.
3.2 Commentary
WalnutRegistry is for use in tracking the changes in a Walnut database. The CurrentWalnutState procedure gives a simple means of checking whether a Walnut database is currently open, eg., if the WalnutWindow StartUp operation has been performed. Clients are notified of other changes by having registered procedures called whenever a relevant change occurs. Changes for which a client can register a procedure to be called include:
1. Basic Walnut events: starting, stopping, reading new mail, completing an expunge,
2. Reading, deleting or undeleting a message,
3. Creating or destroying a message set,
4. Adding or destroying a collection of messages, and
5. Moving a message from one message set to another.
Clients can register a collection of procedures to deal with one or more of these types of events by calling Register; they are returned a unique Registration, which can be used to lookup the procedures or to UnRegister them (if, for instance, the client program is to stop running). If a call of Register specifies and EventProc to be called and the state of Walnut is started or stopped, then the EventProc will be called after registration.
A call to Register must also provide an MBQueue.Queue to be used to queue the procedures when they are called. WalnutRegistry is constructed so that the registered procedures are not called directly when the event occurs, but instead queues them so that they will be called when the queue entry is pulled off. This ensures that a procedure does not delay normal processing by WalnutOps and that it can call any WalnutOps procedure without causing a monitor deadlock (it is WalnutOps that notifies the registry of changes in the database).
4. WalnutOps
4.1 The Interface
DIRECTORY
BasicTime USING [GMT],
GVBasics USING [RName],
IO USING [STREAM],
Rope USING [ROPE],
ViewerTools USING [TiogaContents],
WalnutDefs USING [dontCareDomainVersion, dontCareMsgSetVersion, MsgSet, ServerInfo],
WalnutParseMsg USING [ParseProc, MsgHeaders, MsgHeaderItem];
WalnutOps: CEDAR DEFINITIONS =
BEGIN
Types
ROPE: TYPE = Rope.ROPE;
dontCareDomainVersion: INT = WalnutDefs.dontCareDomainVersion;
dontCareMsgSetVersion: INT = WalnutDefs.dontCareMsgSetVersion;
DomainVersion: TYPE = INT ← dontCareDomainVersion;
MsgSetVersion: TYPE = INT ← WalnutDefs.dontCareMsgSetVersion;
MsgSet: TYPE = WalnutDefs.MsgSet;
MsgSet: TYPE = RECORD[name: ROPE, version: MsgSetVersion];
Message set arguments to WalnutOps procedures take an (optional) version number argument that can be considered a read lock on the message set
ActiveMsgSetName: ROPE;
DeletedMsgSetName: ROPE;
ServerInfo: TYPE = WalnutDefs.ServerInfo;
ServerInfo: TYPE = RECORD[server: ROPE, num: INT];
EnumeratorForMsgs: TYPE = REF EnumeratorForMsgsObject;
EnumeratorForMsgsObject: TYPE;
ParseProc: TYPE = WalnutParseMsg.ParseProc;
ParseProc: TYPE = PROC[fieldName: ROPE] RETURNS[wantThisOne, continue: BOOL];
MsgHeaders: TYPE = WalnutParseMsg.MsgHeaders;
MsgHeaderItem: TYPE = RECORD[fieldName, value: ROPE];
MsgHeaderItem: TYPE = WalnutParseMsg.MsgHeaderItem;
MsgHeaderItem: TYPE = RECORD[fieldName, value: ROPE];
Procedures
Starting and stopping Walnut
Startup: PROC[rootFile: ROPE, wantReadOnly: BOOL ← FALSE]
RETURNS[isReadOnly, newMailExists: BOOL, mailFor: GVBasics.RName, key: ROPE];
Errors (WalnutDefs.Error):
already started ($AlreadyStarted).
database does not match log ($DatabaseIncomplete).
database has become inaccessible ($DatabaseInaccessible)
Shutdown: PROC;
Scavenge: PROC[rootFile: ROPE] RETURNS[newMailExists: BOOL, mailFor: GVBasics.RName, key: ROPE];
Information about the Walnut database
ReadOnly: PROC RETURNS[readonly: BOOL];
GetRootInfo: PROC RETURNS[createDate: BasicTime.GMT, rootFile, mailFor: ROPE];
FileName: PROC RETURNS[dbFileName: ROPE];
SizeOfDatabase: PROC RETURNS[messages, msgSets: INT];
LogLength: PROC RETURNS[length: INT];
MsgSetsInfo: PROC RETURNS[version: DomainVersion, num: INT];
RegisterReporter: PROC[reportStream: IO.STREAM];
UnregisterReporter: PROC[reportStream: IO.STREAM];
Adding new messages to the database
StartNewMail: PROC[] RETURNS[newMailStream: IO.STREAM];
Get the right to write to the New Mail log. If newMailStream is NIL, either it could not be opened or it was already in use.
EndNewMail: PROC[];
Release the new mail log
GetNewMail: PROC[activeVersion: MsgSetVersion, proc: PROC[msg, TOCentry: ROPE, startOfSubject: INT] ← NIL ] RETURNS[responses: LIST OF ServerInfo, complete: BOOL];
GetNewMail takes a version of the active message set and enumerates the messages that are currently "unaccepted" This is solely a database query and does not change the state of the Walnut database (subsequent GetNewMail operations will return the same result until an AcceptNewMail is done). It may fail either because the of a version stamp mismatch or because of database unavailability. If complete is true, then all of the new messages available were retrieved; if false, there is still new mail in the system.
AcceptNewMail: PROC[activeVersion: MsgSetVersion];
Accept accepts all currently unaccepted messages and resets the server response entries in the database -- it also increments the version number of the active message set.
RecordNewMailInfo: PROC[logLen: INT, server: ROPE, num: INT];
The length given must be the length of a legal tail of a newMailLog
CreateMsg: PROC [msgName: ROPE, body: ViewerTools.TiogaContents];
Log a message from some agent other than Grapevine and add it to the database. Body is expected to conform to the syntax for messages. The first message on the log with a given name wins. Any subsequent messages with the same name are dropped on the floor.
Removing unreferenced messages
ExpungeMsgs: PROC[deletedVersion: MsgSetVersion] RETURNS[bytesInDestroyedMsgs: INT];
Removes all messages from the Deleted message set.
GetExpungeInfo: PROC RETURNS[firstDestroyedMsgPos, bytesInDestroyedMsgs: INT];
returns the the log position of the first destroyed msg and the number of bytes in destroyed msgs. The actual amount of space reclaimed during an expunge will probably be more (entry overhead, other entries referring to destroyed msgs)
CopyToExpungeLog: PROC[];
The log is re-written, starting from the beginning
Primitive message set operations
CreateMsgSet: PROC [name: ROPE, msDomainVersion: DomainVersion];
Create a message set with name msgSet. If the named message set exists, this is a noop.
MsgSetExists: PROC [name: ROPE, msDomainVersion: DomainVersion] RETURNS [exists: BOOL, version: MsgSetVersion];
SizeOfMsgSet: PROC [name: ROPE] RETURNS [messages: INT, version: MsgSetVersion];
EmptyMsgSet: PROC [msgSet: MsgSet] RETURNS [someInDeleted: BOOL];
Remove all messages from msgSet. Messages that are no longer in any message set will be added to "Deleted".
DestroyMsgSet: PROC [msgSet: MsgSet, msDomainVersion: DomainVersion] RETURNS [someInDeleted: BOOL];
Remove the specified message set from the database.
EnumerateMsgsInMsgSet: PROC[name: ROPE] RETURNS[enum: EnumeratorForMsgs];
a lazy enumerator for the messages in the msgSet name, at the time of the call. The enumerator is valid until the next time the msgSet's version changes (by adding or deleting msgs); its use after that will cause a WalnutDefs.Error with code $InvalidEnumerator. See NextMsg for details.
EnumerateMsgs: PROC RETURNS[enum: EnumeratorForMsgs];
a lazy enumerator for the messages in the database, at the time of the call. The enumerator is valid until the next expunge; its use after that will cause a WalnutDefs.Error with code $InvalidEnumerator. See NextMsg for details.
NextMsg: PROC[enum: EnumeratorForMsgs] RETURNS [msgID: ROPE, msList: LIST OF ROPE, headers: REF TEXT];
a lazy enumerator for the messages in the database or in a msgSet, at the time of the call. When the end of the messages is reached, msgID will be NIL. msList is the list of message sets the msg belongs to, at the time of the call. headers contains just the headers information from the message. Be warned - subsequent calls to NextMsg may reuse the REF TEXT.
MsgsEnumeration: PROC[alphaOrder: BOOL ← TRUE] RETURNS[mL: LIST OF ROPE];
MsgSetNames: PROC[alphaOrder: BOOL ← TRUE] RETURNS[mL: LIST OF ROPE, msDomainVersion: DomainVersion];
MsgsInSetEnumeration: PROC [name: ROPE, fromStart: BOOL← TRUE]
RETURNS [mL: LIST OF ROPE, msVersion: MsgSetVersion];
EnumerateMsgSets: PROC [alphaOrder: BOOL ← TRUE, proc: PROC[msgSet: MsgSet] ← NIL] RETURNS [msDomainVersion: DomainVersion];
NOTE: proc may not do ANY database calls
EnumerateMsgsInSet: PROC [
name: ROPE,
fromStart: BOOL← TRUE,
proc: PROC[msg, TOCentry: ROPE, hasBeenRead: BOOL, startOfSubject: INT] ← NIL]
RETURNS [msVersion: MsgSetVersion];
for each msg in msgSet, will call proc with the indicated information about the msg. NOTE: proc may not do ANY database calls;
Parsing a message
ParseHeaders: PROC [headers: ROPE, proc: ParseProc] RETURNS[msgHeaders: MsgHeaders];
proc is called for each fieldName encountered in the headers; if proc is NIL then all headers are returned. It is ok to take the REF TEXT returned by NextMsg or GetMsgText and use it, wrapped in a RefText.TrustTextAsRope.
Primitive message operations
MsgExists: PROC [msg: ROPE] RETURNS[exists: BOOL];
GetCategories: PROC [msg: ROPE] RETURNS[LIST OF ROPE];
GetDisplayProps: PROC [msg: ROPE] RETURNS[hasBeenRead: BOOL, TOCentry: ROPE, startOfSubject: INT];
GetMsgDate: PROC [msg: ROPE] RETURNS[date: BasicTime.GMT];
returns the date used to index the message
GetMsg: PROC [msg: ROPE] RETURNS[contents: ViewerTools.TiogaContents, herald, shortName: ROPE];
GetMsgText: PROC [msg: ROPE, text: REF TEXT] RETURNS[contents: REF TEXT];
returns just the contents part of the msg (not the formatting information); note that the contents of comment nodes will NOT be included. Contents will be a new REF TEXT if text was not big enough
GetMsgShortName: PROC[msg: ROPE] RETURNS[shortName: ROPE];
GetHasBeenRead: PROC [msg: ROPE] RETURNS[hadBeenRead: BOOL];
SetHasBeenRead: PROC [msg: ROPE];
More complex message set / message operations
AddMsg: PROC [msg: ROPE, from, to: MsgSet] RETURNS [exists: BOOL];
Add the message (msg) from the message set (from) to the message set (to).
MoveMsg: PROC [msg: ROPE, from, to: MsgSet] RETURNS [exists: BOOL];
Move the message (msg) from the message set (from) to the message set (to).
RemoveMsg: PROC [msg: ROPE, from: MsgSet, deletedVersion: MsgSetVersion] RETURNS [deleted: BOOL];
Remove the message (msg) from the message set (from). If this action results in the message not belonging to any message set, it will be placed in the Deleted message set.
Producing and reading archive files
ReadArchiveFile: PROC [file: ROPE, msgSet: MsgSet] RETURNS[numNew: INT];
Write a "readArchiveFile" log entry; if the archiveFile exists, parses it and writes a separate readArchiveLog with appropriate entries (new msgs and moves if msgSet is not "Active" (NIL defaults to categories specified in the file). The archiveLog is then copied to the tail of the currentLog and the log is replayed. IF the file couldn't be read, numNew = -1; if there were problems parsing file, messages are printed
WriteArchiveFile: PROC [file: ROPE, msgSetList: LIST OF MsgSet, append: BOOL ← FALSE];
Write an archive file that contains the messages from the given message sets. No log entry is written and no updates are made to the database (we just hold the monitor to guarantee that no changes to the message sets occur).
END.
4.2 Commentary
This is the interface that provides all of the procedures to access a Walnut database (ie., the collection of root file, logs and database segment). Most of the procedures are self-explanatory; however, the treatment of "version stamps" in the operations involving message sets is worthy of further description.
Walnut maintains version stamps in the database for each message and for the set of messages sets (ie., the buttons in the Walnut control window). The version stamp of a message set is changed each time a message is added or deleted from the message set, while the version stamp for the set of message sets is updated each time a message set is added or destroyed. The operations that work on message sets (eg., MoveMsg) take as arguments message set names and optional version stamps; if a version stamp is provided and it disagrees with the version stamp actually stored in the database, then the operation fails (it raises a WalnutDefs.Error[$VersionMismatch]).
The Walnut interface code caches the version stamps of all message sets currently being displayed and checks them whenever an operation involves a displayed message set (eg., if both the "to" and "from" message sets were being displayed when a message is moved, then both version stamps would be supplied when calling MoveMsg). This allows WalnutOps clients to share a database without having to have separate transactions: the version stamps play the role of "read locks" on the message sets. Note that since messages are immutable, there is no need to have version stamps on messages. This is not a completely satisfactory replacement for separate transactions, since it is not possible for separate clients to set "write locks" on message sets; if there is a great and pressing need for this sort of capability, then we will attempt to add it.