WalnutImplementationNotes.tioga
Doug Terry, December 3, 1990 3:01 pm PST
Walnut Implementation Notes
Doug Terry
Introduction
This is a collection of notes that describe the implementation of the Walnut mail reader. These notes are very incomplete.
See WalnutDoc.tioga for more information.
Modules
There are two main collection of modules. These are described by the DF files WallTapestry-Suite.df and WallTapestryKernel-Suite.df and their associated configs WalnutControl and WalnutKernel.
The system contains several layers of functionality. The following table lists the layers and their interfaces and exporters (again let me stress that this is incomplete). While I have tried to assign modules to layers such that components within a layer are independent, this is not entirely true. (There are several modules in Walnut that export and import the same interface!) Lower numbered layers are lower in the dependency chain, e.g. level 0 does not depend on anything in level 1. Level 1 (and 1.5) is in WalnutKernel while level 3 is in WalnutControl; level 2 is split between WalnutControl and WalnutKernel.
LEVEL 0:
Interface Exported by Purpose
LoganBerry LoganBerryImpl, etc. database storage and indexing
FS/PFS FSOnPFSImpl, etc. log storage
... ... ...
LEVEL 1:
Interface Exported by Purpose
WalnutStream WalnutStreamImpl deals with streams on log files
WalnutRoot WalnutRootImpl parses root file, transaction management
WalnutSchema WalnutOpsMiscImpl database schema definition
LEVEL 1.5:
Interface Exported by Purpose
WalnutLog WalnutLogImpl writes messages to log file
WalnutMiscLog WalnutLogImpl
WalnutLogExpunge WalnutLogExpungeImpl removes deleted messages from log
WalnutDB WalnutDBOpsImpl database operations
 WalnutDBMsgImpl
 WalnutDBMsgSetsImpl
LEVEL 2:
Interface Exported by Purpose
WalnutNewMail WalnutNewMailImpl reading new mail
WalnutOpsInternal WalnutOpsInternalImpl miscellaneous high-level ops
WalnutOps WalnutOpsImpl top-level operations
WalnutViewer WalnutViewerImpl viewer utilities
LEVEL 3:
Interface Exported by Purpose
WalnutInternal WalnutNotifierImpl a hodge-podge of operations
 WalnutWindowMenuImpl
 WalnutWindowInternalImpl
 WalnutMsgSetButtonsImpl
 WalnutMsgDisplayerImpl
 WalnutMsgSetDisplayerImpl
 WalnutPrintImpl
WalnutWindow WalnutNotifierImpl a hodge-podge of operations
 WalnutWindowCommandsImpl
 WalnutWindowInternalImpl
 WalnutWindowMenuImpl
 WalnutMsgDisplayerImpl
 WalnutMsgSetDisplayerImpl
 WalnutMsgSetButtonsImpl
 WalnutPrintImpl
 LoganWalnutImpl
 WallabyImpl
WalnutWindowSidedoor WalnutMsgSetButtonsImpl more hodge-podge
 WalnutWindowCommandsImpl
LoganWalnut LoganWalnutImpl database entry conversions
Wallaby WallabyImpl message browser
See WalnutCatalog.tioga for a listing of which procedures are exported by which modules.
Using logs
Walnut writes an operation log that keeps track off all notable events. This log contains operations like creating message sets, moving messages between sets, etc. It also contains the text of all messages. For reliablity reasons the log is append-only. The expunge operation writes a new log that contains all of the messages that have not been deleted.
Walnut also maintains a database that indexes information that is in the log. The database is used to enumerate message sets and such. The database can always be completely recovered by scavenging the log.
New mail processing
What happens when mail is retrieved from a mail server?
AutoNewMailProc is called by WatchMailBox, NewMailIdleProc, CheckMailBoxes.
WalnutNotifierImpl.RestartWalnut
Calls WalnutOps.Startup
Checks validity of database and such
Returns newMailExists if NewMailLog is non-empty or servers have mail
Calls WalnutWindowInternalImpl.EnableNewMail
Calls WalnutNewMail.EnableMailRetrieval
Registers NewMailIdleProc
Forks AutoNewMailProc when coming out of idle
Calls MailRetrieve.Create[ msgPollingInterval, WatchMailBox ]
Calls MailRetrieve.NewUser for each name/password
Calls WalnutNewMail.GetLastMailBoxStatus
WalnutNewMailImpl.WatchMailBox
Forks AutoNewMailProc
Calls DoNewMail
Calls MailRetrieve.MailboxState to check state
For each server
Calls DrainServer
Writes messages to NewMailLog
Calls MailRetrieve.Accept
Calls MailFetchDone
Calls MailHandle.notifyProc
WalnutWindowMenuImpl.NewMailProc
Calls WalnutNewMail.CheckMailBoxes if mouseButton = blue
Forks AutoNewMailProc
<see above>
What happens when the user clicks "NewMail"?
Calls WalnutWindow.GetNewMail (WalnutWindowInternalImpl.GetNewMail)
Calls WalnutInternal.AddNewMsgsToActive (WalnutMsgSetDisplayerImpl.AddNewMsgsToActive)
Calls WalnutOps.GetNewMail with proc that calls BuildMsgLineViewer
Calls WalnutOpsInternal.DoNewMail with proc
Checks for operation in progress
Gets length of newmail log
Copies newmail log to current log
Records that a ParseLog operation is in progress
Calls ParseLog
Calls WalnutDB.AddNewMsg (WalnutDBMsgImpl.AddNewMsg) for each message, etc.
Pass message to Tapestry filtering agent
Wake up filtering agent and wait until it is done
Calls WalnutDB.EnumerateUnacceptedMsgs (WalnutDBMsgSetsImpl.EnumerateUnacceptedMsgs) with proc
Calls proc for each "unaccepted" message in database
Gets list of servers and message counts
Calls WalnutOps.AcceptNewMail
Calls WalnutOpsInternal.DoAcceptNewMail
Calls
Records acceptance of new mail
Calls WalnutDB.AcceptNewMail (WalnutDBMsgSetsImpl.AcceptNewMail)
Clears "unaccepted" property for each message
Zeros count for each server
Clears opinprogress, etc.
Increments version number of active message set
Prints summary of number of messages delivered
Expunging old mail
When a user clicks "Expunge":
Calls WalnutWindowMenuImpl.MenuExpunge
Calls WalnutWindowCommandsImpl.Expunge
Calls Quitting
Closes all Walnut viewers, etc.
Reports start of expunge
Calls WalnutOps.ExpungeMsgs
Calls WalnutLog.ExpungeMsgs
Writes "ExpungeMsgs" entry in log
Calls WalnutDBMsgSetsImpl.ExpungeMsgs
Updates size of Deleted message set and total message count
For each message in Deleted
Removes all information about message from database
Writes record of BytesInDestroyedMsgs and FirstDestroyedMsgPos to database
Bumps version number of Deleted
Calls DestroyOrphanAddrsAndSubjs
Does nothing!
Calls WalnutOps.CopyToExpungeLog
Checks quota and space needed for expunge log
Calls WalnutLog.WriteExpungeLog
Writes "WriteExpungeLog" entry in log
Calls WalnutOpsInternal.DoLogExpunge
Until done with all phases of expunge do
Calls WalnutDB.GetLogExpungePhase
Reads expunge phase from database
If phase is idle then
Go to initializingExpungeLog phase
If phase is initializingExpungeLog then
Calls WalnutLogExpunge.StartExpunge
Writes first log record on expunge log
Go to writingExpungeLog phase
If phase is writingExpungeLog then
Calls WriteExpungeLog
Calls WalnutLogExpunge.RestartExpunge
Prepares for expunge
Reports "Copying the head of the log..."
Calls WalnutLogExpunge.CopyBytesToExpungeLog
Copies bytes from current log to expunge log
Reports "Processing the tail of the log..."
Copies some entries from the current log to expunge log and skips others
Calls WalnutLogExpunge.EndExpunge
Calls WalnutRoot.StopExpunge
Closes expunge log
Go to swappingLogs phase
If phase is swappingLogs then
Reports "Swapping Log Files"
Updates expunge info and sets phase to idle
Updates root file version timestamp
Calls WalnutRoot.SwapLogs
Swaps the current log and expunge log in root file
Reports "Finished expunge"
Reports result of expunge
Restarts Walnut
Browsing messages (Wallaby)
Implemented in: LoganWalnut.mesa, LoganWalnutImpl.mesa, Wallaby.mesa, WallabyImpl.mesa.
What happens when a user clicks "Query" in the Walnut control window?
Calls Wallaby.QueryProc
Creates new LoganBerryBrowser.ToolBody
Calls LoganBerry.Open[NIL, wH.opsH.dbName]
Calls LoganBerryBrowser.CreateTool
What happens when a user clicks "Browse" in the Wallaby browser?
Calls WallabyImpl.BrowseProc
Calls DoBrowseProc
Calls BrowseBody
Calls LoganBerryBrowser.ReadEntryForm
Calls LoganQuery.QueryEntries to get query plan (ignores cursor)
Calls LoganBerryBrowser.ReportFeedback
Calls GenerateWalnutEntries
Calls LoganWalnut.GenerateEntriesPlusDate
Calls WalnutOps.GenerateEntriesPlusDate
Calls WalnutDBMsgImpl.GenerateEntriesPlusDate
Calls LoganBerry.GenerateEntries
Builds and returns a LoganBerry.Cursor
Builds and returns a LoganQuery.ComplexCursor
Builds up filters by calling LoganQuery.FilterEntries
Reports "Query in Progress"
Retrieves allmatching entries by calling LoganQuery.NextEntry
Reports "Query Finished"
Changes made for PCedar/Tapestry
Removed dependencies on Alpine and Cypress.
Changed parsing code that looked for \n to check for either '\r or '\l.
Changed names of temp files.
Removed LoganWalnut and private bound in copy of LoganQuery and LoganBerryBrowser.