<<>> <> <> Walnut Implementation Notes Doug Terry Introduction This is a collection of notes that describe the implementation of the Walnut mail reader. These notes are very incomplete. See WalnutDoc.tioga for more information. Modules There are two main collection of modules. These are described by the DF files WallTapestry-Suite.df and WallTapestryKernel-Suite.df and their associated configs WalnutControl and WalnutKernel. The system contains several layers of functionality. The following table lists the layers and their interfaces and exporters (again let me stress that this is incomplete). While I have tried to assign modules to layers such that components within a layer are independent, this is not entirely true. (There are several modules in Walnut that export and import the same interface!) Lower numbered layers are lower in the dependency chain, e.g. level 0 does not depend on anything in level 1. Level 1 (and 1.5) is in WalnutKernel while level 3 is in WalnutControl; level 2 is split between WalnutControl and WalnutKernel. LEVEL 0: Interface Exported by Purpose LoganBerry LoganBerryImpl, etc. database storage and indexing FS/PFS FSOnPFSImpl, etc. log storage ... ... ... LEVEL 1: Interface Exported by Purpose WalnutStream WalnutStreamImpl deals with streams on log files WalnutRoot WalnutRootImpl parses root file, transaction management WalnutSchema WalnutOpsMiscImpl database schema definition LEVEL 1.5: Interface Exported by Purpose WalnutLog WalnutLogImpl writes messages to log file WalnutMiscLog WalnutLogImpl WalnutLogExpunge WalnutLogExpungeImpl removes deleted messages from log WalnutDB WalnutDBOpsImpl database operations WalnutDBMsgImpl WalnutDBMsgSetsImpl LEVEL 2: Interface Exported by Purpose WalnutNewMail WalnutNewMailImpl reading new mail WalnutOpsInternal WalnutOpsInternalImpl miscellaneous high-level ops WalnutOps WalnutOpsImpl top-level operations WalnutViewer WalnutViewerImpl viewer utilities LEVEL 3: Interface Exported by Purpose WalnutInternal WalnutNotifierImpl a hodge-podge of operations WalnutWindowMenuImpl WalnutWindowInternalImpl WalnutMsgSetButtonsImpl WalnutMsgDisplayerImpl WalnutMsgSetDisplayerImpl WalnutPrintImpl WalnutWindow WalnutNotifierImpl a hodge-podge of operations WalnutWindowCommandsImpl WalnutWindowInternalImpl WalnutWindowMenuImpl WalnutMsgDisplayerImpl WalnutMsgSetDisplayerImpl WalnutMsgSetButtonsImpl WalnutPrintImpl LoganWalnutImpl WallabyImpl WalnutWindowSidedoor WalnutMsgSetButtonsImpl more hodge-podge WalnutWindowCommandsImpl LoganWalnut LoganWalnutImpl database entry conversions Wallaby WallabyImpl message browser See WalnutCatalog.tioga for a listing of which procedures are exported by which modules. Using logs Walnut writes an operation log that keeps track off all notable events. This log contains operations like creating message sets, moving messages between sets, etc. It also contains the text of all messages. For reliablity reasons the log is append-only. The expunge operation writes a new log that contains all of the messages that have not been deleted. Walnut also maintains a database that indexes information that is in the log. The database is used to enumerate message sets and such. The database can always be completely recovered by scavenging the log. New mail processing What happens when mail is retrieved from a mail server? AutoNewMailProc is called by WatchMailBox, NewMailIdleProc, CheckMailBoxes. WalnutNotifierImpl.RestartWalnut Calls WalnutOps.Startup Checks validity of database and such Returns newMailExists if NewMailLog is non-empty or servers have mail Calls WalnutWindowInternalImpl.EnableNewMail Calls WalnutNewMail.EnableMailRetrieval Registers NewMailIdleProc Forks AutoNewMailProc when coming out of idle Calls MailRetrieve.Create[ msgPollingInterval, WatchMailBox ] Calls MailRetrieve.NewUser for each name/password Calls WalnutNewMail.GetLastMailBoxStatus WalnutNewMailImpl.WatchMailBox Forks AutoNewMailProc Calls DoNewMail Calls MailRetrieve.MailboxState to check state For each server Calls DrainServer Writes messages to NewMailLog Calls MailRetrieve.Accept Calls MailFetchDone Calls MailHandle.notifyProc WalnutWindowMenuImpl.NewMailProc Calls WalnutNewMail.CheckMailBoxes if mouseButton = blue Forks AutoNewMailProc What happens when the user clicks "NewMail"? Calls WalnutWindow.GetNewMail (WalnutWindowInternalImpl.GetNewMail) Calls WalnutInternal.AddNewMsgsToActive (WalnutMsgSetDisplayerImpl.AddNewMsgsToActive) Calls WalnutOps.GetNewMail with proc that calls BuildMsgLineViewer Calls WalnutOpsInternal.DoNewMail with proc Checks for operation in progress Gets length of newmail log Copies newmail log to current log Records that a ParseLog operation is in progress Calls ParseLog Calls WalnutDB.AddNewMsg (WalnutDBMsgImpl.AddNewMsg) for each message, etc. Pass message to Tapestry filtering agent Wake up filtering agent and wait until it is done Calls WalnutDB.EnumerateUnacceptedMsgs (WalnutDBMsgSetsImpl.EnumerateUnacceptedMsgs) with proc Calls proc for each "unaccepted" message in database Gets list of servers and message counts Calls WalnutOps.AcceptNewMail Calls WalnutOpsInternal.DoAcceptNewMail Calls Records acceptance of new mail Calls WalnutDB.AcceptNewMail (WalnutDBMsgSetsImpl.AcceptNewMail) Clears "unaccepted" property for each message Zeros count for each server Clears opinprogress, etc. Increments version number of active message set Prints summary of number of messages delivered Expunging old mail When a user clicks "Expunge": Calls WalnutWindowMenuImpl.MenuExpunge Calls WalnutWindowCommandsImpl.Expunge Calls Quitting Closes all Walnut viewers, etc. Reports start of expunge Calls WalnutOps.ExpungeMsgs Calls WalnutLog.ExpungeMsgs Writes "ExpungeMsgs" entry in log Calls WalnutDBMsgSetsImpl.ExpungeMsgs Updates size of Deleted message set and total message count For each message in Deleted Removes all information about message from database Writes record of BytesInDestroyedMsgs and FirstDestroyedMsgPos to database Bumps version number of Deleted Calls DestroyOrphanAddrsAndSubjs Does nothing! Calls WalnutOps.CopyToExpungeLog Checks quota and space needed for expunge log Calls WalnutLog.WriteExpungeLog Writes "WriteExpungeLog" entry in log Calls WalnutOpsInternal.DoLogExpunge Until done with all phases of expunge do Calls WalnutDB.GetLogExpungePhase Reads expunge phase from database If phase is idle then Go to initializingExpungeLog phase If phase is initializingExpungeLog then Calls WalnutLogExpunge.StartExpunge Writes first log record on expunge log Go to writingExpungeLog phase If phase is writingExpungeLog then Calls WriteExpungeLog Calls WalnutLogExpunge.RestartExpunge Prepares for expunge Reports "Copying the head of the log..." Calls WalnutLogExpunge.CopyBytesToExpungeLog Copies bytes from current log to expunge log Reports "Processing the tail of the log..." Copies some entries from the current log to expunge log and skips others Calls WalnutLogExpunge.EndExpunge Calls WalnutRoot.StopExpunge Closes expunge log Go to swappingLogs phase If phase is swappingLogs then Reports "Swapping Log Files" Updates expunge info and sets phase to idle Updates root file version timestamp Calls WalnutRoot.SwapLogs Swaps the current log and expunge log in root file Reports "Finished expunge" Reports result of expunge Restarts Walnut Browsing messages (Wallaby) Implemented in: LoganWalnut.mesa, LoganWalnutImpl.mesa, Wallaby.mesa, WallabyImpl.mesa. What happens when a user clicks "Query" in the Walnut control window? Calls Wallaby.QueryProc Creates new LoganBerryBrowser.ToolBody Calls LoganBerry.Open[NIL, wH.opsH.dbName] Calls LoganBerryBrowser.CreateTool What happens when a user clicks "Browse" in the Wallaby browser? Calls WallabyImpl.BrowseProc Calls DoBrowseProc Calls BrowseBody Calls LoganBerryBrowser.ReadEntryForm Calls LoganQuery.QueryEntries to get query plan (ignores cursor) Calls LoganBerryBrowser.ReportFeedback Calls GenerateWalnutEntries Calls LoganWalnut.GenerateEntriesPlusDate Calls WalnutOps.GenerateEntriesPlusDate Calls WalnutDBMsgImpl.GenerateEntriesPlusDate Calls LoganBerry.GenerateEntries Builds and returns a LoganBerry.Cursor Builds and returns a LoganQuery.ComplexCursor Builds up filters by calling LoganQuery.FilterEntries Reports "Query in Progress" Retrieves allmatching entries by calling LoganQuery.NextEntry Reports "Query Finished" Changes made for PCedar/Tapestry Removed dependencies on Alpine and Cypress. Changed parsing code that looked for \n to check for either '\r or '\l. Changed names of temp files. Removed LoganWalnut and private bound in copy of LoganQuery and LoganBerryBrowser.