-- file: NuthatchVoiceFileSystem.txt
-- old description of the voice file interface
-- last edited by Lia on September 30, 1983 3:30 pm
The Nuthatch Voice File System
The Nuthatch program provides an interface allowing Walnut and other applications to make and use recorded voice files. In its first implementation, Nuthatch will permit any message sent by the Walnut mail system to include, along with its text, an attached voice file. Nuthatch maintains on Alpine a directory of all such voice files.
Functions provided to user.
The Nuthatch user can record and play back voice files, which he can attach to Walnut mail messages and send. Walnut messages with voice attachments contain a special attachment field in their headers, so the receiver can know to play them. Such voice attachments may be played by users who use Walnut (and Nuthatch) and have etherphones. Other recipients will be unable to play the messsages but will see the voice attachment field in the header.
All users of Nuthatch must keep their Walnut logs and their Walnut segments on Alpine.
The database, logs, and transactions
The representations of voice files are stored on Bluejay, the voice file server. Voice files are too large to replicate for each recipient, and so the Nuthatch voice file directory provides a way for multiple users to refer to the same voice file without storing it more than once. The Nuthatch directory maintains reference counts for every voice file, and deletes those no longer in use. Whenever a voice flie is created, it is given a default lifetime in the file system of two weeks. A voice file may continue to exist for longer if a user indicates he wants to keep it around. He can do so by explicitly requesting it to be kept by clicking a button, or by moving a Walnut message to which the voice file is attached, into an "interesting" message set; that is, one other than Deleted or Active.
The voice file directory resides on the Alpine file server. It is organized as a Cypress database, described in detail below. In addition to this database, there is a log for each user of Nuthatch. Operations on voice files (like sending, receiving, and deleting them) cause entries to be written to an individual's log. An independent updating procedure reads the log entries and performs the appropriate updates to the voice file directory.
At any time, it is possible to re-create a damaged or absent voice file directory by replaying the individual voice file logs. They may be replayed in any order, but must all be replayed.
Updates to the voice file directory and to the Nuthatch logs are performed using Cypress's atomic transactions.
For the operations of sending a message and updating the database from the log, the transaction is defined by the Nuthatch code and exceptions are handled by it. The database operations performed when getting new messages is done under the transaction defined by Walnut in WalnutOps.NewMail. Likewise, the Nuthatch database operations performed by the AddMessage, RemoveMessage, and Expunge functions are all done under the WalnutDB transactions.
The life of a voice file under Walnut
When a user is in a Walnut Send viewer, he is presented with a dummy new message form. Nuthatch puts three menu buttons into this viewer: Record, Stop Recording, and Attach. When he clicks the Record button, Nuthatch makes a call to the Bluejay, requesting a place to store the utterance about to be recorded. The user is prompted to begin speaking, and recording proceeds until he clicks Stop (or hangs up). When he clicks the Attach button, the Attachment field of the header in the new message form gets filled in with a descriptor (handle? voice file ID?) for the newly created voice file.
None of these actions has modified the state of the voice file directory. When the user clicks Send (or Store Message), Nuthatch informs Bluejay that this utterance is now in use, and sets a default expiration date for it. Sending the message causes Nuthatch to write a log entry containing information about the new message, including its creator, length, and encryption key. The write to the sender's log is not done under any transaction; it need only exclude readers during the course of the write.
Now the message has been sent to some recipients. When it is fetched using the NewMail function, Nuthatch writes a log message for each recipient, indicating that that user now has a reference to that voice message. This log entry will be used to update the recievedList relation in the Nuthatch database. These log entries are written under the aegis of the NewMail transaction called from WalnutOps. Moving a message to an "interesting" message set, by using the AddTo or MoveTo function in Walnut, causes the reference count of the voice message to be increased. Removing it from such a message set decreases the reference count.
Users not running Walnut or Nuthatch will be able to read the text portions of the message but not hear the voice attachment. Those running Nuthatch will have in their message viewers a Play button that, when clicked, plays the piece of voice attached to the Walnut message. The other special button in the menu of this viewer is a Delete Voice button, to be used when the recipient wants to keep the text part of a message but doesn't want to preserve the voice attachment. The effect of deleting the voice part is to write a log entry saying to decrement the reference count of this voice message, if this Walnut message's reference to it had been counted already.
A user can compose a Walnut message and attach to it a voice message he has recieved, by copying the voice file ID from the attachment field of the old message to the attachment field of the new message.
Expunging a Walnut message containing voice causes its tuple in the ReceivedList relation to be removed. This operation is performed under the WalnutDB Expunge transaction.
The updating process
There is a process on each workstation that performs the task of reading the user's Nuthatch log and updating the voice file directory from it. This update is performed as a single transaction involving both the user's log and the voice file directory. This transaction is open for only as long as it takes to read to the end of the log and make the appropriate changes to the database, since various users' updating processes will be competing for access to the database. The current read point and end point of the log are stored at the beginning of the log.
Garbage collection
The garbage collector runs at night, when the use of the voice message system is expected to be low. For each voice message in Bluejay that has exceeded its default expiration time, the garbage collector consults the ReceivedList relation to see if there are any references to it there. If not, the entries in the ReceivedList and VoiceReference relations concerning that voice message can be deleted, and the space on Bluejay used by that message can be reclaimed.
The garbage collector locks the entire Nuthatch database while it is running, and so no voice messages can be sent of received while it is running.
Details of the database:
The database comprises the following two relations.
voiceReferences: Relation :
voiceFileID: Attribute; -- VoiceMessage (rope)
(tuneNumber: Attribute; -- index into Jay table, determined by ID)
(recordingTime: Attribute; -- Time, determined by ID)
creator: Attribute; -- RName
referenceCount: Attribute; -- NaturalNumber (derivable from the total of
the InterestingCopyCounts for this voiceMessageID)
samples: Attribute; -- NaturalNumber
startingSample: Attribute; -- NaturalNumber
expirationDate: Attribute; -- Date
encryptionKey: Attribute; -- Key
time: Attribute; -- Date
type: Attribute; -- (Simple/Structured)
InterestList: Relation :
voiceFileID: Attribute; -- VoiceMessage
user: Attribute; -- RName
IDType, ID; -- Ropes, I guess
Interested: Attribute; -- boolean
Here is how the following Walnut/Nuthatch functions affect the voice file database and Bluejay. The message sets Active and Deleted are considered uninteresting; all others are interesting.
RecordMessage,
Get from Bluejay an index into the tune array. Use returned info to build attachment field.
{StopRecordingMessage,
Keep track of number of samples; update Bluejay info}
AttachMessage,
Tell Bluejay that this tune is now in use? Fill in Attachment field in header.
DeleteVoice, -- request to lose voice attached to a received message
Turn off the interest bit in this tuple of the interestList relation, by calling Nuthatch.RemoveInterest.
A nuthatch log transaction, not embedded in any others.
SendMessage (or StoreMessage)
Tell Bluejay that this utterance is now in use.
Notify Bluejay to update the default expiration time of the utterance in this
voice file (? When?).
If this ID is new to voiceReferences, add (voiceMessageID, referenceCount:0,
encryptionKey, samples, type);
otherwise add the new information to the existing tuple.
A nuthatch log transaction, not embedded in any others.
GetMessage, -- from Grapevine, by doing NewMail
Add receivedList(recipient, voiceMessageID, grapevineMessageID: GVID, counted: false).
If it's already there (Walnut bug -- duplicate message), do nothing.
Done under the Walnut log transaction in WalnutOps.NewMail.
ReadMessage (GVID, voiceMessageID)
Call Nuthatch.MakeInterestEntry to associate the GVID with the voiceMessageID in the voice file directory, for this user.
AddMessage, -- to msg set
If adding it to an interesting msg set, increment the count field in the
receivedList tuple for this message.
If adding to an uninteresting msg set, do nothing.
Done under WalnutDB transaction started by WalnutOps.AddMsgToMsgSet
RemoveMessage, -- from msg set
If removing it from an interesting message set, decrement the count in
receivedList tuple.
If removing it from an uninteresting message set, do nothing.
Done under WalnutDB transaction started by WalnutOps.RemoveMsgFromMsgSet
ExpungeMessage
Remove tuple from receivedList for this message and recipient. If the
count for that tuple isn't zero, there is an inconsistency, since the message
can only be expunged from the deleted set, and so it should not appear in
any interesting sets.
Done under WalnutDB transaction started by WalnutOps.Expunge
Changes to the database are effected by writing entries. to the user's NuthatchLog. These entries are processed by an updating process that reads them and reflects them in the Nuthatch segment.
What is there?
The user does
Run walnut
What happens?
Walnut sees in user.profile that you want Nuthatch started up.
Initially, Nuthatch looks on Alpine for the Nuthatch log for this user, and (if none, then what?) looks on Alpine for the Voice File Directory. If none, reconstruct it from the schema description and the logs.
OK, database is set up. start up an updating process for this user? Could do it now, or wait until some voice-related function happens, to avoid the overhead of spawning the process if it's not needed.
What happens in response to
NewMail: WalnutOps.NewMail is called, fetching from Grapevine all the user's waiting messages. For each mail drop interrogated, Walnut opens a transaction to read in the messages. Within this transaction, Nuthatch writes a log entry for each message retrieved, to update the interestedList relation in the Nuthatch database. The log writes are done within the transaction; if it aborts, they aren't written.
Delete, Create: When a message set is created, Nuthatch doesn't care. When a message set is deleted, all the messages in it are Removed, and so are taken care of in WalnutOps.RemoveMsgFromMsgSet.
SizeOF, Print: These message set operations have no effect on Nuthatch.
NewForm: A Walnut Send Viewer gets created. It may contain an Attachments line in the header, if the user has specified a Recording viewer as the default Send Viewer. The menu contains three new buttons: Record, StopRecording, and Attach.
Attach: If there was no attachments field before, one is created.
If he has just recorded a message, the ID should be moved from where it was into the attachments field.
If not, the attachment field is prepared for type-in.
Record: All the Bluejay connection stuff happens, and a new MsgID gets created and put into a viewer someplace handy.
StopRecording: All the stop recording stuff happens.