This document describes the standard facilities available for file access from a Cedar 5 workstation. These facilities are available through the interfaces FS, FSBackdoor, and FSExtras. FS contains facilities of interest to all users. FSBackdoor contains facilities of interest to experts. FSExtras contains items that will be added to FS or FSBackdoor the next time they can be recompiled. In cases where the descriptions of functions in the interface files differ from the descriptions in this document, this document should be viewed as the truth. All interfaces are available through /Indigo/PreCedar/Top/FS.df. The implementation of FS is contained in the Cedar 5 boot file.

All interfaces are available through /Indigo/PreCedar/Top/FS.df. The implementation of FS is contained in the Cedar 5 boot file.

FS is a file system for use on a Cedar workstation. It provides access both to remote file servers and to the local disk. Remote files accessible from FS must reside on a file server that supports the FTP protocol, in particular IFS's and Alpine servers. These file servers may be accessed from many Cedar instances on different workstations at the same time. Local files are accessed through an abstraction called the local server. The local server is the set of logical volumes on the local disk of a Cedar instance. One of these logical volumes may be designated the system volume. FS provides a directory for each volume, and a cache for remote files on the system volume.

FS allows clients to provide the implementation of open files. Open files obtained from client packages may be presented to FS procedures that operate on open files. Client-provided open files may be local or remote, or many not be disk files at all. Direct, page-at-a-time access to Alpine servers is provided through this mechanism.

FS also contains facilities for creating IO.STREAM's on files, for binding local names to remote files, for finding the version of a file that was created at a particular time, and for limiting the number of extant versions of local files.

FS is a successor to the Cedar Interim File System (CIFS) that was developed by Dave Gifford with help from Larry Stewart. The design of FS benefited from advice offered by Andrew Birrell, Mark Brown, Butler Lampson, Roy Levin, Roger Needham, Eric Schmidt, Larry Stewart, Paul Rovner, and Ed Taft. The main designer and implementor of FS was Michael Schroeder.

Most clients of FS simply want to create a file stream with which to read or write a file. If that's all you want to do then this section contains all you need to know about FS. Suppose that r is a ROPE containing a file name. If your program needs to read characters from the file named by r (local or global file name, probably with no version part) from a STREAM "s", it should call

If your program will provide all new contents for the file named r (it ignores any existing versions of the file and creates a new version), it should call

If your program is logging output to the file named by r (it updates the file only by adding new characters to the end, and creates a new file of that name if none exists), it should call

Finally, if your program will both read and write the file named by r (it treats the file as an extendable, random-access sequence of bytes that it updates "in place"), it should call

It is an unusual program that requires $write mode, while $create mode is used frequently. The last three forms only work for local files.

If you do not wish to land in the debugger when the file name r is misspelled or otherwise garbled, your calls on FS.StreamOpen should be protected with a catch phrase something like

The moral of this tale: doing simple things is simple. You do not need to understand the multitude of parameters to FS.StreamOpen and FS.StreamFromOpenFile, since they default correctly for most purposes. You do not need to use any procedure from this interface except FS.StreamOpen unless you are doing something special. There is only one signal to catch.

Having created a file stream, you may have questions about the semantics of the generic IO operations when applied to this stream: will IO.SetIndex extend the file, and what does IO.Reset do? To answer these questions, read the "File Streams" section below.

It is also possible that you are not interested in streams at all, but want to do something else like enumerate a directory. Read on, the answer is here somewhere.

Client's of FS should only have to catch the single ERROR FS.Error. In particular, errors from the File package and the STP package are mapped into FS.Error's. Other packages that hand out FS.OpenFile's (see section 10) are expected to raise FS.Error.

FS error codes are partitioned into five groups: bugs, environment errors, lock conflicts, client errors, and user errors. A bug usually indicates that something unexpected has gone wrong inside the implementation of FS or lower packages. An environment error means that something in the external environment has changed that requires human intervention. A lock conflict means that some other client has the locks that this client needs to get its job done. A client error is an attempted illegal operation that probably is caused by a bug in a client program. A user error is an attempted illegal operation that probably is caused by a human user's actions. The distinction between the last two groups is somewhat arbitrary.

An error description reports the group, as well as a specific code ATOM indicating exactly what went wrong. In the descriptions of procedures that follow, all codes from the error groups "client" and "user" that can be raised are specifically listed. Errors in the other groups should be expected as appropriate. The "explanation" is intended to be a description of the error suitable for human consumption; it frequently contains the name of the file being operated upon.

Catching all FS.Error's with error.group = user and reporting the error.explanation to the human user is usually sufficient to deal with file names that are mistyped by humans or that do not correspond to existing files.

The FS implementation is careful never to raise FS.Error while locks are held. Any FS procedure can be called from a catch phrase for FS.Error.

Error codes are ATOM's so that packages implementing FS.OpenFile's via FSBackdoor.CreateProcsOpenFile can extend the set of codes. Following is a complete list of the codes generated by the FS implementation. The list is partitioned by the FS.ErrorGroup.

Names for FS files are called "FNames". An FName consists of (in order) a server, a root directory, zero or more subdirectories, a simple name, and a version. An FName may not exceed 120 characters excluding the version part. The version part will always fit in 6 characters, so the maximum space an FName can occupy is 126 characters. Any name discovered that contains more than 120 characters before the version part will cause FS.Error[user, $illegalName] to be raised.

Both FTP bracket syntax and CIFS slash syntax for FNames are accepted, although bracket syntax is used internally, in all FNames returned from FS procedures, and in FS.Error explanations. The following constructs are equivalent:

The structural characters that delimit the server part of an FName are "[" and "]", or "/". Non-structural characters allowed in the server part are all alphameric characters plus ".", "$", "-", "+" and "#". The structural characters allowed in the rest of an FName are "<" and ">", or "/". The non-structural characters are the same as for the server part with the exception of "#". The additional structural character "!" is used as the version part prefix. The rest of the version part can be a number in the range [1 .. 65534], or one of the letters "L" or "H". Case is not significant in FNames.

An empty server part means the local server. Otherwise the server part is presumed to name a remote file server.

The names of files on remote servers are called "GNames", for global names. The detailed requirements on the structure of a GName are defined by the type of server named. The most common remote servers (IFS and Alpine) require the presence of a root directory part and include version parts in all file names.

The names of files on the local server are called "LNames", for local names. In an LName, the root directory part is the name of a volume; an empty root directory part refers to the "system" volume (the one from which Cedar was booted). All LNames have version parts. For example, the following are complete LNames:

The working directory mechanism described below eliminates the need for the server and root directory part on most LNames presented to FS procedures.

Fine points:

The local server and most remote servers include version numbers in file names. Version numbers allow for multiple instances of a file to exist at one time. FS enforces the policy that the version on any newly created file is one larger than the largest existing version, the so-called "next" or !N version. It is not possible to create any other version with FS. When accessing an existing file the version part of an FName may be a number, may be a variable, or may be omitted. The variables allowed are !L, meaning the lowest existing version number, and !H, meaning the highest existing version number. When omitted, the version part defaults to !L or !H, depending on the operation being performed.

An FName that does not start with a server part is called a partial name and will be interpreted relative to a working directory. The working directory may be specified as the argument named "wDir" to FS procedures, or may be defaulted.

To expand the arguments "name" and "wDir" of an FS procedure into a full FName, FS checks the first character of "name". If it is "[" ("/") then "name" is the full FName. If not then FS looks for a working directory name in three places in order:

The first ROPE found whose length is not 0 is preprended to "name" to form the full FName. A working directory obtained from 1 or 2 can be local or global. If it does not end with "]" or ">" ("/") then a ">" ("/") is appended. The default working directory must be local.

The default working directory can be set with the procedure:

The default working directory can be read with the procedure:

FS contains utilities for parsing and constructing FNames.

File properties provided by FS are page count, byte count, created-time, and keep. The page count is the number of pages allocated to the file. The byte count is the number of meaningful bytes in a file; the byte count is kept up-to-date by explicit client calls on FS procedures. The created-time is the date and time when the contents of the file were created (or changed). The keep specifies how many versions of a file to keep around.

Procedures are provided for converting between pages counts and byte counts or word counts:

A property for each LName is the "keep", a CARDINAL that specifies the number of versions of an LName that should be kept around. Whenever an LName is created, its "keep" is inherited from the existing !H version or set from an argument to the procedure doing the creation. The keep of the existing !H version can be changed with FS.SetKeep.

Keep processing occurs when creating a new version of an LName. In this case, FS will enumerate existing version in decreasing order from !H. After "keep - 1" versions are encountered in this enumeration, additional versions will be deleted if not open. Thus, the keep is the steady state count of versions to be retained. The local volume file of a deleted version will be reused as the disk file for the new version being created. For example, if the only existing version of a file is named Example.bcd!4, it has a keep of 1, and no client has it open, then FS.Create[Example.bcd] will cause Example.bcd!4 to be deleted and its disk file to be reused for Example.bcd!5.

Calling FS.SetKeep will have the side effect of deleting files that are obsolete relative to the new keep, even when the keep is not changed.

Note that setting a keep to LAST[CARDINAL] will result in all versions being kept, since there can never be that many versions. A keep of 0 is illegal.

No remote server implements keeps right now.

For certain FS procedures the identification of which global file to operate on can include the version variables !L or !H, or can be defaulted to the variables !L or !H (depending on the operation). Binding a version variable to a particular version would be straightforward if the remote server were always accessible and if the client were always willing to incur the cost of accessing it. In some circumstances, however, a client may wish to complete the binding based on the possibly incomplete information available from the cache. The "remoteCheck" argument to FS.FileInfo, FS.Open and FS.Copy allows the client to control the scope of the version search. (FS.Rename and FS.Delete do not include a "remoteCheck" argument because renaming and deletion occur synchronously on the remote server, so there is no option.) The "remoteCheck" argument is evaluated only when a GName is specified. When "remoteCheck" is TRUE then FS binds a version variable by searching the versions on the remote server. The operation fails if the remote server is inaccessible. When "remoteCheck" is FALSE then FS binds the version variable by searching the versions in the cache; only if no cached version is found is the remote server interrogated.

All of the FS procedures that accept an FName argument also include a "wantedCreatedTime" argument. This time provides an alternative way to name a file. If "wantedCreatedTime" is not equal to BasicTime.nullGMT, then the file to be operated upon is determined by a search for the version with the specified created-time. Any version part included in the name is treated as a hint. The created-time of a file is treated as a unique identifier of the content. If a version of a global file with the wanted created-time is found in the cache, then FS will not bother to check with the remote server (even if "remoteCheck" is TRUE) to see if the same created-time to version binding exists there.

FS includes facilities for attaching an LName to a GName and created-time. An attachment is a name binding. Attachment is a way to give an LName to a global file, and is useful as a cheap way to copy global files into local files. "BringOver" and "SModel" are the primary clients of the attachment mechanism.

Attachments are created using certain modes of FS.Copy. For most FS procedures, an attached LName behaves as though the global file were in fact copied into the local file when FS.Copy was invoked. There are four exceptions. First, the various procedures for obtaining information about files inform the client when an LName is attached and provide the GName of the attachment. Second, opening an attached LName requires opening the global file; if the global file is not in the cache and the remote server is down or the file has been deleted on the remote server, then the open will fail. Third, for a "read" open both the local and global names are read locked. Fourth, for a "write" open, the attachment is broken and the contents of the global file are copied into a local file; the local file assumes the LName and the GName is forgotten.

For access to remote servers FS uses the credentials obtained from the UserCredentials interface. If these do not work then FS.Error[environment, $badCredentials] is raised. FS currently has no facilities for setting or using secondary credentials. Thus, access to files in password protected directories on servers is not possible.

There are two types of locks that can be set on files: "read" and "write". These locks are local to a Cedar instance. Whenever the required lock cannot be obtained, FS raises FS.Error[lock, $lockConflict].

A "read" lock provides shared access to a file. Holders of "read" locks for a file may only read its data or properties. A "read" lock can be set anytime no "write" lock for that name is held. Thus, multiple "read" locks may be held. While a "read" lock is held no other client can delete the file or change its name.

A "write" lock provides exclusive access to a file. The holder of a "write" lock may alter the file's contents and properties. An implicit "write" lock is required to rename or delete a file. While a "write" lock is held no other client can open the file, delete it, or alter its name. A "write" lock can be set only when no other locks are held.

Enumerations are specified with a "pattern" argument, which is an FName in which the character "*", meaning "match anything", may appear zero or more times. "*" may not appear in the server part, or in the root directory of an LName. The presence or absence of a server part in "pattern" controls the use of working directories as usual. The version part of a "pattern" may be !H, !L, !*, or ! followed by digits; or may be omitted (defaults to !*). It cannot contain a mixture of "*" and digits. The order of the enumeration may be different for different sorts of servers. For the local server and IFS's, the enumeration is in lexical order of LNames expressed in bracket syntax (lower cases letters are mapped to upper case) but without the version part, and then in numeric order by version number.

Monitor locks are not held during an enumeration. Thus, the client may call FS.Delete from an FS.InfoProc or an FS.NameProc. Fine point: deleting or storing (by FS.Copy) ahead of the current position in a remote enumeration can produce unexpected results; the change may not be reflected in the enumeration.

An FS.OpenFile may be presented to other procedures to get access to the file's contents and properties. All global files opened using the FS procedures defined below are accessed via the FTP protocol and the cache.

Other packages can include procedures that return FS.OpenFile's. In such cases, the arguments to the opening procedures and the detailed semantics of operations on the client-provided FS.OpenFile's are defined by that package. (The operations are defined by providing procedures to FSBackdoor.OpenFileFromProcs). For example, the FS.OpenFile's obtained by calling the Alpine package directly do permit use of Alpine's special facilities for transaction, locking, and page access. Generation of such FS.OpenFile's is via Alpine procedures and completely bypasses FS's locking and cache.

An existing file may be opened using the following procedure:

There are two important restrictions on file creation. First, no version may be specified; FS always creates the !N version. Second, global files may not be created; only FS.Copy and FS.Rename can cause new global file to come into existence.

The following procedure is provided as a convenience for clients that wish to open an existing file for write, but create one if none exists.

A file is a sequence [0..fileLen) of mutable bytes; the length (fileLen) of a file can be changed.

The state of a file stream is a file (perhaps opened under a transaction trans) with length fileLen, an index streamIndex IN [0..fileLen], plus a number of readonly flags (accessRights, streamOptions) and a BOOL variable closed which jointly determine the effect of stream procs.

Here is what the generic stream operations do when applied to a file stream:

File streams provide no interlocks of their own to control concurrent access to files. Instead, they rely on the underlying file system for concurrency control. FS uses locking to provide concurrency control.

Individual file streams are independent objects, so there is never any need to synchronize calls to their procedures. If two processes are to share a single file stream, they must synchronize their accesses to that stream at a level above the file stream calls; individual file stream calls (PutChar, PutBlock, etc.) are not guaranteed to be atomic.

An FS file is represented as a sequence of disk pages, each containing FS.BytesForPages[1] bytes. There is a single leader page (overhead) that is used to hold file properties such as the byte count and create time. There is no functional relationship between the byte count of a file and the allocated size of that file in pages.

Allocating disk pages to a file is an expensive operation, especially when done incrementally (one page at a time). Extending a file many times can result in a fragmented file. If a client program can estimate the eventual length of a file it is creating, it should use the "createByteCount" parameter to FS.StreamOpen to allocate enough space to hold the entire length all at once; if a client program is updating an existing file then it should create a stream and then extend the file using IO.SetLength. If the file stream implementation must extend a file it will always extend it by several pages; if asked to shorten a file it will simply adjust the byte count field in the leader page without freeing any pages. If the streamOption "truncatePagesOnClose" is specified (the default), then extra pages are freed when the stream is closed.

The following TYPE's and constants are used in conjunction with creating file streams.

File STREAM's can be open from either a file name or an FS.OpenFile.

It is also possible to get the FS.OpenFile back from a file STREAM.

Finding out about errors from generic STREAM procedures is a two step process. First IO.Error[$Failure, self] is raised and is caught by the client. The client then calls FS.ErrorFromStream to obtain more detailed information about the ERROR.

When copying files, only the !N version of the target file can be created. Attachments are created by setting the "attach" argument of FS.Copy to TRUE. Copying from one remote server to another is implemented by retrieving the "from" file into the cache on the system volume.

Only the !N version of a file may be created by FS.Rename. Renaming from one server to another is allowed. In the case of the "from" and "to" names being on the same remote server or the same local volume, remaining occurs without copying. Otherwise, FS.Rename is like FS.Copy followed by FS.Delete.

Only the keep of the !H version may be set. The "name" argument to FS.SetKeep cannot contain a version part.

The interface FSBackdoor.mesa contains facilities that should be of interest only to experts.

FS provides a way for clients to implement objects with the TYPE FS.OpenFile. The client provides procedures that implement the operations on an FS.OpenFile. Refer to section 7 above for the semantics of these procedures. Note that no locking is done by FS on client-provided FS.OpenFile's. FS produces FS.Error[client, $notImplemented] when an attempt is make to use any client procedures that are NIL.

The client procedures should raise FS.Error as appropriate. Client procedures should generate the code ATOM's used by FS where meaningful, but can produce additional code ATOM's if needed. FS.Error's with existing code ATOM's should be generated using FSBackdoor.ProduceError (see below). For any new code ATOM's, the client is responsible for providing the appropriate FS.ErrorGroup.

FS provides access to the File package procedures for setting the volume freeboard on the system volume and reading the size, free and freeboard counts from a named volume.

The directory of LNames and the cache for GName are implemented together as a BTree on a local volume. The following procedures and TYPE's allow direct access to this BTree.

A "nameBody" is an FName without a version part. LNames are recorded in the directory/cache without the "[]<>" prefix. Thus the nameBody for the LName "[]<>foo.mesa!5" will be "foo.mesa", and the nameBody for the GName "[indigo]<Cedar>Top>FS.df!47" will be "[indigo]<Cedar>Top>FS.df". The version part is represented as a scalar. The version for a GName that includes no version is "noVersion", which value otherwise will not occur.

"highestVersion" and "lowestVersion" are used internally as the representation of !L and !H.

An entry in the directory/cache can be gotten at directly via FSBackdoor.Enumerate and the following TYPE's.

*************************************************************
WARNING: You should treat FSBackdoor.Entry's as READONLY
*************************************************************

FSExtras contains two procedures for reporting the occurance of events. FSExtras.NextRemoteEvent reports all actions on remote files, and is intended for use by clients like the "WatchTool". FSExtras.NextCreateEvent reports all creations of new LName - to - data bindings, and is intended for use by clients like "Tioga" that may want to know when new versions of local files come into existence.

The following is the obsolete facility for reporting on remote events.