[Indigo]<Cedar>FS>FS.mesa!1

**** SECTION HEADINGS AND ALL MESA STATEMENTS ARE AT THE TOP LEVEL ****

FS.mesa -- Interface for the Cedar File System

See also FSBackdoor.mesa

File system defined by M. D. Schroeder

File stream creation facilities defined by Mark Brown

Last Edited by:

Schroeder, September 22, 1983 8:32 am

Taft, June 9, 1983 10:46 am

Birrell, July 8, 1983 2:18 pm

MBrown, September 21, 1983 10:16 pm

Levin, September 22, 1983 12:53 pm

DIRECTORY

BasicTime USING [ GMT, nullGMT ],

IO USING [ STREAM ],

Rope USING [ ROPE ];

FS: CEDAR DEFINITIONS =

BEGIN

ROPE: TYPE = Rope.ROPE;

STREAM: TYPE = IO.STREAM;

EVERYTHING YOU REALLY WANTED TO KNOW ABOUT FILE ACCESS

Most clients of FS simply want to create a file stream with which to read or write a file. If that's all you want to do then this section contains all you need to know about FS. Suppose that r is a ROPE containing a file name. If your program needs to read characters from the file named by r (local or global file name, probably with no version part) from a STREAM "s", it should call

s ← FS.StreamOpen[r];

If your program will provide all new contents for the file named r (it ignores any existing versions of the file and creates a new version), it should call

s ← FS.StreamOpen[r, $create];

If your program is logging output to the file named by r (it updates the file only by adding new characters to the end, and creates a new file of that name if none exists), it should call

s ← FS.StreamOpen[r, $append];

Finally, if your program will both read and write the file named by r (it treats the file as an extendible, random-access sequence of bytes that it updates "in place"), it should call

s ← FS.StreamOpen[r, $write];

It is an unusual program that requires $write mode, while $create mode is used frequently. The last three forms only work for local files.

If you do not wish to land in the debugger when the file name r is misspelled or otherwise garbled, your calls on FS.StreamOpen should be protected with a catch phrase something like

! FS.Error => IF error.group = user THEN {
YourProcedureToShowRopeToUser[error.explanation]; CONTINUE }

The moral of this tale: doing simple things is simple. You don't need to understand the multitude of parameters to StreamOpen and StreamFromOpenFile, since they default correctly for most purposes. You don't need to use any procedure from this interface except FS.StreamOpen unless you are doing something special. There is only one signal to catch.

Having created a file stream, you may have questions about the semantics of the generic IO operations when applied to this stream: will SetIndex extend the file, and what does Reset do? To answer these questions, read the "File Streams" section below.

It is also possible that you aren't interested in streams at all, but want to do something else like enumerate a directory. Read on, the answer is here somewhere.

INTRODUCTION TO FS

FS is a file system for use on a Cedar workstation. It provides access both to remote file servers and to the local disk. It includes procedures for creating IO.STREAM's on files. In addition to the normal file system facilities for manipulating named files, FS also contains facilities for cacheing remote files, for binding local names to remote files, for finding the version of a file that was created at a particular time, for clients to provide the implementation of open files, and for limiting the number of extant versions of local files.

FS is a successor to the Cedar Interim File System (CIFS) that was developed by Dave Gifford with help from Larry Stewart. The design of FS benefited from advice offered by Andrew Birrell, Mark Brown, Butler Lampson, Roy Levin, Roger Needham, Eric Schmidt, Larry Stewart, Paul Rovner, and Ed Taft.

FSBackdoor.mesa is an additional interface to the Cedar File System that contains facilities intended for use by experts.

ENVIRONMENT

FS provides access to files on any remote file server that supports the FTP protocol, in particular IFS's and Alpine server's. These file servers may be acccessed from many Cedar instances on different workstations at the same time.

The local disk is accessed through an abstraction called the local server. A Cedar instance is associated with a set of logical volumes on the local disk, one of which may be designated the system volume. FS provides a directory for each volume. It also provides on the system volume a cache for remote files.

FS allows clients to provide the implementation of open files (see FSBackdoor). Open files obtained from client packages may be presented to FS procedures that operate on open files. Client-provided open files may be local or remote, or many not be disk files at all.

ERRORS

Client's of FS should only have to catch the single ERROR FS.Error. In particular, errors from the File package and the STP package are mapped into FS.Error's. Other packages that hand out FS.OpenFile's (see FSBackdoor.CreateProcsOpenFile) are expected to raise FS.Error.

FS error codes are partitioned into five groups: bugs, environment errors, lock conflicts, client errors, and user errors. A bug usually indicates that something unexpected has gone wrong inside the implementation of FS or lower packages. An environment error means that something in the external environment has changed that requires human intervention. A lock conflict means that some other client has the locks that this client needs to get its job done. A client error is an attempted illegal operation that probably is caused by a bug in a client program. A user error is an attempted illegal operation that probably is caused by a human user's actions. The distinction between the last two groups is somewhat arbitrary.

An error description reports the group, as well as a specific code ATOM indicating exactly what went wrong. In the descriptions of procedures that follow, all codes from the error groups "client" and "user" that can be generated are specifically listed. Errors in the other groups should be expected as appropriate. See the section ERROR CODES at the end of this document for a complete list of the code ATOM's that the FS implementation can generate. The "explanation" is intended to be a description of the error suitable for human consumption; it frequently contains the name of the file being operated upon.

Catching all FS.Error's with error.group = user and reporting the error.explanation to the human user is usually sufficient to deal with file names that are mistyped by humans or that do not correspond to existing files.

The FS implementation is careful never to raise FS.Error while locks are held. Any FS procedure can be called from a catch phrase for FS.Error.

Error: ERROR [error: ErrorDesc];

ErrorDesc: TYPE = RECORD [group: ErrorGroup, code: ATOM, explanation: ROPE] ← [ok, NIL, NIL];

ErrorGroup: TYPE = {
ok, -- initial group for a new FS.ErrorDesc
bug, -- caused by an internal bug
environment, -- something's wrong in the environment; human intervention required
lock, -- conflict over locks
client, -- illegal operation, probably due to bug in client program
user -- illegal operation, probably due to user action
};

FILE NAMES

Names for FS files are called "FNames". An FName consists of (in order) a server, a root directory, zero or more subdirectories, a simple name, and a version. An FName may not exceed 120 characters excluding the version part. The version part will always fit in 6 characters, so the maximum space an FName can occupy is:

maxFNameLength: CARDINAL = 126;

Any name discovered that contains more than 120 characters before the version part will cause FS.Error[user, $illegalName].

Both FTP bracket syntax and CIFS slash syntax for FNames are accepted, although bracket syntax is used internally, in all FNames returned from FS procedures, and in FS error explanations. The following constructs are equivalent:

[server]<rootDirectory>subDirectory>simpleName!version

/server/rootDirectory/subDirectory/simpleName!version

The structural characters that delimit the server part of an FName are "[" and "]", or "/". Non-structural characters allowed in the server part are all alphameric characters plus ".", "$", "-", "+" and "#". The structural characters allowed in the rest of an FName are "<" and ">", or "/". The non-structural characters are the same as for the server part with the exception of "#". The additional structural character "!" is used as the version part prefix. The rest of the version part can be a number in the range [1 .. 65534], or one of the letters "L" or "H". Case is not significant in FNames.

An empty server part means the local server. Otherwise the server part is presumed to name a remote file server.

The names of files on remote servers are called "GNames", for global names. The detailed requirements on the structure of a GName are defined by the type of server named. The most common remote servers (IFS and Alpine) require the presence of a root directory part and include version parts in all file names.

The names of files on the local server are called "LNames", for local names. In an LName, the root directory part is the name of a volume; an empty root directory part refers to the "system" volume (the one from which Cedar was booted). All LNames have version parts. For example, the following are complete LNames:

[]<>FS.mesa!3

///FS.mesa!3

[]<Debugger>Temp>DebugTool.bcd!12

//Debugger/Temp/DebugTool.bcd!12

The working directory mechanism described below eliminates the need for the server and root directory prefix on most LNames presented to FS procedures.

Fine points:

1. Certain remote servers, e.g., Gateways, Grapevine servers, do not have a directory structure or versions. Such servers require GNames of the form "[server]simpleName".

2. The root directory in an LName may also be identified by its unique VolumeID. The syntax for the root directory part in this case is "#" followed by a 20-digit hexadecimal string.

3. The Cedar debugger interface makes a remote debuggee's disk available to a debugger as if its disks were attached to the debugger's machine. This is essentially transparent to FS; volumes of the debuggee's disk will appear to be part of the local server, i.e., files will be LNames, not GNames. This is entirely different from the situation in which the remote machine is operating normally and is running, say, an FTP server.

WORKING DIRECTORY

An FName that does not start with a server part is called a partial name and will be interpreted relative to a working directory. The working directory may be specified as the argument named "wDir" to FS procedures, or may be defaulted.

To expand the arguments "name" and "wDir" of an FS procedure into a full FName, FS checks the first character of "name". If it is "[" ("/") then "name" is the full FName. If not then FS looks for a working directory name in three places in order:

1) the "wDir" argument;

2) the value of the $WorkingDirectory property of the process property list;

3) the default working directory (set with FS.SetDefaultWDir).

The first ROPE found whose length is not 0 is preprended to "name" to form the full FName. A working directory obtained from 1 or 2 can be local or global. If it does not end with "]" or ">" ("/") then a ">" ("/") is appended. The defualt working directory must be local.

SetDefaultWDir: PROC [dir: ROPE ← NIL];

GetDefaultWDir: PROC RETURNS [ROPE];

If Rope.Length[dir] is 0 then the default working directory is set to "[]<>", meaning the system volume for this Cedar instance. "[]<>" also is the initial value for the default working directory. If "dir" does not end with ">" ("/") a ">" ("/") is appended.

Client errors: none

User errors: $illegalName, $patternNotAllowed, $badWorkingDir

PARSING AND CONSTRUCTING FNAMES

Position: TYPE = RECORD [start, length: CARDINAL];

ComponentPositions: TYPE = RECORD [server, dir, subDirs, base, ext, ver: Position];

ExpandName: PROC[name: ROPE, wDir: ROPE ← NIL] RETURNS [fullFName: ROPE, cp: ComponentPositions, dirOmitted: BOOLEAN];

The full FName determined by "name" and "wDir" is converted to brackets syntax and syntax checked. The converted FName is returned as "fullFName". The positions of the components in "fullFName" are returned in "components", expressed as starting index and length. No check is made to see if a file named "fullFName" actually exists. A version number variable or missing version part is not bound.

The "server" is the server part without surounding "[ ... ]". The "dir" is the root directory or volume part without the surounding "> ... >". The "subDirs" is the all the subdirectory part without without surounding "> ... >" but including interior ">"s. The "base" is the portion of the simple name up to but not including the final "." The "ext" is the portion of the simple name following the final "." If there are no "."s in the simple name then it is all considered to be "base". The "ver" is everything after the "!".

If the "fullFName" is a GName of the form "[server]simpleName", i.e., it has no directory part (not just the empty directory part "<>"), then "dirOmitted" returns TRUE; otherwise it returns FALSE.

If "fullFName" is an LName then "components.server.length" will be 0; if "fullFName" is an GName then "components.server.length" will be greater than 0.

For a Position "p" from a ComponentPositions "c", Rope.Substr[fullFName, c.p.start, c.p.length] is a ROPE containing that component.

An empty or missing component is reported as [start,0], where "start" is the index where the component would have appeared if present.

Client errors: none

User errors: $illegalName, $patternNotAllowed

ComponentRopes: TYPE = RECORD [server, dir, subDirs, base, ext, ver: ROPE ← NIL];

ConstructFName: PROC [cr: ComponentRopes, omitDir: BOOLEAN ← FALSE] RETURNS [ROPE];

Constructs an FName from the component ROPE's in "cr". No syntax checking is done. The only structual character that should appear in components is ">" within the subdirectory component.

The construction algorithm is as follows:

name ← Rope.Cat[ "[", cr.server, "]" ];

IF NOT omitDir THEN name ← Rope.Cat[ name, "<", cr.dir, ">" ];

IF Rope.Length[cr.subDirs] > 0 THEN name ← Rope.Cat[ name, cr.subDirs, ">" ];

name ← Rope.Cat[ name, cr.base ];

IF Rope.Length[cr.ext] > 0 THEN name ← Rope.Cat[ name, ".", cr.ext ];

IF Rope.Length[cr.ver] > 0 THEN name ← Rope.Cat[ name, "!", cr.ver ];

RETURN [name];

Client errors: none

User errors: none

FILE PROPERTIES

File properties provided by FS are page count, byte count, created-time, and keep. The page count is the number of pages allocated to the file. The byte count is the number of meaningful bytes in a file; the byte count is kept up-to-date by explicit client calls on FS procedures. The created-time is the date and time when the contents of the file were created (or changed). The keep specifies how many versions of a file to keep around (see the next section).

WordsForPages: PROC[pages: INT] RETURNS [words: INT];

BytesForPages: PROC[pages: INT] RETURNS [bytes: INT];

PagesForWords: PROC[words: INT] RETURNS [pages: INT];

PagesForBytes: PROC[bytes: INT] RETURNS [pages: INT];

VERSIONS AND KEEPS

Most servers include version numbers in file names. Version numbers allow for multiple instances of a file to exist at one time. FS enforces the policy that the version on any newly created file is one larger than the largest existing version, the so-called "next" or !N version. It is not possible to create any other version with FS. When accessing an existing file the version part of an FName may be a number, may be a variable, or may be omitted. The variables allowed are !L, meaning the lowest existing version number, and !H, meaning the highest existing version number. When omitted, the version part defaults to !L or !H, depending on the operation being performed.

A property for each LName is the "keep", a CARDINAL that specifies the number of versions of an LName that should be kept around. Whenever an LName is created, its "keep" is inherited from the existing !H version or set from an argument to the procedure doing the creation. The keep of the existing !H version can be changed with FS.SetKeep.

Keep processing occurs when creating a new version of an LName. In this case, FS will enumerate existing version in decreasing order from !H. After "keep - 1" versions are encountered in this enumeration, additional versions will be deleted if not open. Thus, the keep is the steady state count of versions to be retained. The local volume file of a deleted version will be reused as the disk file for the new version being created. For example, if the only existing version of a file is named Example.bcd!4, it has a keep of 1, and no client has it open, then FS.Create[Example.bcd] will cause Example.bcd!4 to be deleted and its disk file to be reused for Example.bcd!5. Calling FS.SetKeep will have the side effect of deleting files that are obsolete relative to the new keep, even when the keep is not changed.

Note that setting a keep to LAST[CARDINAL] will result in all versions being kept, since there can never be that many versions. A keep of 0 is illegal.

Most remote servers do not implement keeps.

THE CACHE

FS manages a cache on the system volume of a Cedar instance. The cache contains complete copies of global files. When the File package underlying FS decides that the volume is getting too full it calls out to FS. FS responds by removing an unopen global file from the cache, using an approximate LRU algorithm to order removals. FSBackdoor contains procedures for manually managing the cache.

Cacheing files introduces the potential for different Cedar instances to see different name to content mappings for global files. When one Cedar instance changes a name to content mapping, the cacheing could delay or prevent propagation of the change to the server or to the caches in other Cedar instances. (Such problems do not arise for local files, of course, since they are not cached.) These problems are mitigated by restrictions on the operations that can be performed. Global files cannot be opened for writing with FS.Open. New global files cannot be created via FS.Create. FS.Copy and FS.Rename can be used only to create the !N version of a global file. Thus, cached global files never need to be written back to the server. (FS.Copy and FS.Rename can overwrite the existing file in the case of a server that does not inplement version numbers.) The next two sections outline other measures for dealing with inconsistencies between caches and servers.

BINDING VERSION VARIABLES

For certain FS procedures the identification of which global file to operate on can include the version variables !L or !H, or can be defaulted to the variables !L or !H (depending on the operation). Binding a version variable to a particular version would be straightforward if the remote server were always accessible and if the client were always willing to incurr the cost of accessing it. In some circumstances, however, a client may wish to complete the binding based on the possibly incomplete information available from the cache. The "remoteCheck" argument to FS.FileInfo, FS.Open and FS.Copy allows the client to control the scope of the version search. (FS.Rename and FS.Delete do not include a "remoteCheck" argument because renaming and deletion occur synchronously on the remote server, so there is no option.) The "remoteCheck" argument is evaluated only when a GName is specified. When "remoteCheck" is TRUE then FS binds a version variable by searching the versions on the remote server. The operation fails if the remote server is inaccessible. When "remoteCheck" is FALSE then FS binds the version variable by searching the versions in the cache. Only if no cached version is found is the remote server interrogated.

FILE NAMING USING CREATED-TIMES

All of the FS procedures that accept an FName argument also include a "wantedCreatedTime" argument. This time provides an alternative way to name a file. If "wantedCreatedTime" is not equal to BasicTime.nullGMT, then the file to be operated upon is determined by a search for the version with the specified created-time. Any version part included in the name is treated as a hint. The created-time of a file is treated as a unique identifier of the content. If a version of a global file with the wanted created-time is found in the cache, then FS will not bother to check with the remote server (even if "remoteCheck" is TRUE) to see if the same created-time to version binding exists there.

ATTACHMENTS

FS includes facilities for attaching an LName to a GName and created-time. An attachment is a name binding. Attachment is a way to give an LName to a global file, and is useful as a cheap way to copy global files into local files. "BringOver" and "SModel" are the primary clients of the attachment mechanism.

Attachments are created using certain modes of FS.Copy. For most FS procedures, an attached LName behaves as though the global file were in fact copied into the local file when FS.Copy was invoked. There are three exceptions. First, the various procedures for obtaining information about files inform the client when an LName is attached and provide the GName of the attachment. Second, opening an attached LName is like opening the global file; if the global file has been deleted and is not in the cache, then the open will fail; it is the global file that is read locked. Third, when a local file that is attached to a global file is opened for writing, the attachment is broken and the contents of the global file are copied into a local file; the local file assumes the LName and the GName is forgotten.

LOCKS

There are two types of locks that can be set on files: "read" and "write". These locks are local to a Cedar instance.

Lock: TYPE = {read, write};

A "read" lock provides shared access to a file. Holders of "read" locks for a file may only read its data or properties. A "read" lock can be set anytime no "write" lock for that name is held. Thus, multiple "read" locks may be held. While a "read" lock is held no other client can delete the file or change its name.

An "write" lock provides exclusive access to a file. The holder of a "write" lock may alter the file's contents and properties. An implicit "write" lock is required to rename or delete a file. While an "write" lock is held no other client can open the file, delete it, or alter its name. A "write" lock can be set only when no other locks are held.

CREDENTIALS

For access to remote servers FS uses the credentials obtained from the UserCredentials interface. If these do not work then FS.Error[environment, $badCredentials] occurs.

FILE INFORMATION

FileInfo: PROC [name: ROPE, wantedCreatedTime: BasicTime.GMT ← BasicTime.nullGMT, remoteCheck: BOOLEAN ← TRUE, wDir: ROPE ← NIL] RETURNS [fullFName, attachedTo: ROPE, keep: CARDINAL, bytes: INT, created: BasicTime.GMT];

Returns information about the file specified by "name", "wantedCreatedTime", and "wDir". A missing version part defaults to !H. If "wantedCreatedTime" is not BasicTime.nullGMT then any version number in "name" is treated as a hint; the information returned is for the file with the wanted created-time, found by searching all versions of the named file as necessary.

If the matching file is a local file not attached to a global file, then the full FName including version part is returned as "fullFName". The keep, byte count, and created-time also are returned. "attachedTo" will be NIL and the "remoteCheck" argument is ignored.

If the matching file is a local file that is attached to a global file, then the full FName of the local file including version part is returned as "fullFName". The keep of the local file is returned. The full FName of the attached global file is returned as "attachedTo". The byte count and created-time of the global file also are returned. In this case, if the "remoteCheck" argument is FALSE then the byte count will be returned as -1, thus eliminating the need to open the global file from the cache or check with the remote server to determine this information. If "remoteCheck" is TRUE then the byte count of the global file is determined from the cache or remote server as necesary and returned. Any FS.Error's encountered when trying to determine the byte count of an attachment are suppressed and "bytes" is reported as -1. Whenever a valid byte count for an attachment is reported then the version part in the "attachedTo" name is the true version number that corresponds to the created-time for the attachment; otherwise this version part will be whatever was recorded by FS.Copy when the attachment was made.

If the matching file is a global file its full FName including version part is returned as "fullFName". The keep returns as 0. The byte count and created-time of the global file are returned. "attachedTo" returns as NIL. If "name" ends with a version variable and no wanted created-time is specified, then "remoteCheck" controls access to the remote server. If "remoteCheck" is TRUE, then the remote server is accessed for the file information. If "remoteCheck" is FALSE, then the version variable is bound relative to the set of versions in the cache; the remote server is interrogated only if no version appears in the cache.

Client errors: none

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $unknownCreatedTime, $illegalName, $patternNotAllowed

ENUMERATION

Enumerations are specified with a "pattern" argument, which is an FName in which the character "*", meaning "match anything", may appear zero or more times. "*" may not appear in the server part, or in the root directory of an LName. The presence or absence of a server part in "pattern" controls the use of working directories as usual. The version part of a "pattern" may be !H, !L, !*, or ! followed by digits; or may be omitted (defaults to !*). It cannot contain a mixture of "*" and digits. The order of the enumeration may be different for different sorts of servers. For the local server and IFS's, the enumeration is in lexical order of LNames expressed in bracket syntax (lower cases letters are mapped to upper case) but without the version part, and then in numeric order by version number.

Monitor locks are not held during an enumeration. Thus, the client may call FS.Delete from an FS.InfoProc or an FS.NameProc. Fine point: deleting or storing (by FS.Copy) ahead of the current position in a remote enumeration can produce unexpected results; the change may not be reflected in the enumeration.

InfoProc: TYPE = PROC [fullFName, attachedTo: ROPE, created: BasicTime.GMT, bytes: INT, keep: CARDINAL] RETURNS [continue: BOOLEAN];

EnumerateForInfo: PROC [pattern: ROPE, proc: InfoProc, wDir: ROPE ← NIL];

"proc" is called for each FName selected by the pattern. Returning "continue" as FALSE stops the enumeration. "fullFName" reports the complete FName including version part of each name encountered. The keep, byte length, and created date of a local or global file are reported. When an LName attached to a GName is encountered, "fullFName" is the LName and "attachedTo" is the GName. In this case, the "keep" reported is that of the LName but the "bytes" and "created" are that of the GName. If any FS.Error's are encountered when trying to determine the byte count of an attachment, they are suppressed and "bytes" is reported as -1. Whenever a valid byte count for an attachment is reported then the version part in an "attachedTo" name is the true version number that corresponds to the created-time for the attachment; otherwise this version part will be whatever was recorded by FS.Copy when the attachment was made.

Client errors: none

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $illegalName

NameProc: TYPE = PROC [fullFName: ROPE] RETURNS [continue: BOOLEAN];

EnumerateForNames: PROC [pattern: ROPE, proc: NameProc, wDir: ROPE ← NIL];

Because only names are provided to "proc", this enumeration can go faster than the previous one, especially for GNames.

Client errors: none

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $illegalName

OPENING FILES

An FS.OpenFile may be presented to other procedures to get access to the file's contents and properties. All global files opened using the FS procedures defined below are accessed via the FTP protocol and the cache.

Other packages can include procedures that return FS.OpenFile's. In such cases, the arguments to the opening procedures and the detailed semantics of operations on the client-provided FS.OpenFile's are defined by that package. (The operations are defined by providing procedures to FSBackdoor.OpenFileFromProcs). For example, the FS.OpenFile's obtained by calling the Alpine package directly do permit use of Alpine's special facilities for transaction, locking, and page access. Access to such FS.OpenFile's is via Alpine procedures and completely bypasses FS's locking and cache.

OpenFile: TYPE = RECORD [REF];

nullOpenFile: OpenFile = [NIL];

Open: PROC [name: ROPE, lock: Lock ← $read, wantedCreatedTime: BasicTime.GMT ← BasicTime.nullGMT, remoteCheck: BOOLEAN ← TRUE, wDir: ROPE ← NIL] RETURNS [OpenFile];

An FS.FileInfo[name, wantedCreatedTime, remoteCheck, wDir] is done. If an FS.Error does not occur, then the file described by the result of FS.FileInfo is opened. If an "attachedTo" is returned then that global file is the one opened. "lock" specifies the lock to be set. Opening a file with an "write" lock does not change the created-time property; maintaining this property is the client's responsibility. When a local file that is attached to a global file is opened for writing, the attachment is broken and the contents of the global file are copied into a local file, which local file assumes the LName. Multiple calls on FS.Open produce different FS.OpenFile's.

Client errors: $noCache, $globalWriteLock

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $unknownCreatedTime, $illegalName, $patternNotAllowed

Create: PROC [name: ROPE, setPages: BOOLEAN ← TRUE, pages: INT ← 0, setKeep: BOOLEAN ← FALSE, keep: CARDINAL ← 1, wDir: ROPE ← NIL] RETURNS [OpenFile];

A new local file with the specified name is created and opened with a "write" lock. The created-time is set to now. No version part may be specified. FS will assign the version number that is one larger than the existing !H version, or !1 if no versions exist. If !1 is being created or "setKeep" is TRUE then the keep is set to "keep"; otherwise the keep for the new file is that from the existing !H version. If "setPages" is TRUE then the number of pages in the created file is set to "pages". If "setPages" is FALSE then the number of pages in the new file is the same as the reused file if any, otherwise it set to "pages".

Client errors: $zeroKeep

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $illegalName, $patternNotAllowed, $versionSpecified, $globalCreation

OpenOrCreate: PROC [name: ROPE, keep: CARDINAL ← 1, pages: INT ← 5, wDir: ROPE ← NIL] RETURNS [OpenFile];

This procedures is a convenience for a client that wishes to open an existing file for write, as is the case for log files. Performs FS.Open[name: name, lock: write, wDir: wDir]. If FS.Error[$unknownFile] results then FS.Create[name: name, pages: pages, keep: keep, wDir: wDir] is called.

Client errors: $noCache, $globalWriteLock, $zeroKeep

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownCreatedTime, $illegalName, $patternNotAllowed, $versionSpecified, $globalCreation

OPERATIONS ON OPEN FILES

GetClass: PROC [file: OpenFile] RETURNS [ATOM];

Returns an ATOM that identifies the implementation for operations on "file". If "file" was obtained by calling FS.Open, FS.Create, or FS.OpenOrCreate, then "$FS" is returned. If "file" was obtained by calling some other package that manufactures FS.OpenFiles via FSBackdoor.CreateProcsOpenFile, then the ATOM returned is that provided by that package.

Client errors: $invalidOpenFile, $notImplemented

User errors: none

SameFile: PROC [file1, file2: OpenFile] RETURNS [BOOLEAN];

Returns TRUE iff "file1" and "file2" are for the same files. Two FS.OpenFile's obtained from an FS procedure are the same if they refer to the same File.FP.

Client errors: $invalidOpenFile, $notImplemented

User errors: none

GetName: PROC [file: OpenFile] RETURNS [fullFName, attachedTo: ROPE];

The full name of the opened file, in brackets syntax with any version number, is returned as "fullFName". If the name that was opened was an LName attached to a GName, then the name of the attached global file is returned as "attachedTo", which otherwise is NIL.

Client errors: $invalidOpenFile, $notImplemented

User errors: none

GetInfo: PROC [file: OpenFile] RETURNS [keep: CARDINAL, pages, bytes: INT, created: BasicTime.GMT, lock: Lock];

The keep (if not defined for the server in question then 0 is returned), number of pages, byte count, created-time, and the lock held are returned. If the name that was opened was an LName attached to a GName, then the pages, byte count, created-time, and lock are for the global file.

Client errors: $invalidOpenFile, $notImplemented

User errors: none

SetPageCount: PROC [file: OpenFile, pages: INT];

Extends or contracts the file until the number of pages is as specified.

Client errors: $wrongLock, $unknownPage, $invalidOpenFile, $notImplemented

User errors: none

SetByteCountAndCreatedTime: PROC [file: OpenFile, bytes: INT ← -1, created: BasicTime.GMT ← BasicTime.nullGMT];

Sets the byte count and created-time properties of "file". If "created" is BasicTime.nullGMT then the created-time of the file is not altered. If "bytes" is -1 then the byte count is not altered.

Client errors: $wrongLock, $badByteCount, $invalidOpenFile, $notImplemented

User errors: none

Read: UNSAFE PROC [file: OpenFile, from, nPages: INT, to: LONG POINTER];

Copies "nPages" pages from the file starting at "from" to the virtual memory interval starting at "to". "to" must be page-aligned. Note that ERROR's caused by "to" being an invalid address are not mapped to FS.Error's.

Client errors: $unknownPage, $invalidOpenFile, $notImplemented

User errors: none

Write: PROC [file: OpenFile, to: INT, nPages: INT, from: LONG POINTER];

Copies "nPages" pages from the virtual memory interval starting at "from" to the file starting at page number "to". "from" must be page-aligned. Note that ERROR's caused by "from" being an invalid address are not mapped to FS.Error's.

Client errors: $wrongLock, $unknownPage, $invalidOpenFile, $notImplemented

User errors: none

Close: PROC [file: OpenFile];

Releases the locks associated with the open "file". Any further operations on copies of this FS.OpenFile will produce an FS.Error[client, $invalidOpenFile]. The locks will also be released by finalization when all copies of the FS.OpenFile disappear, but this occurs at an unspecified future time.

Client errors: $invalidOpenFile, $notImplemented

User errors: none

FILE STREAMS

File Stream Model

A file is a sequence [0..fileLen) of mutable bytes; the length (fileLen) of a file can be changed.

The state of a file stream is a file (perhaps opened under a transaction trans) with length fileLen, an index streamIndex IN [0..fileLen], plus a number of readonly flags (accessRights, streamOptions) and a BOOL variable closed which jointly determine the effect of stream procs.

Here is what the generic stream operations do when applied to a file stream:

IO.GetChar: PROC [self: STREAM] RETURNS [CHAR]

If streamIndex = fileLen then ERROR IO.EndOfStream. Else return file[streamIndex], and set streamIndex ← streamIndex + 1.

IO.CharsAvail: PROC [self: STREAM, wait: BOOL] RETURNS [INT]

Return INT.LAST.

IO.EndOf: PROC [self: STREAM] RETURNS [BOOL]

Return streamIndex = fileLen.

IO.PutChar: PROC [self: STREAM, char: CHAR]

If streamIndex = fileLen then fileLen ← fileLen + 1. Then set file[streamIndex] ← char, streamIndex ← streamIndex + 1.

IO.Flush: PROC [self: STREAM]

Write current value of file[0..fileLen) to be written to disk; write fileLen to disk. If commitAndReopenTransOnFlush then first commit trans, then begin a new transaction as trans.

IO.Reset: PROC [self: STREAM]

If accessRights = $read then set streamIndex ← fileLen. If accessRights = $write and streamIndex and fileLen were initially 0, then set streamIndex ← fileLen ← 0.

IO.Close: PROC [self: STREAM, abort: BOOL ← FALSE]

If NOT abort, then write current value of file[0..fileLen) to be written to disk and write fileLen to disk. If truncatePagesOnClose, then discard unused pages from end of file. If finishTransOnClose, then commit or abort trans depending on the state of abort. Invalidate self (all operations on self other than Reset and Close will raise ERROR IO.Error[StreamClosed]; these two do nothing).

IO.GetIndex: PROC [self: STREAM] RETURNS [INT]

Return streamIndex.

IO.SetIndex: PROC [self: STREAM, index: INT]

ERROR IO.EndOfStream if index > fileLen. Set streamIndex ← index.

IO.GetLength: PROC [self: STREAM] RETURNS [INT]

Return fileLen.

IO.SetLength: PROC [self: STREAM, length: INT]

Set fileLen ← length, then set streamIndex ← MIN[streamIndex, fileLen]. The contents of file[oldFileLen .. fileLen) are undefined.

Concurrency

File streams provide no interlocks of their own to control concurrent access to files. Instead, they rely on the underlying file system for concurrency control. FS and Alpine use locking to provide concurrency control.

Individual file streams are independent objects, so there is never any need to synchronize calls to their procedures. If two processes are to share a single file stream, they must synchronize their accesses to that stream at a level above the file stream calls; individual file stream calls (PutChar, PutBlock, etc.) are not guaranteed to be atomic.

Pragmatics

An FS file is represented as a sequence of disk pages, each containing FS.BytesForPages[1] bytes. There is a single leader page (overhead) that is used to hold file properties such as the byte count and create time. There is no functional relationship between the byte count of a file and the allocated size of that file in pages.

Allocating disk pages to a file is an expensive operation, especially when done incrementally (one page at a time). Extending a file many times can result in a fragmented file. If a client program can estimate the eventual length of a file it is creating, it should use the "createByteCount" parameter to FS.StreamOpen to allocate enough space to hold the entire length all at once; if a client program is updating an existing file then it should create a stream and then extend the file using IO.SetLength. If the file stream implementation must extend a file it will always extend it by several pages; if asked to shorten a file it will simply adjust the byte count field in the leader page without freeing any pages. If the streamOption "truncatePagesOnClose" is specified (the default), then extra pages are freed when the stream is closed.

CREATING FILE STREAMS

StreamOptions: TYPE = PACKED ARRAY StreamOption OF FalseBool;

FalseBool: TYPE = BOOL ← FALSE;

StreamOption: TYPE = {

tiogaRead,

-- if accessOptions = $read and file is in Tioga format, read only plain text portion of the file (ignore "looks" and nodes with the comment property). If accessOptions # $read and file is in Tioga format, raise FS.Error[$cantUpdateTiogaFile].

commitAndReopenTransOnFlush,

-- each IO.Flush call "checkpoints" the transaction being used by the stream.

truncatePagesOnClose,

-- each IO.Close call causes extra pages of the file to be freed.

finishTransOnClose,

-- each IO.Close call causes the transaction to be committed or aborted, according to the abort parameter to IO.Close.

closeFSOpenFileOnClose

-- each IO.Close call performs FS.Close on the underlying FS.OpenFile.

};

defaultStreamOptions: StreamOptions = ALL[TRUE];

StreamBufferParms: TYPE = RECORD [vmPagesPerBuffer: INT [1 .. 128], nBuffers: INT [1 .. 4]];

defaultStreamBufferParms: StreamBufferParms = [vmPagesPerBuffer: 16, nBuffers: 2];

minimumStreamBufferParms: StreamBufferParms = [vmPagesPerBuffer: 1, nBuffers: 1];

Controls the buffering strategy used by a stream.

ByteCount: TYPE = INT;

ExtendFileProc: TYPE = PROC [--current--ByteCount] RETURNS [--new--ByteCount];

Called by a stream when a file is to be extended. Returns a suggested new allocated byte count for the file; the stream may extend file by more than the requested amount.

AccessOptions: TYPE = {read, create, append, write};

StreamOpen: PROC [fileName: ROPE, accessOptions: AccessOptions ← $read, streamOptions: StreamOptions ← defaultStreamOptions, keep: CARDINAL ← 1, createByteCount: ByteCount ← 2560, streamBufferParms: StreamBufferParms ← defaultStreamBufferParms, extendFileProc: ExtendFileProc ← NIL] RETURNS [STREAM];

Create a new stream by (1) generating an FS.OpenFile from fileName, keep, and createByteCount, then (2) creating a stream on this FS.OpenFile.

(1) If accessOptions = $read, perform FS.Open[name: fileName]. If accessOptions = $create, perform FS.Create[name: fileName, keep: keep, pages: FS.PagesForBytes[createByteCount]]. If accessOptions = $append, perform FS.OpenOrCreate[name: fileName, keep: keep, pages: FS.PagesForBytes[createByteCount]]. If accessOptions = $write, perform FS.Open[name: fileName, lock: $write, keep: keep, pages: FS.PagesForBytes[createByteCount]]. Any FS.Error raised by a call to FS.Open, FS.Create, or FS.OpenOrCreate will propogate through FS.StreamOpen.

(2) Perform FS.StreamFromOpenFile[ ... , streamOptions, streamBufferParms, extendFileProc] on the resulting FS.OpenFile. The value of accessOptions determines the set of operations allowed on the stream (disallowed operations raise IO.Error[$NotImplementedForThisStream, stream] when called), and the initial streamIndex, as follows:

read: PutChar, PutBlock, UnsafePutBlock, and SetLength are disallowed, and the initial streamIndex is 0.

create: all operations are allowed, the length is set to zero at stream creation time, and the initial streamIndex is 0.

append: all operations are allowed, and the initial streamIndex is the file's byte count.

write: all operations are allowed, and the initial streamIndex is 0.

Client errors: $noCache, $globalWriteLock, $zeroKeep

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $illegalName, $patternNotAllowed, $versionSpecified, $globalCreation, $cantUpdateTiogaFile

InitialPosition: TYPE = {start, end};

StreamFromOpenFile: PROC [openFile: OpenFile, accessRights: Lock ← $read, initialPosition: InitialPosition ← $start, streamOptions: StreamOptions ← defaultStreamOptions, streamBufferParms: StreamBufferParms ← defaultStreamBufferParms, extendFileProc: ExtendFileProc ← NIL] RETURNS [STREAM];

Create a new stream on the open file. If AccessRights = $read then PutChar, PutBlock, UnsafePutBlock, and SetLength are disallowed on the resulting stream. InitialPosition specifies the initial value of streamIndex: start means 0, end means the file's byte count.

Client errors: $wrongLock

User errors: $cantUpdateTiogaFile

OpenFileFromStream: PROC [self: STREAM] RETURNS [OpenFile];

Will raise IO.Error[$NotImplementedForThisStream, self] if self is not a stream on an FS.OpenFile.

ErrorFromStream: PROC [self: STREAM] RETURNS [ErrorDesc];

Will raise IO.Error[$NotImplementedForThisStream, self] if self is not a stream on an FS.OpenFile.

A client can this procedure after self raises IO.Error[$Failure, self] to get more details about the error.

GENERAL FILE MANIPULATIONS

Copy: PROC [from, to: ROPE, setKeep: BOOLEAN ← FALSE, keep: CARDINAL ← 1, wantedCreatedTime: BasicTime.GMT ← BasicTime.nullGMT, remoteCheck: BOOLEAN ← TRUE, attach: BOOLEAN ← FALSE, wDir: ROPE ← NIL];

The version of the "to" file created is one larger than the existing !H version, unless "to" is a GName on a server that does not support version numbers, in which case an existing global file will be overwritten.

Case 1: "to" is a GName and "attach" is FALSE — FS does an FS.Open[from, read, wantedCreatedTime, remoteCheck, wdir] and stores the contents and properties of the opened file directly in the new "to" file on the remote server.

Case 2: "to" is an LName and "attach" is FALSE — FS does an FS.Open[from, read, wantedCreatedTime, remoteCheck, wDir] to generate a "from" open file, and FS.Create[to, setKeep, keep, wDir] to generate a "to" OpenFile. The contents and properties are copied from the "from" OpenFile to the "to" OpenFile, and both are closed. (In the case of copying from an uncached global file, the global file is not added to the cache. The only pages allocated on the local volume are those needed to hold the target local file.)

Case 3: "to" is a GName, "attach" is TRUE, and "from" is a LName — Things proceed as in case 1, except a "write" lock is acquired for the "from" file. Once the transfer is completed, the LName is attached to the GName. Case 3 is used by "SModel".

Case 4: "to" is an LName, "attach" is TRUE, and from" is a GName — Like case 2 except that instead of an actual transfer of contents and properties, the LName is attached to the GName / created-time. If no "wantedCreatedTime" is specified, or if "remoteCheck" is TRUE, then FS.FileInfo[to, wantedCreatedTime, TRUE] is performed first to determine/check the version number and created-time for the GName and any resulting FS.Error is passed on to the client. When "remoteCheck" is FALSE (but a wantedCreatedTime" is specified) then the attachment is made to the "from" and "wantedCreatedTime" arguments without checking either the remote server or the cache. Case 4 is used by "BringOver".

Case 5: both "from" and "to" are LNames, or both are GNames, and "attach is "TRUE" — FS proceeds as though "attach" were FALSE (see cases 1 and 2 above).

Client errors: $zeroKeep

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $unknownCreatedTime, $illegalName, $patternNotAllowed, $versionSpecified (on "to" name)

Delete: PROC [name: ROPE, wantedCreatedTime: BasicTime.GMT ← BasicTime.nullGMT, wDir: ROPE ← NIL];

A missing verion part defaults to !L. If "name" is an GName then the file is deleted directly from the remote server (and removed from the cache). An FS.Error[lock, $lockConflict] occurs if the specified file is currently open. If "name" is an LName that is attached to some GName, then just the LName is deleted; the global file is not deleted. Note that deleting the !H verison may cause a previous verison to become the new !H version, and therefore cause the keep on the previous version to become the controlling keep for the set of versions.

Client errors: none

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $unknownCreatedTime, $illegalName, $patternNotAllowed

Rename: PROC [from, to: ROPE, setKeep: BOOLEAN ← FALSE, keep: CARDINAL ← 1, wantedCreatedTime: BasicTime.GMT ← BasicTime.nullGMT, wDir: ROPE ← NIL];

A missing version part defaults to !H for the "from" name. FS.Rename is the same as FS.Copy followed by FS.Delete, with the performance optimization that in many cases the actual transfer of bits can be skipped. An FS.Error[lock, $lockConflict] occurs if the specified file is currently open. Renaming that implies copying from one server to another is allowed. Renaming (and implied copying or deletion) on global servers occurs synchronously.

Client errors: $zeroKeep

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $unknownCreatedTime, $illegalName, $patternNotAllowed, $versionSpecified (on "to" name)

SetKeep: PROC [name: ROPE, keep: CARDINAL ← 1, wDir: ROPE ← NIL];

Setting the keep causes any unopen versions that are beyond the new keep to be deleted. Setting the keep to "0" leaves the current keep, but does the keep processing.

Client errors: none

User errors: $nonCedarVolume, $unknownServer, $unknownVolume, $unknownFile, $unknownCreatedTime, $illegalName, $patternNotAllowed, $versionSpecified, $noKeeps

ERROR CODES

Error codes are ATOM's so that packages implementing FS.OpenFile's via FSBackdoor.CreateProcsOpenFile can extend the set of codes. Following is a complete list of the codes generated by the FS implementation. The list is partitioned by the FS.ErrorGroup associated for each code.

Bugs

$inconsistent -- a local volume's permanent data structures are inconsistent

$software -- label check encountered while reading or writing pages on the local volume

$badFP -- File.FP from directory/cache doesn't correspond to a file on a local volume

Environment Errors

$wentOffline -- local volume is no longer accessible

$hardware -- hard error while reading or writing pages on a local volume

$volumeFull -- no more free pages on a local volume

$fragmented -- file required too many non-contiguous areas on a local volume

$noMoreVersions -- next version number would be too big

$serverInaccessible -- couldn't get a response from a remote server

$connectionRejected -- remote server rejected the connection attempt

$connectionTimedOut -- connection to a remote server timed out

$badCredentials -- remote server rejected the user name or password

$accessDenied -- remote server access controls succeeded

$quotaExceeded -- disk space quota on remote server exceeded

$invalidPropertyPage -- unrecognized property page format for a file on a local volume

$badBTree -- directory/cache BTree is malformed and cannot be opened

Lock Errors

$lockConflict -- required lock cannot be obtained

$fileBusy -- remote file currently being used in an incompatible way

Client Errors

$noCache -- couldn't operate on global file because there is no system volume or cache

$wrongLock -- operation requires a write lock and only a read lock is held

$globalWriteLock -- tried to open a GName with a write lock

$zeroKeep -- keep of 0 was specified

$badByteCount -- tried to set a byte count that is beyond end of file or negative

$unknownPage -- tried to read or write a page that is beyond end of the file

$invalidOpenFile -- FS.OpenFile is closed, is FS.nullOpenFile, or is an unrecognized TYPE

$notImplemented -- called procedure not provided for this FS.OpenFile

User Errors

$nonCedarVolume -- named volume is not a cedar volume

$unknownServer -- couldn't get the network address for the named server

$unknownVolume -- named local volume cannot be found

$unknownFile -- FName not in the implied directory

$unknownCreatedTime -- no version of the given FName had the specified created-time

$illegalName -- file name, directory name or pattern has illegal syntax or characters, or is too long,

$patternNotAllowed -- name presented contained a "*" and procedure doesn't allow patterns

$versionSpecified -- tried to create an FName with a specific version

$globalCreation -- tried to create a GName

$badWorkingDir -- default working directory specified is global or has no volume part

$noKeeps -- tried to set the keep on a server that does not implement keeps

$cantUpdateTiogaFile -- tried open a file stream to write on a .tioga file

END....