Abstract
This document contains a proposal for the use of local directories in Cedar 5.
Introduction
I think it will be possible to make use of the local directory structure available with the FS directory system in Cedar 5 without treading on much, if any, unknown ground.
FS provides a local naming environment similar to that of an IFS. How should this facility be used to the benefit of Cedar users?
The objective of local directories is first to reduce the mental working set of Cedar users. In general, the command List * should display only those files that are really worth thinking about right now.
Because FS decouples the size of the local disk from the number of file names in the local directory, there will be so many LNames that subdirectories will be needed just for organization. Users of Public workstations will be especially hard hit.
Local directories make it possible to use a push-pop model of the world. One will be able to interrupt development of a package in one directory, establish a new state in another directory, work for a while, then switch back and resume without the overhead of restoring the previous state. It will still be there! One could even switch back and forth among several projects with very low overhead.
FS provides the ability to have many LNames attached to the same GName. This may be useful in cross-referencing.
Primary Principles
The directory structure of the local disk should make it easy for multiple users to share a machine, even at the same time.
The directory structure of the local disk should reduce the mental working set of Cedar users. In general, the command List * should display only those files that are really worth thinking about right now.
It is ok if LNames are bound into code, but it is not ok if GNames are bound into code.
Distinct shortnames should not be needed so often, leading to less need for name coordination.
The directory structure of the local disk should make it possible to work on multiple projects in parallel, rather than serially, as we have now.
Secondary Principles
There should be few if any entries in the root directory /// except other directories.
The contents of a directory should, where sensible, reflect the contents of a single DF file.
The contents of a directory should, where sensible, be the same kind of item, perhaps one each from many DF files.
Naming
The name /Ivy/Stewart/Temp/Foo.mesa!3 should be read "Ivy Stewart Temp Foo dot mesa bang 3."
The name ///Stewart/Temp/Foo.mesa!3 should be read "root Stewart Temp Foo dot mesa bang 3." Slash slash slash is too hard to say.
How things work
This section provides a tour through the use and interpretation of local directories by various subsystems.
CommandTool
There are actually two notions, one relating to file lookup and one relating to command lookup. Let us first consider file naming.
If one does not supply a full path name to FS, then FS will look for the given name in the current working directory. The current working directory of a CommandTool is shown in the CommandTool herald. Any command run from the CommandTool will aquire the same current working directory. The primary command for manipulating the working directory is CD (ChangeDirectory). Through the use of CommandTool.ResolveRelativePath, CD interprets some additional directory name syntax.
The name "." refers to the current working directory.
The name ".." refers to the directory which is the parent of the current working directory.
If the current working directory is ///a/b/, then the name "../c/foo.mesa" refers to /a/c/foo.mesa. In other words, ".." means "parent".
CD with no argument at all currently means "CD ///". This will eventually change to mean "CD users-home-directory" for the logged-in user. This memo proposes user-home-directories of the form ///Users/Stewart.pa/.
Command lookup in the CommandTool proceeds by first looking in the current working directory and then in each directory on the CommandTool search rules list. These search rules apply only to command name lookup and not to ordinary file lookup. The commands PrintSearchRules, SetSearchRules, and AddSearchRule are used to manipulate the search rules.
DFTool
The DFTool has a field called Working Directory. Bringover will bring files over into the working directory and SModel will look in the working directory for files to store. The DFTool can be made to work in the old way by setting the working directory to "///" or "[]<>".
A future change to the syntax of DF files will make it possible to specify both a particular local directory for files in a DF file. At that time, there will be two ways to use Bringover: either Bringover will believe the information in the DF file and retrieve each file to its particular local directory, or Bringover will ignore information in the DF file and retrieve all files in the DF file to the current working directory. There will be two corresponding versions of SModel. Until the DF file sytax is revised, we will get be as described in the last section of this document.
Compiler and Binder
The Compiler and Binder no nothing of working directories. However, since the working directory is a property of a Cedar process, the Compiler and Binder do not need to know. They traffic in short names and FS takes care of attaching the working directory. In general, the Compiler and Binder will have access only to files in the current working directory and will create files only in the current working directory. When the Compiler and Binder are run from the CommandTool, this will all happen automatically.
Viewers and Tioga
In Cedar 5, Viewers and Tioga traffic in full path names and in general have few notions of a current working directory. However, to make path names easier to type, Viewers and Tioga provide a few facilities. Something like a working directory is the result.
The principle is that one should be able to tell what will happen by looking at the display -- the directory context will be visible.
An empty viewer will display something like "///Indigo/Cedar/Top/" rather than the old "No Name". This new layout means that the viewer will evaluate a shortname like Rope.df as though it were ///Indigo/Cedar/Top/Rope.df. The herald of the viewer indicates the context. The context is applied if a shortname is selected and Get or GetImpl are clicked, or if a shortname is typed into the viewer and linefeed (Load) is typed.
Of course, if a full path name is typed or selected, then the viewer context is ignored (and indeed, changed to that of the new full path).
Typing a directory name (ending in '/ or '>) to a viewer and typing linefeed will simply change the context of the viewer, without loading anything.
A non-empty viewer will display a full path name in the herald. If a shortname is selected and Get or GetImpl are clicked, then the directory part of the full path name in the herald will be used as the context.
If Clear is middle-clicked (to create a new empty viewer), then the new viewer aquires the context of the old.
New and Open buttons
The New and Open buttons in the Cedar herald will have contexts of "///". The New and Open buttons (and commands) in a CommandTool will have the context of that CommandTool.
The various buttons in viewers that take filenames as "arguments" will have the context of the viewer, as indicated in the viewer herald.
Debugging
In Cedar 4, the flat name space made life easy for the debugger, any symbols file would be in the single directory if it existed at all. The debugger would look for a symbols file in <Client>Sysdir> and then it would look in the version map for a remote symbols file.
In Cedar 5, the rich local directory structure makes life for the debugger a bit more interesting. The debugger maintains a search list of directories. When a symbols file is referenced, the debugger will look in each directory on its search list and then in the version map. There are debugger commands to change the search list.
This is intended as a temporary measure. The correct solution is to expand the concept of the version map so that all loaded software is listed in the map, not just software in the Cedar release.
Since boot-essential symbols files must always be available, the directory ///BootEssentials/ will always be on the deubugger search list.
How will this facility be used? A user of Cedar not engaged in devoping or testing software would only encounter software represented in the release version map. A software developer, having just bound and loaded an experimental configuration, might encounter a need for the symbols for that config. A debugger command like "AddDebuggerSearchRule ///Users/Stewart.pa/ProjectX/" would make all symbols for ProjectX available. A software tester, encountering a bug in an experimental version of Walnut, might issue a command like "AddDebuggerSearchRule /Indigo/PostCedar/Walnut/" in order to aquire access to the symbols.
As a convenience, the Run command in the CommandTool, (actually CommandTool.Run) adds to the debugger search rules the name of the directory from which it loads something.
Programming
The current working directory is a ROPE carried around as the $WorkingDirectory property on the process properties list of an individual process. The CommandTool makes sure that this property exists for each command so started, but it does not propagate through FORK. Programmers whose systems use multiple processes or which multiplex the operation of several tools through the same process will have to understand working directories and manage them by hand.
Viewers and Tioga is an example of a system in which the notion of working directory is properly attached to viewers and not to processes. Viewers and Tioga handle this by dealing only in full path names or by explicitly remembering the working directory. The DF tool is an example of a system that multiuplexes operations with different working directories through the same process. The DF software manully changes the process property when appropriate.
Programmers should beware that FS operations with "wdir" defaulted will use the working directory found on the process properties list. If a system FORKs or multiplexes, programmers are advised to use extra caution. If a system uses a single process started from a CommandTool, then it is likely that working directories will behave themselves automatically.
A useful interface for all this is FileNames, which is found in CommandTool.df.
Directories
The following sections discuss various classes of directories.
Temporary Directories
Temporary directories contain only files that never need to be backed up. These directories should never contain attachments.
///Temp/
The contents of ///Temp/ would be deleted every time the system is booted or rolled-back (unless a switch is set).
Programs which create temporary files should create them in the ///Temp/ directory with unique names. A facility should be provided for creating a unique name with a given constant part: UniqueName["WalnutTempLog."] might return "WalnutTempLog.1725376".
Things like log files should go in the ///Temp/ directory.
The reason for having a switch to disable the delete-on-rollback facility is that when the system wedges, one might want to have a chance to roll-back and then copy some log file to a permanant location before deleting it.
///BootTemp/
The contents of ///BootTemp/ would be deleted every time the system is booted (unless a boot switch is set).
Things like TipC files might go in the ///BootTemp/ directory.
The reason for having a switch to disable the delete-on-boot facility is that when the system wedges, a wizard might want to examine some of the temporary files.
Other notions
Rovner has expressed the opinion that there are only a few classes of "temporary" files and perhaps each should be given its own directory: ///TipC/, ///Logs/, etc.
Read-only directories
Read-only directories should never contain anything that is not an attachment.
I propose the following directories as a starting point:
///Fonts/
Viewers and Tioga should keep attachments to fonts, styles, etc in the ///Fonts/ directory. There is usually no need for them to be visible to users.
///BootEssentials/
The boot file should live here, as should other boot essential symbols files, the instream files, etc.
///Top/
This directory should be the home of a complete copy of /Indigo/Cedar/Top/ plus /Indigo/PostCedar/Top/. It should provide a consistant place to find the "appropriate" DF file.
///Commands/
This directory should be the home of all .load files in the system. ///Commands/ should be on the command search rules of nearly every CommandTool.
Subdirectories of ///Commands/ should hold the runnable .bcds and other required files for the various packages.
///Packages/
This directory should be the home of a collection of subdirectories for packages. Examples might be Finch or AlpineUser.
///Doc/
This directory should be the home of a collection of documentation files. It has a short name so that it is easy and fast to "CD ///Doc/".
///Interfaces/
Referenced farthur down. (As one of three proposals.)
Roots of subtrees
Subtree root directories are directories which contain only other directories. I see a need for only two such.
///Projects/
This directory should be the home of a collection of working subdirectories for major system components. Development of Viewers might proceed in the ///Projects/Viewers/ directory.
///Users/
This directory should be the home of a collection of subdirectories for users. These directories, like ///Users/Stewart.pa/ or ///Users/Levin.pa/ would be the default working directory after a Login. This directory would be known as a user's "Home" directory. Users would use subdirectories of their home directory for work on private projects, documents, etc.
User directories
Each user of a given machine should have an individual "home" directory in which to develop documents and private software. This notion will support the ability of more than one person sharing a Cedar workstation.
What will be the internal structure of a home directory? Consider the case of ///Users/Stewart.pa/. There will be a number of top-level files and a number of subdirectories:
Stewart.profile -- my user profile. This is the logical place to put it.
Commands/ -- a directory of private .load files and runnable .bcds. I would add ///Users/Stewart/Commands/ to my command tool search rules (probably in front, so that my private versions of commands would be found first).
Papers/ -- a directory of papers and memos in progress
WLList/ -- a directory for the development of the WLList program, which may someday be integrated with the DA software.
There are a number of advantages to use of this kind of directory subtree for users.
Users of public machines will find it easier to share the machine if they are confident they can create files without fear of overwriting other files.
It will be easy to tell, through the use of List -u ///Users/Stewart.pa/*, whether I have backed up all my personal files.
It will be easy for users of public machines to identify large groups of files which may be safely deleted. (It seems likely there is some performance advantage to telling FS about large classes of files which you know will not be used again in the near future.)
Those who borrow private machines will have a much easier time cleaning up, though the use of List -u to identify not-backed up files, and then Delete ///Users/Stewart.pa/* to clean up.
Project directories
It makes more sense to work on components of a multi-person projects in a private directory than in a subdirectory of one's own personal directory.
What will be the internal structure of a project directory? Consider the case of ///Projects/Voice/. There will be a number of top-level directories:
Doc/ -- a directory of voice project documentation.
Bluejay/ -- a directory for the development of the Bluejay voice file server program.
There are a number of advantages to use of this kind of directory subtree for projects.
Members of a project can sit down at a project machine or at one anothers personal machines and be able to find things.
It will be easy to tell, through the use of List -u ///Projects/Voice/*, whether all the files are backed up.
It will be easy for users of public machines to identify large groups of files which may be safely deleted.
Program development
The material above proposes that the development of a particular project, probably represented by a single DF file, proceed in a private directory.
Suppose I decided to work on Bluejay, the voice file server. Since it is a component of the voice project, I would first change my working directory to ///Projects/Voice/, then to Bluejay/ (a relative move making my working directory ///Projects/Voice/Bluejay/. I would then do a Bringover /a of /Indigo/Voice/Top/Bluejay.df to the current working directory. This would establish a complete context for editing, compiling, binding, and testing Bluejay.
Notice that throughout the system, there might be many LNames of the form ///Projects/Voice/Bluejay/Rope.bcd, but they would all be attached to the same GName, /Indigo/Cedar/Rigging/Rope.bcd, so the FS cache would maintain only one copy.
Notice that this scheme makes it possible to rapidly switch between development of a program using /Indigo/Cedar/Rigging/Rope.bcd and one using /Indigo/PreCedar/Rigging/Rope.bcd without running into the name conflicts that complicate life in a flat namespace.
Notice that this scheme does not require search rules for FS.Open or specification of more than short names in a module DIRECTORY.
There is at least one problem. Suppose that I wish to look at Rope.mesa? In our flat universe, it is quite likely that Rope.mesa is already lying around. Here are three proposals which effectively solve this problem in the world of project directories:
Create another top-level directory called ///Interfaces/ in which all boot file interface source files can be found. Here again, attachments give this a reasonable cost. Source files for the interfaces of packages might be kept in either ///Interfaces/ or in ///Packages/Interfaces/.
Openr would usually do the right thing.
A bringover switch could be created such that Bringover /o Rope.mesa Bluejay.df would work. (Normally Rope.mesa would not be in the explicit imports list in Bluejay.df, but the switch under discussion would ignore the imports lists.)
Debugging
How should the runtime type system work when the current working directory is not ///? At present, there is a temporary kludge that causes the runtime type system to look for a symbols file in /// if it is not found in the current working directory. I suspect this is an adequate solution for the problem of boot-esential symbols files except that they should be in ///BootEssentials/ rather than in /// directly.
It is important that the debugger have access to the symbols for any piece of code that is loaded into VM. How should this be accomplished when bcds are run from a variety of local directories? Obviously it is not going to work for the debugger to look for symbols only in the current working directory. Here are two possible solutions:
The version map machinery could be expanded so that incremental additions are made whenever code is loaded into VM.
Whenever code is loaded into VM, the symbols could be copied to ///Temp/ or some other known place.
The debugger will maintain search rules. Rather than do individual bringovers of necessary symbols files as we do now, one might simply add ///Users/Stewart.pa/WLList/ or /Indigo/Cedar/AM/ to the debugger search rules.
It seems clear that the version map is the correct solution. Until it can be designed and implemented, search rules for the debugger are simple and easy to implement.
Commands and Packages
There is a proposal above that there should be directories ///Commands/ and ///Packages/.
The top level of the ///Commands/ directory should contain the .load files for every CommandTool command in the Cedar release. Each of these .load files would do a bringover of the appropriate code and other support files for the command. The appropriate bcds would then be run, resulting in the registration of the command. The .load file for walnut might look like this:
-- Walnut.load
-- Willie-Sue November 23, 1983 3:33 pm
-- Cedar mail user interface
Bringover /a /p ///Top/Walnut.df
Run ///Packages/AlpineUser/AlpineUserImpl.bcd
Run ///Packages/Walnut/Walnut.bcd
The last comment line is what will be printed by the Help command for something that is not yet loaded. The lines with Run commands know that DF files for packages put their code into subdirectories of the ///Packages/ directory.
FS
The following material assumes a familiarity with the comments in FS.mesa.
FS is a file system for use on a Cedar workstation. It provides access both to remote file servers and to the local disk. In addition to the normal file system facilities for manipulating named files, FS also contains facilities for cacheing remote files, and for binding local names to remote files. The local disk is accessed through an abstraction called the local server. A Cedar instance is associated with a set of logical volumes on the local disk, one of which may be designated the system volume. FS provides a directory for each volume. It also provides on the system volume a cache for remote files.
File Names
Names for FS files are called "FNames". An FName consists of (in order) a server, a root directory, zero or more subdirectories, a simple name, and a version. An empty server part means the local server. Otherwise the server part is presumed to name a remote file server.
The names of files on remote servers are called "GNames", for global names. The names of files on the local server are called "LNames", for local names. In an LName, the root directory part is the name of a volume; an empty root directory part refers to the "system" volume (the one from which Cedar was booted).
The following are complete LNames:
///FS.mesa!3
//Debugger/Temp/DebugTool.bcd!12
The following are complete GNames:
/Ivy/Stewart/Temp/FS.mesa!3
/Indigo/Cedar/Top/BasicPackages.df!4
It is convenient to think of FNames, excepting the server and root directory parts, as just long file names. A name like /server/root/subA/subB/file.ext may be thought of in several ways:
as file subA/subB/file.ext in directory /server/root/
as file subB/file.ext in directory /server/root/subA/
as file file.ext in directory /server/root/subA/subB/
Attachments
FS includes facilities for attaching an LName to a GName. An attachment is a name binding. Attachment is a way to give an LName to a global file, and is useful as a cheap way to copy global files into local files. The effects of attachments can be illustrated through several examples of calls to FS.Open. Suppose "attach.txt" is an attachment to /Ivy/Stewart/Temp/attach.txt.
Open["/Ivy/Stewart/Temp/attach.txt", $read] uses the cache to obtain the bits of the file.
Open["/Ivy/Stewart/Temp/attach.txt", $write] is an error, since FS does not support write access to remote files.
Create["/Ivy/Stewart/Temp/attach.txt"] is an error, since FS does not support direct creation of remote files.
Open["attach.txt", $read] traces the attachment and uses the cache to return the bits of the file /Ivy/Stewart/Temp/attach.txt.
Open["attach.txt", $write] traces the attachment and copies the bits of the file /Ivy/Stewart/Temp/attach.txt to the local disk under the name attach.txt. In other words, writing an attachment breaks the attachment and turns the file into an actual local file.
Create["attach.txt"] breaks the attachment and creates an actual local file.
It is perfectly all right to have more than one LName attached to the same remote file. Due to the magic of the cache, even if more than one of the LNames is FS.Opened, only one copy of the bits of the file will exist on the local disk.
The only way to create new information on a remote server is to first create the information in an actual local file and then FS.Copy or FS.Rename the LName to be a GName.
Working directories
An FName that starts with a server part is called a full path name. An FName that does not start with a server part is called a partial name and will be interpreted relative to a working directory. The working directory may be specified as the argument named "wDir" to
FS procedures, or may be defaulted. FS looks for a working directory name in three places in order:
1) the "wDir" argument;
2) the value of the $WorkingDirectory property of the process property list;
3) the default working directory (set with FS.SetDefaultWDir).
The $WorkingDirectory property will be particularly relevant to this memo.
The cache
FS manages a cache on the system volume of a Cedar instance. The cache contains complete copies of global files. Global files cannot be opened for writing with FS.Open. New global files cannot be created via FS.Create. FS.Copy and FS.Rename can be used only to create the !N version of a global file. Thus, cached global files never need to be written back to the server.
Temporary Expedients
It will not possible to implement this grand vision of the future all at once:
There will be places in Cedar that break when local directories are used, an example might be a system that creates a temporary file with a shortname only. If the file is created from one process and referenced from another, it is likely that different working directories will be used. Such systems should in general be changed to use full path names. Icon files, fonts, and the like all present potential problems. With luck, most can be fixed before the release.
Until the version map machinery can be extended to accomodate code not in the release, the debugger will use a search list to aid in finding symbols files.
A more serious impediment to the full implementation of this proposal is the need for compatibility with old DF files. For the moment, we will have to do without local "scatter-gather" abilities. It will be easy to do a bringover to a particular local directory, but difficult to scatter the individual files referenced in a single DF file to multiple local directories. This presents no difficulties for the program development model above, but what of releases?
For the moment, there will be no subdirectories of ///Commands/ or ///Packages/. All .load files and runnable .bcds will live in ///Commands/, and all files required for packages will live at the top level of ///Packages/. In order to install a subsystem or package one merely needs to do a bringover to ///Commands/ or ///Packages/ as appropriate.
The directories like ///Interfaces/ and ///Doc/ and ///Top/ described above are examples of cases where it would be desirable for a bringover of a single DF file to retrieve documentation to ///Doc/, interfaces to ///Interfaces/, etc. This problem will be solved in the near term by special "orthogonal" DF files. For example, Doc.df would be a single DF file which referenced documentation files from many different packages and systems. These special DF files will be an operational bother, but will work.