{note The file package was written by W. Teitelman, and extended by L. M. Masinter and R. M. Kaplan.  The notion of a typed definition, and the designing and implementing of type-indepedent ways of manipulating the name-definition-type-file associations, belong to L. M. Masinter.}

{note  I think we should consistently refer to "symbolic" files instead of "source" files--R. Kaplan}


Most implementations of Lisp treat symbolic files as unstructured text, much as they are treated in most conventional programming environments.  Function definitions are edited with a character-oriented text editor, and then the changed definitions (or sometimes the entire file) is read or compiled to install those changes in the running memory image.  Interlisp incorporates a different philosophy.  A symbolic file is considered as a database of information about a group of data objects---function definitions, variable values, record declarations, etc.  The text in a symbolic file is never edited directly.  Definitions are edited only after their textual representations on files have been converted to data-structures that reside inside the Lisp address space.  The programs for editing definitions inside Interlisp can therefore make use of the full set of data-manipulation capabilities that the environment already provides, and editing operations can be easily intermixed with the processes of evaluation and compilation.

Interlisp is thus a "resident" programming environment, and as such it provides facilities for moving definitions back and forth between memory and the external databases on symbolic files, and for doing the bookkeeping involved when definitions on many symbolic files with compiled counterparts are being manipulated.  The file package provides those capabilities.  It removes from the user the burden of keeping track of where things are and what things have changed.  The file package also keeps track of which files have been modified and need to be updated and recompiled.


The file package is integrated into many other system packages.  For example, if only the compiled version of a file is loaded and the user attempts to edit a function, the file package will attempt to load the source of that function from the appropriate symbolic file.  In many cases, if a datum is needed by some program, the file package will automatically retrieve it from a file if it is not already in the user's working environment.


Some of the operations of the file package are rather complex.  For example, the same function may appear in several different files, or the symbolic or compiled files may be in different directories, etc.  Therefore, this chapter does not document how the file package works in each and every situation, but instead makes the deliberately vague statement that it does the "right" thing with respect to keeping track of what has been changed, and what file operations need to be performed in accordance with those changes.


{note  ***** philosophy of file package -- "don't edit file; edit fn and let file package worry about file"}

{note ***** a file is a "database" of info about fns and other objects}

{note ***** database file has particular format:  PPed lisp statements plus filemap, so can be listed, but people don't manipulate it --- there are functions for manipulating files}

{note ***** simple def of LOAD, FILES?, MAKEFILE, CLEANUP, with many examples}

For a simple illustration of what the file package does, suppose that the symbolic file {lisp FOO} contains the functions {lisp FOO1} and {lisp FOO2}, and that the file {lisp BAR} contains the functions {lisp BAR1} and {lisp BAR2}.  These two files could be loaded into the environment with the function {fn LOAD}:

{lispcode
_ (LOAD 'FOO)
FILE CREATED  4-MAR-83 09:26:55
FOOCOMS
{bracket DSK}FOO.;1
_ (LOAD 'BAR)
FILE CREATED  4-MAR-83 09:27:24
BARCOMS
{bracket DSK}BAR.;1}


Now, suppose that we change the definition of {lisp FOO2} with the editor, and we define two new functions, {lisp NEW1} and {lisp NEW2}.  At that point, the file package knows that the in-memory definition of {lisp FOO2} is no longer consistent with the definition in the file {lisp FOO}, and that the new functions have been defined but have not yet been associated with a symbolic file and saved on permanent storage.  The function {fn FILES?} summarizes this state of affairs and enters into an interactive dialog in which we can specify what files the new functions are to belong to.

{lispcode
_ (FILES?)
FOO...to be dumped.
    plus the functions: NEW1,NEW2
want to say where the above go ? Yes
(functions)
NEW1  File name: BAR
NEW2  File name: ZAP
  new file ? Yes
NIL}


The file package knows that the file {lisp FOO} has been changed, and needs to be dumped back to permanent storage.  This can be done with {fn MAKEFILE}.

{lispcode
_(MAKEFILE 'FOO)
{bracket DSK}FOO.;2}


Since we added {lisp NEW1} to the old file {lisp BAR} and established a new file {lisp ZAP} to contain {lisp NEW2}, both {lisp BAR} and {lisp ZAP} now also need to be dumped.  This is confirmed by a second call to {fn FILES?}:

{note  Mike, please check to make sure this example is verbatim, commas, dots, periods, etc.}

{lispcode
_ (FILES?)
BAR, ZAP...to be dumped.
FOO...to be listed.
FOO...to be compiled
NIL}

We are also informed that the new version we made of {lisp FOO} needs to be listed (sent to a printer) and that the functions on the file must be compiled.

Rather than doing several {fn MAKEFILE}s to dump the files {lisp BAR} and {lisp ZAP}, we can simply call {fn CLEANUP}.  Without any further user interaction, this will dump any files whose definitions have been modified.  {fn CLEANUP} will also send any unlisted files to the printer and recompile any files which need to be recompiled.  {fn CLEANUP} is a useful function to use at the end of a debugging session.  It will call {fn FILES?} if any new objects have been defined, so the user does not lose the opportunity to say explicitly where those belong.  In effect, the function {fn CLEANUP} executes all the operations necessary to make the user's permanent files consistent with the definitions in his current core-image.

{lispcode
_ (CLEANUP)
FOO...compiling {bracket DSK}FOO.;2
   .
   .
   .
BAR...compiling {bracket DSK}BAR.;2
   .
   .
   .
ZAP...compiling {bracket DSK}ZAP.;1
   .
   .
   .
}


{note ***** typed definitions: each file info has a NAME, TYPE, VALUE, and FILE}

In addition to the definitions of functions, symbolic files in Interlisp can contain definitions of a variety of other types, e.g. variable values, property lists, record declarations, macro definitions, hash arrays, etc.  In order to treat such a diverse assortment of data uniformly from the standpoint of file operations, the file package uses the concept of a {it typed definition}, of which a function definition is just one example.  A typed definition associates with a name (usually a litatom), a definition of a given type (called the file package type).  Note that the same name may have several definitions of different types.  For example, a litatom may have both a function definition and a variable definition.  The file package also keeps track of the files that a particular typed definition is stored on, so one can think of a typed definition as a relation between four elements: a name, a definition, a type, and a file.


{note ***** access database through variables: file root name, filecoms}
{note  Mike:  could you check to see what ROOTFILENAME does on FOO.BAR?  I don't remember whether it strips extensions or not.}

Symbolic files on permanent storage devices are referred to by names that obey the naming conventions of those devices, usually including host, directory, and version fields.  When such definition groups are noticed by the file package, they are assigned simple {it root names} and these are used by all file package operations to refer to those groups of definitions{index root name of a file}.  The root name for a group is computed from its full permanent storage name by applying the function {fn ROOTFILENAME};{index ROOTFILENAME Fn} this strips off the host, directory, version, etc., and returns just the simple name field of the file.  For each file, the file package also has a data structure that describes what definitions it contains.  This is known as the commands of the file, or its "filecoms".{index filecoms}  By convention, the filecoms of a file whose root name is {arg X} is stored as the value of the litatom {lisp {arg X}COMS}.  For example, the value of {lisp FOOCOMS} is the filecoms for the file {lisp FOO}.  This variable can be directly manipulated, but the file package contains facilities such as {fn FILES?} which make constructing and updating filecoms easier, and in some cases automatic.  See {PageRef Tag ManipulatingFileComs}.



{note ***** FP housekeeping: noticing files, marking changes}

The file package is able to maintain its databases of information because it is notified by various other routines in the system when events take place that may change that database.  A file is "noticed" when it is loaded, or when a new file is stored  (though there are ways to explicitly notice files without completely loading all their definitions).  Once a file is noticed, the file package takes it into account when modifying filecoms, dumping files, etc.  The file package also needs to know what typed definitions have been changed or what new definitions have been introduced, so it can determine which files need to be updated.  This is done by "marking changes".  All the system functions that perform file package operations ({fn LOAD}, {fn TCOMPL}, {fn PRETTYDEF}, etc.), as well as those functions that define or change data, ({fn EDITF}, {fn EDITV}, {fn EDITP}, DWIM corrections to user functions) interact with the file package.  Also, {it typed-in} assignment of variables or property values is noticed by the file package.  (Note that modifications to variable or property values during the execution of a function body are not noticed.)  In some cases the marking procedure can be subtle, e.g. if the user edits a property list using {fn EDITP}, only those properties whose values are actually changed (or added) are marked.

All file package operations can be disabled with {var FILEPKGFLG}.

{VarDef {Name FILEPKGFLG}
{Text
The file package can be disabled by setting {var FILEPKGFLG}{index *PRIMARY* FILEPKGFLG Var} to {lisp NIL}.  This will turn off noticing files and marking changes.  {var FILEPKGFLG} is initially {lisp T}.
}}




{note ***** description of rest of chapter}

The rest of this chapter goes into further detail about the file package.  Functions for loading and storing symbolic files are presented first, followed by  functions for adding and removing typed definitions from files, moving typed definitions from one file to another, determining which file a particular definition is stored in, and so on.