DFFiles
DFFiles documentation
Release as [Indigo]<Cedar>Documentation>DFFiles.tioga
Last edited Russ Atkinson on June 8, 1983
Abstract This documentation describes the software that handles DF files. Of particular interest are the programs BringOver, SModel and VerifyDF.
Table of Contents
Introduction 2
Simple DF File Format 2
Individual Commands
BringOver 3
SModel 7
VerifyDF 10
DFDisk 12
DFDiff 12
DeleteAll 12
DFDelete 13
RemoteDeleteAll 13
ReleaseTool 13
Other Information 14
Cedar Release Procedures 14
Complete Syntax Description of DF Files 18
Semantics of DF Files 18
XEROX
Xerox Corporation
Palo Alto Research Center
3333 Coyote Hill Road
Palo Alto, California 94304
DRAFT — For Internal Xerox Use Only — DRAFT
Introduction
The programs described below comprise a general package for file management with explicit version control. Each program takes some number of DF files, which are lists of filenames, one per line, specifying a remote host and directory where the file is normally stored, followed by the file's creation date. (In certain situations some of this information can be omitted.)
The DF files system was initially used by people running Mesa on public Dorados who wanted to guarantee they had the correct version of files they needed and as an easy way to save changed versions of file without unnecessary copying. It is now being used to partially automate the Cedar and Mesa group release process as well. Eric Schmidt was primarily responsible for the programs and the initial versions of this documentation.
Of the commands described in this document, these three are most heavily used:
BringOver - will insure that the files listed in a DF file are retrieved from their remote file servers, possibly overwriting different versions already on the local disk.
SModel - once BringOver has been run and a few files have been changed (possibly by editing them), SModel will store the newest versions on the remote file servers and produce a new DF file containing references to the newest versions.
VerifyDf is applied to a Cedar .Bcd file and a DF file and verifies that all files needed to re-build that .Bcd are listed in the DF.
This document is intended as the definitive documentation for the commands and procedures described here. Please send messages with comments, questions, and discrepancies to CedarSupport^.PA. Messages about changes to the DF files system are sent to CedarImplementors^.PA and/or CedarUser^.PA.
You might also read "Cedar Releases: Policies and Procedures" by Roy Levin, stored on [Indigo]<Cedar>Documentation>ReleaseProcedures.Press and .Bravo, for information on use of DF files in the Cedar release process.
Simple DF File Format
A simple DF file is composed of some control lines and lines of the form:
Directory [host]<directory-path> <file name and version> <create-date>
For example,
Directory [Indigo]<Poplar>Cedar>pl.mesa!13 2-Oct-80 15:43:09 PDT
specifies a specific version number for the file server, and also lists the file's create-date. Note that a file cannot have the name Directory, and upper- and lower-case differences are ignored.
Blank lines are OK and lines are treated as comments if they begin with "//" or "--".
The format of DF files is similar to that produced by the Local-List or Remote-List sub-commands in the FileTool. You may make it almost identical by setting the options in the FileTool so that only the file name and file create dates are listed.
Example DF File:
An example file might be named "Poplar.DF" and contain:
-- File Poplar.DF, created 5-Oct-80 19:48:32
Directory [Indigo]<Poplar>Cedar>
poplar.bcd!16 3-Oct-80 13:26:28
checkpoint.bcd!3 24-Sep-80 11:14:26
AFiles.bcd!5 2-Oct-80 17:17:32
overviewdefs.bcd!6 2-Oct-80 17:16:16
coredefs.bcd!7 2-Oct-80 17:16:20
filepageusedefs.bcd!5 2-Oct-80 17:17:28
filepageusedefs.mesa!1 26-Oct-79 12:45:39
filesystemdefs.bcd!5 2-Oct-80 17:17:31
filesystemdefs.mesa!1 22-Oct-79 13:21:59
BringOver
If the name of your file is "Poplar.DF", you type
BringOver Poplar.DF
to the Executive.
BringOver works as follows: It reads the DF file one line at a time. It takes the remote file name, strips off the directory information and looks to see if it is on the local disk. One of three things can happen:
(1) If the file is not on the local disk, BringOver will try to retrieve it.
(2) If the file is on the local disk, BringOver looks at the version on the local disk. If the create-date on the local file differs from the the create date listed in the DF file, BringOver will try to retrieve the remote version.
(3) If the create-date is omitted from the DF file, BringOver will ALWAYS try to retrieve the file.
If you omit the IFS version number (e.g. "!3"), BringOver will enumerate all the versions of that particular file looking for one with the correct create time. If there are no versions of the file you list in the DF file on the remote host in the directory you specify, BringOver will give you a warning message. If there are files with the same name and none of the create-dates avaliable match that listed in the DF file, BringOver will give you a warning and offer to retrieve the latest version.
After running BringOver you can be sure the files listed in the DF file are on your local disk, and that their create-dates agree with the create dates listed in the DF file, or BringOver will have printed out error messages.
Normally BringOver will list each file to be retrieved and will ask for confirmation. (You may reply "y" or CR to confirm, "n" to skip retrieval of this file, "q" to quit the BringOver program altogether, and "a" to retrieve this file and transfer subsequent files without further confirmation.) BringOver will transfer all the files without confirmation if you provide the /a switch:
BringOver.~ /a Poplar.DF
BringOver can read a remote DF file as easily as a local one:
BringOver.~ [Indigo]<Poplar>Poplar.DF
reads Poplar.DF from the remote file server as specified. (If the host file system is "Indigo", the "[Indigo]" can be omitted.)
As files are brought over, a property (called the RemoteFileName property) is added to their leader page recording where the remote copy can be found. These properties can be printed out by the DFDisk program (see below.)
Comments may be indicated by lines beginning with // or --, as in the Poplar.DF example above. In addition, BringOver will print comments at the end of the file after the last line listing files when it is run. This way distributors of software can print out useful reminders to their users.
Instead of giving create dates
If the create date entry is a "~=" rather than a normal date, BringOver will retrieve the file only if the version on the remote server is different than the version on the local disk. If the file is not on the local disk, it will be retrieved. For example,
Foo.Mesa ~=
If the create date entry is a ">" rather than a normal date, BringOver will retrieve the file only if the version on the remote server is newer (or more recent) than the version on the local disk. If the file is not on the local disk, it will be retrieved. For example,
Foo.Mesa >
will retrieve from the remote server only if there is a newer version on the remote server.
Indirect Use of Other DF Files
There are two common, distinct uses of one or more DF files by another. The first is syntactic inclusion, similar to macro-expansion, where a DF file references another and all programs treat these DF files as if they were a single DF file with the text of the inner DF file substituted in the outer one at the point of reference. The second use is as an "Importer" of another system, package, or set of files controlled by someone else. By analogy with these concepts in Mesa, there are Exports as well.
The DF file syntax allows both forms of use, which are indicated by different keywords before the name and date of the nested DF file.
Syntactic Inclusion
A line of the form
Include [host]<path>file.DF Of <date>
will cause BringOver to invoke itself on file.DF at the point it encounters the Include statement. If the included file itself has an Include statement, then BringOver will again invoke itself on the inner DF file, and so on, in a recursive fashion. Furthermore, the DF file is retrieved using the usual BringOver rules before the recursive call.
Examples of
Include:
Include [Indigo]<Cedar>CompilerSources.DF Of 2-Oct-81 15:43:09 PDT
Include [Indigo]<Cedar>CompilerObjects.DF Of >
(The Of keyword is optional, and the Include statement may be split on multiple lines.)
Exports and Imports
DF files users are encouraged to use a single DF file for packages they maintain. Some, but usually not all, of the files an implementor needs are also needed by users, or clients, of the package. Files that are useful to both users and implementors are said to be
exported. These files are indicated by being listed below a statement similar to the
Directory statement:
Exports [host]<directory>
<list of files of use to users and the implementor>
Directory [host]<directory>
<list of files of use only to the implementor>
BringOver normally examines all the files in a DF file (i.e. it assumes you are an implementor.) If you are a user, you can get only those marked
Exports by giving BringOver the /p switch:
BringOver /p Poplar.DF
Users often need to specify the /p and /a switches. The correct way to invoke BringOver is
BringOver.~ /a /p Poplar.DF
(Note the space before the /a and between /a and /p.)
When a DF file needs to refer to other DF files as a user, it may have an
Imports statement
Imports [host]<path>file.df Of <date>
which causes BringOver to (1) retrieve file.df to the local disk if necessary and (2) examine all files marked Exports in file.df and retrieve them if necessary. Of course file.df may have Include or Imports statements, so this is a recursive algorithm.
Sometimes the
Exports files are too few or too many files and the user of a package would like to list the files he/she needs from the imported DF file. Appending a
Using clause to the
Imports statement, analogous to the one in the Mesa language, gives the user explicit control over which files he/she needs:
Imports [host]<path>Package.DF
Of <date>
Using [list of files, separated by commas]
(The Of <date> clause can be omitted.)
Use of the Using clause is strongly recommended, but is not required.
Examples of
Imports:
Imports [Indigo]<Cedar>Pilot>PilotMesaFriends.DF
Imports [Indigo]<Cedar>CoPilot>CoPilot.DF
Of 24-Sep-81 11:14:26 PDT
Using [CPSwapDefs.Bcd, CPSwap2.Bcd]
As with the Include clause, the Imports statement (and Using clause) may span multiple lines.
In the CoPilot.DF example above, CPSwapDefs.Bcd and CPSwap2.Bcd can appear anywhere in CoPilot.DF. (It does not matter whether they are Exports or not when a Using clause is used to access them.)
The files referred to by an Imports statement may themselves be exported to the next outer DF file by preceding the word Imports by the word Exports. This is useful when users of a package you maintain need to have files from some other package in order to (e.g.) compile their system. You can import and export these files to assure your users will have them.
Using Clause Semantics
Since, in general, CoPilot.DF (in the example above) may have internal structure (i.e. it references other DF files with
Include and
Imports lists), CPSwapDefs.Bcd and CPSwap2.Bcd may be defined in some of the DF files referenced by CoPilot.DF. To avoid an explosion in search time, these restrictions apply when using
Using lists on a DF file with internal structure:
(1) A using list file (e.g. CPSwapDefs.Bcd) may be retrieved from a DF file included by an imported DF file (e.g. included by CoPilot.DF). (In other words, included DF files are treated as if they were macro-substituted inline in the top DF file.)
(2) A using list file may be retrieved from a DF file imported (without a Using list) by an imported DF file (e.g., imported without a Using list by CoPilot.DF) as long as the file is exported in the inner DF file. (I.e., imports are significant one level below the top DF file, CoPilot.DF.)
(3) A using list file may be retrieved from a DF file imported with a Using list by an imported DF file (e.g. imported with a Using list by CoPilot.DF) as long as the using list file being retrieved also appears on the inner using list. (I.e., multiple, nested Using lists must intersect.)
Useful Modes
A option (/v) will run BringOver in verify mode, where it will verify that the files listed in the DF file are present on the remote servers. (No files are retrieved in this mode.) If the file is listed correctly and a local copy exists, BringOver will add the RemoteFileName property to the leader page. Note for large DF files the verify option takes a few minutes. BringOver will inform the user if there are newer versions of files on the remote file servers, and will make a new DF file (on a file called "NewDFFile.DF") listing the newer versions of files listed in the original DF file. If any files were listed in the DF file without their IFS version numbers (e.g. !5), BringOver will make a new DF file with those version numbers filled in (also on "NewDFFile.DF").
An option (/o) will instruct BringOver to only BringOver the file(s) listed after the /o. For example,
BringOver /o x.mesa files.df
will examine only the file "x.mesa" in files.df, even if other files should be brought over. If no "." is given, BringOver will append ".Mesa", so this is equivalent to the preceding example:
BringOver /o x files.df
More than one file can be listed after /o, in which case the last DF file on the command line is searched.
Five options are available to reduce the number of files that are retrieved when BringOver is run. /b causes BringOver to retrieve only those files that are "derived" from source files, i.e. files that end in ".Bcd", ".Signals", ".Boot", and ".Press". /s causes BringOver to retrieve only non-derived files, i.e. all files that do not end in ".Bcd", ".Signals", ".Boot", and ".Press". /r causes BringOver to retrieve files that are in directories marked ReadOnly in the DF file and to retrieve files that are imported from other DF files. /w is the inverse of /r: only files that are not marked ReadOnly or imported are retrieved. Finally, /u causes BringOver to retrieve a file if a version is already on the local disk and the version listed in the DF file differs from the version on the local disk.
The /f option causes BringOver to ignore local files when it decides to retrieve a version of a file, thus /f "forces retrieval" of files on the local disk.
Other information about BringOver
When the filename on a line in a DF file is preceded by a tilde ("~"), BringOver will retrieve the indicated version if it is not present on the disk. If there is already a version on the disk, then it will not be overwritten, regardless of its create date.
BringOver listens for Control-DEL. When typed, BringOver will abort itself.
BringOver can handle DF files with fewer than 500 files.
Should you need to supply a connect name and password, BringOver will prompt you for them. When a CR for the connect name and a CR for the connect password are typed, BringOver will ask if you want to use a different login name or abort the program.
BringOver can be given a list of DF files on the command line. It will behave as if each file had been given separately.
BringOver can be used to retrieve files that are not listed in DF files by putting them on the command line
BringOver [Ivy]<Schmidt>User.CM!3 [Indigo]<Cedar>Lister>Lister.Bcd
will retrieve those two files. If a file ends in ".DF", BringOver will retrieve the contents as well.
If the version of a file on the local disk is newer than the one BringOver is retrieving, BringOver will first rename the local version by appending "$$" to the filename and then retrieve the local file. If the file ends in ".Bcd", BringOver does not bother to rename the file.
Summary of options to BringOver:
/a Transfer files without requesting confirmation.
/b Retrieve only "derived" files.
/f Retrieve files to the local disk even if create time of local copy agrees with the create time of the remote copy.
/o Transfer only the following files (extension defaults to ".Mesa"). The last file in the list is the DF file.
/p Transfer only those files marked Exports or Public.
/r Transfer only those files marked ReadOnly or imported from other DF files.
/s Transfer only those files that are not "derived".
/u Transfer only those files that are already on the local disk, but the local version differs from the remote version (update mode).
/v Verify mode; No files are transferred but checks the DF file for accuracy.
/w Transfer only those files not marked ReadOnly and not imported from other DF files.
SModel
The SModel command (SModel stands for "Simple Modeller") can be used to store back new versions of files you've changed since you last ran the BringOver command to fetch files under control of a DF file. For example, if your DF file is called "Poplar.DF" and you've already run the "BringOver" command, the command
SModel Poplar.DF
will execute as follows: The files listed in Poplar.DF are checked on the local disk. If any have different create dates SModel will store them on the remote servers specified in Poplar.DF. Then SModel copies the DF file to one ending in "$" (e.g. Poplar.DF$), and produces a new Poplar.DF file with the new create dates and remote file system version numbers (e.g. !4).
If a file on the local disk is listed without any create date in the DF file, SModel will fill in the create date from the version on the local disk and then store the file out on the remote directory. If the file is listed in the DF file followed by a ">" or "~=", this file is ignored and will not be transferred.
If the DF file contains a reference to itself (e.g. System.DF lists "System.DF" as one of the files you want), then SModel will also store a new version of the DF file on the remote server. Since SModel has to write out a new DF file before it can store the DF file, SModel cannot put the IFS version number (e.g. !5) on the DF file name in the case of self-referencing DF files. However, BringOver will always get the correct version of the DF file since it will use the create-date of the DF file instead.
SModel will pause before storing each file and ask if it should go ahead. You may respond with "y", "n", "q", or "a". "y" means go ahead and store it, "n" means do not store this file and keep running, "q" means do not store this file and stop running immediately, and "a" is like "y" except subsequent questions will not be asked, and the "y" answer will be assumed. You can give the "/a" option on the command line to SModel, e.g.
SModel /a Poplar.DF
and all questions will automatically be answered "y".
ReadOnly files
When you don't want any of the files in a directory to be changed by SModel, regardless of any changes you may make, you can indicate the files in the directory are read-only by preceding the
Directory keyword with
ReadOnly. For example:
ReadOnly Directory [Indigo]<Mesa>System>
indicates the files listed below should NEVER be stored back on <Mesa>System>.
Maintaining Consistency
It is important to remember that SModel NEVER enumerates the remote file system, so SModel may not detect that certain files listed in a DF file are not present on remote servers.
A common mistake is to assume that if you run SModel on a DF file successfully, and then simply change a Directory in the DF file, then all the files will be transferred (again) to the new directory. This is WRONG! After SModel has been run the first time the create dates listed in the DF file will agree with the dates of the files on the local disk. Since SModel checks the create dates listed in the DF file, on the second invocation SModel will not detect that any files need be transferred EVEN though the Directory has been changed.
To resolve these problems, SModel will take a /v option (/v stands for verify.) If given this /v option, SModel will not only apply the algorithm described above to store files, but if it decides a file does not need to be stored, it will look on the remote file server and check that in fact the file does not need to be stored. If the file is not on the remote directory, or the version listed in the DF file is not on the remote directory, then SModel will store the file. In this way SModel /v will try to force the remote directory to agree with the DF file.
Include and Imports
SModel will invoke itself recursively on DF files that are specified in Include statements (and non-ReadOnly DF files preceded by @.) SModel will NOT invoke itself on DF files listed in Imports statements (or on DF files listed in ReadOnly directories preceded by @.)
If the Include or Imports statement is not followed by an Of <date> clause, SModel will insert such a clause in its output DF file with <date> replaced by the create date of the file on the local disk.
Release Option
If the /p option is given, SModel will store all files listed in the DF file on subdirectories of [Indigo]<PreCedar>. The subdirectory will be the same as the subdirectory given by the ReleaseAs statement. If the files listed in the DF file are normally stored on a working directory that is not <PreCedar>, the copy of the DF file on the local disk is not rewritted by SModel /p. If the normal working directory of the files in the DF file is <PreCedar>, then /n is equivalent to /v.
Other Options
1. The "/n" option instructs SModel to do everything it would normally do BUT it never transfers any files. For example,
SModel /n Poplar.DF
2. If you give the "/s" option to SModel, it will store onto the remote server(s) all the files you've listed in the DF file, except those marked
ReadOnly, whether or not local create dates match the ones in the DF file. For example,
SModel /s Poplar.DF
3. If the /r option is given, then SModel will store files it decides should be stored by the above algorithm even if the directory they are stored onto in marked ReadOnly.
Getting Started with SModel
New users are often confused about the relationship of the entries in a DF file to the local and remote directories, and what SModel will do in certain cases. The easiest way to understand it is that SModel assumes 1) that the DF file was an accurate description of the remote directory at some point in the past and 2) files with differing create dates that it finds on the local disk are the "Truth" and that it should transfer any files that are different. However, assumption #1 allows SModel to assume that files with the same create date in the DF file and on the local disk ALSO exist on the remote server. This assumption is often not true when a new DF file is being created. To get started, there are at least these things you can do:
1. Make a DF file of just file names (no create dates) and run SModel on it. The files will be transferred and the create dates will be filled in.
2. Use VerifyDF (see below) to automatically generate a DF file, edit the DF file as you need to, and use the /s option to store all the files out onto remote servers.
3. If you have a complete consistent world out on remote servers, simply retrieve all those files onto the local disk and make a DF file with all their filenames but no create dates. Run SModel with the /n option to get the DF file filled in with all the create dates. (/n will not needlessly make extra copies of the files on your remote directories).
If you are maintaining a system for others, you may need to have entries in the DF file that are
ReadOnly (such as for <Pilot>Defs>, etc.) Filling in all the create dates by hand is painful, so you may combine the /r and /n options to fill in all the create dates in the DF file but not transfer any of the files! For example,
SModel /r /n Foo.DF
will fill in all the create dates for files in Foo.DF but none will be stored.
CameFrom handling
SModel normally changes entries like
Directory [host2]<directory2> CameFrom [host1]<directory1>
into lines like
Directory [host1]<directory1> ReleaseAs [host2]<directory2>
and changes
Imports like
Imports [host1]<directory>file.DF
Of <date>
CameFrom [host2]<directory2>
into
Imports [host1]<directory>file.DF Of <date>
since the CameFrom is inserted by the Release Tool. If you want to prevent SModel from making these changes, give SModel the /f option.
Other Information about SModel
SModel listens for Control-DEL. When typed, SModel will abort itself.
SModel handles Connect user names and passwords exactly as BringOver does (see above).
If a file is transferred, the RemoteFileName property (see above) is added to the leader page of the file on the local disk to record where a remote copy can be found.
Summary of options to SModel:
/a Store files without asking the user for confirmation.
/c Try to write DF files that are compatible with Alto DF software. (No warnings are given if this is impossible.)
/f Do not switch CameFrom clauses to ReleaseAs clauses and do not delete CameFrom clauses from Imports.
/n Do everything it would normally do BUT never transfer any files.
/p Store files on the <PreCedar> directory for the next Cedar release.
/r Ignore the ReadOnly attribute when deciding to store files.
/s Store all the files in the DF file, even if the create dates are correct.
/t Do not invoke SModel recursively on inner, included DF files. Only examine the top DF file given.
/v Verify mode: Check every entry to see if it is correct. Files will be stored onto remote directories if necessary.
VerifyDF
VerifyDF attempts to answer the question: Does this DF file have entries for all the files I need to rebuild this program? VerifyDF will analyze a .Bcd file and compare the files recorded in the .Bcd with entries in a DF file. If a file entered in the .Bcd is not in the DF file, VerifyDF will give a warning message about the omitted file(s).
To the Simple Executive, type
> VerifyDF DFFile.DF Package.Bcd
where DFFile.DF is supposed to be a complete description of all the source files that were compiled and bound to produce "Package.Bcd", and all the object files produced while building "Package.Bcd". VerifyDF will analyze "Package.Bcd" and give warning messages about those files needed to build Package.Bcd not listed in DFFile.DF. If Package.Bcd was produced by the Binder, VerifyDF will complain if the .Config file or if any of the implementation modules listed in the .Config are not also listed in "DFFile.DF". If Package.Bcd was produced by the Compiler, VerifyDF will complain if the .Mesa source file or any of the Definitions Files (.Bcd's) it depends on are not listed. (A list of files omitted is written to the screen and to a file "MissingEntries.DFFile.DF$". Similarly, .Mesa, .Bcd, and .Config files that are not actually required to build "Package.Bcd" are listed.)
Once it has analyzed Package.Bcd, VerifyDF will then analyze any .Bcd's needed by Package.Bcd (described above). This analysis will continue until all .Bcd's that are referenced in the closure of the dependencies among .Bcd's are analyzed.
If the DF file is omitted, VerifyDF will put a list of the top-level dependencies onto "MissingEntries.DF$", which can then be given a more suitable name and VerifyDF can be invoked again. This is a simple way to bootstrap a complete DF file for a package.
VerifyDF will look on remote file servers for the correct version of a Bcd file and will use the Leaf protocol to read it page by page, so the Bcds do not necessarily have to be on the local disk for VerifyDF to do its job.
Other Information about VerifyDF:
The DF file that describes a package may optionally have entries preceded by a "+". If VerifyDF is invoked without a .Bcd file it will look inside the DF file and assume that any file name preceded by a "+" is a top-level Bcd and proceed to analyze it and all its dependencies, as described above. More than one .Bcd may be preceded by a "+" sign. For example, if DFFile.DF had an entry of the form
+Package.Bcd
will analyze "Package.Bcd" and all its dependencies.
The DF file(s) involved do not need to be on the local disk. For example, if DFFile.DF were stored on [Indigo]<Cedar>Top>,
VerifyDF [Indigo]<Cedar>Top>DFFile.DF
is equivalent to retrieving DFFile.DF and invoking VerifyDF on the local copy.
VerifyDF has two options:
When given /f, VerifyDF will write out a single DF file containing (1) the contents of the top-level DF file, and (2) the entries this top-level DF file refers to using Imports and Include statements. Thus /f "flattens" the top-level DF file. (This is useful for generating DF files to input to ad hoc DF parsers, such as the Cedar cross-reference program, etc.)
When given /a, VerifyDF will retrieve any DF files referenced by the DF file being verified without asking for confirmation. If /a is not given, then the user will be asked to respond with either "y", "n", "a", "l", or "q". "y" means retrieve the DF file, "n" means skip the Df file, "a" is like "y" but subsequent DF files are retrieved as if "y" had been typed, "l" means use the copy of the Df file on the local disk in place of the version listed in the DF file, and "q" means stop running immediately.
VerifyDF listens for Control-DEL. When typed, VerifyDF will abort itself.
Summary of options to VerifyDF:
/a Retrieve DF files to the local disk without confirmation..
/f Write out a single DF file containing all the parts included and imported by the top-level DF file.
Summary of command lines for Pilot VerifyDF:
VerifyDF DFFile.DF Package.Bcd -- analyze Package.Bcd
VerifyDF DFFile.DF -- analyze files preceded by "+"
VerifyDF Package.Bcd -- analyze Package.Bcd and construct a DF file
DFDisk
DFDisk will produce a file "Disk.DF" that describes an entire Pilot volume. It lists most of the files on the Pilot volume (except files ending in $ and a few kinds of log files). If present the RemoteFileName property is used to give a remote host and directory where the file may be found. Note reading leader pages is slow, even on a Dorado, so expect DFDisk to take about two minutes.
DFDisk is most useful when used to print out the RemoteFileName property and when trying to save all the files on your Client volume before you re-initialize the volume.
DFDisk listens for Control-DEL. When typed, DFDisk will abort itself.
DFDiff
DFDiff will take two files specified on the command line and compare them. If a file is present in the first but not the second, it says the file is deleted. If a file is present on the second but not the first, it says the file has been added. If the create dates differ for a file listed in both DF files DFDiff will print a message to that effect. For example, if OldPoplar.DF and NewPoplar.DF are respectively old and new versions of the Poplar system,
DFDiff OldPoplar.DF NewPoplar.DF
will compare the two.
This is most useful when used with DFDisk, before and after major changes, and if the file output by DFDisk (the first time it is run) is copied to some other file name like "SaveDisk.DF". Then
DFDisk
DFDiff SaveDisk.DF Disk.DF
will compare the two DF files and tell you what you've added or deleted.
DFDiff listens for Control-DEL. When typed, DFDisk will abort itself.
DeleteAll
The DeleteAll command will delete any file on the local disk that does not match one specified in an exception list. If invoked with no arguments, DeleteAll uses an exception list on BasicClientVolumeFiles.DF or BasicCoPilotVolumeFiles.DF (stored on [Indigo]<CedarLib>DFFiles), depending on which volume DeleteAll is run from. If DeleteAll is given the /n switch, it will skip the Basic...VolumeFiles.DF exception list. Users may themselves specify exception lists. The simplest way is to list the files you do NOT want deleted on the command line. If you have a file of filenames (one local filename per line) or a DF file you can use it as an exception list by typing its filename preceded by a '@ (i.e. quote, at-sign). For example,
DeleteAll /n Foo.Mesa '@Schmidt.DF '@OtherFiles.List
will delete all files on the volume except those specified in either the DF file or the other file list. In the case of DF files, DeleteAll will include the contents of nested DF files in the exception list.
DeleteAll will list each file not eliminated by an entry in the exception list and ask the user to type "y" or CR for yes, delete it, "n" for no, skip it (don't delete it), and "q" to quit. If DeleteAll is given the /a option it will delete all non-eliminated files without further confirmation. For example,
DeleteAll /a '@Schmidt.DF
will delete all files not in the exception list on Basic...VolumeFiles.DF and not in the DF file Schmidt.DF. Use /a very CAREFULLY! Within CSL and ISL it is common practice to
DeleteAll /a
on public Dorados to delete previous user's files or to cleanup after you are finished.
DFDelete
DFDelete is the reverse of DeleteAll. Given a list of DF files, DFDelete will read in all DF files (and nested ones too) and produce a list of all the filenames in the DF file that appear on the local disk (subject to two restrictions.) You may then execute the Delete.~ command to delete these files and free up the space they occupy on the local disk.
DFDelete will not add to the list of files you should delete
(1) any file that is marked ReadOnly, or any file appearing in a DF file nested in another where the outer DF file marked the inner DF file ReadOnly (as in VerifyDF)
(2) any file listed in the DF file with an explicit create-date whose copy on the local disk does NOT agree in create-date.
DFDelete takes one option (/r):
DFDelete /r Package.DF
will list all the files in Package.DF including those marked ReadOnly.
RemoteDeleteAll
RemoteDeleteAll should be used to clean up old files from release directories. It chooses as a candidate for deletion any file that is on the remote directory but is not listed (with the same create-date) in a set of DF files. To do this, it takes a directory name and a list of DF files and then enumerates the remote directory. Two lists are written in files on the local disk: a list of files that will not be deleted and a list of candidates for deletion. It then lists the candidate files on the terminal and the user is given a chance to type "y" to delete it, "n" or "CR" to skip it (not delete it), and "q" to quit. For example,
RemoteDeleteAll [Ivy]<A>B> [Ivy]<A>B>B.DF!5 [Ivy]<A>B>C.DF!6
would offer to delete all files on [Ivy]<A>B> that are not entered in the same name and create date as those listed in [Ivy]<A>B>B.DF!5 and [Ivy]<A>B>C.DF!6. Should a file listed in a DF file be listed without a create date (such as with ~= or >), then no versions of that file will be deleted.
The user may also enter "a" when asked whether to delete this file. When given "a," RemoteDeleteAll will ask if you are sure you want to go ahead and then delete the remaining candidate files without pausing for confirmation for each file.
ReleaseTool
The ReleaseTool should be run by the Cedar Release Master. Contact me for more information.
Other Information
You can find all the commands in Cedar .Bcd form on [Indigo]<Cedar>DFFiles>. All programs, except the RemoteGetTool, will run in the Cedar Viewers and Cedar UserExec world.
You may retrieve the newest released versions of these Cedar DF programs by running
BringOver /p [Indigo]<Cedar>Top>DFFiles.DF
followed by
Run BringOver.Bcd
to load in the new version of BringOver.
Also, you may find newer, experimental versions on [Indigo]<PreCedar>Top>*.DF. Use these at your own risk!
All of these programs listen for control-DEL. When typed, they will abort themselves.
Should you need to supply a connect name and password, any of these programs will prompt you for them. When a CR for the connect name and a CR for the connect password are typed, these programs will ask whether you would like to login using different login credentials or abort the program.
If you get a message that begins with "Schmidt's Debugging:" you may safely ignore it-- these messages are usually due to certain objects in memory not being freed properly.
Cedar Release Procedures
[The following section is an edited version of a set of messages written by Roy Levin and sent to Cedar implementors in September 1981. Although the procedures described below are required of Cedar implementors, many of the extensions added to DF files will be helpful for non-Cedar users as well. Although it represents a restriction on DF file usage, I recommend non-Cedar users follow this style as much as possible.]
Background
Releasing large systems is a tricky business. SDD has known this for a long time and has developed elaborate release procedures to cope with the matter. We're only just discovering how messed up things can get. However, since we have many fewer human cycles to expend on this problem than SDD does, it is imperative that we devise natural, easy-to-use, and efficient (of people time) procedures for our releases of Cedar.
Release Procedures
The [Indigo]<Cedar> directory will be used exclusively for Cedar releases; that is, a file will appear on this directory if and only if it is a part of a (reasonably current) Cedar release. There will be a "release master" (initially Levin) who is responsible for insuring this invariant. Only the release master will be permitted to create or delete files on the release directory.
In the future, we expect to use system modelling facilities to describe and issue releases. Until the necessary software is in service, we will use DF files and a number of manually-executed conventions to achieve a similar effect. The procedures I will describe have the virtue that: (1) they place only a slight additional burden on the individual implementor, (2) they enable the release master to insure the release invariant without building the release files from the original sources, and (3) they are amenable to substantial automation. The software needed to perform this automation is nearly complete.
Since the DF file is the sole descriptive mechanism presently at our disposal, we will use it in a stylized way as a "poor man's system model". Cedar implementors will observe the following conventions in producing components for release as part of the Cedar system:
(1) Each component will be described by a DF file, which may, in turn refer to other DF files. (Indeed, cross-reference is encouraged, as will be obvious shortly.)
(2) A DF file should mention (directly or indirectly) every file needed to build and use the component it describes. This requirement does not apply to the standard construction programs (PGS, Compiler, Binder, TableCompiler, Packager, MakeBoot) except in the DF files that define them, but such programs may be mentioned if circumstances seem to warrant. Thus, it should be possible to build the component by starting with a "bare disk" (i.e., one with nothing but the standard construction programs and BringOver), and, by applying BringOver to the DF file, acquire all the pieces necessary to build the component.
(3) All files that form an intrinsic part of this component (e.g., the .mesa and .config files and their derived .bcds) should appear in the DF file under Directory headings that are NOT ReadOnly. This applies also to miscellaneous command files useful in building the component as well as the DF file for the component itself. All other files mentioned in the DF file should appear under ReadOnly Directory headings. Thus, a ReadOnly Directory heading says, in effect, "the following files are 'imported' from other components", and a non-ReadOnly Directory heading says, "the following files comprise a part of this component".
(4) It is perfectly permissable to mention a superset of the files actually needed to build the component. For example, instead of painstakingly discovering all of the Pilot public definitions files needed by your program, you could simply include
Imports [Indigo]<Cedar>Pilot>PilotMesaPublic.df Of ~=
in the DF file. There are several convenient DF files of this sort available, e.g.,
[Indigo]<Cedar>Pilot>
PilotMesaPublic.df
PilotMesaFriends.df
ComSoftPublic.df
[Indigo]<Cedar>Tajo>
TajoPublic.df
TajoFriends.df
and others may appear soon.
(5) The "end result" of building a component is usually a single BCD that implements the component. You should use a program, named VerifyDF, which takes a BCD and a DF file and determines whether or not the latter mentions all the files necessary to build the former. For the purposes of checking the completeness of a release component, it is useful to have a DF file indicate the BCD(s) named within it that represent(s) the "end result". The DF file syntax will allow the character "+" to appear immediately before a file name. BringOver, SModel, etc. all ignore this character, but the release-checking software uses it to prepare the input for VerifyDF. All "top-level" BCDs in a DF file defining a release component should be marked with a "+" (more than one per DF file is permitted). The author/maintainer of a component should apply VerifyDF to it before submitting it to the release master.
(6) When the component has been built, the (non-
ReadOnly)
Directory headings in the DF file identify the location of the files intrinsic to the component. However, when the component is released, this location is changed to (an appropriate subdirectory of) the release directory. The DF file syntax has been extended to enable
Directory headings to identify both locations, e.g.,
Directory [Indigo]Levin>SpyBuild ReleaseAs [Indigo]<Cedar>Spy>
The release automation software uses this information to move the files under this heading to the indicated place on the release directory and to update DF files (including others that reference the original directory with ReadOnly) accordingly. The ReleaseAs clause is only appropriate for non-ReadOnly Directory headings, and all DF files involved in a release must have ReleaseAs clauses in all of their non-ReadOnly Directory headings.
(7) It is highly desirable to have a single DF file identify which of the files that comprise a release component are of interest to the client of the component and which files are of interest to the implementor. Several syntactic changes to DF files and the BringOver command line together provide the facilities to do this. They are best explained by an example. Consider the following fragment of a DF file for the Spy:
-- Spy.df
Exports [Ivy]<Levin>SpyBuild>
ReleaseAs [Indigo]<Cedar>Top>
Spy.bcd <date>
SpyNub.bcd <date>
SpyClient.bcd <date>
Directory [Ivy]<Levin>SpyBuild>
ReleaseAs [Indigo]<Cedar>Spy>
Spy.df <date>
SpyImplA.bcd <date>
SpyImplA.mesa <date>
. . .
Imports [Indigo]<Cedar>CoPilot>CoPilot.df Of <date>
Now suppose that a user of the Spy wants to retrieve the public files of the Spy.
The user types:
BringOver /p [Ivy]<Levin>SpyBuild>Spy.df
The /p means "exports only" and will cause Spy.bcd, SpyNub.bcd, and SpyClient.bcd to be retrieved (assuming they aren't already on the local disk). These are the only files whose
Directory headings include
Exports. Since CoPilot.df is imported but not exported, neither it nor any of its contents are retrieved.
Suppose instead that the implementor wants to make a change to the Spy. He types:
BringOver [Ivy]<Levin>SpyBuild>Spy.df
Since no /p appears, all files in Spy.df will be retrieved (unless, of course, they are already present in correct versions on the local disk). This includes CoPilot.df and, since its contents are imported, BringOver will recur on the contents of this DF file. However (the nested instance of) BringOver will retrieve only the files in CoPilot.df that appear under Exports headings. Thus, the implementor of the Spy will get all of the Spy's files, but only the exported portions of CoPilot (which is appropriate since the Spy is a client of CoPilot). To summarize: Exports in place of a Directory heading identifies files that are of interest to the client, i.e., those files that will be retrieved by BringOver /p. Imports causes BringOver to modify its behavior when recurring to retrieve only files identified as Exports. If /p is omitted, all files are retrieved, except that, once BringOver enters "exports only" mode, all nested retrievals of DF files will be exports only, regardless of the subsequent appearance of Include statements in nested DF files. This has the natural, intuitive behavior.
(8) As each implementor completes a component needed for an upcoming release, he/she sends a message to the release master containing the name of the DF file that defines the component. The implementor then must leave the component's constituent files undisturbed until the release master notifies him/her that the files have been safely moved to the release directory.
The following summary of qualifiers that appear in DF files may be helpful:
Qualifier Interpreted by Ignored by
------------ ------------------- ---------------
ReadOnly SModel, VerifyDF, ReleaseTool BringOver
Exports BringOver SModel, VerifyDF, ReleaseTool
Imports BringOver SModel, VerifyDF, ReleaseTool
Include BringOver, VerifyDF, ReleaseTool, SModel
ReleaseAs ReleaseTool BringOver, VerifyDF, SModel
Guidelines for Date Specifications
Within a DF file that is intended to conform to the guidelines for release:
(a) files that are an intrinsic part of the component defined by the DF file (i.e., those appearing under non-ReadOnly Directory headings) should specify explicit dates,
(b) files which the component imports (i.e., those appearing under ReadOnly Directory headings or those imported DF files) should specify "#" (or equivalently "~=").
However, there is an important exception to (b). It occasionally happens that a single person is concurrently developing both the implementation of an interface and a client of the same interface. (Example: a collection of BCD manipulation facilities is imported by both the compiler and the binder. Ed Satterthwaite is the implementor of the BCD facilities as well as these two illustrious clients. When developing a new version of the BCD stuff, Ed frequently and rapidly switches between his roles as implementor-of-BCD-package and implementor-of-client-of-BCD-package.) In such situations, specifying "#" in the client's DF file will cause the wrong thing to happen; it was to accommodate this situation that the ">" form of date specification was originally added to DF files. Implementors who find themselves in this predicament should use ">" instead of "#" where necessary.
It should also be understood that when the ReleaseTool is constructing the DF files to be stored on the release directory, it will uniformly replace "#" and ">" specifications by explicit dates. This is important for a release, since it is intended to be a snapshot of the state of the files at some instant of time, and the loose binding implied by "#" and ">" is inappropriate. During the development process, however, these "dynamically bound" date specifications are very useful, and it is unreasonable and unnecessary to expect people to perform the manual translation to specific dates when release time arrives.
How, then, does this transformation of dates fit in with the ongoing development process? When an implementor finds it necessary to change a component following a release, he/she has the choice of starting from the DF file on the release directory or from the one used to build the component in preparation for the release. (Of course, the latter may validly disappear after the release; only the DF file on the release directory is guaranteed to be retained.) In the former case, the maintainer will have to edit the released DF file to make it convenient for subsequent development, e.g., by adjusting Directory headings to reference working directories and by replacing specific dates with "#" or ">". In the latter case, the DF file is already (still) suitable for development work. If a particular component is frequently changed and usually by the same person, the latter approach will likely be followed. If the component is infrequently modified or if many different people have occasion to change it, the former approach will be less error-prone. The choice should be made on a package-by-package basis.
Complete Syntax Description of DF Files
Directory [host]<path>
Exports [host]<path>
ReadOnly [
host]<
path>
Any of the above can be followed by ReleaseAs [host]<path> or CameFrom [host]<path>. Exports, ReadOnly, and Directory can be combined on one line. (Public is a synonym for Exports.)
filename <
date>
Multiple lines consisting of filenames and (optionally) create dates may follow Exports, ReadOnly, and Directory headings. filename may be preceded by "+", "@", and/ or {PublicOnly}. The filename may have a directory path (but no host), a short filename, possibly followed by a version number. The <date> can have the usual form, "~=", "#", ">", or be omitted.
Include
[
host]<
path>
file.df!version
Of
<date>
The Include statement may be followed by a ReleaseAs [host]<path> or CameFrom [host]<path> statement. The Of <date> is optional.
Imports
[
host]<
path>
file.df!version
Of
<date>
Using
[ list of filenames, separated by commas ]
The Imports can be preceded by Exports, the Of <date> and Using [ ] can be omitted, and the Using clause may be preceded by a CameFrom statement.
Semantics of DF Files
All DF programs in this set use a consistent view of what a line in a DF file means. In summary, here is a list of ways to specify files in a DF file:
filename!vers <date>
Any file on the remote directory with create time <date>. If necessary, Df programs like BringOver will enumerate the remote server looking for such a file. The version number vers (if present) is used as a hint.
filename!vers >
filename!vers #
filename!vers ~=
The highest version (!H). If ~= or # is given, a program like BringOver will retrieve filename from the remote directory if its create time differs from that of the file on the local disk. The version number vers (if present) is used as a hint. > is like ~= except that the file will be retrieved only if its create time is newer.
filename!vers
When no date is present, and a version # is given, then this refers to the file filename on the remote server with version number !vers.
filename
When no date and no version number is present, then it is treated like ~=. (Except by SModel, which fills in the local create time.)
The cases for DF files that are included or imported are exactly analogous.
For example,
Directory [Indigo]<Cedar>Top>
Spy.Bcd!2 1-Jan-82 11:11:11
Refers to a file created on 1 Jan 82. Version # is a hint.
Spy.Bcd!3 ~=
Refers to the highest version on the directory. The version # is ignored.
Spy.Bcd!4
Refers to version 4.
Spy.Bcd
Refers to the highest version on the directory.