Command Tool
To  Cedar Folk  Date December 6, 1983 2:15 pm
From Stewart.pa  Location PARC
Subject Cedar CommandTool Organization CSL
Release as /Ivy/Stewart/Cedar/CommandToolStructureDoc.tioga
Came from /Ivy/Stewart/Cedar/CommandToolStructureDoc.tioga
Last edited by Stewart.pa, December 16, 1983 9:18 pm
Abstract This document describes the CommandTool for Cedar 5. The document first describes the command line syntax supported by the CommandTool, then discusses in some detail what actions are taken with a command line. Additional details about the CommandTool implementation are also included, as is some material on the "standard" properties to be found on the process and Commander property lists.
Introduction
See CommandToolDoc.tioga.
CAUTION! This document contains some obsolete information, but we have not yet had time to revise it. Verify any details before relying on them. {Russ Atkinson}
Command line syntax
The first token on the command line is interpreted as the name of the command. The first token is defined to be anything after the leading spaces and tabs are stripped away up to but not including the first character in the set {SP, TAB, CR, '|, '&, and ';}. The command is looked up in a database and found to be either "interpreted" or "uninterpreted."
The processing of the two types of commands are quite different from then on.
Uninterpreted Commands
The command line for an uninterpreted command extends from the character after the name up to (and includes) the first CR. The command tool makes no assumptions about the command line syntax of uninterpreted commands, except that the name of the command must be isolated by the same mechanism that finds the name of an interpreted command.
As a consequence of this strict view of "uninterpreted," uninterpreted commands cannot participate in many of the facilities provided by the command tool.
Interpreted Commands
The command line of an interpreted command is a string of tokens, some of which are interpreted specially by the command tool. Some of the tokens may participate in substitutions. After substitutions, special tokens are interpreted directly by the command processor. Remaining tokens are passed to the command as arguments.
Quoting: \
\xxx, where xxx is exactly three octal digits IN [000..377], will be replaced by the character with that octal code.
\n, \r, \t will be replaced by CR, CR, and TAB, respectively.
These are the Cedar ROPE quoting conventions. The interpretation is made by Convert.RopeFromLiteral.
A string contained within " marks is considered to be a single token. A '" within a quoted string must be represented as "", so that a"b would be written "a""b".
Characters with special significance for the command processor must be quoted if a command is to receive them as part of its command line. (But see the section on Command parsing below.) There is no specification for how this is to be done yet.
Substitutions: @, &, $, *
@file[SP] will be replaced by the contents of file followed by SP.
@file[CR] will be replaced by the contents of file followed by CR.
@file@ will be replaced by the contents of file
If file does not exist, file.cm will be tried.
&name will be replaced by the value of the environment variable name. An environment variable is a name-value mapping maintained by the command processor and kept on a property list. Examples of environment variables are the current user and the working directory. Conversion to the atom which names the environment variable is made by the procedure Atom.MakeAtom[name]. The lookup will be made first on the Commander property list and then on the process properties list. If no value exists or the value found is not a Rope.ROPE, then an empty string is returned.
$n (n an integer), when it occurs in the text of a command file, will be replaced by the nth argument to the command file.
* has its usual meaning in the context of the file system.
*.mesa would expand to a list of all mesa files in the current working directory: a.mesa b.mesa c.mesa . . .
///stuff/a* would expand to a list of all files in the ///stuff/ directory whose names begin with a: ///stuff/acme.mesa ///stuff/azzz.txt . . .
In the context of star expansion, relative paths are interpreted before expansion:
"./a/b/c*" is converted to Concat[$WorkingDirectory, "a/b/c*"]
"../a/b/c*" is converted to Concat[parent-of-$WorkingDirectory, "a/b/c*"]
At this writing (November 4, 1983), the default is to not do star expansion. Commands which might reasonably expect to be called with lists of files must call the star expansion facility themselves (See the section on the CommandTool interface.).
I/O Redirection: <, >, >>, |
> file directs the standard output of a command to file (create)
>> file appends the standard output of a command to file
< file sets the standard input of a command to read from file
These redirection directives must appear as separate tokens, the ">" or ">>" or "<" must appear surrounded by whitespace.
commandA | commandB sets the standard output of commandA to connect to the standard input of commandB (pipe). CommandA will be started "in the background" as though it were started with &.
Spawning of separate activities: &
Command & starts command as a separate activity in its own window. (E. g. the command is FORKed). & must be the last token on the command line before the activating character. A command started with & gets a new viewer to run in. If a command has output redirected and is forked, then no separate window is created and input comes from IO.noInputStream unless otherwise redirected. (See the section below on IO streams)
Terminating characters: ;, CR, |, & (These are not activation characters in the ReadEvalPrint sense!)
; serves to separate multiple commands, with the additional semantics that if a command fails, succeeding commands will not be executed (quit-on-failure). When a command fails, the command processor reads until a CR is encountered or until end of file on the input stream. In the stream sense, '; is not an activating character. Stream based editing may continue until CR is found.
If a Commander.CommandProc returns the atom $Failure, it is deemed to have failed.
CR separates commands. Each command is executed in turn without regard to the success or failure of previous commands.
| and & start commands in the background. If a command within a pipeline fails, subsequent commands receive EndOfFile on their inputs. (Earlier commands should be flushed, but at present, the CommandTool has no memory and spins off each command as it is encountered.)
Command files
The first token on the command line may be the name of a command file. A new instance of the command processor is started, with input stream connected to the command file, but output and error streams connected to the original viewer.
Standards for command authors
File names use / syntax.
Standard switch characters are - and +
Neither of these matters is of especial interest to the command processor, but we should encourage some degree of uniformity among applications.
Command parsing and processing
This section is an extremely detailed discussion of what the code in CommandToolImpl does. It could be boiled down, but . . .
Obtaining the string corresponding to a specific command.
The first stage of the command processor runs as the ClientProc of a ReadEvalPrint main loop -- CommandTool.EachCommand. This ClientProc does not get to look at the command line until the deliverWhen procedure passed to the ReadEvalPrint package detects an activation character. At present, the only activation character is CR. This means that the users gets to edit the command line at will provided that it doesn't include a CR.
At a future time, the ReadEvalPrint package will start to use the new EditedStreams interface and some command line parsing will be done on the fly (e.g. ^X and ESC processing).
When a CR is found, reading from the input stream stops and ReadEvalPrint passes control to CommandTool.EachCommand along with the accumulated text from the input stream. The string accumulated is called the initial command line. This string may correspond to more than one command, because of internal @files, or special characters such as ';.
EachCommand uses a procedure called Pass1 to scan the initial command line. Pass1's job is to isolate the part of the initial command line in front of one of a set of "terminating characters". In order to do this, @files are expanded, backslash sequences are resolved, and double-quoted strings are skipped. Pass1 returns the first part of the initial command line, the terminating character, and the rest of the initial command line. Pass1 also returns a flag that indicates whether any @file expansion has taken place.
Pass1 is actually used twice. On the first occasion, it is used only to locate the name of the command and on the second occasion it is used to find the end of the arguments for the command. The "termination characters" which cause Pass1 to return are different in these two cases.
Pass1 scanning proceeds sequentially. Backslash sequences are resolved as they are encountered. Double-quoted strings are skipped over with their contents uninterpreted (except that backslash sequences inside quoted strings are resolved). @file expansion proceeds sequentially (and recursively) with the contents of the @file replacing the appropriate portion of the initial command line. @file expansion which does not result in a terminating character is halted after (at present) 20 levels on the grounds that it would probably go on forever.
Obtaining the name of the command
When Pass1 is used to isolate the name of a command, it interprets SP, TAB, CR, '|, '&, and '; as terminating characters.
The terminating character is in a kind of limbo at this point because it is not immediately evident whether the terminating character forms a token of the command language by itself or whether it is merely the first character of some longer token. The decision on this point is deferred until the next time that Pass1 is called. When that happens, the terminating character is passed back in to Pass1 for further deliberation.
In the event that the "command name" is the name of a command file, then the command file name is itself passed back to Pass1 for use as the initial part of the command arguments. In this case, the command name is changed to refer to the command file processor. In other words, the "automatic command files" facility works by rewriting the command line to make it appear that the user explicitly invoked the command file handler.
Command Lookup
The command name returned by the first invocation of Pass1 is used to find the command proc which will execute the command. Along with the commandProc itself, the lookup process returns a flag specifying whether the CommandTool's usual processing is to take place or whether the command wants an uninterpreted command line (interpreted vs uninterpreted commands). At this writing, the Commander interface does not include a facility for marking CommandProcs as interpreted or not. The default is "interpreted", clients wishing to register CommandProcs as uninterpreted must call CommandExtras.MakeUninterpreted as well as Commander.Register.
See the section on Command Lookup to see how all this is done. In brief, a Commander.Handle is allocated and the cmd.command field is filled in with the name of the command. The cmd.property list is set to the property list of the CommandTool. The cmd.err field is set to the CommandTool viewer so that Lookup procedures can print if they wish. A successful command lookup will fill in the cmd.procData field. If the cmd.commandLine field is filled in, it is passed back to Pass1 for use as the initial part of the command arguments.
If a commandProc cannot be found, then the uninterpreted flag is set.
Obtaining the rest of the command line
If the uninterpreted flag is set, the rest of the command line is the text up to the first command line. Since some @ files may have been expanded to get this far, one must analyze on a case by case basis. If no @ files are involved, then the text is taken from the original command line received from ReadEvalPrint.
If the uninterpreted flag is not set, Pass1 is called for the second time to obtain the rest of the command line. It is passed the first character separately because it was the "terminating character" returned above. Pass1 is also passed the uninterpreted flag.
Pass1 uses CR, '|, '&, and '; as terminating characters, although '& is treated specially. A CR, '|, or '; is returned right away (by Pass1) as the terminating character of a command. If Pass1 encounters an '&, the next character is checked; if it is whitespace or ';, then '& is returned as the terminator. This special treatment of '& is necessary because the syntax &token is used elsewhere in the command language. As before, Pass1 resolves backslash sequences, skips double-quoted strings, and expands @files until it finds a terminating character.
When Pass1 finds a terminating character, it returns the leading part of the initial command line (the arguments for the current command), the terminating character, the remainder of the initial command line, and the flag which indicates whether any @file expansion took place.
EachCommand saves away the remaining part of the initial command line in its partially processed state as the "buffered command line." After completion of processing of the "first command", EachCommand loops and calls Pass1 again with the buffered command line. It proceeds in this fashion until the buffered command line is used up before returning to ReadEvalPrint.
Processing of an individual command
If Pass1 has set the flag which indicates that some @file expansion has occurred, the command line which will actually be used is printed within [[double-brackets]].
At this point, only the cmd.command, cmd.propertyList, and cmd.procData fields are filled into the Commander.Handle. The CommandTool now proceeds to fill in the remaining fields by processing the arguments for the command and the terminating character.
Processing of an uninterpreted command line
The cmd.in, cmd.out are set to "Insulated" copies of the CommandTool viewer streams. The err stream is set to the top level CommandTool viewer output stream. A property called $ErrorInputStream is set to the top level CommandTool viewer input stream (this will be changed when the Commander interface is next changed.) The cmd.propertyList is just the CommandTool's property list with no changes. Refer to the section on Command Execution to see what happens next.
Processing of an interpreted command line
The command is examined. The terminating character returned by Pass1 is examined (see below).
The cmd.commandLine is initially set to the rope returned by Pass1. This rope will be processed by the several phases of processing described below. Each phase will generally alter the commandLine rope and may also fill in other Commander.Handle fields and make notes about the desired command execution environment.
Processing of the terminating character
The CommandTool inspects the terminating character and makes some notes to itself. It does not act on these notes immediately, but continues with processing.
Effect on Streams:
CR, '; -- streams will be the command tool viewer streams unless redirected
'& -- streams will be the viewer streams of a new viewer unless redirected
'| -- the output stream will be a pipe connected to the input of the next command.
Note that if the terminating character of the previous command was '| then the input stream of the current command will be affected.
Effect on Environment:
CR, '; -- The command will run in the CommandTool's process.
'&, '| -- The command will run in a new process, leaving the CommandTool free to continue.
Effect on CommandTool state:
'; -- "quitOnFailure" bit set
'& -- "background" and "create-viewer" bits set
'| -- "pipe" bit set and background bit set
Note that there are a set of these bits for the previous command as well as the current command.
Processing of the argument string: cmd.commandLine
The following actions are taken in processing the command line arguments:
& substitution
The command line is scanned for tokens of the form &name. The scanner skips over double-quoted strings. "name" may include lower and upper case letters and digits. When a name is found, the CommandTool property list is scanned for a property with the key $name (an Atom). If it is not found or is not a rope, then the process property list is tried. If the value of the property is a rope, then the value will replace the string "&name" on the command line. If the value is not a rope then the string "&name" is deleted from the command line.
$ substitution
The command line is scanned for tokens of the form $number. The scanner skips over double-quoted strings. "number" may include only digits. When a number is found, the CommandTool property list is scanned for a property with the key $CommandFileArguments. If the value of the property is a CommandTool.Argv, then the string "$number" is replaced on the command line by the value of argv[number] (where number is converted to a NAT). If the value is not a CommandTool.Argv, or the number is larger than a NAT or larger than argv.argc then the string "$number" is deleted from the command line.
* expansion
At this writing, star expansion is done explicitly by those CommandProcs which want it.
The command line is scanned for tokens which include the character '*. Tokens are things surrounded by whitespace. Each such token is first passed through CommandTool.ResolveRelativePath and then through FS.Enumerate. The notion is to replace the token which includes a '* by the string consisting of the list of full FS path names separated by spaces.
CommandTool.ResolveRelativePath interprets a small set of convenient extensions to FS naming. If ///A/B/C/ is the current working directory then CommandTool.ResolveRelativePath makes the following conversions:
"." becomes "///A/B/C/" -- the current working directory
".." becomes "///A/B/" -- the parent directory
"./xxx" becomes "///A/B/C/xxx" -- a path extending from the current working directory
"../xxx" becomes "///A/B/xxx" -- a path extending from the parent directory
It is important to note that ResolveRelativePath as used by the * expansion phase has no effect on command line tokens that do not include a *. The first two conversions above could not happen. ResolveRelativePath also does not affect bracket syntax file names.
FS.EnumerateForNames is used to do the * expansion. The names, if any, returned by FS are concatenated together with separating spaces and the whole thing replaces the original command line token.
In order for all commands to behave in a consistant fashion, any command which expects a command line as an argument should use CommandTool.ResolveRelativePath in order to handle the above .. syntax. At some point, this relative syntax may be absorbed into FS.
IO redirection
The command line is scanned for IO redirection directives. Such directives are the tokens "<", ">", and ">>". These tokens must be surrounded by whitespace. (The reason for this conservatism is to avoid confusing an IO redirection directive and a bracket syntax FS filename.) The token after the redirection directive must be a valid filename. "<" means set the cmd.in stream of the command to read from the fgiven file. ">" means to create the given file and send output from the command to the file. ">>" meand append output from the command to the given file.
Some error checking and special case handling takes place here:
It is an error to redirect the input from a file which cannot be opened.
It is an error to redirect the output if the pipe bit is set.
It is an error to redirect the input if the pipe bit for the previous command is set.
It is an error to get an FS error while trying to create the appropriate file stream.
If the create-viewer bit is set and output is redirected, then clear the create-viewer bit and redirect input from IO.noInputStream unless otherwise redirected.
IO Streams
The table below describes what streams are set up in what circumstances. The right hand four columns describe the streams for input (cmd.in), output (cmd.out), error input ($ErrorInputStream), and error output (cmd.err). The left hand four columns describe the previous terminating character, the present terminating character, whether the present command has input redirected, and whether the present command has output redirected.
Rows marked ERROR represent non-sensical command combinations.
Abbreviations:
IRD -- input redirected from specified file
ORD -- output redirected to specified file
CT -- CommandTool stream
tlCT -- top level CommandTool viewer stream
vwr -- stream attached to a new viewer
Pipe -- pipe stream
nil -- IO.noInputStream
Terminating IRD ORD in out error in error out
Character
last this
---- ---- --- --- -- --- -------- ---------
CR,& CR CT CT tlCT tlCT
CR,& CR X IRD CT tlCT tlCT
CR,& CR X CT ORD tlCT tlCT
CR,& CR X X IRD ORD tlCT tlCT

CR,& & vwr vwr vwr vwr
CR,& & X IRD vwr vwr vwr
CR,& & X vwr ORD vwr vwr
CR,& & X X IRD ORD tlCT tlCT

CR,& | CT Pipe tlCT tlCT
CR,& | X IRD Pipe tlCT tlCT
CR,& | X ERROR
CR,& | X X ERROR

| CR Pipe CT tlCT tlCT
| CR X ERROR
| CR X Pipe ORD tlCT tlCT
| CR X X ERROR

| & Pipe vwr vwr vwr
| & X ERROR
| & X Pipe ORD tlCT tlCT
| & X X ERROR

| | Pipe Pipe tlCT tlCT
| | X ERROR
| | X ERROR
| | X X ERROR
It would be nice to have all the error streams of a whole pipeline attached to the new viewer constructed for it, but I don't see how to easily implement it.
When a command is passed the CommandTool input stream, the first character available from the stream is the character after the activating character for ReadEvalPrint, (the character after the CR). If the running command is the first in a series of commands separated by '|, the standard input does not contain the later command strings. The command does not get to see any of the characters in the command processors "buffered command line" but does get to read the next character in the command processors input stream.
Since commands generally process up to EOF on the standard input, we adopt a standard keyboard character, control-shift-DEL, as the EOF character. This requires a special TIP table until the standard TIP tables are changed. (Not implemented) At present, typeing DEL in the CommandTool viewer causes the signal ViewerIO.Rubout, which is a partial and temporary substitute.
Property List
The cmd.propertyList field is initialized to the property list of the command processor.
At this point, the CommandTool is ready to construct an environment and execute the command. Refer to the section on Command Execution.
Command Lookup
General
The CommandTool does command lookup in a fairly general way which allows users and clients to alter its behavior. Basically, the $Lookup property on the commander handle property list is interpreted as a list of CommandProcs. The Commander.Handle for the command (with only the command, propertyList, and err fields filled in) is passed to each of these CommandProcs in turn. As soon as one of them fills in the procData field of the Commander.Handle, the process stops and command processing resumes. If the successful lookup proc also fills in the cmd.commandLine field of the Commander.Handle, the rope will be prepended onto the argument string. This facility is intended for use by generic lookup procedures which might want to invoke some common CommandProc with the original command name as its first argument. (The automatic command file facility uses this feature.)
If there are no procedures on the $Lookup list for some reason, then the built-in procedure LookupWithSearchRules is tried (see below).
The CommandTool starts each top level Commander with a list of three Lookup procedures: LookupWithSearchRules, LoadAndRunWithSearchRules, and CommandFileWithSearchRules, which are invoked in that order.
Lookup procedures which fail to find anything should return result: NIL. Lookup procedures which want to terminate the lookup process entirely should return result: $Failure. In any case, if a Lookup procedure returns a non-NIL msg, it will be printed on the cmd.err stream.
The CommandTool Lookup procedures make use of the working directory and search rules. The working directory is a rope attached to the $WorkingDirectory property of the process properties list. Search rules are a CommandTool specific data structure. They are a LIST OF Rope.ROPE attached to the $SearchRules property of the Commander property list. The general idea is that Lookup procedures will first look for the command name as given, and then, if the command name is not a full path name, Lookup procedures will look for the command in the current working directory and then in each directory on the search rules list.
Note 1: These search rules are not intended to be used for opening of files in general, only for locating commands.
Note 2: The Commander registry stores full path names of commands. Its structure will generally parallel the naming structure of the FS directory system.
In what follows, suppose the name typed by the user is "root", suppose the working directory is "wDir", and suppose the search rules are ("dirA" "dirB").
LookupWithSearchRules
This Lookup procedure looks for Commander.CommandProcs in the Commander registry. LookupWithSearchRules tries the following names in the following order before giving up:
Commander.Lookup[root]
If root is not a full path name then
Commander.Lookup[Cat[wDir, root]]
Commander.Lookup[Cat[dirA, root]]
Commander.Lookup[Cat[dirB, root]]
Commander.Lookup[Cat[root, *]] (try for a unique match)
If root is not a full path name then
Commander.Lookup[Cat[wDir, root, *]] (try for a unique match)
Commander.Lookup[Cat[dirA, root, *]] (try for a unique match)
Commander.Lookup[Cat[dirB, root, *]] (try for a unique match)
If LookupWithSearchRules fails all these tries, then it reports either straight failure, if none of the pattern matches produced any results, or a list of the first collection of ambiguous matches found. LookupWithSearchRules gives up on the first ambiguous match it finds. It might report "Ambiguous command: (/dirA/rootXXX /dirA/rootYYY)" even though dirB has a single file named /dirB/rootZZZ.
LoadAndRunWithSearchRules
If both LookupWithSearchRules and CommandFileWithSearchRules fail to find a CommandProc but neither terminates lookup, LoadAndRunWithSearchRules gets control. LoadAndRunWithSearchRules looks in the FS directory system for a batch of instructions on how to load a certain command, then retries the lookup using LookupWithSearchRules. The idea is to run a command file which will load the code for the command. The START code should register the command so that the second lookup attempt will succeed. These special load command files have the extension ".load". LoadAndRunWithSearchRules maintains a list of every command file it runs and never runs one twice, this is so that if a command file fails to result in the registration of the desired command, it won't be fruitlessly loaded multiple times.
The .load file is put on the I've-run-this-one list before the command file is run. If the command file fails without registering the command, the attemp can nevertheless not be repeated.
LoadAndRunWithSearchRules looks in the following places:
Cat[root, *, ".load"] (try for a unique match)
Cat[wDir, root, *, ".load"] (try for a unique match)
Cat[dirA, root, *, ".load"] (try for a unique match)
Cat[dirB, root, *, ".load"] (try for a unique match)
If all of these fail to work, LoadAndRunWithSearchRules will report failure. LoadAndRunWithSearchRules will give up after finding a set of ambiguous matches.
At this writing, LoadAndRunWithSearchRules does NOT do pattern matching. You must type the name of the command correctly for the "autoload" to work.
If a load command file is found, LoadAndRunWithSearchRules will first run it as a command file in an isolated environment (a separate instance of the CommandTool). Normally, one would expect a load command fule to "Run" some module whose START code registers one or more commands. After the command file completes, LoadAndRunWithSearchRules retries the Lookup by using LookupWithSearchRules.
CommandFileWithSearchRules
If LookupWithSearchRules fails to find a CommandProc but doesn't terminate the lookup, CommandFileWithSearchRules gets control. CommandFileWithSearchRules looks in the FS directory system to find a command file with the right name. This command file is then run with the rest of the command line serving as "command file arguments". The following (FS) places are tried:
root
If root is not a full path name then
Cat[wDir, root]
Cat[dirA, root]
Cat[dirB, root]
Cat[root, ".cm"]
If root is not a full path name then
Cat[wDir, root, ".cm"]
Cat[dirA, root, ".cm"]
Cat[dirB, root, ".cm"]
Command Execution
The CommandTool includes a mechanism for impelementing statistics gathering procedures. The Commander property list includes properties $Before and $After. The values of these properties are lists of Commander.CommandProcs. Each CommandProcs will be called in turn with the Commander.Handle intended for the eventual command. The only difference made is that the cmd.out stream is temporarily set to the ReadEvalPrint output stream for the duration of the calls on these CommandProcs. The Statistics command, for example, attaches a $Before procedure which records the starting time of a command and an $After procedure which prints the running time for the command.
Before and After procedures are always given a Commander.Handle which has the CommandTool output stream as its cmd.out. Thus the statistics information is not redirected along with the output from a command. At present, if the command is FORKed by & or |, then the Handle passed to the Before procedures is the same one that is given to the command, but the Handle passed to the After procedures is a different one.
The $Before list is called before execution of the command.
The command is now ready for execution, awaiting only a suitable environment. The procedure ExecuteCommand handles matters from here, its arguments are the Commander.Handle and a boolean which says wheter to run the command in the background.
At this point in command processing, the Commander.Handle is completely assembled with command, commandLine, streams, and property list.
If the command is to be run in the background, a new process properties list is created. See the section on copying property lists.
The $CommanderHandle property of the process properties list is set and cmd.started is set to TRUE. (Or will be as soon as the next round of Commander changes are done)
If the command is to run in the background, a new process is forked and what follows is executed in the new process. If the command is run if the foreground, what follows is executed in the CommandTool process. What follows is also executed under the protection of an UNWIND catch phrase. The catch phrase flushes cmd.out and cmd.err, and closes cmd.in and cmd.out.
The CommandProc for the user's command is now called. If it returns, the message is printed and the result is put on the Commander property list as the $Result property.
If the CommandProc returns, cmd.out and cmd.err are flushed and closed, and cmd.in is closed.
When the call to ExecuteCommand returns to EachCommand, the $After list is called. EachCommand now returns to execution of remaining commands from the same line of user input or itself returns to ReadEvalPrint.
Command line editing
This section will be re-written after the command tool starts to use the new EditedStream interface.
The command processor is a stream oriented program. It reads commands from a stream and directs output to a stream.
^X expands the command line in place, completing the substitution phase in a visible way, but without starting the command. Control-X does @file expansion, & and $ substitution and * expansion, but does not do IO redirection. (Not implemented)
ESC tries to complete the current token, which is taken to be a file name. This works for command names as well, since they live in the directory system. (Not implemented)
BS, ^A, ^W, ... -- standard editing
Spelling correction. (Not implemented)
Process management and property lists
A new process will be created for the execution of each command run "in the background". This happens when the command is started from a command line ending with & or if the command is a member of a pipeline (other than the last element of the pipeline). When commands are run in the background, the command processor will not wait for the command to finish.
There are two property lists involved: the process properties property list and the commander handle property list. In the interests of spartan living, only a small number of properties are kept on the process properties list while a larger number may inhabit the commander property list. While a property list may use an arbitrary REF ANY as a key, in the interests of standardization, it is recommended that the keys on property lists be ATOMs.
The process properties property list
This list is manipulated using the ProcessProps interface. There are two standard properties:
$WorkingDirectory -- a Rope containing a working directory acceptable to the FS file system. The working directory is inspected by all FS procedures invoked by any procedure running in the process.
$CommanderHandle -- a Commander.Handle which is the one passed to the CommandProc currently running. The CommanderHandle property is a convenience so that the authors of complex commands may avoid passing the Commander.Handle as an explicit argument to many different places. A low level procedure wishing to print error information visible to the user may use the err stream in the Commander.Handle on the process properties list. The procedure CommandTool.GetHandle returns the Commander.Handle from the properties list or raises a signal if there isn't one (not implemented -- use NARROW[List.Assoc[key: $CommanderHandle, aList: ProcessProps.GetPropList[]], Commander.Handle].err). The absence of a $CommanderHandle property is a programming error somewhere, a FORK without setting the process properties list.
The commander handle property list
There are a variety of standard properties on this list, and the programmer is also encouraged to use this list for internal communications within a complex command and for communications among a collection of related commands. The property list is also used to store methods used internal to the command tool. This is an attempt to construct an open command interpreter. Here are the standard properties:
$Prompt -- the ROPE printed by the command processor before the next command is read.
$ReadEvalPrintHandle -- the command processor uses the ReadEvalPrint package for its "main loop." This property is the ReadEvalPrint.Handle in question. In the case of a command tool processing a command file, this will be a ReadEvalPrint.Handle of the STREAM type, so any use of this record must be prepared for the "viewer" field to be NIL.
$ErrorInputStream -- This property is an input IO.STREAM from which an application can get characters from the user. It will be replaced by a field in the Commander.Handle next time that interface is changed.
$ParentCommander -- The Commander.Handle representing the instance of the command processor under whose control the present command processor was invoked. This will be NIL in the case of a top level command tool run from the Cmd button in the message window.
$CommandFileArguments -- A CommandTool.Argv represnting the arguments with which the presently running command file was invoked. It is from this property that the values for $n substitution are obtained. This will be NIL if there is no command file presently running.
$SearchRules -- A LIST OF REF ANY. Each element is a ROPE. The list represents the directories to be searched when looking for commands. The current working directory is always treated as the first element of the search rules. Search rules are ignored if full path names are given.
$Result -- An atom representing the "result" of the previous command. The atom $Failure is interpreted specially (see above). This property retains its value through the execution of the following command, so it can be used for communications between commands.
$Lookup -- A LIST OF REF ANY. Each element is a Commander.CommandProcHandle. The CommandProcs on the list will be called in turn with the Commander.Handle intended for the eventual command. The elements of the PreLookup list are free to alter the Commander.Handle. One might envision using this facility to implement command name spelling correction by altering the command name. If one of the CommandProcs fills in the procData field of the Commander.Handle, that will be the CommandProc called to execute the current command. This is how commands are looked up in the command registration database.
$Before -- A LIST OF REF ANY as above. The CommandProcs will be called in turn with the Commander.Handle intended for the eventual command just before the CommandProc for that command is called. Together with the $After list, this facility may be used by pairs of procedures which gather statistics about command execution. For example, the Statistics command attaches a $Before procedure which records the starting time of a command and an $After procedure which prints the running time for the command.
$After -- A LIST OF REF ANY as above. The CommandProcs will be called in turn with the Commander.Handle passed to the command just after the CommandProc for that command returns.
On the copying of property lists, and local and global properties
A property list is a List.AList (same as Atom.PropList). It is a LIST OF DottedPair; a DottedPair is a REF RECORD [key, val: REF ANY]. Let us suppose that all keys are ATOMs. All instances of a key are therefore the same since ATOMs are global. In order for the $Thing property on one property list to be bound to a different value than the $Thing property on another property list, the two $Thing properties must be represented by different DottedPair records. Copying a property list means that the old and the new copies initially contain the same bindings of properties to values, but that after the copy operation concludes, the properties on each list may be rebound independently. In order to implement these semantics of copy, both the CONS cells of the property list itself and the DottedPair cells of the properties must be copied. Because the DotCons cells are copied, one can change the key-value mappings in the new list without affecting the mappings in the old list. Because the CONS cells are copied, one can alter the structure of the new list without affecting the structure of the old list.
Property lists are singly linked. This means that if a program is holding a REF to the middle of a property list, only the properties further down the list are accessible. This fact may be used to implement "local" and "global" properties. When a command tool is first started, it creates a property list with the initial set of properties listed above. The REF to this "initial" property list is held for the lifetime of the command tool and handed in turn to each of the commands invoked by the command tool. This implementation has several consequences. First, a command may change the binding of one of the initial properties or may truncate the property list by setting one of the rest fields to NIL. This is generally discouraged (except that this is the way to change the prompt!). Second, a command may add a new property to the end of the property list. This new property will become part of the property list handed to later commands invoked by the same command tool. Commands may use these "global properties" in order to communicate to later commands or to "customize" a command tool by adding some application specific property. Programmers are encouraged to customize an otherwise standard command tool rather than write a private command interpreter. Third, a command may insert new properties at the head of the property list. Since the command tool is holding a ref which points into the middle of the new property list, the new properties will be lost when the command returns. Programmers are encouraged to use these "local properties" for communications within a command.
On when property lists are copied
The process properties property list is for each command run in the background. The commander handle property list is copied whenever a new instance of the command tool is created (such as when a command file is run). When a new instance of the command tool is created, more than just the property list is copied. Certain standard properties which are normally bound to lists are also copied: $SearchRules, $Lookup, $Before, and $After.
Command registration: Commander.mesa
The Commander interface defines the type of a CommandProc, its arguments and return values. The Commander interface also defines the form of the command registry, which is a database of mappings between string names for commands and the CommandProcs which implement them.
The CommandTool also makes heavy use of the CommandProc type for things which do not implement commands, but which process other commands. For example, the command lookup procedures on the $Lookup list accept a partially filled out Handle and try to complete it.
Types
A CommandProc is passed a Commander.Handle and returns a REF ANY result and a ROPE msg. Let us first consider the return arguments, because they are simpler. The result is stored by the CommandTool on the command tool property list under they key $Result. If the result is the distinguished atom $Failure, then further commands on the command line are not executed. The msg is printed on the standard output stream. Thus commands which wish to display a rope to the user may simple return it.
A CommandProc is passed cmd: Commander.Handle. The argument cmd is a REF Commander.CommandObject, which is a record containing the following fields:
in, out, err: IO.STREAM
in and out are the standard input and output streams and may be redirected. err is an output stream that always allows communication with the user. (Even if out is redirected, err connects to the CommandTool viewer)
commandLine: Rope.ROPE
commandLine is the command line with the first "token" (the name of the command) removed; commandLine ends with CR. If the command is interpreted, then things like IO redirection directives have alredy been removed from the rope.
command: Rope.ROPE
The first token on the command line, used as a name to find the CommandProc. This rope will be the actual name under which the current CommandProc was registered. Although the lookup procedures do pattern matching to find a unique command corresponding to what the uyser typed, the lookup procedure which finally succeeded replaced the original name by the actual name.
propertyList: List.AList
Name-value pairs. See the section on property lists for the details of the "standard" properties. A CommandProc may alter the property list either by appending to it, in which case the changes are permanent (seen by later commands), or by prepending, in which case the changes will be discarded when the CommandProc returns. Changes made to the value fields of existing properties are always permanent.
procData: Commander.CommandProcHandle
This is a REF to the Commander.CommandProcObject which contains the information registered with the current CommandProc. The clientData field may be of interest to CommandProcs which are registered under different names and expcted to behave slightly different.
Procedures
Register is used to make an entry in the command data base. The database stores full path names of commands. A command can be reached from its short name only if the "directory" it is in is on the search rules list. Someday Register may make the entry in the current working directory if it is called with a short name rather than a full path.
Lookup and Enumerate are used by CommandTool lookup procedures to find the one or more CommandProcs which might correspond to a user's command line.
PutProperty and GetProperty are modified versions of List.PutAssoc and List.Assoc. Whereas List.PutAssoc and List.Assoc match property list keys using "Eq," PutProperty and GetProperty check whether the key is a ROPE and in that case match keys using Rope.Equal[ . . . , case: TRUE]. Two ropes may have the same contents but different REFs, the List procedures would call them different keys, but PutProperty and GetProperty will call them the same. These procedures are candidates for removal during the next revision of Commander because I have come to the conclusion that it is not reasonable to use anything but ATOMs as property list keys.
Hints on usage
I recommend that commands be registered with full path names. For command intended for general use by the Cedar community, the directory ///Commands/ is recommended. For commands intended for personal or project use, use an appropriate directory name. Don't forget to add any new directories used to your search path, so that the commands will be easily accessible.
This directory business is intended to promote the use of the FS directory system. There should be nothing in the local root directory (///) except other directories.
The data registered along with a CommandProc includes a documentation rope and a REF ANY field called clientData. It is recommended that the documentation rope be short enough to fit on one line, as it is printed by the Help command when displaying lists of commands. ClientData is not interpreted by the CommandTool. It is intended to permit commands which differ ony slightly to share a common CommandProc. For example, the GetProperties and GetProcessProperties commands differ only in which property list is used. Both commands are registered with the same CommandProc, but one of them has clientData: NIL while the other has clientData: $Process. The common code checks the clientData field to determine the desired function.
CommandTool programmer's interface: CommandTool.mesa
The CommandTool interface includes procedures useful to the implementors of commands as well as many interesting procedures used by the implementation of the CommandTool.
Facilities for implementors of Commands
Types
Argv. History buffs may recognize this descendant of the Unix (tm) command line data structure. Argv is basically an array (sequence) of command line tokens. Argv[0] is the name of the command and other elements are the later command line tokens. An application requiring direct access to command line tokens can use Argv.
Signals
Failed. Procedures which scan the command line or execute commands may raise Failed with some message of explanation.
Procedures for processing the command line
StarExpansion. Until such time as star expansion is better integrated with the CommandTool, you must call it yourself. StarExpansion rewrites cmd.commandLine with tokens containing '* expanded through use of FS.Enumerate. If your command expects one or more filenames on the command line, then it is a good idea to call StarExpansion before calling one of the Parse procedures.
ParseToList. ParseToList breaks up cmd.commandLine into a list of ROPE tokens. Tokens are generally separated by whitespace, but a "double-quoted string" is a single token. If a switchChar is given, it is always considered to be the first character of a new token (except when inside a double quoted string). For example if '+ were the switchChar, then the string foo+bar would be returned as the two tokens foo and +bar. If the switchChar is '-, then it must be preceded by whitespace. This is to preserve the validity of '- as a character in filenames. If switchChar=SP, the switch character facility is disabled. ParseToList may raise Failed for things like mismatched "double-quotes".
Parse. Parse does the same job as ParseToList, but returns an Argv structure. In accordance with historical precedent, cmd.command is returned as argv[0] and the "real" arguments start at argv[1].
Procedures for running commands
DoCommand. DoCommand provides access to the top level facilities of the CommandTool. If a program wants to run a command as though a user had typed it, this is the way to do it. The commandLine is processed by the same routines that process user input.
DoCommandRope. DoCommandRope is similar to DoCommand except that the standard input stream given to the command is an IO.RIS from the client provided rope. Similarly, the standard output from the command is returned to the client as a rope.
These two procedures are more sensible interfaces to CommandTool.EachCommand.
One interesting application of DoCommand is to use it in the implementation of commands which take other commands as arguments. Check out the implementation of the Time command in InitialCommandsImpl.mesa in CommandTool.df:
CommandTool.DoCommand[commandLine: cmd.commandLine, parent: cmd];
Run. Run is a programmers interface to the Run command. It is useful for loading and starting bcds.
Procedures for file manipulation
CurrentWorkingDirectory. Returns the currentworking directory, if there is one, otherwise returns the FS default working directory. Converts to slash format. (If any conversion was necessary, stores back the new version as the working directory.). This procedure uses the process properties list to do its work.
ResolveRelativePath. ResolveRelativePath interprets some extensions to the standard FS filename syntax that have been found historically valuable. ResolveRelativePath does not affect any bracket syntax file names. ResolveRelativePath performs the following translations.
. becomes the name of the current working directory.
.. becomes the name of the parent of the current working directory.
./path becomes the full path name consisting of Rope.Concat[CurrentWorkingDirectory, path]
../path becomes the full path name consisting of Rope.Concat[parent-of-CurrentWorkingDirectory, path]
ConvertToSlashFormat. Converts FS name strings to slash format, but makes no consistancy checks. Since FS accepts both slash and bracket syntax, it is not clear that this is too useful, but sine the FS default is brackets, a program that wishes to print FS enumeration results in slash format would be an example.
AddSearchRule. Adds a directory to the end of the CommandTool search rules. If the argument is NIL, deletes all search rules. This procedure is used by the AddSearchRule command and by the SetSearchRules command.
Facilities for the implementation of the CommandTool
Property lists
PutLocalProperty. PutLocalProperty is used to set a "local" property on a list. OrigList is intended to point into the middle of aList. If key is found on the part of aList in front of origList, then the old binding is changed and aList is returned. If key is not found on aList before origList is encountered, then a new property is added to the head of aList and the new list is returned. If a CommandProc wants to use the property list for internal communications, here is a polite way to do it. The standard call would be cmd.propertyList ← PutLocalProperty[key: $MyPropertyKey, val: myValue, aList: cmd.propertyList, origList: cmd.propertyList];
CopyAList. CopyAList copies the CONS cells of the list itself and also copies the DotCons cells which are the elements of the list. Because the DotCons cells are copied, one can change the key-value mappings in the new list without affecting the mappings in the old list. Because the CONS cells are copied, one can alter the new list without affecting the old list
CopyListOfRefAny. CopyListOfRefAny searches aList for the binding of the given key. If it is a LIST OF REF ANY, then it is List.CopyTopList-ed and put back. This procedure is used by the CommandTool creation code to copy the Lists-of-CommandProcs ($Lookup, $Before, $After) when creating a new, independent, instance of the CommandTool.
CallList. CallList uses the given property name (typically an ATOM) to search cmd.propertyList. The value of the property should be LIST OF REF ANY. Each of the elements of the list should resolve to a Commander.CommandProcHandle. Each of the command procs found will be called with cmd as its argument. If proc is not NIL, it is called with the return values from each of the CommandProcs called. proc can stop CallList from proceeding by returning stop = TRUE; CallList is used by the CommandTool to manage the $Before, $After, and $Lookup facilities.
AddProcToList. RemoveProcFromList. These procedures are used to construct or alter a LIST OF REF ANY whose elements are actually Commander.CommandProcHandles. This is the way to add $Lookup procedures or to use the $Before and $After facility. Look at the implementation of the Statistics command.
Command line processing
FileWithSearchRules. FileWithSearchRules uses the working directory and the search rules to try to translate the short name of a file into a full path name. FileWithSearchRules calls ResolveRelativePath first. NIL is returned if the file cannot be found. The following places are tried:
IF root is a full path name then tries root then tries Concat[root, defaultExtension]
ELSE {
tries root (automatically tries $WorkingDirectory)
tries root with wdir: each element of search rules
tries Concat[root, defaultExtension] (automatically tries $WorkingDirectory)
tries Concat[root, defaultExtension] with wdir: each element of search rules
}
FileWithSearchRules is used by the Lookup procs to locate command files anywhere along the search rules.
AmpersandSubstitution. Performs the AmpersandSubstitution phase of command processing. AmpersandSubstitution rewrites cmd.commandLine with tokens of the form &name replaced by the value of the property $name on cmd.propertyList. If there is no such property, or its value is not a rope, then the &name token is deleted.
DollarSubstitution. Performs the DollarSubstitution phase of command processing. DollarSubstitution rewrites cmd.commandLine with tokens of the form $number replaced by the number-th argument to the currently running command file. If there is no command file running, or it didn't have a number-th argument, then the $number token is deleted.
IORedirection. Performs the IORedirection phase of command processing. IORedirection rewrites cmd.commandLine with IO redirection directives removed and cmd.in or cmd.out filled in as appropriate.
Insulate. Insulate creates streams for which close appears to work, but which do not close the backing stream. It creates a stream layered upon the given stream. The streamProcs are the IO package defaults except for close, which is IOUtils.closedStreamProcs.close.
Pass1. Pass1 is heavily discussed elsewhere in this document. Pass1 is used to isolate the part of a commandLine that is the command name (when nameOnly = TRUE) or to find that part of a command line containing arguments (when nameOnly = FALSE).
Lookup procedures and others
LookupWithSearchRules, CommandFileWithSearchRules, RunWithSearchRules. These procedures are described in detail in the section on command lookup.
CommandFile. This is the procedure used to create new top-level CommandTools or to execute command files. If there are no arguments, then a new top level viewer is created, otherwise, the name of the command file ahould be the first token on the command line. Other tokens will be arguments to the command file. If clientData is $Source, the command file will be run in the callers property list context.
ExecuteCommand. Execute the given command. The Commander.Handle should be fully set up (procData filled in, etc.). If background is TRUE then the command will be FORKed and the call to ExecuteCommand will return right away.
EachCommand. This is the ReadEvalPrint.ClientProc which is the top level of a CommandTool. The clientData of the ReadEvalPrint.Handle must be a Commander.Handle. EachCommand will use its propertyList and streams. EachCommand calls ExecuteCommand above after doing all the command line processing described elsewhere in this document.
LookupCommand. LookupCommand is the procedure that EachCommand uses to look for the CommandProc corresponding to a given name. It calls each of the $Lookup procedures on cmd.propertyList. It expects cmd to have the err, command, and propertyList fields set up. It may print on err. If it returns with cmd.procData filled in, that is it. Some $Lookup procedures run command files, so you may get more than you bargained for! If cmd.commandLine is filled in on return, then the procData corresponds to some generic commandProc which takes the original command name as its first argument. Generally, cmd.command will have been replaced by the correctly spelled version.