[Indigo]<KRL>document>doc-editlog!3

March 29, 1979 9:38 AM: Last week in the locker room I had a conversation with Bill Paxton in which I claimed that the problems to be faced in the next round of text preparation aids were those of "editing in the large". One can argue for hours about the virtues of modeless vs. modeful editors, page oriented vs. scroll oriented layout, etc., but in actuality the things which dominate (and frustrate) my editing activity involve the manipulation of larger-scale chunks: documents and pieces of documents, versions and files, etc. As of this week, I am embarked in a major editing activity -- taking the collection of stray KRL-1 documents and producing a single comprehensive report. (see file <???> for details). This morning on the way to Palo Alto I realized that I could use that experience in a systematic way to clarify what the problems are. I plan to keep a log, describing my editing-in-the-large activities throughout the project. At the end, I may write up a summary, or may just leave the log as a piece of data for those interested in the problem.

This is a worst-case project. The raw material with which I start consists of 103 files, containing documentation and examples (these do not include the source files for the system). They are on two file servers (MAXC2 and IVY) in 4 directories. The IVY directories are broken up into several sub-directories. Each of these subdirectories contains stray material in addition to the things I want. I wanted to do a count of the total length of all the files, but realized it would be a tedious hand project to log in to the various servers, ask for file directories with counts, eliminate the irrelevant files, then add up the numbers by hand (perhaps by going back to my ALTO and getting the transcript into BRAVO). They print out as 749 hardcopy pages (see below).

The files span over a period of two years, and bear all possible relationships to each other. Some are updated versions of others. Some take pieces from several others and combine them in an updated form (note that this leaves me with the problem of getting back the pieces that weren’t included in the combined version, while using the most recent version of those which were). They were generated in three major passes, each of which had hopes (unfulfilled) of being a complete documentation of KRL as it was at that date. Other files have been generated at odd times when something was in particular need of documentation. In some cases several small but separate things have been included in a single file in order to save file shuffling. In others, a single large document has been broken into pieces to avoid the BRAVO 60K limitation. Because the files were created and transferred using many systems over a long period of time, I don’t trust the CREATED and WRITTEN dates (besides which I can’t see them either on hardcopy or when I am in BRAVO). Fortunately, almost all of the files follow the convention of incorporating date and time information into the BRAVO header, so I can use those (assuming, of course, that nobody every slipped and forgot to update that information manually before doing a PUT).

The first thing I did was have Diane print out hardcopies of the entire set of files, and to lay them all out on my table. It would be hopesless to browse through them or do large scale organization on line. First of all, only a fraction would fit on my ALTO disk at any one time, and second of all the overhead in going from file to file in BRAVO is far too high. I got the total page figure above by simply going through them one by one adding up the final page numbers. I also created a bravo file (KRL.FILES) with all of the file names, and am editing it to reflect the fate of each file as I work with it. This means bringing it in to BRAVO in a separate window whenever I want to make a note in it.

I have decided to put the reorganized stuff (plus the new text which I need to write to pull it all together) into a single IVY subdirectory, using a tree-structure for the document itself. This way I can avoid numbering sections and having to reorganize everything if I add or remove one. Unfortunately, if I use the standard subdirectory convention, I cannot retrieve things to my ALTO without losing all but the terminal node of the file name. In general, I will want to retrieve an entire subsection at one time to work on it (i.e. everything below a certain node in the tree). In order to make sure that all the files get to and from my ALTO, I either need a different naming convention or have to manually make sure the tree gets reconstructed properly after each session, sending and naming one file at a time. I am hoping to get around this by creating my own tree structure using a separator character (probably "-") which can be part of an ALTO file name. If that works, the only problem will be keeping the tree-node-names short enough so the entire path name doesn’t exceed the file-name-length limit on the ALTO (I believe it is 30 characters). Since I really like having understandable mnemonic names, this is bound to be an annoyance, but one with which I can live.

Yesterday I started through the files (the hardcopy -- so far, none of the actual text files are on my ALTO disk). 3 of them were simply slated for renaming and placed in the outline. Cross-references to them were put at other places in the outline, since they contain material which will eventually be spread to different places. Two files were so heterogeneous that I went through them (marking with ink on the hardcopy) marking sections as to their destinations. At some point I need to get all the appropriate files (sources and destinations) on my ALTO and do the necessary redistribution, but I think it will be better to batch that activity. So far, all of the actual text editing has been on my outline file and on KRL.FILES, and none of the renaming activity has been done (I’ll batch that too, since it involves file transfers, dealing with multiple servers, etc.). My hope is that if I am religious enough about recording things carefully in KRL.FILES, I can use that as a kind of shopping list to do the actual work later.

March 29, 1979 11:21 AM: In going through more files I realized that I wanted to add annotations to a directory (e.g. "this is Paul’s old KRL0-version of cryptarithmetic, modified to the new notation"). I can put a collection of such annotations in a file, but then there is the updating problem as the files change. It would be very nice to have annotation available linked to the actual directory. I found two files with no dates, one of which seemed to be a pure subset of the other. I managed to verify that this was almost true (only two minor changes) by using the old-fashioned method of source-compare---holding pages together up to a bright light and looking for differences. If there were an appropriately smart source compare (i.e. can deal with significant chunks missing, etc.) which was fast enough (and could get hold of the files in their various homes) it would have saved some effort. I also established the convention of keeping a small window at the bottom of the BRAVO screen to make notes to myself to remember things for this log.

April 3, 1979 12:49 PM: I’ve spent a couple of hours going through files successfully (reading hardcopy and making notes in the outline and file list, but not actually doing any file manipulations). A certain amount of time lost in fussing over names so that files deep in the tree like "structures-levels-memory-compaction-declarations" become acceptably short (in this case "str-lev-mem-comp-decl"). The problems will come later if I can’t remember whether "comp" stands for compiling, compaction, complete, computational, etc. etc. and don’t have sufficient context.

April 17, 1979 12:54 PM: In manipulating the file list, I have been using fonts and faces to indicate the status of the various files (e.g. smaller font for files which turn out to be empty or irrelevant, bold for those which need further processing, etc.) This makes it easy to scan down the list and get my bearings. Some sort of fancier property marking might be useful. Color displays would open up lots of possibilities. The output of all my work so far is the file [IVY]<winograd>document>file.shuffle, which contains commands for FTP, PUPFTP, etc. I plan to use it in performing all of the activities. The files being brought over to the alto are for splitting up into other files, according to the list I have compiled in [IVY]<winograd>document>krl.files. I am hoping they will all fit on the ALTO disk at once, so I can do all the text shuffling then transfer them all back to IVY.

1). My routine involves working with four windows -- one for the outline of the final document (which I update after each activity as I work, to maintain a consistent notion of where I am), one for the list of files I am working from (which is marked up to indicate just which pieces of which are going where) and two for the files themselves, (one the source and one the destination). I can only work for about 15 minutes (bringing in and writing out 3 to 5 files) before I get a "formatting space running low" message, and have to quit and restore the whole thing. I automated this by writing a do file which copies a piece of BRAVO transcript (one which is too long to be included as an INIT macro) into the bravo.ts file, then does a bravo/r. This means that the save-restore cycle doesn’t take much thinking, but occupies a couple of minutes.

2) (not really Bravo’s fault). I ran out of disk space in the middle of a PUT. I had hoped to be able to do all of the shuffling without deleting sources, but failed. This in itself wasn’t so bad, but in trying to gain disk space, I did a ’Delete *$’. This was fine, except for the one file which BRAVO was writing out when it ran out of space. The old version had been renamed OUTLINE$, so was deleted. The new version contained just as far as it has gotten in the file. Fortunately, by reading that in (using ↑Z since the end was not in bravo format), and getting a backup version (several days old) from IVY, I was able to almost reconstruct it in ten or fifteen minutes. Unfortunately it was my outline file, and I think I have now lost track of some file names which I entered into it. I will later do a complete sweep of all the file names on my ALTO disk, to see if there are any strays.

3) The inability to switch contexts is very annoying. I am in the middle of editing, with four windows going, three of them containing documents which have not yet been written out. I just realized that as a consequence of one of the file manipulations I just did, I need to change the name of a file currently on IVY. I want to get into FTP, do a simple RENAME and get out. In order to do this, I will have to write out each of my files, quit bravo, do the ftp command, get back into bravo, and restore the state. The alternative is to write myself a note (on paper, or in the form of a little command file, which I could write out if I had a spare window available -- Bravo only allows 4) and hope that I remember it later. Given my experience with my own memory reliability (and the speed with which papers on my desktop become lower sedimentary strata) I think I’ll do the former. It’s a good time to take a break for some coffee anyway (maybe that will help avoid things like the ’delete *$’ fiasco).

April 28, 1979 4:11 PM: Finished! The subdirectory [IVY]<krl>document> now contains 94 files containing 811 disk pages. I have created a large message for Diane, specifying a collection of file transfers and archiving steps. It is all somewhat clumsy, because the archive system cannot deal with IFS style file names. There are cases of files whose names would be identical if transferred with PUPFTP (e.g. <krl>specs>matcher.bravo and <krl>oldspecs>matcher.bravo). I traced them all down (using BRAVO on directories printed out by CHAT into transcript files) and am simply changing names so there will be no repeats. I had one more BRAVO out-of-space crash (it would be nice if it at least kept you informed of disk space, like FTP does!), but this time managed to recover and make some room without stepping on myself.

One of the problems with the files is that a number of them were renamed, and thus their internal bravo header is inconsistent with their file name. Part of what Diane will do includes bringing them one-by-one to her ALTO, changing the header, and putting them back. It seems that an automated "filename" variable in headers would prevent this (and other kinds of errors which often occur). The same is true for date. I manually change a date in the header every (well, almost every) time I write out a file, but it would be much more convenient for the header to simply use the real file-write date.

April 29, 1979 9:57 AM: Yesterday as I was out running, I thought of a problem that often comes up in trying to maintain files. I have two very different kinds of files (at least). One is a version of a piece of text. It has a structure which is fairly well set, and changes tend to be in the form of modifications and additions to existing pieces. The other kind is more chronological, with changes consisting mainly of the addition of entries to the end. I have various "notes-to-myself" files (probably a dozen or so on different topics), and often while I am in the middle of doing something else (usually in BRAVO) I want to append a short note to one such file. I also do this with bibliography entries I run across (adding them to one of several different bibliography files), files like this one, etc. The problem is that it is often too high an overhead to bring in the entire file (which often is on non-local storage), add to the end and write the whole thing out. What is needed is some kind of more message-like system where I can send a message to be appended a local or non-local file (and do the message sending without leaving the editor). As I got ready to write this note this morning, I realized I was faced with the exact problem I wanted to describe. I want to add this to my Editlog, but I am using my Textbook-disk, instead of my KRL-disk. I’m not even sure whether the version of the editlog on IVY is up to date, or whether the current version is the one on my KRL-disk. Rather than go through a sequence of disk load/unloads, I am making this a separate file, putting it on IVY, and sending myself a message (which I hope I will read when I happen to have the other disk loaded) to remember it.

Looking back at the file shuffling, I am very glad that I had as much redundancy as I did. There were two files describing what was there (one organized by source, and one organized by destination). These, along with the file names, and the file headers provided a variety of cross-checks. In fact, no one was consistent. There were mistakes in names, headers, and in both directory files (since all of the updating was manual, hence errorful), but there was no case (at least that I am aware of) where there wasn’t enough information around to patch things up. A better system would provide some combination of redundancy and automatic cross-validating, so the errors wouldn’t happen.