1 XEROX COMMON LISP DESIGN DOCUMENT PACKAGES 1 Packages 6 Ron Fischer, Bill van Melle 29-Mar-86 This is an implementation of the symbol package mechanism of Common Lisp, as described in chapter 11 of "Common Lisp: The Language" by Guy Steele (1984, Digital Press). The Interlisp-D type litatom is substantially the same as the Common Lisp type symbol, with one significant exception: symbols are partitioned into separate name spaces, called packages; a symbol has an extra attribute, its package cell, which is a pointer to the package in which the symbol resides. A package is essentially an object that maps from strings to symbols. Within a single package, a given string maps uniquely to a single symbol; however, the same string can represent different symbols in different packages. Packages can inherit symbols from one another so as to make easily accessible in one package context a wide collection of symbols, even though those symbols actually reside in other packages. To refer to symbols not accessible in the current package context, one must qualify the symbol with its package name. The symbols in a package can either be internal or external. External symbols are "public" and usually constitute the documented interface of some module; internal symbols are private and require a deliberate effort to access from outside the package. The primary purpose of packages is to prevent the inadvertant conflict of symbols used by separately developed programs. They have an additional advantage of making explicit the public interface of a module. 2 Issues 1 New datatype vs. extend Interlisp litatoms. One could imagine having symbol be a new datatype, and have the Common Lisp reader create symbols instead of litatoms. However, the requirement for free intercalling between Interlisp and Common Lisp effectively rules this out%Common Lisp programmers would have to have some way of creating Interlisp litatoms in order to call Interlisp functions, which would at best be awkward and confusing. So this is a non-issue: we extend Interlisp litatoms. Interlisp read/print syntax for symbols. The Interlisp printer and reader must be able to print symbols and read them back. As we extend litatoms to become symbols, the printer and reader must be extended in some fashion to print package prefixes where needed and read them back. An additional issue with Common Lisp symbols, unrelated to packages, is that symbols are permitted to have numeric print names. Therefore, the Interlisp printer must recognize numeric print names and correctly escape them, and the reader must correctly intern such escaped symbols. File package read/print environment. Ideally, one should be able to specify a package environment for a file so that symbols print nicely and still read back in. If random access to files is to work with files having an explicit package environment, functions such as LOADFNS need to know about it. A similar issue exists for DEdit: in what package environment should a function being edited be prettyprinted and user type-in be read? Initial package structure of the system. Not only are we implementing the package feature for Common Lisp, we are planning to make use of packages to resolve inherent conflicts between Interlisp and Common Lisp functions of the same name. That is, we want to make use of packages as part of the distinction between "Interlisp mode" and "Common Lisp mode". In addition, it makes sense to migrate our implementation's symbols into some sensible package structure; e.g., large programs such as Sketch and TEdit may want to have their own packages; internal system functions may want to become internal symbols in a system or "si" package. Which leads to another thorny issue... Compatibility with Interlisp's flat symbol name space. The question is, can Interlisp users continue to load and run old code unchanged? Consider, for example, a user text-editing application that loads TEdit and calls several of its functions. If TEdit lives in its own package, the user program cannot gain access to the TEdit functions it needs without using a package qualifier or importing the desired names. At first blush, this requirement would appear to prevent loadable modules (e.g., the Library) from making rational use of packages. This is discussed in greater detail under "compatibility issues". Litatom limitations. Currently, Interlisp symbols are limited in number to 64K, and they are not garbage collected. Common Lisp will introduce a large number of new symbols and possibly make heavier use of gensyms and other non-interned symbols. 2 Symbol Representation 1 To make the Interlisp LITATOM data type serve as the SYMBOL data type it requires a new "package cell", pointing at the package in which each symbol lives. This is conceptually a separate pointer field in each LITATOM "structure". However, Interlisp LITATOMs are not represented as contiguous structures, but rather as indices into a set of pre-allocated tables, one for each major slot (function cell, value cell, property list, pname). Rather then redesign the LITATOM structure (which is deeply wired), or pre-allocate another 2 segments (256KB) of the address space for a package cell table, we choose to assign a spare 8-bit field in one of the existing parallel tables as a "package cell index". The index is an indirection to a vector of 256 packages. Note that this limits the total number of packages to 256, including initial system packages and the null (uninterned symbol) package. It is believed that this is adequate for the initial implementation. A separate paper (to be written) discusses a redesign of symbol structures that would relieve both this limitation and the 64K limitation on total number of symbols. The accessfns LITATOM is extended with a field PACKAGEINDEX, which is BITS 8, and a field PACKAGE obtained by indexing into \PKG-INDEX-TO-PACKAGE, a global 256-entry array of package pointers. When a new package is created, it takes the next available index in the array. Packages are never destroyed, so indices are never re-used. There are 2 pre-allocated package numbers, that of the Interlisp package, and the null package (NIL, the package cell for uninterned symbols). 2 Package implementation 1 The basic package facility will be implemented by importing and testing the Spice Lisp package code. The major change required is that Interlisp strings are potentially full 16-bit strings, not 8-bit strings. This may affect hashing and matching algorithms. With Interlisp's multiprocessing environment, there is also an issue of making sure that critical sections in the package code are appropriately protected. Performance points 1 Interning symbols in a package system is inherently slower than Interlisp's MKATOM, because potentially many packages need to be searched for the desired symbol. The performance impact will be most significant on LOAD. Performance of the hash algorithms used is important. To the extent that Spice Lisp uses multiple values in performance-critical sections, it may be necessary to recode if an efficient multiple values implementation does not exist. 2 Package Hierarchy 1 Common Lisp demands at least the following well-known packages: LISP%the external symbols of this package are exactly all the (non-keyword) symbols in the language specification. USER%the default "current package" when the system starts, which inherits from LISP. Random user code is read into this package. KEYWORD%all keywords live here. SYSTEM%unspecified. The user establishes a sort of "symbol environment" by the setting of the current package (the special variable *PACKAGE*). Choosing a package that inherits LISP, such as USER, puts the user in an environment where only "pure" Common Lisp programs can be written (at least without typing package qualifiers). There are at least two other symbol environments we might wish to present: XCL%LISP plus all the public symbols of the "Xerox Lisp Environment", however that might be defined. This is the package one would expect to inherit in Xerox "Common Lisp mode". INTERLISP%the fully backward-compatible symbol environment, where old Interlisp programs can be run without change. Two other possibly different environments suggest themselves as well: XCLIL%XCL plus all the non-conflicting symbols of Interlisp. ILCOMMON%INTERLISP plus all the non-conflicting symbols of LISP. The former is for the Common Lisp user who wants convenient access to the Interlisp "language" as a source of additional functions. The latter is for the Interlisp user wanting to view Common Lisp as a source of a large set of new functions to play with (at least those with different names), or who is tiptoeing slowly into Common Lisp%thus, part of a migration path for existing Interlisp users. The XCLIL package seems of dubious utility. Common Lisp provides equivalent functionality to most of the "language" aspects of Interlisp. Furthermore, existing Common Lisp users presumably understand how to use packages; if they desire to use some parts of the Interlisp "language", they can either use package qualifiers or explicitly import the desired symbols. The ILCOMMON package differs from the INTERLISP package only in the importation of Common Lisp symbols not in conflict with Interlisp. We could choose to define the INTERLISP package itself this way. However, there is a good reason not to, the same reason packages exist: name conflicts. Interlisp users may well have defined functions having the same popular names as those chosen by the Common Lisp designers. Therefore, the Interlisp user ought not be placed by default in a package where Common Lisp definitions could be inadvertantly trashed. The proposed package structure is shown graphically as follows. Each box represents a package or collection of symbols. Arrows indicate that the "pointed to" package inherits the "pointed from" package. Because inheritance via Common Lisp use-package is only one-level deep, "X Inherits Y" is defined as "X imports all the external symbols of Y, then exports them." A dotted line indicates that only non-conflicting symbols are used. The bold boxes indicate the packages that a user-created package might choose to :USE; the other boxes denote a conceptual division of symbols not necessarily related to the actual package(s) in which the symbols reside. ((SKETCH "figure from {DSK2}PACKAGE-DESIGN.TEDIT;9" VERSION 3 SKETCHCONTEXT ((ROUND 1 BLACK) (GACHA 10 (MEDIUM REGULAR REGULAR)) (CENTER BASELINE) (CURVE 18.0 8) NIL NIL (CENTER CENTER) (NIL NIL) T NIL NIL 1.0)) ((.036 38.0 (PRI 0)) (TEXTBOX (24.0 128.0 76.0 36.0) ("(Common) " "" "Lisp") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((33.0 153.0 58 14) ( 62.0 139.0 0 14) (51.5 125.0 21 14)) BLACK (ROUND 3 BLACK) NIL (NIL NIL))) ((0.0 34.0 (PRI 0)) (WIRE ((56.0 . 128.0) (56.0 . 60.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((56.0 . 60.0) (53 . 70) (59 . 70))))) ((0.0 42.0 (PRI 0)) (WIRE ((72.0 . 128.0) ( 156.0 . 60.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) (2 4) 1.0 NIL (NIL ((156.0 . 60.0) (147 . 64) (151 . 68))))) ((.036 38.0 (PRI 0)) (TEXTBOX (128.0 152.0 76.0 36.0) ("Lisp " "Environment") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((155.5 170.0 21 14) (130.0 156.0 72 14)) BLACK (ROUND 1 BLACK) NIL (NIL NIL))) ((0.0 46.0 (PRI 0)) (WIRE ((156.0 . 152.0) (76.0 . 60.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((76.0 . 60.0) (80 . 69) (85 . 65))))) ((0.0 14.0 (PRI 0)) (WIRE ((180.0 . 152.0) (188.0 . 124.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((188.0 . 124.0) (182 . 132) (188 . 134))))) ((.036 38.0 (PRI 0)) (TEXTBOX (24.0 24.0 76.0 36.0) ("Xerox" "Common" "Lisp") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((45.5 49.0 33 14) (37.0 35.0 50 14) (51.5 21.0 21 14)) BLACK (ROUND 3 BLACK) NIL (NIL NIL))) ((.036 38.0 (PRI 0)) (TEXTBOX (128.0 24.0 76.0 36.0) ("Common Interlisp") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((117.0 35.0 98 14)) BLACK (ROUND 3 BLACK) NIL (NIL NIL))) ((.036 38.0 (PRI 0)) (TEXTBOX ( 232.0 152.0 76.0 36.0) ("Interlisp " "Language") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((247.5 170.0 45 14) ( 242.5 156.0 55 14)) BLACK (ROUND 1 BLACK) NIL (NIL NIL))) ((0.0 18.0 (PRI 0)) (WIRE ((264.0 . 152.0) (228.0 . 124.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((228.0 . 124.0) (234 . 132) (237 . 127))))) ((.036 38.0 (PRI 0)) (TEXTBOX (176.0 88.0 76.0 36.0) ( "Interlisp") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((191.5 99.0 45 14)) BLACK (ROUND 3 BLACK) NIL (NIL NIL))) ((0.0 14.0 (PRI 0)) (WIRE ((208.0 . 88.0) (180.0 . 60.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((180.0 . 60.0) (185 . 69) (189 . 65)))))) (15.0 19.0 308.0 174.0) 1.0 4.0 INTERLISP is the backward-compatible Interlisp-D package. Its symbols are divided up somehow between "language" and "environment" (break package, editors, window system, etc). The latter are exported to/from the Xerox Common Lisp (XCL) package. Note that there is a fair amount of duplication implied here. For example, ILCOMMON's external name table is a superset of INTERLISP's. This could be expensive if we decide that the INTERLISP package's external symbols are, for example, every symbol in the Koto sysout, as opposed to every symbol in the Interlisp Reference Manual. There is also an inheritance problem with keeping ILCOMMON up to date with additions to INTERLISP. Similarly, XCL is a superset of LISP. The set of symbols in LISP is smaller than the set of "every symbol in the Koto sysout", but size could still be an issue. An alternative approach is not to provide a single package analogous to LISP that others :USE, but rather provide a collection of packages from which new packages can use some subset. Thus, provide a package that exports the environment, say XAIE (but choose a better name), and assure that XAIE and LISP have no conflicts. Then if you want to define a package that is in "Xerox Common Lisp mode", then you would make the package use (via use-package) both LISP and XAIE. Similarly, to inherit "Interlisp mode", use both IL (the language) and XAIE. Finally, to inherit "Common Interlisp", use IL and XAIE, shadow the conflicting IL symbols, and use LISP. To hide some of that messiness from the user, and to provide some flexibility as to whether XAIE is a single package or several, we could extend use-package to know about the "pseudo-packages" XCL, INTERLISP and ILCOMMON. This gives the following structure, where again the heavy boxes denote actual packages and the light boxes denote the pseudo-packages, or "mode" in which a package using the connected packages would be running: ((SKETCH "figure from {DSK2}PACKAGE-DESIGN.TEDIT;9" VERSION 3 SKETCHCONTEXT ((ROUND 1 BLACK) (GACHA 10 (MEDIUM REGULAR REGULAR)) (CENTER BASELINE) (CURVE 18.0 8) NIL NIL (CENTER CENTER) (NIL NIL) T NIL NIL 1.0)) ((.036 38.0 (PRI 0)) (TEXTBOX (24.0 136.0 76.0 36.0) ("LISP") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((51.0 147.0 22 14)) BLACK (ROUND 3 BLACK) NIL ( NIL NIL))) ((.036 38.0 (PRI 0)) (TEXTBOX (128.0 36.0 76.0 36.0) ("Common Interlisp") 1.0 ( CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((117.0 47.0 98 14)) BLACK (ROUND 1 BLACK) NIL (NIL NIL))) ((.036 38.0 (PRI 0)) (TEXTBOX (24.0 36.0 76.0 36.0) ("Xerox" "Common" "Lisp") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((45.5 61.0 33 14) (37.0 47.0 50 14) (51.5 33.0 21 14)) BLACK (ROUND 1 BLACK) NIL (NIL NIL))) ((.036 38.0 (PRI 0)) (TEXTBOX (232.0 36.0 76.0 36.0) ("Interlisp") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR) ) ((247.5 47.0 45 14)) BLACK (ROUND 1 BLACK) NIL (NIL NIL))) ((.036 38.0 (PRI 0)) (TEXTBOX ( 232.0 136.0 76.0 36.0) ("IL") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) (( 265.5 147.0 9 14)) BLACK (ROUND 3 BLACK) NIL (NIL NIL))) ((0.0 40.0 (PRI 0)) (WIRE ((260.0 . 136.0) (180.0 . 72.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((180.0 . 72.0) (185 . 80) (189 . 76))))) ((.036 38.0 (PRI 0)) (TEXTBOX (128.0 136.0 76.0 36.0) ( "XAIE") 1.0 (CENTER CENTER) (MODERN 10 (MEDIUM REGULAR REGULAR)) ((152.5 147.0 27 14)) BLACK ( ROUND 3 BLACK) NIL (NIL NIL))) ((0.0 32.0 (PRI 0)) (WIRE ((184.0 . 136.0) (244.0 . 72.0)) ( ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((244.0 . 72.0) (235 . 77) (240 . 81 ))))) ((0.0 32.0 (PRI 0)) (WIRE ((164.0 . 136.0) (164.0 . 72.0)) (ROUND 1 BLACK) (NIL ( SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((164.0 . 72.0) (161 . 82) (167 . 82))))) ((0.0 36.0 ( PRI 0)) (WIRE ((148.0 . 136.0) (76.0 . 72.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((76.0 . 72.0) (81 . 81) (85 . 76))))) ((0.0 38.0 (PRI 0)) (WIRE ((72.0 . 136.0) (148.0 . 72.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) (2 4) 1.0 NIL (NIL ((148.0 . 72.0) (139 . 76) (143 . 80))))) ((0.0 32.0 (PRI 0)) (WIRE ((56.0 . 136.0) (56.0 . 72.0)) (ROUND 1 BLACK) (NIL (SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((56.0 . 72.0) (53 . 82) (59 . 82 ))))) ((0.0 32.0 (PRI 0)) (WIRE ((276.0 . 136.0) (276.0 . 72.0)) (ROUND 1 BLACK) (NIL ( SOLID 18.0 10.0)) NIL 1.0 NIL (NIL ((276.0 . 72.0) (273 . 82) (279 . 82)))))) (13.0 23.0 310.0 156.0) 1.0 4.0 Interlisp Name Space Partitioning 1 A first task is to determine which symbols in Interlisp are part of the environment, as opposed to the language, so that they are available for making up the XCL package (be it a real or pseudo package). A question remains as to what other packages should the Interlisp/Common Lisp implementation be divided into. Should DEdit have a package? The window system? Network code? We have the opportunity to broaden the "system internal" name space from its current flat state of "all system internal functions have names starting with \". In particular, if system internals are in the system (or si) package, then we can think about making some of the symbols of that package external as a semi-formal way of providing below-the-line entries that one might expect educated system hackers to use; e.g., SI:ALLOCBLOCK, or SI:GETBASE. 2 Compatability issues 1 Can Interlisp users continue to load and run old code unchanged? Can we even load up the system without it breaking? (See discussion under bootstrapping.) Interlisp files that don't specify a package environment are defaultly loaded in the INTERLISP package. What symbols are accessible in INTERLISP? The most obvious choices are (a) all symbols in the released Koto sysout or (b) all symbols in the Interlisp Reference Manual. The argument in favor of (a) is rationality%those symbols constitute Interlisp's documented interface%and compactness of the INTERLISP and ILCOMMON packages if we make those real, rather than pseudo, packages. The argument in favor of (b) is that many users have used undocumented system functions, and their code will now "gratuitously" break, even if those undocumented system functions have not changed a bit, other than to have their names moved into a different name space. Can Library modules make use of packages? For a concrete example, suppose that the TEdit module lives in the TEDIT package. It exports all the symbols in the TEdit manual, its public interface. In a Common Lisp program, one would then write a call, say, to TEDIT:OPENTEXTSTREAM, or if one made heavy use of TEdit functions, one would either use the TEDIT package (via use-package) or explicitly import the desired TEDIT symbols. However, for the Interlisp user who does not know about packages, or at least has not yet converted all her old programs to take packages into account, all these external TEDIT symbols are inaccessible. It's not possible for the INTERLISP package to know ahead of time that the user is going to load TEdit and wants access to those symbols. This suggests that any Library module wishing to make use of packages needs to, at least in the short term, perform some special operation to make its public symbols accessible by importing them into the INTERLISP package. However, the issue is more complicated than that. A problem arises in the order in which symbols are interned. Consider, for example, a user text-editing application module MYEDIT that loads TEdit and calls several of its functions. The way such applications are typically written, somewhere in MYEDITCOMS, often at the end, appears a command (FILES TEDIT). Unfortunately, when MYEDIT is loaded, by the time this command is executed, the system has already read all of the symbols used by MYEDIT functions, including the ones that are trying to be calls to TEdit's interface. When TEdit is finally loaded, its external symbols land in the TEDIT package and are different symbols from the ones used in the MYEDIT functions. Furthermore, if TEdit tries to import its external symbols into INTERLISP, it will get an import error because of the name clash. If it instead uses SHADOWING-IMPORT, it won't get an error, but MYEDIT will still have the wrong symbols in it. A possible solution is to formalize the "import symbols into the INTERLISP package" by providing the following function: (export-to-interlisp symbols) [Function] Same as export, except in addition it makes the symbols accessible in the INTERLISP package. Specifically, for each element sym of symbols, it checks to see if a symbol of the same name is present in the INTERLISP package. If not, it imports sym into the INTERLISP package. If, however, there is such a symbol already in INTERLISP, it uninterns sym, imports the INTERLISP symbol into sym's former package, and sets the symbol's home package to be sym's former package. There might want to be a third case, where the symbol found in the INTERLISP package has some other package as its home, in which case export-to-interlisp should probably leave it alone%there's already a name clash. In other words, if one of the symbols had already been created, but in the wrong package, it is moved to the right package. export-to-interlisp is one of the forms that is part of a module's "package environment definition" that must appear at the beginning of the module, before any of its functions (etc) are read. How to put a module into a package 1 The problem is to take a program from a flat name space and convert it into one that lives in its own package, revealing only its public interface. Using TEdit as an example: 1. Define the new package, inheriting from INTERLISP. For example, (in-package "TEDIT" :use "INTERLISP"). 2. Load the module's source file(s) in this package. All the symbols read from the file and now in memory are in the TEDIT package unless they were already defined in INTERLISP (i.e., calls to Interlisp functions, etc). 3. Decide what the external symbols will be. Produce a "file environment definition" (more on this under file package) containing at least something like (in-package "TEDIT" :use "INTERLISP") (export-to-interlisp . public-symbols) 4. Write the files out and compile them. If at some point you decide to migrate the module into Xerox Common Lisp, you can (while in the TEDIT package) say (unuse-package "INTERLISP") (use-package "XCL"), edit the environment definition, and dump the files again. Testing issues 1 Test that one can load an Interlisp file (e.g., a LispUsers application) and call functions in it without getting into package trouble. Especially test this for applications that load Library modules that have non-trivial package environments. 2 Interlisp reader & printer integration 1 Interlisp read tables are extended with a syntax class "package delimiter". The old Interlisp read tables, used to read old code, have no character with this class, and hence no package specifiers in what they read. The new Interlisp read tables, used to print Interlisp code in the new world, have some not too inconvenient character as the package delimiter (up-arrow and bang are possibilities). Note that Common Lisp does not have a "package delimiter" read syntax: the character colon is hard-wired as the package delimiter, and if you ever changed colon's read syntax from constituent to something else, you'd have a very hard time reading package-qualified symbols. However, there is nothing stopping us from introducing an Interlisp read syntax of "package delimiter" in order to provide a compatibility path for old users. Interlisp's internal read function, \SUBREAD is changed to call INTERN (or some internal form of it) with the appropriate package information when a character of this syntax is encountered. The litatom printer \LITPRIN is changed to print a package prefix, using the appropriate delimiter, when in "prin2" mode and the symbol is not accessible in *package*. \LITPRIN also checks for numeric pnames and escapes them appropriately. If the numeric check is too expensive to test on every print, we might consider caching in the symbol a couple bits saying whether the pname is/isnot/maybe numeric. The Common Lisp printer has essentially the same test to make, with one extra complication: the determination of "potential number" is dependent on the current input radix (*read-base*), so if the radix is 16, such "obviously" non-numeric symbols as DEFACE become numeric. This is not an issue for the Interlisp reader/printer, which does not believe in an input radix. But for the Common Lisp printer it means that a "numericp" flag in the symbol is not quite sufficient. A possibility is to use the flag to mean "numeric in radix 10" and do the full test on every print when *read-base* is greater than 10. 2 Bootstrapping 1 MAKEINIT knows a lot about the present LITATOM structure. If symbols change now so that they all live in separate package hash tables, then the global atom hash table goes away, and MAKEINIT has to be able to create packages and symbols in them. If we're willing to insist that all files that go into the init.sysout have symbols in only one package, or at least no symbols that clash, then a simpler bootstrap structure is possible: all symbols go in one hash table in a well-known location, and code that runs inside the init.sysout takes care of moving the symbols into the correct package structures. That non-clashing constraint is fairly strong. An alternative is to create clashing symbols as special gensyms that completely get renamed when running init.sysout. TeleRaid also has to know about packages. Dorado's Raid may be a lost cause, which effectively kills Raid (but note that Dorado TeleRaid does not currently work). If we change system code to live in a variety of packages, we run into the problem of forward reference. For example, we probably have many cases of code living in files loaded earlier than WINDOW that call window functions. It might be necessary to provide separate package definition files for such system packages, loaded early in the loadup. A related problem is code that conditionally calls a function if that function exists. For example, where today one might write (if (GETD 'OPENTEXTSTREAM) then (OPENTEXTSTREAM --)), one would have to write something like the following if TEdit entrypoints lived in their own package: (if (FIND-PACKAGE "TEDIT") then (FUNCALL (INTERN "OPENTEXTSTREAM" "TEDIT") --)) 2 The File package 1 The problem is that the contents of a file cannot be read without knowing what the "reader environment" of the file is, in particular, its read table and package environment. For files simply being loaded sequentially, the package environment, at least, is not an issue, as files are required to start with any expressions needed to correctly establish the file's package environment. However, for random access to files, such as with LOADFNS or the "copy the unchanged definitions" feature of MAKEFILE, it is necessary to have a means of easily determining the reader environment. Even if we decide that the file package simply does not work on Common Lisp files, we'd still have the read table problem: since the standard file read table must be extended with a package delimiting character, we must have a way of distinguishing which read table a file was made with. Proposal: every file starts with an expression defining the reader environment read table and package setup; the package expressions are the ones described in the manual for how to arrange your modules ("Put In Seven Extremely Random User Interface Commands"). This has to be before the FILECREATED expression, since you can't even read the "changes" part of that expression without knowing the reader environment. LOADFNS, MAKEFILE etc read this expression to decide what read table, package, etc to read or print with. To be more concrete, each file begins with an expression of the following form: (reader-info (provide --) (in-package --) (shadow --) (export --) (require --) (use-package --) (import --) (in-readtable --) ) The form need not contain every element shown. The new function in-readtable is defined below. The reader-info expression is read with package USER and the standard Common Lisp read table. Old-style Interlisp source files (those beginning directly with the FILECREATED expression), are treated as if they began with (reader-info (in-package "INTERLISP") (in-readtable "OLD-INTERLISP") ) Any other file is not considered LISPSOURCEFILEP and thus cannot be correctly read with LOADFNS or READFILE. Registering read tables 1 In order to retain sanity when dealing with files printed with different read tables, it is necessary to associate with each file the read table used to print it, so that it can be read correctly. The in-readtable expression in the reader-info form is intended to accomplish this: (in-readtable tablename) [Function] Sets the primary read table to be the one whose name is tablename. (name-readtable readtable tablename) [Function] Registers the read table readtable as having the name tablename. A read table is permitted to have more than one name. (find-readtable tablename) [Function] Returns the read table whose name is tablename. We expect that the following read tables will be registered in Xerox Common Lisp: OLD-FILERDTBL%the read table that was the value of FILERDTBL in Koto. All old Interlisp files are read with this table. ORIG%the "original" Interlisp read table. Keep this for the benefit of users who have been programming their own read tables, starting with (COPYREADTABLE 'ORIG). LISP%the standard Common Lisp read table. INTERLISP%the "new" Interlisp read table. This read table will be the value of the read tables T, NIL, FILERDTBL and CODERDTBL in the next release. It will be designed to be similar to Common Lisp's default read table. It differs from ORIG in the following ways: | multiple escape. # macro dispatch (formerly |). HPRINT either uses #{ or suitable Common Lisp syntax. tbd package delimiter. ^A thru ^F separators (for pspool font info). ' quote. ` and , backquote expressions. Users can register their own custom read tables with this mechanism and thus be assured of not confusing other readers of their files%either the reader will know about the read table specified in the reader-info expression or it will complain (rather than read the file incorrectly, as it would today). An open question is whether INTERLISP should be the same readtable as LISP. In the current proposal, INTERLISP differs from LISP in the following ways: % single escape (rather than \). tbd package delimiter (rather than :). ^A thru ^F separators (rather than alphabetic). And in addition, INTERLISP is case-sensitive. The case-sensitivity and the formatting characters are the principal reasons to have different read tables (the formatting characters could be punted if we bite off implementing the long-proposed LISTFILES that does its own prettyprinting, so that MAKEFILE need not pretty print). Reading the reader-info expression 1 There are some subtleties here. The require expressions in general imply the need to read at least the reader-info from the required file, even if we are not actually loading the original file. Might also want a way of indirecting the info to a separate file, so that groups of files with the same complex reader environment could specify it in one place rather than duplicate it on each file. Whoever handles the reader-info expression needs to do some smart caching. 2 DEdit 1 DEdit has to know in what package/read table environment to prettyprint and read user input. Proposal: package is the home package of the symbol being edited; read table is Common Lisp iff the package uses LISP. If it is not a symbol being edited, but rather some random expression, edit according to the environment in which DEdit was called.(LIST ((PAGE NIL (PAPERSIZE Letter FOLIOINFO (ARABIC)) (0 0 612 792) ((HEADING NIL (HEADINGTYPE FOOTINGR) (72 27 540 36) NIL) (TEXT NIL NIL (72 72 504 648) NIL))) (PAGE NIL (PAPERSIZE Letter FOLIOINFO (ARABIC)) (0 0 612 792) ((FOLIO NIL (PARALOOKS (QUAD RIGHT) CHARLOOKS (SUPERSCRIPT 0 INVISIBLE OFF SELECTPOINT OFF PROTECTED OFF SIZE 10 FAMILY MODERN OVERLINE OFF STRIKEOUT OFF UNDERLINE OFF EXPANSION REGULAR SLOPE REGULAR WEIGHT MEDIUM INVERTED OFF USERINFO NIL STYLE NIL) FORMATINFO (ARABIC)) (288 12 288 36) NIL) (HEADING NIL (HEADINGTYPE FOOTINGR) (72 27 540 36) NIL) (HEADING NIL (HEADINGTYPE RECTOHEAD) (72 762 540 36) NIL) (TEXT NIL NIL (72 72 504 648) NIL))) (PAGE NIL (PAPERSIZE Letter FOLIOINFO (ARABIC)) (0 0 612 792) ((FOLIO NIL (PARALOOKS (QUAD RIGHT) CHARLOOKS (SUPERSCRIPT 0 INVISIBLE OFF SELECTPOINT OFF PROTECTED OFF SIZE 10 FAMILY MODERN OVERLINE OFF STRIKEOUT OFF UNDERLINE OFF EXPANSION REGULAR SLOPE REGULAR WEIGHT MEDIUM INVERTED OFF USERINFO NIL STYLE NIL) FORMATINFO (ARABIC)) (288 12 288 36) NIL) (HEADING NIL (HEADINGTYPE FOOTINGR) (72 27 540 36) NIL) (HEADING NIL (HEADINGTYPE RECTOHEAD) (72 762 540 36) NIL) (TEXT NIL NIL (72 72 504 648) NIL)))))1(xx(xx (( .`` (dd ( /``T/T/ T)T)2T(`` )``T)`` TB PAGEHEADING RECTOHEADA PAGEHEADINGFOOTINGR TERMINALTERMINAL MODERNMODERNMODERN MODERN MODERNMODERNMODERN  HRULE.GETFNMODERN  "   HRULE.GETFNMODERN    HRULE.GETFNMODERN&  3 \ ' A     HRULE.GETFNMODERN   HRULE.GETFNMODERN*   ' # ' 54    HRULE.GETFNMODERN   HRULE.GETFNMODERN   "  " t             HRULE.GETFNMODERN   HRULE.GETFNMODERN    HRULE.GETFNMODERN   L t   HRULE.GETFNMODERN   HRULE.GETFNMODERN@   3  8   '  /   /   5   r    q  )   H  s   4   Q  ? n (    SKIO.GETFN.2MODERN     G   SKIO.GETFN.2MODERN    "  HRULE.GETFNMODERN    W     HRULE.GETFNMODERN   HRULE.GETFNMODERN j ) y | ) H < a  + &  M y  }  i e $ <  ? >  #  HRULE.GETFNMODERN   C &   & *  s  ;    HRULE.GETFNMODERN     HRULE.GETFNMODERN '  HRULE.GETFNMODERN  $     @ S BD    HRULE.GETFNMODERN   HRULE.GETFNMODERN'  g   \  f 8  HRULE.GETFNMODERN   HRULE.GETFNMODERNI  S      P   0  "! (      HRULE.GETFNMODERN      8       9   %   R  O x   *            o   W  /  !  W   !    " ( 1    +   #  HRULE.GETFNMODERN      HRULE.GETFNMODERN   HRULE.GETFNMODERNY @z