Common Lisp Issues version: March 10, 1986 These notes are derived from notes of the meeting on 3/4/86 and 3/6/86. This is filed on {ERIS}CML>Doc>IssuesOutline.tedit & will be revised as needed. 0. Global Issues Slow/Buggy/Nonportable Spice code We recognize that the Spice code is in places inefficient, buggy, and possibly not portable. We have to live with this. We had no general words of wisdom, except: "don't trust that Spice implements what the book says. Read the book as the last resort." Some proposed actions include: acquire additional copies of Common Lisp sources/implementations to check ours against. Possibilities include: working version of Lucid Common Lisp, HP Common Lisp, Sun Common Lisp. Action assigned to: ??? CommonLoops Using CommonLoops is percieved both as a possible boon (uniform implementation of defstruct, an additional way of getting more structure types) and a risk (Common Loops is buggy, incomplete.) Gregor was reluctant to have us use Common Loops, as he believed it would slow us down. Our initial implementation will not use Common Loops, although there may be specific pieces we will want to import (e.g., the ability to have more than 256 DEFSTRUCT-types) and some code for SUBTYPEP. Strategy for decoupling implementation parts, e..g., lexicality, packages Our original plan outlined a couple of key enabling pieces and the rest as a reasonably straightforward port of Spice Lisp. Our belief is that of the initial requirements (namely, lexical scoping, multiple values, packages) that only the multiple value code was truely necessary to begin porting Spice Lisp code. We described the current "hack" in the file CMLREAD where some symbols being read in are translated using the list CMLTRANSLATIONS to an equivalent CL: name. Once packages are fully implemented, we will have to go back into all of our CML sources and retranslate them back into the new package structure. However, work on other sections can begin. (The sooner we have packages implemented, the less we will have to reconvert.) (Action: Fischer/van Melle to report on current status & prognosis for package implementation.) It is our current belief that there are few if any places that depend upon lexical scoping and lexical closures for them to work correctly. There is a risk this isn't true. Implementors are cautioned to examine the code they are converting carefully for such dependencies. (Action: entire group). It is our current belief that there are few if any other dependencies in the implementation of Spice Lisp. There is a larger risk that this isn't true,. Implementors are asked to be aware of these dependencies and broadcast their findings, so that we can deal with the problems as they arise. (Action: entire group) Deep bound variable performance, e.g., looking up *package*, *ibase* One of the major differences between our architecture and spice lisp is that our variable lookup for dynamic variables uses deep binding rather than shallow binding. The ramification is a significant difference in performance profile. There are some pervasive dynamic variables in Common Lisp, which are frequently accessed: *package* and *ibase* come to mind, but there are many others. We >hope< that these turn out not to significantly impact performance. If they do, we can possibly resort to a more complex mechanism of per-process shallow binding for those variables, or a hybred deep/shallow binding mechanism. (several have been proposed.) This is an area for research & development. (Action: ???). Status of CML:CML needs to be exmained and its contents examined for completeness. Parts will certainly have to be reimplemented. Additional resources required: We need someone to keep/maintain "internal" documentation. We need a small number of additional 1186's for testing & borrowing for working at home on weekends. (Perhaps trading 1108s for 1186's would allow people to bring their machine home for weekends?) Additional toolbuilding: the tools for importing Spice code need to be updated & maintained. Organization structure: It was felt that smaller subgroups of at least two people should be assigned to the tasks. An informalized process of design review was suggested, wherein the task groups would present their designs for their part of the system in the tech meetings. Discussion would follow, with revisions and implementation afterwards. Code review of one's code by another of the team members was suggested. (Gregor suggested that the Spice sources be broken into major functional areas--the number mentioned was about "60" pieces.) ACTIONS: Bill and Ron will document and try to make useable the Spice transport tool; Larry will fix CL:LAMBDA to strip comments (DONE), Larry will remove the CMLPretty code from CML (done). A new sysout will be created which is the latest FULL.SYSOUT plus the CML files. Strategy for future releases. We presume we are not only doing a first release, but also laying the groundwork for subsequent releases. Some items can be done "first version", but we need a clear story on when the second version is going to come out, and what shortfalls of the first version we can tolerate. (E.g., performance problems, etc.) None of the current planning documents extend beyond the first version release. Benchmarks, testing: After the meeting, a group of us discussed the issues of benchmarks & testing. We separated out testing along the following dimensions: System vs unit test: System tests test the functioning of the entire system. A typical Common Lisp system test would be to test the proper functioning of someones complete Common Lisp program. A unit test tests an individual function. A unit test might be: does PROGN pass back multiple values both compiled and interpreted? Performance vs functionality: Our product has a definite performance goal. We need to know how close we reach that goal. The system performance goal is that it run our customers Common Lisp applications with reasonable performance in the first release, and better performance in the second. The unit performance goals include doing well on the Gabriel performance benchmarks, as well as a few others. Acquired vs generated We believe that a significant percentage of tests can be acquired from other sources, where the main job of the test group is to organize the test & place them in a framework where the development group can get at them, run them on incrementally generated systems. Sources include: a) our customers b) ARs on Common Lisp (from the development group) c) other implementors of Common Lisp (see Guy Steele's list of CML implementations). (To get these, we will probably have to "trade"). d) ISI (contact Balzer@ISI) d) current Interlisp diagnostics (MACROTEST, COMPTEST) converted to Common Lisp. 3. Lexical Scoping Lexical variable ref Non-local control transfer (must invalidate on exit of dynamic context) Discussion of these issues was deferred; Bill van Melle is preparing a memo. (Action: van Melle, distribute memo). 4. Type System There is, in CML, a preliminary implementation of the type system. At this point, we're less sure of its stability, completeness. For example, we believe that SUBTYPEP, a necessary primitive, is not implemented, and incorrect. There are a couple of areas of the type system which are performance sensitive -- typep, type-of, coerce. (Action: Andy Daniels will report within a week on the current status of the type system.) In the long-term, the DECL package of Interlisp-D, which is a precursor of the Common Lisp type system, needs to be withdrawn and a CommonLisp compatible equivalent put in its place. As discussed previously, the type system is one area where some judicious borrowing from CommonLoops can give us some leverage later. 5. Program StructureFunctions CML has a "hack" implementation of Common Lisp argument list function call features, (&REST, &OPTIONAL) resting on Interlisp's lambda-spread functions. This implementation seems to meet the functional requirements of Common Lisp: (there is some risk that not all features are implemented. The debugger (BT) and the break package do not perform correctly given Common Lisp lambda lists. The implementation rests on LAMBDATRAN, which is not fully integrated into the rest of the system. The performance of &REST and &OPTIONAL and &KEY words is very poor. This is a performance-critical part of Common Lisp. We agreed to live with the current performance, although the integration of the lambda lists into the rest of the system will have to be carried further. (Action: integrate LAMBDATRAN with functions it advises. Fix ARGLIST, ?=, SMARTARGLIST etc to know about Common Lisp style argument descriptors. Design uncertain here. Common Loops contains some functions for dealing with these argument lists, but currently they are undocumented.) The fact that &REST arguments require CONSing is likely to be the most serious implication; we might be able to do some compiler optimizations to detect the most stylized uses of these. The current implementation does *not* check for argument number mismatch. This is allowed in the Common Lisp specification, although almost all implementations *do* in fact check for argument number mismatch. (The book says "it is an error."). We will live with this too until there is a microcoded implementation of Common Lisp function call. (Action: design for new style function call to be done by ???? Bane?/Masinter?/vanMelle?) A major hole in our implementation is the lack of anonymous functions & closure function objects. Action: This will be addressed in the design memo by Bill van Melle. 6. Predicates We are uncertain as to the completeness of the CML implementation of the predicate chapter. Action: Bane is to load CML and see, generating ARs & test cases where appropriate. 7. Control Structure This chapter of CLtL contains a number of unrelated topics. The parts having to do with special forms are discussed in the section "Interpreter" below. setf: CML contains an implementation of SETF imported from Spice Lisp. (In a message subsequent to the meeting, Kelly Roach reported having independently ported SETF. ) The CML and Roach versions of SETF need to be examined for completeness and functionality, and some test cases generated. A number of functions which should be SETF-able are not, and this is fairly serious. The current implementation is flakey, and doesn't generate good quality code. Gregor says "you need a codewalker to implement SETF", even though the current version of SETF doesn't use a code walker. call-argument-limit: we little belief that the current implementation deals reasonably with the number of arguments asserted by call-argument-limit. multiple values: the multiple value implementation is documented elsewhere. There are some performance issues to work out. Many functions in CML which *should* return multiple values do not. progv: we had some concern that might be difficult to implement. There is currently no design, although Gregor asserted that progv was not performance critical (& described how kyoto Common Lisp had gotten it wrong). return-from, block namescompiler & interpreter work are required for these, although they are not believed to be "hard". iteration construct? (for, do, loop). Common Lisp does not define any iteration constructs. We agreed that in our implementation we would use Interlisp's "for" iteration construct, although there is some question about whether any problems will arise. Integration of unwind-protect & RESETLST. The discussion of these issues was postponed to a discussion of the Error system and Andy Daniels error system design. Non-local go: see {ERIS}CML>Doc>ControlDesign.tedit. 8. Macros [I know we discussed this, viz: should we get rid of CLISPARRAY & not cache macros?] Not fully implemented, compiler doesn't compile macros environment args Must have macro def for everything in book labeled "macro" Must they be compiled? (performance issue) Live in function definition cell? (Can't have both function and macro) 9. Declarations Doing more with declarations requires compiler work. The work on handling floating point declarations is indicative of the difficulty. One of the more important declarations to deal with is "inline", since various other packages want to declare things inline (e.g., generated functions from defstruct.) 10. Symbols 11. Packages We implement Symbols as Interlisp Litatoms, with an addition of an 8-bit "package cell" which is an indirect to the package of a symbol. The Spice code which implements the package structure has been ported by Ron Fisher, and integration is underway. Implementing packages requires modifications to Read & print.Reclaimable? Gensyms Initial hierarchy? Environment issues Modules 12. Numbers Integers (done; bignum performance?) Floatp (need more subtypes? probably not) Unportable fns (decode-float, etc) Rationals Complex *random-state* (= RANDSTATE? Must be printable) Performance issues Implementation source? (not all available in Spice) 13. Characters Donedocument (char-font-limit = char-bits-limit = 1) 14. Sequences Open compilation 15. Lists (Same datatype) 16. Hash tables mostly done? IL vs CL slightly incompatible Good sxhash 17. Arrays There is a full (?) implementation of Common Lisp arrays in the file CMLARRAY. It is in widespread use. Common Lisp arrays subsume BITMAPs and STRINGs and ARRAYs of Interlisp. There are microcoded operations designed and being implemented. Design documents are to be written and distributed. Open issues include: is the performance of the bit vector functions adequate? How far do we use CMLarrays in rest of system? Can we change bitmaps in the system instead to use the CML functions, to allow other common lisp array types to know about bitblt? (The design permits this, and might lead to some important leverage.) 18. Strings The string functions need to work on extended Common Lisp strings (using CMLARRAY facilities). 19. Structures Structures (e.g. defstruct) currently done as datatypes, but there hard limit of 255 datatypes. DEFSTRUCT currently generates a record declaration. There are some alternatives dealing with defstruct First, we need to get rid of the "record package" calls. The record package is an unnecessary level of indirection. Second, we need to increase the number of available datatypes. Currently, our plans are to live with the limit. However, we will likely need to move to CommonLoops format for more datatypes. Doing "vector types" seems like a week substitute. 20. The Evaluator Special formsinteraction with interpreter. One of the primary issues in the Common Lisp design is the handling of the interpreter and special forms. We discussed three possibilities, but reached no consensus: (a) There are two separate interpreters, (e.g., IL:EVAL and CL:EVAL). IL:EVAL remains unchanged. SYSTEM:EVAL differs in that it is relative to a lexical environment. LAMBDA implies using IL:EVAL, while CL:LAMBDA implies using SYSTEM:EVAL. Gregor reported that originally, Symbolics had two evaluators, discovered that it was a mess, and converted to a single evaluators. In this scenario, we wouls port the Spice Lisp interpreter, presuming its mechanism for handling special forms and lexical environments would handle our needs. In this scenario, we can retain a single "version" of symbols that are special forms in Common Lisp, because the Common Lisp interpreter need not use the NLAMBDA definition of the Interlisp special forms. (b) change the Interlisp interpreter so that it could interpret relative to a lexical environment. In this scenario, we might have to have separate IL:LET and CL:LET to account for the fact that CL:LET needs to establish a lexical environment rather than a dynamic one. We will have to examine existing code to determine how important this is. (c) run all (or most) code compiled. This simplifies the interface between compiled an interpreted code, and gives us a single set of code to maintain. This is obscured by various issues having to do with compile time caching, how the debugger would work, etc. Some differences betwen Common Lisp's EVAL and Interlisp's have to do with the handling of errors (IL:EVAL goes thru FAULTEVAL & DWIM, while DWIM is likely to be confused about Common Lisp constructs), the handling of evalhook (the Interlisp feature and the Common Lisp one are incompatible), and on whether EVAL of "random" datatypes is an error. Handling of special forms Multiplex by package (IL:LET vs CL:LET) or by kind of interpreter (IL:LAMBDA vs CL:LAMBDA) Incompatible names: block, if, (setq) Incompatible semantics because of lexicality: declare, function, let, let*, (prog), setq, the (why is it special?) Question: (and, or, cond, prog1), go, progn, quote Not in Interlisp: catch, compiler-let, eval-when, flet, labels, macrolet, multiple-value-call, multiple-value-prog1, progv, return-from, tagbody, throw, unwind-protect Unbound global variablesrepresentation 21. Streams Same, but need to extend (new types) Multiple byte sizes required? Need multiple streams per file 22. Input/Output Read syntax Extend IL:READ vs. completely new CL:READ (packages, symbol escapes, case sensitivity) Read tables (different; allow IL:READ to take cl read table?) Implementation of packages requires modifications to read & print. 23. File System Interface Pathnames 24. Errors Andy is preparing a description of the error system for the next meeting. 25. Miscellaneous Features Compiler Package environment Extend current vs Write new #, Lexicality Multiple values (e.g., floor) Declarations, type inference Microcode Function call Multiple value return (might want to at least optimize returning to single-value receiver) New ops AREF EQL, EQUAL Environment DEdit Package environment (infer from symbol's pkg? From lambda type?) File Package Package environment. The file package needs to be redesigned and modified to work with "packages." Read tables Masterscope On-line manual BREAK on lambda*, common lisp functions. 4H0H( 4N*<4N*< TIMESROMANGACHA ?1(DEFAULTFONT 1 (GACHA 10) (GACHA 8) (TERMINAL 8)) TIMESROMAN TIMESROMAN TIMESROMAN TIMESROMAN �#� �KF.A3 �v =�_"��HB v7� EIu6]�� =m�vXo� w@V8<,I�� F &.% 5.&P� `��] �z 1 ) & .EK \ DO +G�z�