Date: Tue, 16 Apr 85 17:20 EST
From: Daniel L. Weinreb <DLW@TENEX.SCRC.Symbolics.COM>
Subject: Compilation and package system
To summarize this discussion, the desired solution that would satisfy
everyone would have the following properties:

(1) Package specifications should not use the -*- stuff at all, because
there's nothing about -*- in the Common Lisp manual.  Not only does this
make it easier to port programs, but it's necessary to allow the
building of portable programs that understand files of Common Lisp code
(e.g. cross referencers).  It's OK to have the -*- in ADDITION to the
portable forms that specify package setups, but it's purely optional
from the point of view of the Common Lisp evaluator and compiler.
(Possibly we could back off and have a yellow-pages package that
Symbolics provides and other implementations pick up.)

(2) The editor needs as much information about packages as the compiler
and the loader.  All programs that read Lisp forms from files must get
the package information from the same place, and work consistently.
Furthermore, it is out of the question for you to have to manually
evaluate forms out of the buffer before you can edit properly.

(3) It should not be required that complex package setups be in their own
file, since you want to be able to use packages flexibly even in
programs that fit in one file.  Use of packages shouldn't force you to
have to split your program into many files, with associated pain in
copying them, etc, etc.

(4) If Symbolics Common Lisp and its editor are to work correctly on
ported unmodified Common Lisp source files, it must be prepared to deal
with CMU files, which use -*- lines that refer to packages that aren't
created until that very file is evaluated.  Right now, the CLCP user
at CMU gets an error.  Strictly speaking, the program is valid Common
Lisp, because Common Lisp doesn't ever look inside comments, and so
the CLCP is technically in error.

As far as I can tell, the only way to meet the first three constraints
is to require the editor to parse through the entire file, finding all
of the package-related forms such as IMPORT, when the file is read in.
It's a bit harder than that, since those forms might be inside of
EVAL-WHEN forms.  Or COND forms.  Or there might be a function defined
in the file and later called at top-level that does an IMPORT.  Worse
yet, the "package environment" might be different at different parts of
the file, as various package-related forms are sprinkled among the
function definitions.

I don't think this is feasible.

Zetalisp has basically taken the stance that each file has exactly one
"package environment" that's the same throughout the file, and the
information that sets up this environment is essentially declarative,
rather than imperative.  These restrictions have not turned out to be a
problem, based on large volumes of sophisticated code turned out over
several years.  In exchange for them, we get point (2) above.  Perhaps
it is our disagreement over (2)'s importance that is really at the
bottom of this discussion.

Date: 17 Apr 85 17:20 PST
From: masinter.pa@Xerox.ARPA
Subject: "lexical" packages
To: Common-Lisp@SU-AI.ARPA

Rebuttals:

Fahlman:

"When we're speaking of the scope variables, lexical scoping means that
the scope of the variable is one contiguous chunk in the file being
read, and once you leave the lexical block you can never again get hold
of that variable by using its name (though function closures defined
within the block can still get at the variable).  You cannot sneak in
one more function later that refers to the same X.  That's awkward at
times, but also an advantage, since the compiler can be sure that after
it has reached the end of a block, it knows all about how a lexical
variable in that block can be referenced."

Of course this isn't true, lest we be reduced to using core dumps to
debug. Of course, the debugger and the environment can reasonably be
expected to provide a way to access "lexical" variables. But such
mechanisms are part of CLTE rather than CLTL. The language has semantics
with invariants (if you (setq x --) here and call a function, you can
still expect x to have the same value), and the environment gives you
tools to break those invariants. Thus, while I propose making the
"package" a lexical property (more on that later), any reasonable
environment would provide ways of dynamically switching.

Moon:
"T has some reasonable responses to these issues, and I think if Common
Lisp were being designed in 1987 we might decide differently after
studying languages such as T and how their users use them.  But Common
Lisp was designed in 1982.  We decided to go with the well-understood
package philosophy that the Lisp machine had been using since about
1978, substantially redesigned to fix its well-understood bugs without
radically changing the whole philosophy.  It's hard to achieve
perfection in this mortal vale.  For additional insight, see the James
Madison quotation on the second page of the manual."

This is one of the weakest arguments to make -- it can be applied to any
of the discussions on this or any of the other mailing lists about
Common Lisp. If there's no chance to remove some of the mistakes in
Common Lisp, there seems to be no grounds for further discussion.

- - - - - - -

Restatement of proposal:

Packages should rightfully be lexical (determined by character sequences
in the file) rather than a dynamic (determined by what you've recently
executed). 

Let me make a specific proposal:

Just as (DECLARE --) is a special form, and "it is an error" to attempt
to evaluate a declaration, I propose that 

provide, in-package, shadow, export, require, use-package, import

all be documented as as special forms; they may only occur at the
beginnings of files. They are special markers that load, the compiler,
and the editor etc. can look for. (The wording for this can be taken
almost directly from the documentation of "declare" in CLTL p 153).

This turns what was formally a convention into a requirement. It means
that editors/file packages/etc. can actually look at a file and
determine (by scanning the beginning of the file) what packages might be
used/exported/etc. by the file. It makes illegal most of the odd (bad
programming practice) things that can go wrong with the package system.
It removes the requirement that the whole file be read, parsed and
executed in order to get a particular function. Many of the caveats in
the package section could be removed.

For those who already have an implementation of the current package
mechanism, the burden of this new scheme is minimal -- they can add
error checking, but need not.


Date: Wed, 17 Apr 1985  20:51 EST
Message-ID: <FAHLMAN.12103995276.BABYL@CMU-CS-C.ARPA>
Sender: FAHLMAN@CMU-CS-C.ARPA
From: "Scott E. Fahlman" <Fahlman@CMU-CS-C.ARPA>
To:   masinter.pa@XEROX.ARPA
Cc:   Common-Lisp@SU-AI.ARPA
Subject: "lexical" packages
In-reply-to: Msg of 17 Apr 1985  20:20-EST from masinter.pa at Xerox.ARPA


Larry,

OK, now I think I understand what you're saying, as far as it goes.
This is very similar to what the Symbolics people were just arguing for:
that all of the package stuff happens at the start of a file, and the
"scope" within which a package is the default for the reader is one
file.

I'm basically in sympathy with the idea that you don't change the
package environment in the middle of a file.  Right now, I can't think
of any situation in which I'd really want to change packages within a
file.  So far, so good.

I think that I would often want to load N files, some of which go into
package A and some of which go into package B in any order.  There's no
problem with this as long as each file going into a certain package has
identical (or maybe just compatible) package stuff at the start.  To me,
this isn't exactly lexical, as I understand the term, but there's no
point in quibbling about terminology.

What I don't understand about your proposal is how one deals with
input to the reader that is not in the form of a file, with its natural
lexical boundaries.  How do we handle terminal input, input from streams
whose source is unknown to the Lisp, and so on, if not dynamically?

If the essence of your proposal is that we handle packages more as less
as we do now, except that IN FILES the package forms must all occur at
the start and the system can take their scope to be the whole file,
that's worth some careful study.  It certainly is good style in all the
situations I can think of, so maybe enforcing it wouldn't be so bad.
However, this sort of messes up the abstraction that Lisp just sucks
streams into the reader and that the reader doesn't care where the stuff
is coming from.

-- Scott


Date: Thu, 18 Apr 85 13:10 EST
From: Daniel L. Weinreb <DLW@TENEX.SCRC.Symbolics.COM>
Subject: Compilation and package system, an addendum
To: TIM@MC.MIT
cc: common-lisp@SU-AI.ARPA
In-Reply-To: <[MIT-MC].459123.850418.TIM>
Message-ID: <850418131043.2.DLW@CHICOPEE.SCRC.Symbolics.COM>

    Date: Thu,18 Apr 85 01:22:26 EST
    From: Tim McNerney <TIM@MIT-MC>

    Yes, there could be an arbitrary amount of hair, but there is a big
    difference between assuming that an entire file will be read in the
    package declared by the -*- line and supporting lusers who want to
    generate the screw cases you allude to above.  Once the package
    environment is established, simply having the ZMACS keep a package
    attribute for each section will support editing files like patch files
    which need to be read in a number of different packages.

You seem to be saying that the high cost can be avoided by not
supporting "lusers" who generate the "screw cases".  In other words, the
editor should not actually parse the entire file, because that costs too
much, and it's OK if it does the wrong thing for the screw cases.  I
could be convinced of that principle.  So, what is your counterproposal?
Exactly what will the editor do when a file is read in, such that it
does all the right things except in "screw cases"?  How does it know
where to stop parsing?

And if you expect the editor to work correctly on files like patch
files, with different package environments in each section, then you
certainly do need to scan the entire file.  If you disagree, what's
your countersuggestion?


Date: 18 Apr 1985 1235-PST
From: Rem@IMSSS
Subject: What is a "file"? Lexical package usage?
To:   COMMON-LISP%SU-AI@SCORE

A file signals its end by an end-of-file marker of some kind (I'd have to
look in the CL manual to remind myself what it is in CL). The beginning of
a file is indicated by being right after you have done some kind of OPEN,
and is lexical extent is from that point unti the end-of-file marker occurs.
Perhaps if lexical packages are implemented, other streams can issue package
and other start-of-file stuff right after they are "opened" whatever that
means for the stream, and can signal the end of a lexical segment by an
end-of-file marker? For terminal-input and other infinite streams, the
interpretor can immediately re-open them after noting the lexical break
caused by end-of-file, flushing any info it had about the package state
from the just-now-closed segment and preparing to accept new package
declarations from the new segment to arrive next on the re-opened stream.
Would that solve the problem with "IN A FILE" being a special case?
-------


Date: Sat, 20 Apr 85 00:30 EST
From: Daniel L. Weinreb <DLW@TENEX.SCRC.Symbolics.COM>
Subject: "lexical" packages
To: masinter.pa@XEROX.ARPA, Common-Lisp@SU-AI.ARPA
In-Reply-To: The message of 17 Apr 85 20:20-EST from masinter.pa@XEROX.ARPA
Message-ID: <850420003003.1.DLW@CHICOPEE.SCRC.Symbolics.COM>

If you recall my four-point message (of Tue, 16 Apr 85 17:20 EST,
replying to Skef), your proposal indeed meets all four points.  It also
solves the editor-scanning problem, since as soon as the editor sees any
normal forms it can stop scanning.  It also solves all those other
problems I mentioned about evaluating those things inside various forms
or functions, since they can no longer appear there.

Only one problem comes to mind, and it's not hard to solve.  Sometimes
you have a system made up of many files that are all in the same
package, and all have the same package environement.  (In fact, almost
all of the software that we work on here is big enough that we handle it
this way.)  It would be inconvenient and error-prone to have to
duplicate the package setup information in each of the files, since it
needs to be modified from time to time.  There needs to be some kind of
indirection feature that lets you have one centralized file holding the
package setup.  The editor would have to read this centralized file when
the original file is read in.

This leads to problems inherent in storing absolute pathnames inside of
files.  On the Lisp Machine, we would store logical pathnames, so that
if the whole group of files were installed at some other site, the
embedded pathname would still work.  This is outside the realm of CLTL
as currently defined, but for us it should suffice.  There may be other
problems with embedded pathnames that aren't coming to mind, however.
I'd like to hear other comments.


JonL's comments

Just to review, do we all agree that 
  (1) readtables, 
  (2) packages, 
  (3) dynamic variables *read-base* and *read-suppress* etc
all form part of a global database affecting the reader syntax, and that it
is a unique featue of Lisp that this database may be modified dynamically
by the user?  Perhaps a fourth point ought to be added:
  (4) packages do not solve, nor even address, the problem of lexicality.

Too often in the past several years I've seen persons who "come up" with
a solution to the problem of lexical variables by "putting them in
separate packages".  It takes a lot of explaining to separate out the
difference between syntax parsing and program semantics; perhaps it is
because Lisp is so lividly dynamic that this confusion is so easy to make.
On the other hand, the spate of conjectures as to what "lexical packages" 
might possibly mean leads to a situation, a variant of (4), which could be 
stated as:
  (5) lexicality does not solve, nor even address, the problem of packages.

Given this background, and the fact that the "lexical packages" discussion 
seems to have been started by a misunderstanding of "lexical" or "packages" 
or both, then I think we can best spend our broadcasted mail time focusing
on questions like the following:
  (A) What else besides the default setting of *package* should be added
      to the set of conventions?  is the ZetaLisp "Syntax:" command adequate?
      Of course there is *read-base*, but what else?
  (B) Must the editor settle all parsing/syntax related settings before
      even "loading" in the file ?  or can it wait until a READ operation
      is applied to some part of the buffer?  a "lazy" parsing.
  (C) What are reasonable conventions for scoping a database configuration?
      Is a file the appropriate scope?   What about streams that arent files?

Regarding (B), one may well ask why a line like
    (IN-PACKAGE "FOO")
is any more mysterious to a default-context parser than the line
;;; Package: FOO
Dill seemed to be asking this question in his note last week; one needn't
think that the former line has to be subjected to READ any more than the
latter one.  But the more lispy looking line *can* be interpreted by
the simple-minded read-eval-print loaded.

Weinreb seemed to summarize the differences between th ZetaLisp approach
and the SpiceLisp approach as the amount of importance attached to the
editor context problem.  I hope it will never be the case, such as it
is now in Interlisp, that one can create files which the standard system
tools cannot parse well enough to be able to edit (and, hence to fix
the silly little bug that managed to creep into them).  Admittedly it
doesn't happen often, but on those ocasions one must abandon the standard
program/file editor (in Interlisp) and drop down to some kind of text
editor to repair the damage.  [It is not, repeat not, such a simple 
thing to do to an Interlisp program file, for generally lots of low-level
things you never really wanted to know about will break when you "hand-
edit" a file, and you'll spend half an hour to half a day trying to
figure out random messages about "file maps" and so on.]