Number: 416

Date: 31-Mar-84 14':28':38

Submitter: Sannella.PA

Source: Jellinek.pa, LispCore^

Subject: Want multiple streams to the same file

Assigned To: 

Attn: Lisp

Status: Open

In/By: 

Problem Type: Design - Impl

Impact: Serious

Difficulty: Hard

Frequency: Everytime

Priority: Absolutely

System: Operating System

Subsystem: Generic File Operations

Machine: 

Disk: 

Lisp Version: 

Source Files: 

Microcode Version: 

Memory Size: 

File Server: 

Server Software Version: 

Disposition: [not gonna make it into Harmony]

Description: '
From': Jellinek.pa'
Subject': Lisp': More than one stream for same file'
To': LispSupport.pa'
'
This is probably the n-hundredth (n >= 1) AR on this subject, but oh! how I wish I could have more than one stream open on the same file, for reading, writing, whatever.   It is almost unbelievable that it''s not possible to do so.  Case in point':  I needed both to print and to compile a file of mine.  I started a TCOMPL, realized that I also wanted hardcopy, hit ctrl-B, and ran LISTFILES.  I then typed OK.  Result': hardcopy of the tail of the file, and no compile - LISTFILES had closed the single stream to the file.  Blecch!'
'
	Herb'
'
-----'
'
Date': 18 APR 84 23':35 PST'
From': MASINTER.PA'
Subject': multiple streams per file'
To':   lispcore^'
'
I think one of the kinds of problems that allowing multiple streams per file would cause is that, in some cases, the user WANTS to use the old stream. One way of getting around this problem is as follows':'
'
Lets define a NEW function,'
'
(NEW.STREAM file mode options)'
'
(I think modelling this after the common lisp function; don''t have my common lisp manual handy).'
'
 We will leave OPENSTREAM and OPENFILE with the following semantics':'
if the file named in their argument is already open ONCE, then they  return their old argument. '
If it is  open more than once, cause an error.'
'
I think we should enforce the standard operating system constraints for files tht we have local control of': can have multiple readers with no writer, or only one writer, unless OPTIONS are specified for all streams to allow multiple readers and writers. This is so that if you write a program to read a file, it doesn''t change out from under you.'
'
We then have to go thru the system looking at all of the INFILEs and OUTFILEs and OPENFILEs and OPENSTREAMs and decide if they want a new one or an old one.'
'
LOAD wants a new stream.'
'
A lot of the output guys want old streams if there is one.'
'
-----'
'
Date': 23 Apr 84 13':41 PST'
From': JonL.pa'
Subject': Re': multiple streams per file'
In-reply-to': MASINTER.PA''s message of 18 APR 84 23':35 PST'
To': MASINTER.PA'
cc': lispcore^.PA'
'
The general MacLisp world always gives a new incarnation of the stream when a file is "opened"; there is no system facility to cache the correlation between fully-qualified file names and streams.'
'
I''d prefer that OPENSTREAM have the MacLisp semantics -- always generate a new stream, paying attention to the parameters list for options like multiple readers/writers.  Since OPENSTREAM is relatively new, and not in Interlisp-10, there is a good chance of being able to insure that each system caller can tolerate the proposed semantics.'
'
Since OPENFILE has had the Interlisp semantics associated with it -- i.e., new stream if not already open, otherwise re-initialize the canonically-cached stream, then I suggest leaving it alone.  Note that there may be multiple streams on the same file, but at most one will be in the canonical cache (\OPENFILES).  As a last resort, one of the parameters could specify a "fresh", or unique, stream, which would of course not be cached on \OPENFILES. '
'
Incidentally, the MacLisp GC had the equivalent of finalization forms when a stream was GC''d; the full notion isn''t needed, but merely a step or two to insure that open files got closed when GC''d.  Another way to implement this need is to have two lists like the current \OPENFILES -- the existing one would be the canonical cache, and the second one would be for streams opened that were not intended to be registered as canonical.'
'
-----'
Date': 23 Apr 84 14':10 PST'
From': Jellinek.pa'
'
I second and agree with JonL''s comments regarding OPENSTREAM and OPENFILE; in fact, I was about to propose virtually the same ideas myself.'
'
-----'
'
From': masinter.pa'
Date': 24-Apr-84  8':08':25 PST'
Subject': Re': multiple streams per file'
In-reply-to': JonL''s message of 23 Apr 84 13':41 PST'
To': JonL'
cc': MASINTER, lispcore^'
'
I was hoping to make the transition less painless by not requiring everyone to edit their code to look at all of the OPENSTREAMs before they could run in the new release. That was the main reason for picking a new name rather than changing the old name.'
'
Sometimes you really can''t do that, but whenever you can, we''re a lot better off.'
'
Most of the customers who aren''t doing system programming couldn''t care a fig about multiple streams, but are outraged when perfectly good programs stop working without 6 months notice and a graceful plan for conversion.'
'
Plan':'
'
June 1984': introduce NEWSTREAM (or some other name, what does CommonLisp call this?)'
'
January 1985': remove OPENSTREAM (if it seems useful to do so)'
'
Date': 24 Apr 84 10':09 PST'
From': JonL.pa'
Subject': Re': What does CommonLisp call it?'
In-reply-to': masinter.pa''s message of 24-Apr-84  8':08':25 PST'
To': masinter.pa'
cc': JonL.pa, lispcore^.pa'
'
Ooops, the most important part of my reply to this message didn''t get in the first "reply" -- the only CL function for opening streams/files is merely called OPEN.  '
'
*** Always returns a new stream -- a stream given in place of a filename argument means to get the "truename" of the stream first, then open a new stream'
'
*** A string given as the file argument is used as a file name (hence we needn''t support the string-as-open-file-TOTAL-CROCK!)'
'
However, there are a few incongruities in the OPEN specification':  MacLisp (and I think LispMachineLisp) support APPEND mode directly, but CL specifies it by yet-another-additional-keyword; it does support BOTH mode directly (called IO mode, however).  Also some differences in the NEW file OLD file notion, but I think one could map between OPENSTREAM and OPEN on most usages.'
'
Perhaps implementing OPEN, as compatibly as possible, is the best first strategy.'
'
-----'
Date':  5 Jul 83 09':36 PDT'
From': Sybalsky.pa'
Subject': Re': TEdit Include & Copy trouble'
In-reply-to': vanMelle.pa''s message of 26 Jun 83 22':25 PDT'
To': vanMelle.pa'
cc': Sybalsky, LispSupport'
'
Bill-'
'
	I suspect that we''re seeing the tip of an iceberg here':  Whenever two processes share access to a common file, there will be problems synchronizing who gets to open it and close it.  If one process closes the file before the other is done--Bang!'
	This suggests that perhaps there be a reference count in each stream, which would have to go to zero before the file really gets closed?'
'
	--Joh'
'
'
From': KAPLAN.pa'
Date': 27-Apr-84  9':21':33 PST'
Subject': Re': Ever More on the multiple streams problem'
In-reply-to': MASINTER''s message of 27-Apr-84  0':23':38 PST'
To': MASINTER'
cc': JonL, LispCore^'
'
The major prerequisite, even before examing all the places that open files, is to make sure that all the devices insure protection against incompatible openings that is currently being done at the logical IO level (in the stream mechanism itself).  It wouldn''t do to have some devices (e.g. file servers) prevent inconsistent openings, whle other devices (DSK, CORE) do not.'
'
Distinct openings within a single Lisp  should behave as if people on different machines were attempting to access the same file on a file server.'
'
(We might later add a THAWED option, which might only be implementable on some devices).'
'
The last time I was through there, I reorganized the relation of OPENFILE to OPENSTREAM so that OPENSTREAM need not depend on the name caching of \OPENFILES and could in fact return multiple streams.  But I stopped short of making the obvious transition that is now being talked about because of the need to install protection mechanisms in all the devices.'
'
Are we now ready to do that (an AR ?)?'
'
The other desirable feature, which I think Bill mentioned, is a finalize action for streams.  So that the opening associated with a given stream can be eliminated (and perhaps the file actually closed on the device) when a stream is collected.  My impression is that that is not that hard to do':  we just need to insure that streams do not get reclaimed in microcode, then install a branch in \RECLAIMCELL to handle them specially.  Is''t there a bit in the datatype table that causes the \RECLAIMCELL opcode to punt out?  If so, that bit should be turned on as a first step down the path.'
'
--Ron'
'
-----'
'
Date': 29 Apr 84 00':23 PST'
From': vanMelle.pa'
Subject': more on multiple streams'
To': LispSupport'
cc': LispCore^'
'
Seems like there are a lot of bug reports of the form "I was tediting file x.  In some other process I did something with file x.  While that was going on, I quit the Tedit.  Tedit closed x and screwed up the other process."  There used to be dual reports from the other side--closing a file in another process screwed up Tedit--until Tedit was fixed to be super careful about reopening closed files, something it ought not have to do.'
'
Such reports should clearly be filed under whatever AR it is we have given to "Want multiple open streams on single file."  My Ar 834 is just the most recent such.'
'
	Bill'


Workaround: 

Test Case: 

Edit-By: masinter

Edit-Date: 28-Jun-84 08':50':51