A stream (an instance of type IO.STREAM) is a producer or consumer of a byte sequence. An input stream is a producer, an output stream is a consumer. Cedar's notion of stream is similar to that of Unix and many other systems. The stream abstraction is important for two related reasons.

The first reason is that most of the data storage and communication devices that are part of our computers can be modeled as streams. Disks and tapes store byte sequences, and the Ethernet transmits them. Streams do not represent the capabilities of any of these devices perfectly, and streams model several other devices even less perfectly: consider bitmap displays and unencoded keyboards. But each of these devices can be represented as a stream.

The second reason is that streams are an adequate form of communication for a significant number of applications. For example, many programs are organized in a "read - eval - print" structure. The program parses an input stream and produces values (which are probably not simply character strings). These values are manipulated in some way, and then they are printed (i.e. converted to character strings and sent to the output stream). Such a program can be structured as a procedure taking an input stream and an output stream as its parameters, and is oblivious to the source of its input and the destination of its output. As long as one application can read the printed output of a second application, the two can be coupled in this way and avoid the difficulty of agreeing upon a single internal data format. Thus the stream abstraction forms a "lowest common denominator" for communication with applications and devices.

Cedar makes nearly all devices available as streams. If a device has capabilities that are not adequately modelled by a stream, its streams simply implement some extra procedures that apply only to streams for that device (or for a set of similar devices). For instance, a stream for reading a disk file implements a SetIndex operation that reflects the random-access capability of the disk. By presenting a file as an upward-compatible extension of a stream, the same basic operations can be used by programs that use files as streams and programs that use files as random-access devices.

We call each stream implementation a stream class; a stream class is named by an ATOM. For example, a stream on a file has class $File, while a stream on a ROPE has class $ROPE. For emphasis we sometimes call a particular stream a stream instance. Each stream instance of a particular class and "direction" (input or output) implements the same set of procedures.

IO is a large interface that contains three major components in addition to the generic operations on streams: procedures for creating streams from a few important data types, procedures for reading from a stream and performing input conversion, and procedures performing output conversion and writing the results to a stream. The stream creation procedure for a typical stream class is defined in an interface associated with the class, not in IO; the EditedStream and IOClasses interfaces define some useful stream classes. IOUtils contains procedures that are only of interest to the implementor of an unusually complex stream class.

Why is the IO interface is so large? The usual rule is "one abstraction, one interface", and by this rule the IO interface would only contain the the generic operations on streams. Cedar has chosen to provide many stream procedures in a single large interface as a convenience to programmers: most of the commonly-used procedures that deal in streams can be found in IO, and these procedures can often be called using object notation because they are part of IO.STREAM's cluster.

The IO and IOUtils interfaces contains a certain amount of documentation, but this is structured for use while programming and does not attempt to explain underlying concepts or give full detail. The EditedStream and IOClasses interfaces contain full documentation.

The Convert interface provides conversions from character strings into Cedar values (numbers, times, etc.) and from Cedar values into character strings. The Convert interface does not really involve streams, but it is often used in conjunction with streams, so we document it here.

The remainder of this document is organized as follows:

See Section 3.1.1 of the Cedar Language Reference Manual for a description of the notation used for grammars below.

The fourteen generic stream procedures fall into three groups:

There are three stream varieties: input stream, output stream, and input/output stream. All streams implement the Control procedures, but other procedures are implemented or not as a function of the stream variety. The Cedar type system does not enforce this stream taxonomy; a stream class implementor is responsible for documenting the variety of stream he is implementing, and for providing all of the necessary procedures.

An input stream implements the Input procedures and none of the Output procedures. Informally, an input stream produces a sequence of bytes (returned by calls to GetChar, GetBlock, and UnsafeGetBlock). The end of the byte sequence is signalled by (1) EndOf returning TRUE, (2) GetChar raising EndOfStream, or (3) GetBlock or UnsafeGetBlock returning zero bytes when a nonzero number of bytes was requested. Backup and PeekChar provide a look-ahead capability for use by scanners. CharsAvail allows the caller to estimate the real-time response of the stream to later requests. Reset skips to the end of the input sequence, and Close releases resources that were reserved by the stream (such as a file opened for reading).

An output stream implements the Output procedures and none of the Input procedures. Informally, an output stream consumes a sequence of bytes (passed in calls to PutChar, PutBlock, and UnsafePutBlock). Flush forces the output bytes to be transmitted immediately to their destination. EraseChar "erases" the most-recently written character (most useful for display-like output devices). Reset has no uniformly-defined effect, while Close terminates the output sequence and releases resources that were reserved by the stream (such as a file opened for writing).

An input/output stream implements both the Input and Output procedures. It is not clear that input/output streams are a good idea, and this stream variety may be eliminated in a later version of IO. At present there are two input/output stream classes: read/write file streams, and Pup byte streams. These two have little in common: for file streams, the input and output halves are closely coupled (there is a single "stream index" into the file), while the input and output halves of a Pup byte stream are largely independent. In both cases, the semantics of the input and output halves are similar, and in both cases it seems to make sense for the Control procedures to operate on both halves simultaneously. In contrast, creating a typescript viewer produces two streams, one input stream and one output stream, because the the keyboard and the display have very different properties.

Only two errors are raised by generic stream procedures: EndOfStream and Error. Error passes an element of an enumerated type to describe the specific error. Each stream class maps its errors into this fixed set. Clearly this set cannot truly describe the totality of errors that may arise; Error[$Failure, stream] is raised for any error that cannot be represented explicitly. If a client needs more information on the particular error, it must call a class-specific procedure to get the information. The complete error information can be stored in the stream, which is a parameter to Error.

A simple model of an input stream is that the stream contains a sequence of bytes, the input sequence, as part of its state. GetChar removes the first byte from the input sequence, Backup adds a byte to the front of the sequence, and EndOf tests the sequence for emptiness. GetBlock, UnsafeGetBlock, and PeekChar are expressed in terms of GetChar and Backup.

The validity of this model is not diminished by the fact that for several stream classes, the entire input sequence for a stream is not known at the time a stream is created. The stream must behave, except for performance, as though the input sequence were known in advance and only modified by GetChar and Backup. In particular, once the input sequence is empty (as signified by stream.GetChar[] raising EndOfStream), it remains empty unless stream.Backup[char] is performed. The CharsAvail procedure provides information on how much of the input sequence is actually known to a stream.

A simple model of an output stream is that the stream connects to a sequential output device of some kind. PutChar transmits a single byte to this device. PutBlock and UnsafePutBlock are expressed in terms of PutChar.

In practice, of course, an output stream may contain buffering. The Flush procedure forces a stream to flush its buffers to the underlying device. Some important sequential output devices allow PutChar to be "undone": erase the last character painted to the display, truncate the last character appended to a file. The EraseChar procedure provides this function for devices that support it, and has a simple default behavior for devices that do not.

See the FS interface documentation for basic information on files. FS.StreamOpen creates a stream, given the string name of a file; FS.StreamFromOpenFile creates a stream, given an FS.OpenFile.

The FS interface documentation describes the detailed semantics of generic stream operations as they apply to files; we provide a summary here. A file is a sequence of bytes (numbered from zero), and a stream on a file defines a stream index into this sequence. GetChar on an input stream raises EndOfStream when the stream index equals the file length, and otherwise returns the file byte at the stream index and advances it one byte. PutChar on an output stream extends the file by one byte when the stream index equals the file length, and otherwise replaces the file byte at the stream index and advances it one byte. An input/output file stream has a single stream index, so GetChar and PutChar interact. Several input streams may be opened on a single file. It is also possible to open an output stream and several input streams on a single file (if this works out, we may get rid of input/output file streams).

The following operations apply to file streams (and streams for other file-like devices):

See the Rope interface documentation for basic information on ROPE.

The following operation applies to ROPE output streams:

The following operation applies to TEXT output streams:

See the ViewerIO interface documentation. ViewerIO.CreateViewerStreams creates a pair of streams, one for keyboard input and one for display output, coupled so that input is echoed as it is read.

See the documentation contained in the EditedStream interface. An edited input stream provides simple interactive editing of an underlying input stream. Typically the editing is teletype-style (BS, ^W), but features such as default values and input completion can be implemented by the edited stream client.

Interactive editing is also provided by editable typescripts, and these are generally preferred to edited streams for applications that "live above" Tioga.

IOClasses.CreatePipe[bufferByteCount] creates a pipe, that is, an output stream push and an input stream pull such that the byte sequence written to push can be read from pull. Closing push terminates the sequence. The parameter bufferByteCount specifies the (approximate) maximum number of bytes that can be written to push and not yet read from pull at any instant of time. Typically, one process does the pushing and another process does the pulling.

IOClasses.CreateCatInputStream[input1, input2] creates an input stream s such that the byte sequence produced by s is the byte sequence produced by input1, followed by the byte sequence produced by input2. For instance, CreateCatInputStream[IO.RIS[rope1], IO.RIS[rope2]] is equivalent to IO.RIS[rope1.Cat[rope2]].

IOClasses.CreateDribbleOutputStream[output1, output2] creates an output stream s such that each operation performed on s will be performed in turn on output1 and output2. For instance, s.PutChar[c] is equivalent to { output1.PutChar[c]; output2.PutChar[c] }.

IOClasses.CreateCommentFilterStream[stream] creates an input stream s such that comments contained in stream are filtered out. The comments in stream must follow Cedar syntax, and stream must also follow Cedar conventions for string and character literals. For more detail see documentation in the IOClasses interface.

See the ViewerIO interface documentation. ViewerIO.CreateMessageWindowStream creates an output stream to the Viewers message window.

See the PupStream interface documentation. PupStream.PupByteStreamMake creates an input/output Pup byte stream.

(Soon to be made obsolete by Ed Taft's new FTP implementation: STP.CreateRemoteStream. Used by FS only.)

See Ed McCreight; not part of Cedar release.

Each procedure in this section takes r: ROPE as a parameter, but is perfectly happy to have the client use RefText.TrustTextAsRope to loophole a REF TEXT into ROPE to pass in. This is ok because these procedures do not retain the ROPE or test its type at runtime.

Each procedure in this section raises the following errors:

Each "CardFrom..." or "IntFrom" or "TimeFrom" or "CharFrom" proc raises the following error:

In general, the parsing procedures do not consume the entire input rope, but are willing to stop by reaching the end of the input rope or by seeing whitespace after a valid token.

The following procedures accept the same literal formats accepted by the compiler. This means, for instance, that the octal and hex routines demand that a radix character ('B or 'b for octal, 'H or 'h for hex) be present. Numeric literals are always non-negative, and these procedures do not recognize a leading plus or minus sign. The numeric procedures are willing to skip over leading white space, and will terminate the literal on end of string or on trailing white space.

There are two parallel sets of procs for printing. A proc in the first set returns a rope, while the proc in the second set appends its result to an existing REF TEXT (usually avoiding additional storage allocation) and returns a REF TEXT (usually but not always the same REF as the original).

Each "RopeFrom..." proc raises the following error:

Each "Append..." proc raises the following errors:

Other errors are identical for the two variants of each procedure.

The Put procedure provides a convenient means of printing Cedar values. It is both faster and less flexible than PutF, because it does not accept a format string in addition to the sequence of Cedar values to be printed.

Character and character string (ROPE and REF TEXT) values are simply transmitted to the stream. REF ANY values are converted by calling PrintRefAny[stream, refAny.value, 4, 32, FALSE]. Other values are converted to character strings using standard calls to Convert, then transmitted to the stream:

(The implementation actually uses the "Append" variants of these Convert calls, to reduce allocations, but the effect is the same.)

Scanning is the process of dividing an input stream into character subsequences or tokens. These tokens may then be converted into values of specific type, e.g. numbers or atoms.

Scanning routines that return a result are generally provided in two versions. The first version takes a REF TEXT buffer as a parameter, and may return its result in this buffer, but will allocate a larger one if the buffer fills up. Hence if each token is smaller than the buffer, only a single allocation is required to scan a sequence of tokens. The second version returns its result in a ROPE; hence at least one byte of storage is allocated for each byte of token scanned. In either case, a token cannot exceed NAT.LAST bytes in length.

Here is a sample program fragment to illustrate one style of use for the REF TEXT version of a typical scanning procedure. Note that the value of buffer below is never changed, even though a call to GetCedarToken may return a different REF TEXT.

The following convenience procedures generally call GetCedarToken (with flushComments = TRUE), check that the tokenKind is as expected, and convert the token to a Cedar value of the indicated type. GetInt, GetCard, and GetReal accept an optional leading plus or minus sign. GetReal accepts REAL literals, but also accepts anything that GetInt or GetCard would accept, converting it to REAL. GetTime and GetUnpackedTime are not implemented as just described because there is no standard Cedar syntax for time; they simply consume whatever looks like a time.

The procedures described in this section are designed to tokenize an input stream according to criteria defined by client-supplied procedures.

Several character constants from the Ascii interface are re-defined in the IO interface for convenience:

We represent a stream as a REF to a STREAMRecord:

The streamProcs field of a stream points to an immutable record that can be shared by any number of stream instances. A typical stream class implementation builds this record during module initialization. In contrast, the streamData field of a stream points to a mutable record that is not shared across streams. A stream class implementation builds this record in its stream creation procedure.

Many stream class implementations do not use the propList and backingStream fields of the STREAMRecord. Such a stream class implementation is a module of the form:

Note that IO.CreateStreamProcs allocates and initializes storage for the StreamProcs record, and that IO.CreateStream allocates and initializes storage for the STREAMRecord. Also note that there is no need to monitor accesses to the global variable procs above, because it is constant after initialization and the record it points to is immutable.

A stream class implementation may vary from this template along several dimensions. We summarize them here, and describe them more fully in later sections:

If s is a stream, s.backingStream is called its backing stream. Backing streams are intended to serve a specific purpose: to make it convenient to implement one stream class in terms of another. As mentioned above, CreateStreamProcs supplies default implementations of generic stream procedures -- even the most fundamental ones such as GetChar and EndOf. These default implementations all check to see if a backing stream is present, and if so they "pass the buck" to it. This means that you can implement a stream class by implementing a few procedures, obtaining the rest from CreateStreamProcs, and supplying a backing stream to CreateStream (via its backingStream parameter, which we defaulted to NIL in the template above). The backing stream of an input stream should be an input stream, and the backing stream of an output stream should be an output stream.

On a call to CreateStreamProcs, any procedure parmeters that you pass NIL as the value of (the default) are assigned specific values by CreateStreamProcs. So to understand CreateStreamProcs it is sufficient first to understand what happens when you supply non-NIL values for all procedure parameters, and then to understand the defaults that CreateStreamProcs uses when NIL is supplied.

To first order, all you can do with a REF StreamProcs is to insert it into the streamProcs field of a stream, e.g. by passing it to CreateStream. So the semantics of CreateStreamProcs are understood in terms of the behavior of the resulting stream. Consider the following code fragment:

The call "s.GetChar[]" in the final line is equivalent to "myGetChar[s]". At a more primitive level, the call "s.GetChar[]" translates to "call the proc supplied as the getChar parm to the CreateStreamProcs call that created s.streamProcs". This applies uniformly to all of the procedure-valued parameters to CreateStreamProcs, not just to GetChar.

CreateStreamProcs does not allow its caller to specify an implementation of PeekChar. This reflects the current implementation, in which PeekChar is implemented for all stream classes as a call to GetChar, followed by a call to PutBack.

Now for the defaults:

Consider the problem of implementing a stream class that writes to the Viewer Message Window. The primitives on this window are MessageWindow.Append and MessageWindow.Clear. We want the stream to buffer characters until a Flush is executed, then send them to the window. Stream Reset should clear the buffer and the window.

This implementation demonstrates the full power of backing streams and CreateStreamProcs defaulting. We use a rope output stream to implement the character buffering between Flush calls.

Non-generic stream procedures fall into two classes. The first type is exemplified by IO.RopeFromROS, which applies only to rope output streams (created by IO.ROS) and returns a rope containing the bytes sent to the stream. RopeFromROS simply checks that the stream it has been passed is a rope output stream, computes the result rope, and returns it.

The second type is exemplified by IO.GetLength, which applies to any "file like" stream, including file streams, rope streams, and ref text streams, but does not apply to all streams. Rather than have the GetLength procedure "know" about all of these stream classes, GetLength searches the StreamProcs property list of the stream it has been passed for a $GetLength procedure, then calls it. The $GetLength procedure for the class is added to the property list by CreateStreamProcs when the class is created. If CreateStreamProcs did not take a getLength parameter, it would still be possible to attach a getLength procedure to a StreamProcs property list using IOUtils.StoreProc.

We use the module below, a simplified version of the module that implements rope output streams, to illustrate these ideas.

A rope is an immutable byte sequence that lives in (and dies with) a virtual memory. A rope may be derived from an immutable file, but the file and the rope are still different objects.

A file is a (possibly mutable) byte sequence that lives outside of virtual memory. A file is accessed through an open file, which lives in virtual memory, but the file and the open file are still different objects.

An input stream can be a pointer into an existing byte sequence, such as a file or a rope. It can also deal with byte sequences that are not manifest in the way ropes and files are, e.g. a sequence of keystrokes. An output stream can be used to produce a rope or a file.

Individual streams are independent objects, so there is never any need to synchronize calls to their procedures. If two processes are to share a single stream, they must synchronize their accesses to that stream at a level above the stream procedures; individual stream procedures (PutChar, PutBlock, etc.) are not guaranteed to be atomic. A stream class may use a monitor (probably either a module monitor on the implementation or an object monitor on the instance data) to provide atomicity at this level, but would do so to protect its own integrity, not to synchronize its callers.

A stream may be used to access a system resource that has its own concurrency control. For instance, FS provides locks that control concurrent access to local files, and Alpine maintains a separate locking facility for its files.

The basic byte-transmission procedures are GetChar for input streams and PutChar for output streams. In some circumstances the overhead of one procedure call per byte transmitted is not acceptable. In this case a client should use the corresponding block-transmission procedures GetBlock and PutBlock. These traffic in REF TEXT buffers, TEXT being a standard safe type for representing mutable byte sequences. (Note that TEXT is limited to 32k bytes.) Not all stream classes implement the block procedures more efficiently than the byte procedures, but several important ones do, including file streams and Pup streams.

An application that transmits non-text data (such as a MACHINE DEPENDENT record that must be stored on a file for later use) will find it more convenient to use UnsafeGetBlock and UnsafePutBlock, which traffic in LONG POINTERs and byte offsets. A common error in using these procedures is to pass the SIZE of a datum (a word count), rather than the SIZE times PrincOps.BytesPerWord (a byte count). Since UnsafeGetBlock stores bytes through an untyped LONG POINTER, it has the potential for unbounded destruction of the garbage collector invariants if misused; inspect each use of UnsafeGetBlock carefully as you wrap it in a TRUSTED block.

A particular stream class may have its own efficiency considerations, which should be discussed in the documentation for the class.

This document should contain more information on the efficiency of the basic stream primitives when applied to commonly-used stream classes, and on the efficiency of the scanning and printing packages (Taft).

IOUtils.SetDefaultPFCodeProc and IOUtils.SetDefaultPFErrorProc do not exist (Spreitzer).

It seems advisable to define inlines for the unsafe block operations that take word counts rather than byte counts, because this is a better match to the language's type.SIZE operator (Taft).