ExpressToken.mesa
Breaks an IO STREAM into tokens for the Express package
Created Tuesday, July 17, 1984 9:35 pm PDT
Last edited by Eric Nickell, July 18, 1984 2:30:22 am PDT
DIRECTORY
Express USING [ClientProcList, Symbols],
IO USING [STREAM],
Rope USING [ROPE];
ExpressToken: CEDAR DEFINITIONS ~ {
ROPE: TYPE ~ Rope.ROPE;
Symbols: TYPE ~ Express.Symbols;
ClientProcList: TYPE ~ Express.ClientProcList;
STREAM: TYPE ~ IO.STREAM;
Token: TYPE ~ RECORD [rope: ROPE, kind: ATOM, size: INT];
TokenStream: TYPE ~ REF TokenStreamRec;
TokenStreamRec:
TYPE ~
RECORD [
stream: STREAM,
buffer: LIST OF Token ← NIL,
position: INT ← 0
];
GetNextToken:
PROC [ts: TokenStream, symbols: Symbols, cProcs: ClientProcList]
RETURNS [token: Token];
Retrieves the next token from the stream
PushToken:
PROC [ts: TokenStream, token: Token];
Pushes a token back into the token stream
TokenPosition:
PROC [ts: TokenStream]
RETURNS [position:
INT] ~
INLINE {
RETURN[ts.position];
};
Current position within the IO STREAM
RegisterToken:
PROC [rope:
ROPE, kind:
ATOM];
Creates new parsable tokens, ONLY for things that look like identifiers anyway.
TokenStreamFromIOStream:
PROC [s:
STREAM]
RETURNS [ts: TokenStream];
Turns a vanilla IO STREAM into a token stream. Wasn't that obvious already?
}.