DIRECTORY Express USING [ClientProcList, Symbols], IO USING [STREAM], Rope USING [ROPE]; ExpressToken: CEDAR DEFINITIONS ~ { ROPE: TYPE ~ Rope.ROPE; Symbols: TYPE ~ Express.Symbols; ClientProcList: TYPE ~ Express.ClientProcList; STREAM: TYPE ~ IO.STREAM; Token: TYPE ~ RECORD [rope: ROPE, kind: ATOM, size: INT]; TokenStream: TYPE ~ REF TokenStreamRec; TokenStreamRec: TYPE ~ RECORD [ stream: STREAM, buffer: LIST OF Token _ NIL, position: INT _ 0 ]; GetNextToken: PROC [ts: TokenStream, symbols: Symbols, cProcs: ClientProcList] RETURNS [token: Token]; PushToken: PROC [ts: TokenStream, token: Token]; TokenPosition: PROC [ts: TokenStream] RETURNS [position: INT] ~ INLINE { RETURN[ts.position]; }; RegisterToken: PROC [rope: ROPE, kind: ATOM]; TokenStreamFromIOStream: PROC [s: STREAM] RETURNS [ts: TokenStream]; }. ÐExpressToken.mesa Breaks an IO STREAM into tokens for the Express package Created Tuesday, July 17, 1984 9:35 pm PDT Last edited by Eric Nickell, July 18, 1984 2:30:22 am PDT Retrieves the next token from the stream Pushes a token back into the token stream Current position within the IO STREAM Creates new parsable tokens, ONLY for things that look like identifiers anyway. Turns a vanilla IO STREAM into a token stream. Wasn't that obvious already? ÊƘšœ™J™7Jšœ*™*J™9J™—šÏk ˜ Jšœœ˜(Jšœœœ˜Jšœœœ˜J˜—Jšœœ œ˜#˜Jšœœœ˜Jšœ œ˜ Jšœœ˜.Jšœœœœ˜J˜—Jš œœœœœœ˜9Jšœ œœ˜'šœœœ˜Jšœœ˜Jšœœœ ˜Jšœ œ˜Jšœ˜—J˜šÏn œœ=œ˜fJ™(J™—šž œœ!˜0J™)J™—š ž œœœ œœ˜HJšœ˜J˜Jšœœ™%J˜—šž œœœœ˜-J™OJ˜—šžœœœœ˜DJ™LJ™—J˜J˜—…—:Ð