DIRECTORY ExpressToken, IO USING [BreakProc, EndOfStream, GetTokenRope], Rope USING [Equal, Length]; ExpressTokenImpl: CEDAR PROGRAM IMPORTS IO, Rope EXPORTS ExpressToken ~ { OPEN ExpressToken; DefinedTokens: TYPE ~ LIST OF RECORD [rope: ROPE, kind: ATOM]; definedTokens: DefinedTokens _ LIST[ [NIL, $Id1], [NIL, $Id2], [NIL, $Id3], [NIL, $Id4], [NIL, $Id5], [NIL, $Constant], ["+", $Add], ["-", $Subtract], ["*", $Multiply], ["/", $Divide], ["(", $LPar], [")", $RPar], ["[", $LBracket], ["]", $RBracket], [",", $Comma], ["IF", $If], ["THEN", $Then], ["ELSE", $Else], ["min", $min], ["max", $max], ["or", $or], ["and", $and], ["xor", $xor], [">", $GT], ["<", $LT], ["=", $EQ], [">=", $GE], ["<=", $LE], ["#", $NE], ["~", $not] ]; GetNextToken: PUBLIC PROC [ts: TokenStream, symbols: Symbols, cProcs: ClientProcList] RETURNS [token: Token] ~ { TokenBreak: IO.BreakProc ~ { RETURN[SELECT char FROM IN ['\000 .. ' ] => sepr, '>, '<, '=, '#, '~, '[, '], '(, '), '+, '-, '*, '/, '^, ', => break, ENDCASE => other ]; }; Pos: PROC [size: INT] RETURNS [INT] ~ { ts.position _ ts.position+size; RETURN[size]; }; ids: INT _ 1; --Counts identifiers as they go by... rope: ROPE; size: INT; IF ts.buffer#NIL THEN {token _ ts.buffer.first; ts.buffer _ts.buffer.rest; []_Pos[token.size]; RETURN}; [rope, size] _ IO.GetTokenRope[ts.stream, TokenBreak ! IO.EndOfStream => { token _ ["", $EndOfStream, Pos[size]]; GOTO ReturnEOS; }]; size _ size+Rope.Length[rope]; FOR sym: Symbols _ symbols, sym.rest UNTIL sym=NIL DO IF Rope.Equal[sym.first, rope, FALSE] THEN RETURN [[rope, SELECT ids FROM 1=>$Id1, 2=>$Id2, 3=>$Id3, 4=>$Id4, 5=>$Id5, ENDCASE=>ERROR, Pos[size]]]; ids _ ids+1; ENDLOOP; FOR cProc: ClientProcList _ cProcs, cProc.rest UNTIL cProc=NIL DO IF Rope.Equal[cProc.first.userRope, rope, FALSE] THEN RETURN [[rope, $CProc, Pos[size]]]; ENDLOOP; FOR builtIn: DefinedTokens _ definedTokens, builtIn.rest UNTIL builtIn=NIL DO IF Rope.Equal[builtIn.first.rope, rope, FALSE] THEN RETURN [[rope, builtIn.first.kind, Pos[size]]]; ENDLOOP; RETURN[[rope, $Constant, Pos[size]]]; EXITS ReturnEOS => RETURN; }; PushToken: PUBLIC PROC [ts: TokenStream, token: Token] ~ { ts.buffer _ CONS[token, ts.buffer]; ts.position _ ts.position - token.size; }; RegisterToken: PUBLIC PROC [rope: ROPE, kind: ATOM] ~ { definedTokens _ CONS[ [rope, kind], definedTokens]; }; TokenStreamFromIOStream: PUBLIC PROC [s: STREAM] RETURNS [ts: TokenStream] ~ { RETURN [NEW[TokenStreamRec _ [stream: s]]]; }; }. ¬ExpressTokenImpl.mesa Implementation of tokens for the Express package Created Tuesday, July 17, 1984 9:50 pm PDT Last edited by Eric Nickell, July 18, 1984 2:32:31 am PDT rope is the name found in the input stream. NIL means that there is no text string that can be placed in the input stream that implicitly convert to said operation. kind is the actual operator proc, if it is not NIL, means that the code generated by ExpressProc will be a procedure call, and proc is the procedure to call ["", $], First, see if there is anything in the buffer of the TokenStream Nothing in buffer...build one from IO stream Check against the client-supplied symbol table Check against the client-supplied proc names Check against the built-in symbol table The only other thing it could be is a number...assume so Add a new token to the defined tokens list... Turns a vanilla IO STREAM into a token stream. Wasn't that obvious already? ʘšœ™Jšœ0™0Jšœ*™*J™9J™—šÏk ˜ Jšœ ˜ Jšœœ(˜0Jšœœ˜J˜—šœ ˜Jšœœ˜Jšœ ˜J˜Jšœ˜—J˜š œœœœœœœ˜>J™¥J™J™€šœœ˜$J™Jšœœ ˜Jšœœ ˜Jšœœ ˜Jšœœ ˜Jšœœ ˜Jšœœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ˜Jšœ ˜ Jšœ ˜ Jšœ ˜ Jšœ˜Jšœ˜Jšœ ˜ Jšœ ˜ J˜——J˜šÏn œœœ=œ˜pšž œœ˜šœœ˜Jšœ œ ˜JšœD˜DJšœ ˜J˜—J˜—š žœœœœœ˜'J˜Jšœ˜ J˜—Jšœœ Ïc%˜9Jšœœ˜ Jšœœ˜ J™@šœ ˜JšœJœ˜V—J™J™,šœœ&œ˜JJ˜5Jšœ ˜J˜—˜J˜—J™.šœ"œœ˜5š œœœœ œ˜IJšœ2œœ˜N—J˜ Jšœ˜—J™,šœ,œœ˜AJšœ(œœœ˜YJš˜—J™'šœ6œ œ˜MJšœ&œœœ)˜cJšœ˜—J™8Jšœ˜%š˜Jšœ œ˜—J˜J˜—šž œœœ$˜:Jšœ œ˜#J˜'J˜J˜—J˜š ž œœœœœ˜7J™-Jšœœ˜3Jšœ˜J˜—š žœœœœœ˜NJ™LJšœœ ˜+J˜J˜—Jšœ˜—…— ¿