JunoLexer.mesa (was Lexer + pieces of ParseTable)
Last edited by Stolfi June 11, 1984 9:15:43 pm PDT

OBSOLETE see JunoParseUnparse June 11, 1984 9:15:18 pm PDT.

DIRECTORY

Rope USING [ROPE];

JunoLexer: DEFINITIONS =

BEGIN

END.

- - - - - JUNK

LexerRec: TYPE = RECORD

[maxAtomLength: INTEGER, -- max length of identifiers. Default is 200
 maxStringLength: INTEGER, -- max length of strings. Default is 200
 chType: ARRAY CHAR OF CharType, -- described below
 twoCharOps: LIST OF TwoCharOp -- list of two-character lexemes

 stream: Stream, -- Source of characters to be broken into lexemes.

lex: Lexeme, -- current lexeme, the result of the most recent call to Lex
 eof: BOOL, -- set by Lex when lexeme was not found.
 buf: ROPE, -- contains the substring that lexed to lex
 error: Rope.ROPE -- initially NIL, set to error message on lexical error
];

CharType: TYPE = {letter, digit, blank, quote, op, invalid, other};

Identifiers must start with letter and continue with letters or digits;
blanks are ignored; quotes delimit strings; ops form one- or two-character
operators; invalids are invalid in any context; others are valid only inside strings.

TwoCharOp: TYPE = RECORD
[chars: RECORD [CHAR, CHAR],
atom: ATOM -- same two characters in atom form
 ];

Stream: TYPE = RECORD [pos: INT, rope: ROPE, len: INT]; -- fake character stream

- - - - - LEXING

StartLexer: PROC [lexer: Lexer, rope: ROPE];

Attaches the specified stream to the lexer, and puts the first lexeme into lexer.lex

Lexer.eof and lexer.error have same meaning as for Lex[lexer].
The lexer table and aliases must have been set up before calling StartLexer.
Further lexemes are obtained by calling Lex[lexer] (see below)