M3Scan.mesa
Copyright Ó 1991, 1992 by Xerox Corporation. All rights reserved.
Spreitze, October 17, 1991 12:22 pm PDT
DIRECTORY IO, Rope;
M3Scan: CEDAR DEFINITIONS =
BEGIN
GetM3Token:
PROC [stream:
IO.
STREAM, buffer:
REF
TEXT, flushComments:
BOOL ¬
TRUE]
RETURNS [tokenKind: TokenKind, token:
REF
TEXT, charsSkipped:
INT, error: TokenError];
! (none)
Consumes chars from stream, looking for next Modula3 token. Returns the kind of token found, the characters of the token, and the number of white space characters discarded before reaching the token. If flushComments then the characters of a comment are treated as white space. The error returned is # none only for tokenKind = tokenERROR.
TokenKind:
TYPE = {
tokenERROR, -- token.error describes the scanning error
tokenID, -- an identifier or reserved word
tokenDECIMAL, -- a whole number literal expressed in decimal
tokenBASED, -- a whole number literal with explicit base
tokenREAL, -- a REAL literal
tokenTEXT, -- a TEXT literal
tokenCHAR, -- a CHAR literal
tokenSINGLE, -- a single-character token
tokenDOUBLE, -- a double-character token
tokenCOMMENT, -- a comment
tokenPRAGMA, -- a pragma
tokenEOF -- the end-of-file marker
};
TokenError:
TYPE = {
none, -- no error
extendedChar, -- error following backslash in char or string literal
numericLiteral, charLiteral, stringLiteral, -- error in parsing indicated type
singleChar -- first non-whitespace char is not legal as first char of token
};
GetM3TokenRope:
PROC [stream:
IO.
STREAM, flushComments:
BOOL ¬
TRUE]
RETURNS [tokenKind: TokenKind, token: Rope.
ROPE, charsSkipped:
INT];
! IO.EndOfStream
! IO.Error[SyntaxError]
Calls GetM3Token. If token returned is tokenEOF or tokenERROR, raises an appropriate signal. Otherwise converts the token into a ROPE and returns it.
ScanBack:
PROC [base: Rope.
ROPE, from:
INT]
RETURNS [start:
INT];
Precondition: from is not inside a comment, pragma, char literal or text literal.
Returns a position that is < from and (=0 or at the start of a token).
Tries to return the largest such position, but will fail this optimization in some cases.
END.