DIRECTORY IO, Rope; M3Scan: CEDAR DEFINITIONS = BEGIN GetM3Token: PROC [stream: IO.STREAM, buffer: REF TEXT, flushComments: BOOL ¬ TRUE] RETURNS [tokenKind: TokenKind, token: REF TEXT, charsSkipped: INT, error: TokenError]; TokenKind: TYPE = { tokenERROR, -- token.error describes the scanning error tokenID, -- an identifier or reserved word tokenDECIMAL, -- a whole number literal expressed in decimal tokenBASED, -- a whole number literal with explicit base tokenREAL, -- a REAL literal tokenTEXT, -- a TEXT literal tokenCHAR, -- a CHAR literal tokenSINGLE, -- a single-character token tokenDOUBLE, -- a double-character token tokenCOMMENT, -- a comment tokenPRAGMA, -- a pragma tokenEOF -- the end-of-file marker }; TokenError: TYPE = { none, -- no error extendedChar, -- error following backslash in char or string literal numericLiteral, charLiteral, stringLiteral, -- error in parsing indicated type singleChar -- first non-whitespace char is not legal as first char of token }; GetM3TokenRope: PROC [stream: IO.STREAM, flushComments: BOOL ¬ TRUE] RETURNS [tokenKind: TokenKind, token: Rope.ROPE, charsSkipped: INT]; ScanBack: PROC [base: Rope.ROPE, from: INT] RETURNS [start: INT]; END. ’ M3Scan.mesa Copyright Σ 1991, 1992 by Xerox Corporation. All rights reserved. Spreitze, October 17, 1991 12:22 pm PDT ! (none) Consumes chars from stream, looking for next Modula3 token. Returns the kind of token found, the characters of the token, and the number of white space characters discarded before reaching the token. If flushComments then the characters of a comment are treated as white space. The error returned is # none only for tokenKind = tokenERROR. ! IO.EndOfStream ! IO.Error[SyntaxError] Calls GetM3Token. If token returned is tokenEOF or tokenERROR, raises an appropriate signal. Otherwise converts the token into a ROPE and returns it. Precondition: from is not inside a comment, pragma, char literal or text literal. Returns a position that is < from and (=0 or at the start of a token). Tries to return the largest such position, but will fail this optimization in some cases. Κ3–(cedarcode) style•NewlineDelimiter ™codešœ ™ Kšœ Οeœ7™BK™'—K˜KšΟk œžœ˜K˜KšΟnœžœž œ˜K˜Kšž˜K˜šŸ œžœ žœžœ žœžœžœžœžœžœžœžœ˜©Kšœ™KšœΦ™Φ—K˜šŸ œžœ˜Kšœ Οc+˜7Kšœ  !˜*Kšœ &Πck˜