DIRECTORY IO USING [STREAM], Scheme USING [Any]; SchemeExtras: CEDAR DEFINITIONS ~ BEGIN TokenKind: TYPE ~ { error, -- error describes the scanning error endOfFile, -- the end-of-file marker comment, -- a comment, including the initial semicolon identifier, -- an identifier * moduleReference, -- a structured name "foo#bar" * boolean, -- a boolean * number, -- a Scheme number * character, -- a character literal * string, -- a character string literal * openParenthesis, -- a left parenthesis closeParenthesis, -- a right parenthesis openVector, -- the opening of a vector literal "#(" quote, -- a single-close-quote quasiquote, -- a single-open-quote unquote, -- a comma unquoteSplicing, -- a comma-at-sign dot, -- a period primitiveSyntax -- one of the special "#!" tokens * }; ScanningError: TYPE ~ { none, -- no error earlyEOF, -- an end-of-file in the middle of a token unknownCharacterName, -- "#\" followed by an illegal character name unknownPrimitiveSyntax, -- "#!" followed by an illegal primitive syntax name unknownHashDispatch, -- "#" followed by an illegal character unknownStringEscape, -- "\" followed by an illegal character in a string illegalInitialCharacter, -- the first non-whitespace character cannot begin a token badIdentifier, -- illegally-formed identifier badNumber -- illegally-formed number }; GetToken: PROC [stream: IO.STREAM, buffer: REF TEXT, flushComments: BOOL _ TRUE, buildValue: BOOL] RETURNS [tokenKind: TokenKind, token: REF TEXT, error: ScanningError, value: Scheme.Any]; END. ° SchemeExtras.mesa Copyright Σ 1989, 1990, 1992 by Xerox Corporation. All rights reserved. Last changed by Pavel on March 2, 1990 3:48 pm PST Scanning Scheme code In the cases marked with an asterisk above, GetToken can return the Scheme value corresponding to the token. Consumes chars from stream, looking for the next Scheme token. Returns the kind of token found, the characters of the token, and, optionally, the Scheme value corresponding to the token, if appropriate. If flushComments then the characters of a comment are treated as white space. If tokenKind = tokenERROR, then error # none, token is an appropriate error message, and value is an appropriate culprit. Κ/•NewlineDelimiter –(cedarcode) style™code•Mark outsideHeaderšœ™Kšœ Οeœ=™HK™2K™—šΟk ˜ Kšžœžœžœ˜Kšœžœ˜—K˜KšΠbl œžœž ˜Kšœž˜head™šœ žœ˜KšœΟc%˜-Kšœ ˜%Kšœ  -˜7Kšœ  ˜Kšœ  ˜1Kšœ  ˜Kšœ  ˜Kšœ  ˜$Kšœ  ˜(Kšœ ˜&Kšœ ˜(Kšœ  '˜4Kšœ ˜Kšœ  ˜#Kšœ   ˜Kšœ ˜#Kšœ  ˜Kšœ #˜3Kšœ˜Kšœl™lK™—šœžœ˜Kšœ  ˜Kšœ  *˜5Kšœ -˜CKšœ 4˜LKšœ '˜