TokenKind:
TYPE ~ {
error, -- error describes the scanning error
endOfFile, -- the end-of-file marker
comment, -- a comment, including the initial semicolon
identifier, -- an identifier *
moduleReference, -- a structured name "foo#bar" *
boolean, -- a boolean *
number, -- a Scheme number *
character, -- a character literal *
string, -- a character string literal *
openParenthesis, -- a left parenthesis
closeParenthesis, -- a right parenthesis
openVector, -- the opening of a vector literal "#("
quote, -- a single-close-quote
quasiquote, -- a single-open-quote
unquote, -- a comma
unquoteSplicing, -- a comma-at-sign
dot, -- a period
primitiveSyntax -- one of the special "#!" tokens *
};
In the cases marked with an asterisk above, GetToken can return the Scheme value corresponding to the token.
ScanningError:
TYPE ~ {
none, -- no error
earlyEOF, -- an end-of-file in the middle of a token
unknownCharacterName, -- "#\" followed by an illegal character name
unknownPrimitiveSyntax, -- "#!" followed by an illegal primitive syntax name
unknownHashDispatch, -- "#" followed by an illegal character
unknownStringEscape, -- "\" followed by an illegal character in a string
illegalInitialCharacter, -- the first non-whitespace character cannot begin a token
badIdentifier, -- illegally-formed identifier
badNumber -- illegally-formed number
};
GetToken:
PROC [stream:
IO.
STREAM, buffer:
REF
TEXT, flushComments:
BOOL ←
TRUE, buildValue:
BOOL]
RETURNS [tokenKind: TokenKind, token:
REF TEXT, error: ScanningError, value: Scheme.Any];
Consumes chars from stream, looking for the next Scheme token. Returns the kind of token found, the characters of the token, and, optionally, the Scheme value corresponding to the token, if appropriate. If flushComments then the characters of a comment are treated as white space. If tokenKind = tokenERROR, then error # none, token is an appropriate error message, and value is an appropriate culprit.