Loyc.Syntax.Lexing.TokensToTree Class Reference

A preprocessor usually inserted between the lexer and parser that converts a token list into a token tree. Everything inside brackets, parens or braces is made a child of the open bracket. More...

---

Source file:

• /Core/Loyc.Syntax/Lexing/TokensToTree.cs

Inheritance diagram for Loyc.Syntax.Lexing.TokensToTree:

Remarks

A preprocessor usually inserted between the lexer and parser that converts a token list into a token tree. Everything inside brackets, parens or braces is made a child of the open bracket.

The close bracketis not treated as one of the children of the opening bracket.

Public Member Functions
	TokensToTree (ILexer< Token > source, bool skipWhitespace)
override Maybe< Token >	NextToken ()
	Returns the next (postprocessed) token. This method should set the _current field to the returned value. More...
Public Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
	LexerWrapper (ILexer< Token > sourceLexer)
SourcePos	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...
virtual void	Reset ()

Additional Inherited Members
Properties inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
ILexer< Token >	Lexer [get, set]
ISourceFile	SourceFile [get]
virtual IMessageSink	ErrorSink [get, set]
int	IndentLevel [get]
UString	IndentString [get]
int	LineNumber [get]
int	InputPosition [get]
Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile	SourceFile [get]
	The file being lexed. More...
IMessageSink	ErrorSink [get, set]
	Event handler for errors. More...
int	IndentLevel [get]
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...
UString	IndentString [get]
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...
int	LineNumber [get]
	Current line number (1 for the first line). More...
int	InputPosition [get]
	Current input position (an index into SourceFile.Text). More...
Protected Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
void	WriteError (int index, string msg, params object[] args)
Protected fields inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
Maybe< Token >	_current

Member Function Documentation

override Maybe<Token> Loyc.Syntax.Lexing.TokensToTree.NextToken ( )

inlinevirtual

Returns the next (postprocessed) token. This method should set the _current field to the returned value.

Implements Loyc.Syntax.Lexing.LexerWrapper< Token >.