Loyc.Syntax.Lexing.IndentTokenGenerator< Token > Class Template Reference

A preprocessor usually inserted between the lexer and parser that inserts "indent", "dedent", and "end-of-line" tokens at appropriate places in a token stream. More...

---

Source file:

• /Core/Loyc.Syntax/Lexing/IndentTokenGenerator.cs

Inheritance diagram for Loyc.Syntax.Lexing.IndentTokenGenerator< Token >:

Remarks

A preprocessor usually inserted between the lexer and parser that inserts "indent", "dedent", and "end-of-line" tokens at appropriate places in a token stream.

This class will not work correctly if the lexer does not implement ILexer{T}.IndentLevel properly.

This class is abstract because it doesn't know how to classify or create tokens. The derived class must implement GetTokenCategory, MakeEndOfLineToken, MakeIndentToken and MakeDedentToken. IndentTokenGenerator is a non-abstract version of this class based on Loyc.Syntax.Lexing.Token structures, with several properties that can be customized.

Creation of indent, dedent, and end-of-line tokens can be suppressed inside brackets, i.e. () [] {}. This is accomplished by recognizing brackets inside your implementation of GetTokenCategory.

TokensToTree can be placed in the pipeline before or after this class; if it is placed afterward, anything between Indent and Dedent tokens will be made a child of the Indent token.

Note: whitespace tokens (TokenCategory.Whitespace) are passed through and otherwise unprocessed.

Note: EOL tokens are not generated for empty or comment lines, and are not generated after a generated indent token, although they could be generated after a pre-existing indent token that was already in the token stream, unless that token is categorized as TokenCategory.OpenBracket.

Partial dedents and unexpected indents, as in

if Condition:
    print("Hello")
  print("Hello again")
else:
    print("Goodbye")
      print("Goodbye again")

will cause an error message to be written to the ILexer{Tok}.ErrorSink of the original lexer.

Please see IndentTokenGenerator for additional remarks and examples.

Public Types
enum	TokenCategory {   TokenCategory.OpenBracket = 1, TokenCategory.CloseBracket = -1, TokenCategory.IndentTrigger = 0x10000,   TokenCategory.Whitespace = 0x20000, TokenCategory.Other = 0 }

Properties
int	BracketDepth [get]
int	CurrentIndent [get]
IListSource< int >	OuterIndents [get]
Properties inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
ILexer< Token >	Lexer [get, set]
ISourceFile	SourceFile [get]
virtual IMessageSink	ErrorSink [get, set]
int	IndentLevel [get]
UString	IndentString [get]
int	LineNumber [get]
int	InputPosition [get]
Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile	SourceFile [get]
	The file being lexed. More...
IMessageSink	ErrorSink [get, set]
	Event handler for errors. More...
int	IndentLevel [get]
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...
UString	IndentString [get]
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...
int	LineNumber [get]
	Current line number (1 for the first line). More...
int	InputPosition [get]
	Current input position (an index into SourceFile.Text). More...

Public Member Functions
	IndentTokenGenerator (ILexer< Token > lexer)
	Initializes the indent detector. More...
abstract TokenCategory	GetTokenCategory (Token token)
	Gets the category of a token for the purposes of indent processing. More...
override void	Reset ()
override Maybe< Token >	NextToken ()
	Returns the next (postprocessed) token. This method should set the _current field to the returned value. More...
Public Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
	LexerWrapper (ILexer< Token > sourceLexer)
SourcePos	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...

Protected Member Functions
abstract Maybe< Token >	MakeIndentToken (Token indentTrigger, ref Maybe< Token > tokenAfterward, bool newlineAfter)
	Returns a token to represent indentation, or null to suppress generating an indent-dedent pair at this point. More...
abstract IEnumerator< Token >	MakeDedentToken (Token tokenBeforeNewline, ref Maybe< Token > tokenAfterNewline)
	Returns token(s) to represent un-indentation. More...
abstract Maybe< Token >	MakeEndOfLineToken (Token tokenBeforeNewline, ref Maybe< Token > tokenAfterNewline, int?deltaIndent)
	Returns a token to represent the end of a line, or null to avoid generating such a token. More...
virtual bool	IndentChangedUnexpectedly (Token tokenBeforeNewline, ref Maybe< Token > tokenAfterNewline, ref int deltaIndent)
	A method that is called when the indent level changed without a corresponding indent trigger. More...
virtual SourcePos	IndexToLine (Token token)
virtual void	CheckForIndentStyleMismatch (UString indent1, UString indent2, Token next)
Protected Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
void	WriteError (int index, string msg, params object[] args)

Additional Inherited Members
Protected fields inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
Maybe< Token >	_current

Member Enumeration Documentation

enum Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.TokenCategory

Enumerator
OpenBracket	An open bracket, inside of which indent triggers should be ignored.
CloseBracket	A close bracket, whch reverses the effect of an open bracket.
IndentTrigger	This token may trigger an indentation token, with an unindent token to be generated later, when a line is encountered that is not indented in comparison to this line.
Whitespace	A whitespace token, which should be passed though and otherwise ignored.
Other	None of the other categories apply to this token.

Constructor & Destructor Documentation

Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.IndentTokenGenerator ( ILexer< Token >  lexer )

inline

Initializes the indent detector.

Parameters

lexer

Original lexer (either a raw lexer or an instance of another preprocessor such as TokensToTree.)

Member Function Documentation

abstract TokenCategory Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.GetTokenCategory ( Token  token )

pure virtual

Gets the category of a token for the purposes of indent processing.

Implemented in Loyc.Syntax.Les.LesIndentTokenGenerator.

virtual bool Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.IndentChangedUnexpectedly ( Token  tokenBeforeNewline, ref Maybe< Token >  tokenAfterNewline, ref int  deltaIndent  )

inlineprotectedvirtual

A method that is called when the indent level changed without a corresponding indent trigger.

Parameters

tokenBeforeNewline	Final non-whitespace token before the newline.
tokenAfterNewline	First non-whitespace token after the newline. Though it's a Maybe{T}, it always has a value, but this function can suppress its emission by setting it to NoValue.Value.
deltaIndent	Amount of unexpected indentation (positive or negative). On return, this parameter holds the amount by which to change the CurrentIndent; the default implementation leaves this value unchanged, which means that subsequent lines will be expected to be indented by the same (unexpected) amount.

Returns

true if MakeEndOfLineToken should be called as usual, or false to suppress EOL genertion. EOL can only be suppressed in case of an unexpected indent (deltaIndent>0), not an unindent.

The default implementation always returns true. It normally writes an error message, but switches to a warning in case OuterIndents[OuterIndents.Count-1] == OuterIndents[OuterIndents.Count-2], which this class interprets as a single unindent.

Reimplemented in Loyc.Syntax.Les.LesIndentTokenGenerator.

abstract IEnumerator<Token> Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.MakeDedentToken ( Token  tokenBeforeNewline, ref Maybe< Token >  tokenAfterNewline  )

protectedpure virtual

Returns token(s) to represent un-indentation.

Parameters

tokenBeforeNewline	The last non-whitespace token before dedent
tokenAfterNewline	The first non-whitespace un-indented token after the unindent, or NoValue at the end of the file. The derived class is allowed to change this token, or delete it by changing it to NoValue (LesIndentTokenGenerator does this).

This class considers the indented block to be "over" even if this method returns no tokens.

Implemented in Loyc.Syntax.Les.LesIndentTokenGenerator.

abstract Maybe<Token> Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.MakeEndOfLineToken ( Token  tokenBeforeNewline, ref Maybe< Token >  tokenAfterNewline, int?  deltaIndent  )

protectedpure virtual

Returns a token to represent the end of a line, or null to avoid generating such a token.

Parameters

tokenBeforeNewline	Final non-whitespace token before the newline was encountered.
tokenAfterNewline	First non-whitespace token after newline.
deltaIndent	Change of indentation after the newline, or null if a dedent token is about to be inserted after the newline.

This function is also called at end-of-file, unless there are no tokens in the file.

Implemented in Loyc.Syntax.Les.LesIndentTokenGenerator.

abstract Maybe<Token> Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.MakeIndentToken ( Token  indentTrigger, ref Maybe< Token >  tokenAfterward, bool  newlineAfter  )

protectedpure virtual

Returns a token to represent indentation, or null to suppress generating an indent-dedent pair at this point.

Parameters

indentTrigger	The token that triggered this function call.
tokenAfterward	The token after the indent trigger, or NoValue at EOF.
newlineAfter	true if the next non-whitespace token after indentTrigger is on a different line, or if EOF comes afterward.

Implemented in Loyc.Syntax.Les.LesIndentTokenGenerator.

override Maybe<Token> Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.NextToken ( )

inlinevirtual

Returns the next (postprocessed) token. This method should set the _current field to the returned value.

Implements Loyc.Syntax.Lexing.LexerWrapper< Token >.