Indent postprocessor for Loyc Expression Syntax More...

Source file:

/Core/Loyc.Syntax/LES/LesIndentTokenGenerator.cs

Inheritance diagram for Loyc.Syntax.Les.LesIndentTokenGenerator:

Remarks

Indent postprocessor for Loyc Expression Syntax

LES's "Python mode" is comparable to Python's rules, but I want LES with Python mode to be able to parse most of the code that was designed for LES without Python mode. For example, in LES (without indentation processing) you could write a statement like

if foo "foo "
    + "detected " +
    "here";

And I'd like this code to still parse OK when IndentTokenGenerator is inserted into the pipeline. Therefore, LesIndentTokenGenerator will allow unexpected indentation and treat it as a way of continuing the previous statement.

Code like

if condition:
   then();
if condition:
   then();
:else:
   otherwise();
foo
: bar;

is translated to this (where [ and ] represent Indent and Dedent):

if condition: [
   then(); };
] ;
if condition: [
   then();
] else [
   otherwise(); 
] ;
foo bar;

Notice that dedents are normally followed by semicolons, and that colons are suppressed if they appear at the beginning of a line AND block the generation of a semicolon. A colon is not suppressed if it does not have this effect; forexample

:
    hi();
    : Foo;

means

{
    hi();
    <tt>:</tt> Foo;
};

Note: originally I allowed code such as

if foo: if bar: print(foo + bar);

print(foo only);

as equivalent to

if foo { if bar { print(foo + bar);

print(foo only); }; };

But I decided it would be better to treat code such as

x = a:b;

as a simple statement instead. This means, however, that

if Foo: x = 5;

does not have the meaning you would expect! You must use braces for this:

if Foo { x = 5; };

Public Member Functions
	LesIndentTokenGenerator (ILexer< Token > lexer)

override TokenCategory	GetTokenCategory (Token token)
	Gets the category of a token for the purposes of indent processing. More...

Public Member Functions inherited from Loyc.Syntax.Lexing.IndentTokenGenerator< Token >
	IndentTokenGenerator (ILexer< Token > lexer)
	Initializes the indent detector. More...

override void	Reset ()

override Maybe< Token >	NextToken ()
	Returns the next (postprocessed) token. This method should set the `_current` field to the returned value. More...

Public Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
	LexerWrapper (ILexer< Token > sourceLexer)

SourcePos	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...

Protected Member Functions
override Maybe< Token >	MakeIndentToken (Token indentTrigger, ref Maybe< Token > tokenAfterward, bool newlineAfter)
	Returns a token to represent indentation, or null to suppress generating an indent-dedent pair at this point. More...

override IEnumerator< Token >	MakeDedentToken (Token tokenBeforeDedent, ref Maybe< Token > tokenAfterDedent)
	Returns token(s) to represent un-indentation. More...

override bool	IndentChangedUnexpectedly (Token tokenBeforeNewline, ref Maybe< Token > tokenAfterNewline, ref int deltaIndent)
	A method that is called when the indent level changed without a corresponding indent trigger. More...

override Maybe< Token >	MakeEndOfLineToken (Token tokenBeforeNewline, ref Maybe< Token > tokenAfterNewline, int?deltaIndent)
	Returns a token to represent the end of a line, or null to avoid generating such a token. More...

Protected Member Functions inherited from Loyc.Syntax.Lexing.IndentTokenGenerator< Token >
virtual SourcePos	IndexToLine (Token token)

virtual void	CheckForIndentStyleMismatch (UString indent1, UString indent2, Token next)

Protected Member Functions inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
void	WriteError (int index, string msg, params object[] args)

Additional Inherited Members
Public Types inherited from Loyc.Syntax.Lexing.IndentTokenGenerator< Token >
enum	TokenCategory { TokenCategory.OpenBracket = 1, TokenCategory.CloseBracket = -1, TokenCategory.IndentTrigger = 0x10000, TokenCategory.Whitespace = 0x20000, TokenCategory.Other = 0 }

Properties inherited from Loyc.Syntax.Lexing.IndentTokenGenerator< Token >
int	BracketDepth `[get]`

int	CurrentIndent `[get]`

IListSource< int >	OuterIndents `[get]`

Properties inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
ILexer< Token >	Lexer `[get, set]`

ISourceFile	SourceFile `[get]`

virtual IMessageSink	ErrorSink `[get, set]`

int	IndentLevel `[get]`

UString	IndentString `[get]`

int	LineNumber `[get]`

int	InputPosition `[get]`

Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile	SourceFile `[get]`
	The file being lexed. More...

IMessageSink	ErrorSink `[get, set]`
	Event handler for errors. More...

int	IndentLevel `[get]`
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...

UString	IndentString `[get]`
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...

int	LineNumber `[get]`
	Current line number (1 for the first line). More...

int	InputPosition `[get]`
	Current input position (an index into SourceFile.Text). More...

Protected fields inherited from Loyc.Syntax.Lexing.LexerWrapper< Token >
Maybe< Token >	_current

Member Function Documentation

override TokenCategory Loyc.Syntax.Les.LesIndentTokenGenerator.GetTokenCategory ( Token token )

inlinevirtual

Gets the category of a token for the purposes of indent processing.

Implements Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.

References Loyc.Syntax.Lexing.Token.Value.

override bool Loyc.Syntax.Les.LesIndentTokenGenerator.IndentChangedUnexpectedly	(	Token	tokenBeforeNewline,
		ref Maybe< Token >	tokenAfterNewline,
		ref int	deltaIndent
	)

inlineprotectedvirtual

A method that is called when the indent level changed without a corresponding indent trigger.

Parameters

tokenBeforeNewline	Final non-whitespace token before the newline.
tokenAfterNewline	First non-whitespace token after the newline. Though it's a Maybe{T}, it always has a value, but this function can suppress its emission by setting it to NoValue.Value.
deltaIndent	Amount of unexpected indentation (positive or negative). On return, this parameter holds the amount by which to change the CurrentIndent; the default implementation leaves this value unchanged, which means that subsequent lines will be expected to be indented by the same (unexpected) amount.

Returns: true if MakeEndOfLineToken should be called as usual, or false to suppress EOL genertion. EOL can only be suppressed in case of an unexpected indent (deltaIndent>0), not an unindent.

The default implementation always returns true. It normally writes an error message, but switches to a warning in case OuterIndents[OuterIndents.Count-1] == OuterIndents[OuterIndents.Count-2], which this class interprets as a single unindent.

Reimplemented from Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.

override IEnumerator<Token> Loyc.Syntax.Les.LesIndentTokenGenerator.MakeDedentToken	(	Token	tokenBeforeNewline,
		ref Maybe< Token >	tokenAfterNewline
	)

inlineprotectedvirtual

Returns token(s) to represent un-indentation.

Parameters

tokenBeforeNewline	The last non-whitespace token before dedent
tokenAfterNewline	The first non-whitespace un-indented token after the unindent, or NoValue at the end of the file. The derived class is allowed to change this token, or delete it by changing it to NoValue (LesIndentTokenGenerator does this).

This class considers the indented block to be "over" even if this method returns no tokens.

Implements Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.

override Maybe<Token> Loyc.Syntax.Les.LesIndentTokenGenerator.MakeEndOfLineToken	(	Token	tokenBeforeNewline,
		ref Maybe< Token >	tokenAfterNewline,
		int?	deltaIndent
	)

inlineprotectedvirtual

Returns a token to represent the end of a line, or null to avoid generating such a token.

Parameters

tokenBeforeNewline	Final non-whitespace token before the newline was encountered.
tokenAfterNewline	First non-whitespace token after newline.
deltaIndent	Change of indentation after the newline, or null if a dedent token is about to be inserted after the newline.

This function is also called at end-of-file, unless there are no tokens in the file.

Implements Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.

override Maybe<Token> Loyc.Syntax.Les.LesIndentTokenGenerator.MakeIndentToken	(	Token	indentTrigger,
		ref Maybe< Token >	tokenAfterward,
		bool	newlineAfter
	)

inlineprotectedvirtual

Returns a token to represent indentation, or null to suppress generating an indent-dedent pair at this point.

Parameters

indentTrigger	The token that triggered this function call.
tokenAfterward	The token after the indent trigger, or NoValue at EOF.
newlineAfter	true if the next non-whitespace token after `indentTrigger` is on a different line, or if EOF comes afterward.

Implements Loyc.Syntax.Lexing.IndentTokenGenerator< Token >.

References Loyc.Syntax.Lexing.Token.EndIndex.

Documentation moved to ecsharp.net

Remarks

Public Member Functions

Protected Member Functions

Additional Inherited Members

Member Function Documentation