Lexer for EC# source code. More...

Source files:

Inheritance diagram for Loyc.Syntax.Les.LesLexer:

Remarks

Lexer for EC# source code.

See also: ILexer{Token}, TokensToTree

Public fields
bool	AllowNestedComments = true

bool	SkipValueParsing = false
	Used for syntax highlighting, which doesn't care about token values. This option causes the Token.Value to be set to a default, like '\0' for single-quoted strings and 0 for numbers. Operator names are still parsed. More...

Public Member Functions
	LesLexer (UString text, IMessageSink errorSink)

	LesLexer (ICharSource text, string fileName, IMessageSink sink, int startPosition=0)

override void	Reset (ICharSource source, string fileName="", int inputPosition=0, bool newSourceFile=true)

LesLexer	Clone ()

override Maybe< Token >	NextToken ()
	Scans the next token and returns information about it. More...

bool	TDQStringLine ()

bool	TSQStringLine ()

bool	MLCommentLine (ref int nested)

Public Member Functions inherited from Loyc.Syntax.IIndexToLine
SourcePos	IndexToLine (int index)
	Returns the position in a source file of the specified index. More...

Static Public Member Functions
static string	UnescapeQuotedString (ref UString sourceText, Action< int, string > onError, UString indentation=default(UString))
	Parses a normal or triple-quoted string that still includes the quotes. Supports quote types '\'', '"' and '`'. More...

static void	UnescapeQuotedString (ref UString sourceText, Action< int, string > onError, StringBuilder sb, UString indentation=default(UString))
	Parses a normal or triple-quoted string that still includes the quotes (see documentation of the first overload) into a StringBuilder. More...

static bool	UnescapeString (ref UString sourceText, char quoteType, bool isTripleQuoted, Action< int, string > onError, StringBuilder sb, UString indentation=default(UString))
	Parses a normal or triple-quoted string whose starting quotes have been stripped out. If triple-quote parsing was requested, stops parsing at three quote marks; otherwise, stops parsing at a single end-quote or newline. More...

static string	ParseIdentifier (ref UString source, Action< int, string > onError, out bool checkForNamedLiteral)
	Parses an LES-style identifier such as `foo`, , `<tt>foo` or `–punctuation–`. More...

static object	ParseNumberCore (UString source, bool isNegative, int numberBase, bool isFloat, Symbol typeSuffix, out string error)
	Parses the digits of a literal (integer or floating-point), not including the radix prefix (0x, 0b) or type suffix (F, D, L, etc.) More...

Protected Member Functions
override void	Error (int lookaheadIndex, string message, params object[] args)

sealed override void	AfterNewline ()

override bool	SupportDotIndents ()

Protected fields
Dictionary< UString, Symbol >	_idCache = new Dictionary<UString,Symbol>()

Additional Inherited Members
Properties inherited from Loyc.Syntax.Lexing.ILexer< Token >
ISourceFile	SourceFile `[get]`
	The file being lexed. More...

IMessageSink	ErrorSink `[get, set]`
	Event handler for errors. More...

int	IndentLevel `[get]`
	Indentation level of the current line. This is updated after scanning the first whitespaces on a new line, and may be reset to zero when NextToken() returns a newline. More...

UString	IndentString `[get]`
	Gets a string slice that holds the spaces or tabs that were used to indent the current line. More...

int	LineNumber `[get]`
	Current line number (1 for the first line). More...

int	InputPosition `[get]`
	Current input position (an index into SourceFile.Text). More...

Member Function Documentation

override Maybe<Token> Loyc.Syntax.Les.LesLexer.NextToken ( )

inline

Scans the next token and returns information about it.

Returns: The next token, or null at the end of the source file.

Implements Loyc.Syntax.Lexing.ILexer< Token >.

References Loyc.Syntax.Lexing.Operator, and Loyc.Syntax.Lexing.Spaces.

static string Loyc.Syntax.Les.LesLexer.ParseIdentifier	(	ref UString	source,
		Action< int, string >	onError,
		out bool	checkForNamedLiteral
	)

inlinestatic

Parses an LES-style identifier such as foo, , <tt>foo or –punctuation–.

Parameters

source	Text to parse. On return, the range has been decreased by the length of the token; this method also stops if this range becomes empty.
onError	A method to call on error
checkForNamedLiteral	This is set to true when the input starts with @ but doesn't use backquotes, which could indicate that it is an LES named literal such as or .

Returns: The parsed version of the identifier.

static object Loyc.Syntax.Les.LesLexer.ParseNumberCore	(	UString	source,
		bool	isNegative,
		int	numberBase,
		bool	isFloat,
		Symbol	typeSuffix,
		out string	error
	)

inlinestatic

Parses the digits of a literal (integer or floating-point), not including the radix prefix (0x, 0b) or type suffix (F, D, L, etc.)

Parameters

source	Digits of the number (not including radix prefix or type suffix)
isFloat	Whether the number is floating-point
numberBase	Radix. Must be 2 (binary), 10 (decimal) or 16 (hexadecimal).
typeSuffix	Type suffix: F, D, M, U, L, UL, or null.
error	Set to an error message in case of error.

Returns: Boxed value of the literal, null if total failure (result is not null in case of overflow), or CodeSymbols.Sub (-) if isNegative is true but the type suffix is unsigned or the number is larger than long.MaxValue.

static string Loyc.Syntax.Les.LesLexer.UnescapeQuotedString	(	ref UString	sourceText,
		Action< int, string >	onError,
		UString	indentation = `default(UString)`
	)

inlinestatic

Parses a normal or triple-quoted string that still includes the quotes. Supports quote types '\'', '"' and '`'.

Parameters

sourceText	input text
onError	Called in case of parsing error (unknown escape sequence or missing end quotes)
indentation	Inside a triple-quoted string, any text following a newline is ignored as long as it matches this string. For example, if the text following a newline is "\t\t Foo" and this string is "\t\t\t", the tabs are ignored and " Foo" is kept.

Returns: The decoded string

This method recognizes LES and EC#-style string syntax. Firstly, it recognizes triple-quoted strings (''' """ ```). These strings enjoy special newline handling: the newline is always interpreted as
regardless of the actual kind of newline ( and
newlines come out as
), and indentation following the newline can be stripped out. Triple-quoted strings can have escape sequences that use both kinds of slash, like so: / / \'/ "/ \0/. However, there are no unicode escapes (/ is NOT supported).

Secondly, it recognizes normal strings (' " `). These strings stop parsing (with an error) at a newline, and can contain C-style escape sequences: \' " \0 etc. C#-style verbatim strings are NOT supported.

static void Loyc.Syntax.Les.LesLexer.UnescapeQuotedString	(	ref UString	sourceText,
		Action< int, string >	onError,
		StringBuilder	sb,
		UString	indentation = `default(UString)`
	)

inlinestatic

Parses a normal or triple-quoted string that still includes the quotes (see documentation of the first overload) into a StringBuilder.

References Loyc.Localize.Localized().

static bool Loyc.Syntax.Les.LesLexer.UnescapeString	(	ref UString	sourceText,
		char	quoteType,
		bool	isTripleQuoted,
		Action< int, string >	onError,
		StringBuilder	sb,
		UString	indentation = `default(UString)`
	)

inlinestatic

Parses a normal or triple-quoted string whose starting quotes have been stripped out. If triple-quote parsing was requested, stops parsing at three quote marks; otherwise, stops parsing at a single end-quote or newline.

Returns: true if parsing stopped at one or three quote marks, or false if parsing stopped at the end of the input string or at a newline (in a string that is not triple-quoted).

This method recognizes LES and EC#-style string syntax.

References Loyc.Localize.Localized().

Member Data Documentation

bool Loyc.Syntax.Les.LesLexer.SkipValueParsing = false

Used for syntax highlighting, which doesn't care about token values. This option causes the Token.Value to be set to a default, like '\0' for single-quoted strings and 0 for numbers. Operator names are still parsed.

Documentation moved to ecsharp.net

Remarks

Public fields

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected fields

Additional Inherited Members

Member Function Documentation

Member Data Documentation