Class Lexer<TToken>

Performs lexical analysis of the source code in particular language Allows to enumerate over source as typed Token stream - depending on implementation enumeration may do lexical analysis token-by-token or in chunks with every enumerator advance

Inheritance
System.Object
CommonCodeProcessor
SourceRefCommonCodeProcessor
Lexer<TToken>
CSLexer
JSONLexer
LaconfigLexer
Implements
System.Collections.Generic.IEnumerable<TToken>
System.Collections.IEnumerable
Inherited Members
System.Object.Equals(System.Object)
System.Object.Equals(System.Object, System.Object)
System.Object.GetHashCode()
System.Object.GetType()
System.Object.MemberwiseClone()
System.Object.ReferenceEquals(System.Object, System.Object)
Namespace: Azos.CodeAnalysis
Assembly: Azos.dll
Syntax
public abstract class Lexer<TToken> : SourceRefCommonCodeProcessor, IConfigurable, ILexer, ICodeProcessor, IEnumerable<TToken>, IEnumerable where TToken : Token
Type Parameters
Name Description
TToken

Constructors

Lexer(IAnalysisContext, ISourceText, MessageList, Boolean)

Declaration
protected Lexer(IAnalysisContext context, ISourceText source, MessageList messages = null, bool throwErrors = false)
Parameters
Type Name Description
IAnalysisContext context
ISourceText source
MessageList messages
System.Boolean throwErrors

Lexer(IAnalysisContext, SourceCodeRef, ISourceText, MessageList, Boolean)

Declaration
protected Lexer(IAnalysisContext context, SourceCodeRef srcRef, ISourceText source, MessageList messages = null, bool throwErrors = false)
Parameters
Type Name Description
IAnalysisContext context
SourceCodeRef srcRef
ISourceText source
MessageList messages
System.Boolean throwErrors

Lexer(ISourceText, MessageList, Boolean)

Declaration
protected Lexer(ISourceText source, MessageList messages = null, bool throwErrors = false)
Parameters
Type Name Description
ISourceText source
MessageList messages
System.Boolean throwErrors

Fields

m_AllAnalyzed

Declaration
protected bool m_AllAnalyzed
Field Value
Type Description
System.Boolean

m_Tokens

Declaration
protected TokenList<TToken> m_Tokens
Field Value
Type Description
TokenList<TToken>

Properties

AllAnalyzed

Returns true when the whole input source has been analyzed. This property is always true for lexers that do not support lazy analysis

Declaration
public bool AllAnalyzed { get; }
Property Value
Type Description
System.Boolean

Source

References source code text that was lexed

Declaration
public ISourceText Source { get; }
Property Value
Type Description
ISourceText

Tokens

Returns tokenized source as indexable list. Accessing this property causes lexical analysis to complete on the whole source if it has not been completed yet

Declaration
public Tokenized<TToken> Tokens { get; }
Property Value
Type Description
Tokenized<TToken>

TokenStream

Enumerates Token stream - depending on implementation enumeration may do lexical analysis token-by-token or in chunks with every enumerator advance

Declaration
public IEnumerable<Token> TokenStream { get; }
Property Value
Type Description
System.Collections.Generic.IEnumerable<Token>

Methods

AnalyzeAll()

If lexer supports lazy analysis, forces analysis of the whole source

Declaration
public void AnalyzeAll()

DoLexingChunk()

Lexes more source and returns true when EOF has been reached. Depending on particular implementation, a chunk may contain more than one token. Tokens are added into m_Tokens

Declaration
protected abstract bool DoLexingChunk()
Returns
Type Description
System.Boolean

GetEnumerator()

Declaration
public IEnumerator<TToken> GetEnumerator()
Returns
Type Description
System.Collections.Generic.IEnumerator<TToken>

ToString()

Declaration
public override string ToString()
Returns
Type Description
System.String
Overrides
System.Object.ToString()

Explicit Interface Implementations

IEnumerable.GetEnumerator()

Declaration
IEnumerator IEnumerable.GetEnumerator()
Returns
Type Description
System.Collections.IEnumerator

Implements

System.Collections.Generic.IEnumerable<T>
System.Collections.IEnumerable

Extension Methods