Class StringTokenizer

Map source string in to tokens for parser

Hierarchy

Tokenizer
- StringTokenizer

Index

Constructors

Properties

charNumber current lineNumber source start tokens

Methods

addToken advance isAtEnd isDigit isDigitOrDot isIdentifier peek tokenize String

Constructors

constructor

new StringTokenizer(source): StringTokenizer
Parameters
- source: string
Returns StringTokenizer
Overrides Tokenizer.constructor
- Defined in packages/lib/src/tokenizer.ts:25

Properties

`Private` charNumber

charNumber: number = 0

`Private` current

current: number = 0

`Private` lineNumber

lineNumber: number = 0

`Private` source

source: string

`Private` start

start: number = 0

`Private` tokens

tokens: Token[] = []

Methods

`Private` addToken

addToken(tokenType, literal?): void
Parameters
- tokenType: TokenType
- literal: unknown = null
Returns void
- Defined in packages/lib/src/tokenizer.ts:195

`Private` advance

advance(): string
Returns string
- Defined in packages/lib/src/tokenizer.ts:187

`Private` isAtEnd

isAtEnd(): boolean
Returns boolean
- Defined in packages/lib/src/tokenizer.ts:183

`Private` isDigit

isDigit(char): boolean
Parameters
- char: string
Returns boolean
- Defined in packages/lib/src/tokenizer.ts:146

`Private` isDigitOrDot

isDigitOrDot(char): boolean
Parameters
- char: string
Returns boolean
- Defined in packages/lib/src/tokenizer.ts:152

`Private` isIdentifier

isIdentifier(char): boolean
Parameters
- char: string
Returns boolean
- Defined in packages/lib/src/tokenizer.ts:156

`Private` peek

peek(): string
Returns string
- Defined in packages/lib/src/tokenizer.ts:191

tokenize

tokenize(): Token[]
Tokenize source string

Returns Token[]
List of tokens from source
Overrides Tokenizer.tokenize
- Defined in packages/lib/src/tokenizer.ts:33

`Static` String

String(source): StringTokenizer
Parameters
- source: string
Returns StringTokenizer
Inherited from Tokenizer.String
- Defined in packages/lib/src/tokenizer.ts:10

Generated using TypeDoc