Class Tokenizer`Abstract`

Map source input in to tokens for parser

Hierarchy

Tokenizer
- StringTokenizer

Index

Constructors

Methods

tokenize String

Constructors

constructor

new Tokenizer(): Tokenizer
Returns Tokenizer

Methods

`Abstract` tokenize

tokenize(): Token[]
Returns Token[]
- Defined in packages/lib/src/tokenizer.ts:8

`Static` String

String(source): StringTokenizer
Parameters
- source: string
Returns StringTokenizer
- Defined in packages/lib/src/tokenizer.ts:10

Generated using TypeDoc