token

fun <T> LexerContext<in Token<T>>.token(type: T, back: Int = 1, length: Int = back, offset: Int = 0): Token<T>

Creates a token, based off the current state of the LexerContext.

When called with default values, the returned token matches the last character consumed.

Parameters

type

The type of the token to be created.

back

The amount of characters to offset back from the lexer's current index.

length

The amount of characters this section is composed off.

offset

This value added to back and length as an offset.

fun <T> LexerContext<in StringToken<T>>.token(type: T, value: String, back: Int = value.length, length: Int = back, offset: Int = 0): StringToken<T>

Creates a token with a value, based off the current state of the LexerContext.

When called with default values, the returned token matches the last N character consumers, where N is value's length.