README
¶
gex
A generic framework for lexical analysis of UTF-8 text in Go, based on the implementation discussed in "Lexical Scanning in Go" by robpike.
Installation
$ go get -u github.com/bdreece/gex
Usage
import (
"github.com/bdreece/gex"
)
Examples
Coming soon.
MIT License
Copyright (c) 2025 Brian Reece
Documentation
¶
Overview ¶
Package gex provides a generic framework for lexical analysis of UTF-8 text in Go, based on the implementation discussed in "Lexical Scanning in Go" by robpike.
Index ¶
- func Start[T any](input string, init State[T], opts ...Option[T]) <-chan Token[T]
- func StartWithConfig[T any](config *Config[T]) <-chan Token[T]
- type Config
- type Lexer
- func (l *Lexer[T]) Accept(valid string) bool
- func (l *Lexer[T]) AcceptRun(valid string)
- func (l *Lexer[T]) Back() (ok bool)
- func (l Lexer[T]) Current() string
- func (l Lexer[T]) EOF() rune
- func (l *Lexer[T]) Emit(kind T)
- func (l *Lexer[T]) Errorf(format string, args ...any) State[T]
- func (l Lexer[T]) Input() string
- func (l Lexer[T]) Name() string
- func (l *Lexer[T]) Next() rune
- func (l *Lexer[T]) Peek() rune
- func (l Lexer[T]) Pos() int
- func (l *Lexer[T]) Runes() iter.Seq[rune]
- func (l *Lexer[T]) Skip()
- func (l Lexer[T]) Start() int
- func (l Lexer[T]) Width() int
- type Option
- type State
- type Token
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config[T any] struct { // The input text (required). Input string // The initial state (required). Init State[T] // The name of the lexer (defaults to "gex"). Name string // The buffer size of the lexer token stream (defaults to 2). Capacity int // The sentinel `rune` value used to demarcate the end of the // token stream (defaults to `*new(rune)`). EOF rune // The sentinel `T` value used when emitting a synthetic error // token (defaults to `*new(T)`). Error T }
A Config provides additional configuration parameters for the lexer state machine in the form of a struct.
func DefaultConfig ¶
DefaultConfig returns a Config object populated with the default values.
type Lexer ¶
type Lexer[T any] struct { // contains filtered or unexported fields }
Lexer provides utility methods for emitting tokens and directing the state of iteration.
func (*Lexer[T]) Accept ¶
Accept advances the iterator position if the next rune is in the valid set.
func (*Lexer[T]) AcceptRun ¶
func (l *Lexer[T]) AcceptRun(valid string)
AcceptRun advances the iterator position until it encounters a rune not found in the valid set.
func (*Lexer[T]) Back ¶
func (l *Lexer[T]) Back() (ok bool)
Back decrements the current iterator position. Can only be run once per call to Next.
func (Lexer[T]) Current ¶
func (l Lexer[T]) Current() string
Current returns a string slice representing the current token.
func (Lexer[T]) EOF ¶
func (l Lexer[T]) EOF() rune
EOF returns the sentinel EOF rune associated with the lexer.
func (*Lexer[T]) Emit ¶
func (l *Lexer[T]) Emit(kind T)
Emit emits a token with the provided kind to the output stream.
func (*Lexer[T]) Errorf ¶
Errorf formats and emits a synthetic error token using the configured error kind (defaults to the zero value).
func (*Lexer[T]) Next ¶
func (l *Lexer[T]) Next() rune
Next returns the next rune in the input text, advancing the iterator position.
func (*Lexer[T]) Peek ¶
func (l *Lexer[T]) Peek() rune
Peek returns the next rune in the input text without advancing the iterator position.
func (*Lexer[T]) Runes ¶
Runes returns an iterator over the runes in the sequence, until EOF is reached.
func (*Lexer[T]) Skip ¶
func (l *Lexer[T]) Skip()
Skip advances the start cursor to the current iterator position.
type Option ¶
type Option[T any] interface { // contains filtered or unexported methods }
An Option provides an internal mechanism for mutating a Config.
func WithCapacity ¶
WithCapacity returns an option configuring the capacity of the Lexer token stream.
func WithEOF ¶
WithEOF returns an option configuring the sentinel EOF rune associated with the lexer. See Config for more details.
func WithErrorType ¶
func WithErrorType[T any](typ T) Option[T]
WithErrorType returns an option configuring the sentinel error value associated with the lexer. See Config for more details.
type State ¶
A State is a node in the lexer state machine.
Implementors of this function contract can take advantage of utility functions provided through the Lexer object to emit tokens and direct iteration over the input text.
Source Files