Class Tokenizer

Namespace

OpenCvSharp.Dnn

Assembly

OpenCvSharp.dll

Tokenizer for LLM / VLM text (OpenCV 5). Provides a simple API to encode text to token ids and decode token ids back to text (BPE, e.g. gpt2 / gpt4 families).

public class Tokenizer : CvObject, IDisposable

Inheritance

object

CvObject

Tokenizer

Implements

IDisposable

Inherited Members

CvObject.IsDisposed

CvObject.ptr

CvObject.SetSafeHandle(OpenCvSafeHandle)

CvObject.Dispose()

CvObject.Dispose(bool)

CvObject.DisposeManaged()

CvObject.DisposeUnmanaged()

CvObject.ThrowIfDisposed()

CvObject.CvPtr

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

Decode(int[])

Decodes token ids back to text.

public string Decode(int[] tokens)

Parameters

tokens int[]

The token ids.

Returns

string

The decoded UTF-8 string.

Encode(string)

Encodes UTF-8 text to token ids.

public int[] Encode(string text)

Parameters

text string

UTF-8 input string.

Returns

int[]

The token ids.

Load(string)

Loads a tokenizer from a model directory. The directory must contain a config.json (with a supported model_type, e.g. "gpt2" or "gpt4") and a tokenizer.json.

public static Tokenizer Load(string modelConfig)

Parameters

modelConfig string

Path prefix to the model directory. File names are concatenated directly, so it must end with an appropriate path separator.

Returns

Tokenizer

A ready-to-use Tokenizer.