Class Tokenizer

Namespace

OpenCvSharp.Dnn

Assembly

OpenCvSharp.dll

Tokenizer for LLM / VLM text (OpenCV 5). Provides a simple API to encode text to token ids and decode token ids back to text (BPE, e.g. gpt2 / gpt4 families).

public class Tokenizer : CvObject, IDisposable

Inheritance

object

CvObject

Tokenizer

Implements

IDisposable

Inherited Members

CvObject.IsDisposed

CvObject.ptr

CvObject.SetSafeHandle(OpenCvSafeHandle)

CvObject.SetPostReleaseAction(Action)

CvObject.Dispose()

CvObject.Dispose(bool)

CvObject.DisposeManaged()

CvObject.DisposeUnmanaged()

CvObject.ThrowIfDisposed()

CvObject.CvPtr

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

Decode(int[])

Decodes token ids back to text.

Encode(string)

Encodes UTF-8 text to token ids.

Load(string)

Loads a tokenizer from a model directory. The directory must contain a config.json (with a supported model_type, e.g. "gpt2" or "gpt4") and a tokenizer.json.