Module: text/BasicTokenizer

Classes

• BasicTokenizer

Type aliases

BasicTokenizerConfig

Ƭ BasicTokenizerConfig: Object

Copyright (c) Meta Platforms, Inc. and affiliates.

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

format

Type declaration

Name	Type
lowercase?	boolean
neverSplit?	string[]

Defined in

text/BasicTokenizer.ts:10