Skip to main content
Version: 0.2.4

Module: text/BasicTokenizer

Classes​

Type aliases​

BasicTokenizerConfig​

Ƭ BasicTokenizerConfig: Object

Copyright (c) Meta Platforms, Inc. and affiliates.

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

format

Type declaration​

NameType
lowercase?boolean
neverSplit?string[]