Skip to main content
Version: 0.2.3

Module: text/BasicTokenizer

Classes

Type aliases

BasicTokenizerConfig

Ƭ BasicTokenizerConfig: Object

Copyright (c) Meta Platforms, Inc. and affiliates.

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

format

Type declaration

NameType
lowercase?boolean
neverSplit?string[]