budoux.gleam

A Gleam port of BudouX, a standalone, small, and language-neutral phrase segmenter that provides beautiful and legible line breaks.

Line breaks of a Japanese sentence without and with budoux

Install

gleam add budoux

Works on both the Erlang and JavaScript targets.

Usage

import budoux

pub fn main() {
  let parser = budoux.ja()

  budoux.parse(parser, "今日は天気です。")
  // -> ["今日は", "天気です。"]
}

API

budoux

FunctionDescription
parse(parser: Parser, sentence: String) -> List(String)Splits a sentence into phrases.
ja() -> ParserCreates a parser with the Japanese model.
ja_word() -> ParserCreates a parser with the Japanese word model, which splits sentences into words.
zh_hans() -> ParserCreates a parser with the Simplified Chinese model.
zh_hant() -> ParserCreates a parser with the Traditional Chinese model.
th() -> ParserCreates a parser with the Thai model.
new(model: Model) -> ParserCreates a parser from a model.

budoux/model

ItemDescription
ModelThe scores of a BudouX model, grouped in the same way as its JSON file.
empty: ModelA model with no scores.
decoder() -> decode.Decoder(Model)Decodes a model from the JSON format used by BudouX.

Development

The tools are managed in flake.nix, and the versions used in CI in .tool-versions.

Generated files

These files are generated from the BudouX release pinned in flake.nix, and the update workflow regenerates them.

Generated fileSource in BudouXScript
src/budoux/model/*.gleambudoux/models/*.jsoncodegen/models.nu
test/budoux/quality_cases.gleamtests/quality/ja.tsvcodegen/quality_cases.nu

Acknowledgements

The algorithm, the pretrained models, and the test cases come from google/budoux.

License

Apache License 2.0

✨ Search Document