Proper line breaking for Khmer, Thai, Lao, and Burmese (Myanmar)

[chearon]

These scripts don't separate words with spaces, and require a dictionary. In line breaking, iterate scripts as well as everything else. If the script is one of these, switch to a specialized break iterator.

This library can break all 4, but need to look into efficiency. Should probably use a trie.

Wikipedia says the first 3 are related through the Khmer writing system:

• Khmer script, for the language of Cambodia
• Thai script, for the language of Thailand
• Lao script, for the language of Laos

But Burmese is not:

• Burmese alphabet (Myanmar)

---

Sort of related: Vietnam borders Cambodia and Laos, Vietnamese people today use a variant of Latin. I may need to tweak font selection at some point in order to properly select a Vietnamese font for the entire run instead of having a mix of a Latin and Vietnamese font, since I don't think Latin fonts necessarily support all of Vietnamese.

[seanghay]

@chearon I think it's possible to use icu-segmenter (compiled to wasm) for line breaking. icu-segmenter supports these languages.

[chearon]

So it does. If it's not too difficult I may just use that as a reference and write it in JS or Zig so it's easier to integrate into the existing build system. I'm also interested in some of the machine learning-based breaking I've seen around. Might help keeping the .wasm size down.

[seanghay]

@chearon worth checking: https://github.com/unicode-org/icu4x/tree/main/ffi/npm this is wasm port of icu4x which includes icu-segmenter as well.

[seanghay]

I've made a project called Sone: https://github.com/seanghay/sone with support Khmer text wrapping and justification. Learnt a lot building it.

[chearon]

Nice work and welcome to the club! It didn't occur to me to use Intl.Segmenter. I'll have to see if it performs the same as what I have now.