ⵜⴰⵙⵓⵔⵉⴼⵜ ⵜⴰⵎⵣⵡⴰⵔⵓⵜ ⴰⴼⴰⴷ ⴰⴷ ⵜⵔⵎⵙ ⵜⵖⵉⵙⵜ ⵜⴰⵙⵎⴳⵓⵔⵜ ⵜⵓⵜⵍⴰⵢⵜ: ⴰⵎⵙⵎⴰⵡⵍ
ⵙⵓⴳⵏ ⵎⴰⵙ ⵜⵔⵉⴷ ⴰⴷ ⵜⵙⵙⵍⵎⴷⴷ ⵉ ⵓⵎⵙⵙⵓⴷⵙ ⴰⴷ ⵉⵖⵔ ⵢⴰⵜ ⵜⵡⵉⵏⵙⵜ ⵣⵓⵏ ⴷ "ⵉⴷⴷⴰ ⵓⵔⴳⴰⵣ ⵙ ⵜⵡⵓⵔⵉ." ⵉⵎⵙⵙⵓⴷⵙⵏ ⵙ ⵓⴳⴰⵎⴰ ⵏⵏⵙⵏ ⵓⵔ ⴷⴰ ⵜⵜⵔⵎⴰⵙⵏ ⵜⵉⴳⵓⵔⵉⵡⵉⵏ ⵏⵖ ⵜⵉⵡⵉⵏⴰⵙ ⵙ ⵜⵖⴰⵔⴰⵙⵜ ⵏ ⵓⴼⴳⴰⵏ. ⴷⴰ ⵙⵙⵎⵔⴰⵙⵏ ⵉⵎⴹⴰⵏⵏ. ⵉⵍⵎⵎⴰ ⴷⴰⵜ ⴰⴷ ⵉⵖⵉⵢ ⵓⵎⵙⵙⵓⴷⵙ ⴰⴷ ⵉⵙⵎⴽⵍ ⴰⴹⵕⵉⵚ, ⴰⴹⵕⵉⵚ ⴰⵏⵏ ⵉⵇⵇⴰⵏ ⵜ ⵉⴷ ⴰⴷ ⵉⵜⵜⵓⴱⴹⴰ ⵖⴼ ⵜⴳⵣⵎⵉⵏ ⵜⵉⵎⵥⵢⴰⵏⵉⵏ ⵏⵏⴰ ⵉⵥⴹⴰⵕ ⴰⴷ ⵉⵙⵡⵓⴷⴷⵓ. ⵖⵉⴷ ⴰ ⴳ ⴷⴰ ⵉⴽⵛⵛⵎ ⵓⵎⵙⵎⴰⵡⵍ (Tokenizer).
ⴰⵎⵙⵎⴰⵡⵍ ⴳ ⵜⵙⵉⵍⴰ, ⵉⴳⴰ ⵜⴰⵙⵓⵔⵉⴼⵜ ⵜⴰⵎⵣⵡⴰⵔⵓⵜ ⴳ ⵓⵙⵎⵓⵜⵜⴳ ⵏ ⵓⴹⵕⵉⵚ ⴰⴼⴰⴷ ⴰⴷ ⵜ ⵙⵎⴽⵍⵏⵜ ⵜⵅⴰⵡⴰⵔⵉⵣⵎⵉⵢⵉⵏ, ⵙⵍⴰⵡⴰⵏ ⴰⴽⴽⵯ ⴳ ⵓⵙⵎⴽⵍ ⵏ ⵜⵓⵜⵍⴰⵢⵜ ⵜⴰⴳⴰⵎⴰⵏⵜ (ⵙⵜⴳ) ⴷ ⵉⵣⵓⵔⵜⵏ ⵏ ⵓⵍⵎⵎⵓⴷ ⵏ ⵜⵎⴰⵛⵉⵏⵜ. ⵜⴰⵡⵓⵔⵉ ⵏⵏⵙ ⵜⴰⵙⵉⵍⴰⵏⵜ ⵜⴳⴰ ⴰⴷ ⴷ ⵉⵜⵜⴰⵙⵉ ⵉⵙⴷⴷⵉⵜⵏ ⵉⵎⵙⴰⵔⵉⵜⵏ ⵏ ⵓⴹⵕⵉⵚ ⵉⵍⵎⵎⴰ ⵉⴱⴹⵓ ⵜⵏ ⵖⴼ ⵜⵢⵓⵏⵉⵏ ⵉⵎⵢⴰⵍⵍⴰⵏ ⵉⵙⵎ ⵏⵏⵙⵏⵜ ⵉⵎⵡⴰⵍⵏ.
ⵎⴰ ⵉⴳⴰⵏ ⵉⵎⵡⴰⵍⵏ? ⵙ ⵓⵙⴽⵓⵜⵜⵢ ⵖⴼ ⵡⴰⵏⴰⵡ ⵏ ⵓⵎⵙⵎⴰⵡⵍ, ⵉⵎⵡⴰⵍⵏ ⵥⴹⴰⵕⵏ ⴰⴷ ⴳⵏ ⵜⵉⴳⵓⵔⵉⵡⵉⵏ ⵜⵓⴼⵔⵉⴷⵉⵏ, ⵜⵉⵢⵓⵏⵉⵏ ⵜⵉⴷⵓⴳⵓⵔⵉⵡⵉⵏ (ⴰⵎⵎ ⵜⵉⴳⵣⵎⵉⵏ ⵏ ⵜⴳⵓⵔⵉⵡⵉⵏ), ⵏⵖ ⴰⴽⴽⵯ ⵉⵙⴽⴽⵉⵍⵏ ⵓⴼⵔⵉⴷⵏ.
ⵙ ⵓⵎⴷⵢⴰ, ⵜⴰⵡⵉⵏⵙⵜ "ⴰⵣⵓⵍ, ⴰⵎⴰⴹⴰⵍ!" ⵜⵣⵎⵔ ⴰⴷ ⵜⴻⵜⵜⵓⵙⵎⴰⵡⵍ ⵙ ⵓⵎⵣⴰⵔⴰⵢ:
- ⵖⴼ ⵜⴳⵓⵔⵉⵡⵉⵏ: ["ⴰⵣⵓⵍ", ",", "ⴰⵎⴰⴹⴰⵍ", "!"]
- ⵖⴼ ⵉⵙⴽⴽⵉⵍⵏ: ["ⴰ", "ⵣ", "ⵓ", "ⵍ", ",", " ", "ⴰ", "ⵎ", "ⴰ", "ⴹ", "ⴰ", "ⵍ", "!"]
ⵎⴰⴼ ⵜⵅⴰⵜⵔ ⵜⵙⵓⵔⵉⴼⵜ ⴰⴷ ⴽⵉⴳⴰⵏ? ⵜⵉⵎⴰⵛⵉⵏⵉⵏ ⵓⵙⵔⵏⵜ ⴰⵏⴽⵛⵓⵎ ⵉⵎⵙⴰⵙⴰⵏ ⴷ ⵉⵏⵎⴰⵍⴰⵏ. ⵜⵉⴱⴹⵉ ⵏ ⵓⴹⵕⵉⵚ ⵖⴼ ⵉⵎⵡⴰⵍⵏ ⴷⴰ ⵜⴰⴽⴽⴰ ⵜⵓⵥⴽⵉⵡⵜ ⴰⴷ. ⴽⵓ ⴰⵎⵡⴰⵍ ⵉⵖⵉⵢ ⵉⵍⵎⵎⴰ ⴰⴷ ⵉⵟⵟⴼ ⴰⵎⵙⵎⴰⴳⵉ ⴰⵎⵓⵟⵟⵓⵏ ⴰⵎⵢⵉⵡⵏ, ⵉⵏⴼⵍⵖⴰ ⵓⴹⵕⵉⵚ ⵖⵔ ⵢⴰⵏ ⵓⵙⴷⴷⵉ ⵏ ⵉⵎⴹⴰⵏⵏ ⵍⵍⵉ ⵉⵣⵎⵔ ⴰⴷ ⵉⵔⵎⵙ ⴰⴷ ⵉⵙⵎⴽⵍ ⵓⵎⵙⵙⵓⴷⵙ. ⴰⵢⴰⴷ ⵉⴳⴰ ⴰⵎⵖⵜⴰⵙ ⵉ ⵜⵙⵎⵉⴳⴰⵍ ⵣⵓⵏ ⴷ ⴰⵙⵉⵙⵎⵍ ⵏ ⵓⴹⵕⵉⵚ, ⴰⵙⵓⵖⵍ ⵏ ⵜⵎⴰⵛⵉⵏⵜ, ⴰⵚⵍⴰⴹ ⵏ ⵓⵙⵢⴰⴼⴰ, ⴷ ⵜⵓⵚⴽⴰ ⵏ ⵉⵣⵓⵔⵜⵏ ⵉⵎⵇⵔⴰⵏⵏ ⵏ ⵜⵓⵜⵍⴰⵢⵜ.
ⵍⵍⴰⵏ ⵎⵏⵏⴰⵡ ⵡⴰⵏⴰⵡⵏ ⵏ ⵉⵎⵙⵎⴰⵡⵍⵏ, ⴽⵓ ⵢⴰⵏ ⴷⴰⵔⵙ ⵜⵓⵎⵏⵉⴳⵉⵏ ⴷ ⵜⵓⵣⴷⵉⵔⵉⵏ. ⵉⵎⵙⵎⴰⵡⵍⵏ ⵏ ⵜⴳⵓⵔⵉⵡⵉⵏ ⴳⴰⵏ ⵉⵎⴰⴽⴰⵣⵏ ⵎⴰⵛⵛ ⴷⴰ ⵜⵜⵎⵔⵔⴰⵜⵏ ⴷ ⵜⴳⵓⵔⵉⵡⵉⵏ ⵓⵔ ⵉⵍⵍⵉⵏ ⴳ ⵓⵎⴰⵡⴰⵍ (ⵜⵉⴳⵓⵔⵉⵡⵉⵏ ⵍⵍⵉ ⵓⵔ ⵢⴰⴷⵍⵍⵉ ⵥⵕⵉⵏ). ⵉⵎⵙⵎⴰⵡⵍⵏ ⵏ ⵜⴷⵓⴳⵓⵔⵉⵡⵉⵏ, ⵣⵓⵏ ⴷ ⴰⵙⵏⴳⵍ ⵏ ⵜⵢⵓⴳⴰ ⵏ ⵓⴱⴰⵢⵜ (BPE) ⵏⵖ WordPiece, ⴷⴰ ⴱⵟⵟⵓⵏ ⵜⵉⴳⵓⵔⵉⵡⵉⵏ ⵉⴷⵔⵓⵙⵏ ⵖⴼ ⵜⴷⵓⵢⵓⵏⵉⵏ ⵉⵛⵛⴰⵔⵏ, ⴰⵢⵍⵍⵉ ⵉⵙⵡⵓⴷⴷⵓⵏ ⴰⵏⵛⵜ ⵏ ⵓⵎⴰⵡⴰⵍ ⴷ ⵜⴳⵓⵔⵉⵡⵉⵏ ⵜⵉⵎⴰⵢⵏⵓⵜⵉⵏ ⵙ ⵜⵕⵡⵉ. ⵉⵎⵙⵎⴰⵡⵍⵏ ⵏ ⵉⵙⴽⴽⵉⵍⵏ ⴳⴰⵏ ⵓⵏⵣⵉⵍⵏ ⴷ ⵓⵔ ⴷⴰⵔⵙⵏ ⵉⵎⵓⴽⵔⵉⵙⵏ ⵏ ⴱⵕⵕⴰ ⵏ ⵓⵎⴰⵡⴰⵍ, ⵎⴰⵛⴰ ⴷⴰ ⵣⵍⵍⵓⵏ ⴰⵏⴰⵎⴽ ⴰⵙⵏⴰⵎⴽⴰⵏ ⴳ ⵓⵙⵡⵉⵔ ⵏ ⵜⴳⵓⵔⵉ.
ⴰⵙⵜⴰⵢ ⵏ ⵓⵎⵙⵎⴰⵡⵍ ⵉⵏⵎⵏ, ⵉⴱⴷⴷ ⴱⴰⵀⵔⴰ ⵖⴼ ⵜⵙⵎⵉⴳⵍⵜ ⵉⵥⵍⵉⵏ ⴷ ⵜⵓⵜⵍⴰⵢⵜ ⵉⵜⵜⵓⵙⵎⴽⵍⵏ.
ⴳ ⵜⵙⵉⵍⴰ, ⴳⴰⵏ ⵉⵎⵙⵎⴰⵡⵍⵏ ⵉⵏⴱⵔⴰⵣⵏ ⴰⵔⵓⵙⵙⵉⵏⵏ ⵏ ⵓⵙⵎⴽⵍ ⵏ ⵓⴹⵕⵉⵚ. ⴳⴰⵏ ⵉⵎⴰⵙⵙⵏ ⵉⵙⵉⵍⴰⵏⵏ ⵍⵍⵉ ⵉⵙⵙⵓⵖⵓⵍⵏ ⵜⵓⵜⵍⴰⵢⵜ ⵏ ⵓⴼⴳⴰⵏ ⵖⵔ ⵢⴰⵜ ⵜⴰⵍⵖⴰ ⵍⵍⵉ ⵙ ⵖⵉⵢⵏⵜ ⵜⵎⴰⵛⵉⵏⵉⵏ ⴰⴷ ⵙⵡⵓⵔⵉⵏⵜ.
ⴼⵍ ⴰⵖ ⴽⵔⴰ ⵏ ⵓⵖⴼⴰⵡⴰⵍ :)