RTF dictionary export fails with encoding error #1705

Activity

1. mkrnr

   added

   bug

   on Mar 11, 2025
2. mkrnr

   changed the title [-]RTF dictionary fails with encoding error[/-] [+]RTF dictionary export fails with encoding error[/+] on Mar 11, 2025
3. r-tae commented on Mar 11, 2025

   

   r-tae

   on Mar 11, 2025

   Last edited by r-tae

   More actions

   Minimal reproduction:

   1705-reproduction.json

   The issue is with U+0301 : COMBINING ACUTE ACCENT {stress mark; Greek oxia, tonos}, as the error says, although there are other chars that cause it; the first report had the problem as \u1edf (which is from "TPA*": "phở",)
4. r-tae commented on Mar 11, 2025

   

   r-tae

   on Mar 11, 2025

   More actions

   Getting rid of just those two entries makes main.json work again
5. sammdot commented on Mar 11, 2025

   

   sammdot

   on Mar 11, 2025

   Last edited by sammdot

   Member

   More actions

   We know that Plover encodes RTF files using code page 1252, which is the de facto standard code page that professional CAT programs use. It doesn't exactly have a facility for encoding characters outside the code page, unlike UTF-8 which uses multiple bytes.

   According to the RTF 1.6 spec, Unicode characters can be encoded using UTF-16 and the \uN control word. Examples:

   • The character that spawned this issue report, ở U+1EDF, can be "escaped" in RTF as \u7903 (7903 = 0x1EDF).
   • The spec says Unicode values greater than 32767 must be expressed as negative numbers, so 속 U+C18D can be escaped not as \u49549 ❌, but as \u-15987 (49549 – 65536).
   • Unicode codepoints above U+10000 can be expressed in two bytes using surrogate pairs: 😊 U+1F60A would be treated as two bytes, U+D83D U+DE0A, which can be escaped as \u-10179\u-8694.

   I don't think the Python encoder supports any of this, so it'll have to be done on Plover's end.

   The RTF spec also says \uN should be followed by a fallback character — in our case I think ? should suffice for non-Unicode-supporting CAT programs.

   For what it's worth, if we want to break from the RTF spec slightly for convenience, we could simply write \uN with one positive number and not have a fallback, e.g. \u128522 instead of \u-10179 ?\u-8694 ?.
6. jthlim commented on Mar 16, 2025

   

   jthlim

   on Mar 16, 2025

   More actions

   fwiw.. macOS TextEdit.app encodes rtf 你好😀😃 as:

   \uc0\u20320 \u22909 \uc0\u55357 \u56832 \u55357 \u56835 
   

   So they're using surrogate pairs, and they're not using negative numbers like the spec suggests.
7. sammdot commented on Mar 16, 2025

   

   sammdot

   on Mar 16, 2025

   Member

   More actions

   I tested this myself, and found that it accepted both positive and negative versions, as long as you used surrogate pairs:

   {\rtf1\ansi
   \u20320\u22909\u55357\u56832\u55357\u56835\~
   \u20320\u22909\u-10179\u-8704\u-10179\u-8701
   \par
   \u7903 {\*\cxcomment U+1EDF}
   \par
   \u49549\~
   \u-15987 {\*\cxcomment U+C18D}
   \par
   \u55357\u56842\~
   \u-10179\u-8694\~
   \u128522 {\*\cxcomment U+1F60A}
   \par
   A\~\u65601
   }

   你好😀😃 你好😀😃
   ở
   속 속
   😊 😊 
   A A

   The last two lines are curious though — \u128522 was rendered as U+F60A, so it looks like it just clips the numbers to below 65536, and you can see that with \u65601 (0x10041) being equivalent to A (U+0041). Not really relevant to this issue, just an interesting find.

   I'm fine with the approach of using surrogate pairs but positive numbers since Mac supports it, and it's unlikely pro software would even bother supporting Unicode to this extent.
8. mkrnr

   closed this as completedin #1857on Aug 4, 2026