Tomasz claims FASM to have support UTF-8 encoding of sources. It does so just by treating all characters above 0x80 as normal characters allowed in symbols and strings.

Unfortunatelly this is not enough. There are two problems with that:

1. Due to standards, characters with overlength encoding should be refused. That means for example if character is encoded in 2 bytes and could be in one (like zero encoded as C0,00), FASM should display error. This includes checking for incomplete last character.

2. when using du directive with string initializer, FASM should decode UTF-8 to UTF-16 used by Windows (real variable-length character UTF-16, not UCS-2)

I think point 1. could be relatively easily solved in stage where internal tokens are generated, and point 2 shouldn't be big problem aswell. This will give FASM a REAL support for sources in UTF-8. I believe that such internally used tool as FASM undoubtely is deserves good internalization support.

1. I intentionally ignored this rule, as it matters only where security is important, or where you plan to compare the UTF-8 strings etc. I did not see any advantage in signalizing such error.

2. It already does. You may have missed this fragment of UTF8.INC:


3. The real problem with UTF-8 support is that if your editor put BOM as a header to your file, it won't work with fasm, as fasm will treat it as some real symbol.

1.
conformace to standards, for example?

2. sorry, seems i have missed the "encoding" directory completely. Nice to see FASM supports all these encodings, even though doing so by include file is little ... uncommon

3. if you had checking for valid UTF-8 encoding during tokenization, it would be easy to add this feature

I treat it rather as a "security hint". Accepting non-standard encodings of such kind is not really much worse from the fact that fasm accepts any non-standard line breaks regardless of the operating system.


fasm itself is encoding-transparent, that's why the encodings are supported with include files.