ASCII stands for American Standard Code for Information Interchange, only includes 128 characters (not all of them even printable), and is based on the needs of American use circa 1960. It includes nothing related to any Japanese characters. I believe you want the Unicode code points for so
There is a plain-text database of information about every Unicode character available from the Unicode Consortium; the format is described in Unicode Annex #44. The primary information is contained in UnicodeData.txt. Open and close punctuation characters are denoted with Ps (punctuation start) and Pe (punctuation end) in the General_Category field (the third field, delimited by … Read more
The UTF-8 BOM is a sequence of bytes at the start of a text stream (0xEF, 0xBB, 0xBF) that allows the reader to more reliably guess a file as being encoded in UTF-8. Normally, the BOM is used to signal the endianness of an encoding, but since endianness is irrelevant to UTF-8, the BOM is unnecessary. According to the Unicode standard, the BOM … Read more
The following code points exist related to clocks, watches, and other devices to indicate time: You can copy and paste the characters from this page into most editors. At unicode-table.com you might find more useful code points.
CJK(Chinese Japanese and Korean), Hiragana and Katakana(include Halfwidth Katakana) http://www.unicode.org/charts/
What’s up with these unicode characters? That’s a character with a series of combining characters. Because the combining characters in question want to go above the base character, they stack up (literally). For instance, the case of ก้้้้้้้้้้้้้้้้้้้้ …it’s an ก (Thai character ko kai) (U+0E01) followed by 20 copies of the Thai combining character mai tho (U+0E49). How … Read more
You don’t need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here: Just write your data directly to the file, there is no need to encode already-encoded data. If you instead build up unicode values instead, you would … Read more
Try this: On the first record (line), remove the BOM characters. Print every record. Or slightly shorter, using the knowledge that the default action in awk is to print the record: 1 is the shortest condition that always evaluates to true, so each record is printed. Enjoy! — ADDENDUM — Unicode Byte Order Mark (BOM) FAQ includes … Read more
ASCII defines 128 characters, which map to the numbers 0–127. Unicode defines (less than) 221 characters, which, similarly, map to numbers 0–221 (though not all numbers are currently assigned, and some are reserved). Unicode is a superset of ASCII, and the numbers 0–127 have the same meaning in ASCII as they have in Unicode. For example, the … Read more
If you know for sure that is the only character you don’t want, you can .replace it: If you need to handle all non-ascii characters, encoding and replacing bad characters might be a good start…: