Using awk to remove the Byte-order mark

Try this:

awk 'NR==1{sub(/^\xef\xbb\xbf/,"")}{print}' INFILE > OUTFILE

On the first record (line), remove the BOM characters. Print every record.

Or slightly shorter, using the knowledge that the default action in awk is to print the record:

awk 'NR==1{sub(/^\xef\xbb\xbf/,"")}1' INFILE > OUTFILE

1 is the shortest condition that always evaluates to true, so each record is printed.

Enjoy!

— ADDENDUM —

Unicode Byte Order Mark (BOM) FAQ includes the following table listing the exact BOM bytes for each encoding:

Bytes         |  Encoding Form
--------------------------------------
00 00 FE FF   |  UTF-32, big-endian
FF FE 00 00   |  UTF-32, little-endian
FE FF         |  UTF-16, big-endian
FF FE         |  UTF-16, little-endian
EF BB BF      |  UTF-8

Thus, you can see how \xef\xbb\xbf corresponds to EF BB BF UTF-8 BOM bytes from the above table.

Related Posts:

  1. What’s the difference between UTF-8 and UTF-8 without BOM?
  2. u’\ufeff’ in Python string
  3. How to remove this \xa0 from a string in python?
  4. What’s the difference between ASCII and Unicode?
  5. python encoding utf-8
  6. What’s up with these Unicode combining characters and how can we filter them?
  7. Unicode range for Japanese
  8. What Unicode characters represent “time”?
  9. List of all unicode’s open/close brackets?
  10. Japanese ASCII Code
  11. How to resolve TypeError: can only concatenate str (not “int”) to str [duplicate]
  12. How to correct TypeError: Unicode-objects must be encoded before hashing?
  13. How to replace � in a string
  14. Invisible characters – ASCII
  15. How to grep for case insensitive string in a file?
  16. What’s the difference between ASCII and Unicode?
  17. Tallest Unicode character?
  18. What’s the difference between ASCII and Unicode?
  19. How to remove \xa0 from string in Python?
  20. Using multiple delimiters in awk
  21. How to split a delimited string into an array in awk?
  22. How can I use “:” as an AWK field separator?
  23. What does the ‘b’ character do in front of a string literal?
  24. What does the ‘b’ character do in front of a string literal?
  25. (unicode error) ‘unicodeescape’ codec can’t decode bytes in position 2-3: truncated \UXXXXXXXX escape
  26. What are NR and FNR and what does “NR==FNR” imply?
  27. How to remove double-quotes in jq output for parsing json files in bash?
  28. SSIS Convert Between Unicode and Non-Unicode Error
  29. What is the difference between UTF-8 and Unicode?
  30. Printing the last column of a line in a file
  31. UnicodeDecodeError: ‘charmap’ codec can’t decode byte X in position Y: character maps to
  32. (unicode error) ‘unicodeescape’ codec can’t decode bytes in position 2-3: truncated \UXXXXXXXX escape
  33. What’s the complete range for Chinese characters in Unicode?
  34. “Unicode Error “unicodeescape” codec can’t decode bytes… Cannot open text files in Python 3
  35. Python – ‘ascii’ codec can’t decode byte
  36. How to Sum a column in AWK?
  37. UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)
  38. Absolute value in awk doesn’t work?
  39. std::wstring VS std::string
  40. Use of `NF` in awk command
  41. Python string prints as [u’String’]
  42. NameError: global name ‘unicode’ is not defined – in Python 3
  43. Convert a Unicode string to a string in Python (containing extra symbols)python string unicode type-conversion
  44. Best way to reverse a string
  45. Using unicode character u201c
  46. What is the difference between sed and awk?
  47. What is a unicode string?
  48. How well is Unicode supported in C++11?
  49. Convert Unicode to ASCII without errors in Python
  50. TypeError: coercing to Unicode: need string or buffer, int found
  51. How can I do a recursive find/replace of a string with awk or sed?
  52. UnicodeDecodeError, invalid continuation byte
  53. Javascript: Unicode string to hex
  54. What exactly do “u” and “r” string flags do, and what are raw string literals?
  55. How to print last two columns using awk
  56. Byte and char conversion in Java
  57. How to write unicode strings into a file?
  58. What does _T stands for in a CString
  59. JSON and escaping characters
  60. is there any plugin for auto taging with unicode support for wordpress?
  61. Do you have any useful awk and grep scripts for parsing apache logs? [closed]

Leave a Comment

tech