RE error: illegal byte sequence on Mac OS X

A sample command that exhibits the symptom: sed 's/./@/' <<<$'\xfc' fails, because byte 0xfc is not a valid UTF-8 char.
Note that, by contrast, GNU sed (Linux, but also installable on macOS) simply passes the invalid byte through, without reporting an error.

Using the formerly accepted answer is an option if you don’t mind losing support for your true locale (if you’re on a US system and you never need to deal with foreign characters, that may be fine.)

However, the same effect can be had ad-hoc for a single command only:

LC_ALL=C sed -i "" 's|"iphoneos-cross","llvm-gcc:-O3|"iphoneos-cross","clang:-Os|g' Configure

Note: What matters is an effective LC_CTYPE setting of C, so LC_CTYPE=C sed ... would normally also work, but if LC_ALL happens to be set (to something other than C), it will override individual LC_*-category variables such as LC_CTYPE. Thus, the most robust approach is to set LC_ALL.

However, (effectively) setting LC_CTYPE to C treats strings as if each byte were its own character (no interpretation based on encoding rules is performed), with no regard for the – multibyte-on-demand – UTF-8 encoding that OS X employs by default, where foreign characters have multibyte encodings.

In a nutshell: setting LC_CTYPE to C causes the shell and utilities to only recognize basic English letters as letters (the ones in the 7-bit ASCII range), so that foreign chars. will not be treated as letters, causing, for instance, upper-/lowercase conversions to fail.

Again, this may be fine if you needn’t match multibyte-encoded characters such as é, and simply want to pass such characters through.

If this is insufficient and/or you want to understand the cause of the original error (including determining what input bytes caused the problem) and perform encoding conversions on demand, read on below.

The problem is that the input file’s encoding does not match the shell’s.
More specifically, the input file contains characters encoded in a way that is not valid in UTF-8 (as @Klas Lindbäck stated in a comment) – that’s what the sed error message is trying to say by invalid byte sequence.

Most likely, your input file uses a single-byte 8-bit encoding such as ISO-8859-1, frequently used to encode “Western European” languages.

Example:

The accented letter à has Unicode codepoint 0xE0 (224) – the same as in ISO-8859-1. However, due to the nature of UTF-8 encoding, this single codepoint is represented as 2 bytes – 0xC3 0xA0, whereas trying to pass the single byte 0xE0 is invalid under UTF-8.

Here’s a demonstration of the problem using the string voilà encoded as ISO-8859-1, with the à represented as one byte (via an ANSI-C-quoted bash string ($'...') that uses \x{e0} to create the byte):

Note that the sed command is effectively a no-op that simply passes the input through, but we need it to provoke the error:

  # -> 'illegal byte sequence': byte 0xE0 is not a valid char.
sed 's/.*/&/' <<<$'voil\x{e0}'

To simply ignore the problem, the above LCTYPE=C approach can be used:

  # No error, bytes are passed through ('á' will render as '?', though).
LC_CTYPE=C sed 's/.*/&/' <<<$'voil\x{e0}'

If you want to determine which parts of the input cause the problem, try the following:

  # Convert bytes in the 8-bit range (high bit set) to hex. representation.
  # -> 'voil\x{e0}'
iconv -f ASCII --byte-subst='\x{%02x}' <<<$'voil\x{e0}'

The output will show you all bytes that have the high bit set (bytes that exceed the 7-bit ASCII range) in hexadecimal form. (Note, however, that that also includes correctly encoded UTF-8 multibyte sequences – a more sophisticated approach would be needed to specifically identify invalid-in-UTF-8 bytes.)

Performing encoding conversions on demand:

Standard utility iconv can be used to convert to (-t) and/or from (-f) encodings; iconv -l lists all supported ones.

Examples:

Convert FROM ISO-8859-1 to the encoding in effect in the shell (based on LC_CTYPE, which is UTF-8-based by default), building on the above example:

  # Converts to UTF-8; output renders correctly as 'voilà'
sed 's/.*/&/' <<<"$(iconv -f ISO-8859-1 <<<$'voil\x{e0}')"

Note that this conversion allows you to properly match foreign characters:

  # Correctly matches 'à' and replaces it with 'ü': -> 'voilü'
sed 's/à/ü/' <<<"$(iconv -f ISO-8859-1 <<<$'voil\x{e0}')"

To convert the input BACK to ISO-8859-1 after processing, simply pipe the result to another iconv command:

sed 's/à/ü/' <<<"$(iconv -f ISO-8859-1 <<<$'voil\x{e0}')" | iconv -t ISO-8859-1

Related Posts:

  1. grep –ignore-case –only
  2. Using Bash regex match (=~) where regex includes quotes (” characters)
  3. Using the star sign in grep
  4. How can I output only captured groups with sed?
  5. How to replace a whole line with sed?
  6. sed command with -i option failing on Mac, but works on Linux
  7. Error when using ‘sed’ with ‘find’ command on OS X: “invalid command code .”
  8. OR condition in Regex
  9. Using or ‘|’ in regex [duplicate]
  10. What is a Context Free Grammar?
  11. How can I validate an email address using a regular expression?
  12. Regular expression to match a line that doesn’t contain a word
  13. Difference between \b and \B in regex
  14. What does the regex \S mean in JavaScript?
  15. How can I validate an email address using a regular expression?
  16. How do I use a regex in a shell script?
  17. Regex: ignore case sensitivity
  18. Python regex AttributeError: ‘NoneType’ object has no attribute ‘group’
  19. Regular Expression for alphanumeric and underscores
  20. Regular Expression for alphanumeric and underscores
  21. Installing Homebrew on OS X
  22. python re.split() to split by spaces, commas, and periods, but not in cases like 1,000 or 1.50
  23. Regex to test if string begins with http:// or https://
  24. Regular expression negative lookahead
  25. Writing a very simple lexical analyser in C++
  26. Regular Expression.how to add optional character at end of regex
  27. What is a non-capturing group in regular expressions?
  28. Split string on whitespace in Python
  29. What is a non-capturing group in regular expressions?
  30. How to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loops
  31. “sed” command in bash
  32. How to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loops
  33. Python string.replace regular expression
  34. Symbol for any number of any characters in regex?
  35. Grep and Python
  36. How to replace ” \ ” with ” \\ ” in java
  37. A regular expression to exclude a word/string
  38. mvn command not found in OSX Mavrerick
  39. Searching for UUIDs in text with regex
  40. Regular expression to match a word or its prefix
  41. How can I remove punctuation from input text in Java?
  42. Regex optional group
  43. Regex for string contains?
  44. How to remove double-quotes in jq output for parsing json files in bash?
  45. Question marks in regular expressions
  46. Difference between \w and \b regular expression meta characters
  47. Java; String replace (using regular expressions)?
  48. Recursively find all files that match a certain pattern
  49. How to remove the quotes when reading a variable in jq in shell?
  50. What is the grep equivalent in Python?
  51. javascript HTML from document.body.innerHTML
  52. Python re.split() vs split()
  53. How can I write a regex which matches non greedy?
  54. Regex Explanation ^.*$ [duplicate]
  55. Negative matching using grep (match lines that do not contain foo)
  56. How to exclude a specific string constant?
  57. Regex match everything after question mark?
  58. Validating IPv4 addresses with regexp
  59. How to validate phone number using PHP?
  60. Can a URL contain a semicolon and still be valid?
  61. Open an .html file with default browser using Bash on Mac
  62. How is \\n and \\\n interpreted by the expanded regular expression?
  63. -bash: export: `=’: not a valid identifier
  64. Regular Expression to reformat a US phone number in Javascript
  65. Regex optional capturing group?
  66. RegEx to find two or more consecutive chars
  67. Regex to match an optional ‘+’ symbol followed by any number of digits
  68. Regex Match all characters between two strings
  69. psql: command not found Mac
  70. Regular expression to match a word or its prefix
  71. Regular expression for a string that does not start with a sequence
  72. Using SED with wildcard
  73. python regular expression “\1”
  74. Regular expression to stop at first match
  75. Why does cURL return error “(23) Failed writing body”?
  76. /bb|[^b]{2}/ how does it work?
  77. How do I match any character across multiple lines in a regular expression?
  78. Regular expression to remove HTML tags from a string
  79. Regular expression to limit number of characters to 10
  80. My Vim replace with a regex is throwing a `E488: Trailing characters`
  81. How can I do a recursive find/replace of a string with awk or sed?
  82. Regex for “AND NOT” operation
  83. Regular expression that matches valid IPv6 addresses
  84. Escaping Parentheses in Regex
  85. How to run C program on Mac OS X using Terminal?
  86. What is the best regular expression to check if a string is a valid URL?
  87. Regex empty string or email
  88. Regex to validate date formats dd/mm/YYYY, dd-mm-YYYY, dd.mm.YYYY, dd mmm YYYY, dd-mmm-YYYY, dd/mmm/YYYY, dd.mmm.YYYY with Leap Year Support
  89. Regex difference: (\w+)? and (\w*)
  90. Wrap element with in any number of elements with PHP
  91. 404 regular-expression advice needed
  92. How to use a variable as a className inside preg_match_all parameters?
  93. Adding span tags to post titles using regex
  94. Redirect non www to www using redirection plugin regex
  95. Restrict comments with regular expression
  96. Remove Custom HTML Posts Using Regex
  97. Regexes allow only [a-z][A-Z][0-9] and –
  98. Call Shortcode Attribute Value within another function
  99. How to fix old embeded iframes with WP-CLI search-replace
  100. Use Regex Redirection via htsaccess for RSS Feeds WordPress

Leave a Comment