Finding unwanted unicode characters in source code files

Cryptic error messages pointing to source code with no obvious logic or formatting errors can be a nightmare to debug, with the nastiest incidents caused by unicode. Anything from non-printable control characters and lookalikes to entirely broken UTF byte sequences can seriously hinder even the most productive developers.

Text encodings for source code

Most source code today is stored as UTF-8, introducing a few troublesome error conditions and unicode characters that can be difficult to debug without a solid understanding of text encodings.

A good CI pipeline or pre-commit git hook should verify that files are properly encoded as UTF-8 and not an older ASCII-compatible encoding like Latin-1 or Windows-1252, to prevent issues arising from incompatible text encodings. A simple bash command can handle all text files in a directory recursively:

find . -type f -name '*.c' -print0 |
while IFS= read -r -d '' f; do
    enc=$(file -bi "$f" | sed 's/.*charset=//')
    case "$enc" in
        utf-8|us-ascii)
            ;;
        *)
            iconv -f "$enc" -t UTF-8 "$f" > "$f.tmp" && mv "$f.tmp" "$f"
            ;;
    esac
done

The script finds all files with the .c extension (adjust for your target language), checks the current encoding and calls iconv to convert it to UTF-8 if it is not UTF-8 or ASCII (which is technically valid UTF-8). Converting text to UTF-8 is always safe, as it can represent all unicode characters.

Broken unicode characters

Since UTF-8 is a multi-byte encoding, it can end up with "broken characters" if a sequence is not ended properly or byte formatting is off. A simple bit flip from a memory error or disk corruption can cause such errors seemingly out of nowhere, and a broken UTF-8 byte sequence can swallow valid following character bytes, leading to missing characters or weird printed symbols in the output.

Some word processors or WYSIWYG online editors also use custom byte sequences to mark things like paragraphs or syntax highlighting, but using such text outside of the editor will result in broken or otherwise invalid unicode.

You can check a UTF-8 file for invalid bytes with iconv:

iconv  -f UTF-8 -t UTF-8 main.c > /dev/null && echo "Valid UTF-8" || echo "Contains invalid UTF-8 bytes"

The command checks file main.c for invalid UTF-8 sequences and prints "Valid UTF-8" if none are found, or displays every violating byte and its offset, and finishes with the message "Contains invalid UTF-8 bytes".

Errors will look like this:

iconv: illegal input sequence at position 7

Where "position 7" means the violation occurs after the 7th byte in the file. Since unicode may use more than one byte per character, you have to look at bytes specifically to find the offset, for example with tools like xxd.

If you are not concerned with removing broken unicode sequences from the file, you can blindly fix all occurrences automatically:

iconv  -f UTF-8 -t UTF-8 -c main.c > main_clean.c

Note the added -c flag to discard any invalid sequences, writing any legal UTF-8 output to main_clean.c. This is a destructive operation, as it will blindly strip out any invalid bytes, so use with care and run it before any other CI tools and tests when using inside automation pipelines.

Lookalike characters

Many english characters have lookalike variants in other languages, for example e and é are visually similar enough to seem identical at first glance, especially when hidden within larger text blocks like variable names.

While technically identifiable with eyesight alone, the process can be time consuming and frustrating.

The only automated way for normalization is to decompose such unicode characters to NFKD form, then strip the trailing accents:

uconv -x NFKD main.c |  iconv -f UTF-8 -t ASCII//TRANSLIT > main_ascii.c

Since this operation is destructive and applies to string contents as well, it can only be used on files that should be ASCII-only, without unicode in string contents. Also note that while mostly stable on glibc linux, iconv behavior varies by implementation and can produce differing output on musl/bsd/mac; always test with your target system instead of trusting it to work reliably across systems.

A common workaround is to separate translation/localized text contents into config files to keep source code ASCII only, enforced through automation scripts.

Invisible characters

When talking about invisible characters, there are two distinct types to watch out for. The first are genuinely invisible characters that simply have to visual representation, like ASCII control characters or the optional UTF-8 Byte-Order-Mark (BOM). The second are invisible in the sense that they hide in plain sight, like the non-breaking space that looks exactly like a normal space in most editors.

Both types are difficult to catch, luckily modern compilers like clang and python3 can natively catch them:

python3:

  File "/tmp/aa/main.py", line 4
      print("end of block")
    ^
SyntaxError: invalid non-printable character U+00A0

clang:

main.c:2:1: warning: treating Unicode character as whitespace [-Wunicode-whitespace]
    2 |  
      | ^
1 warning generated.

Older compilers can catch them too, but their error messages are often more cryptic:

Here is what a non-breaking space in a C file compiled with gcc looks like:

main.c:2:1: error: stray ‘\302’ in program
    2 | <U+00A0>
      | ^~~~~~~~

Unfortunately, some tools have less helpful error output. Consider this docker-compose.yml file:

services:
  web: 
    build:
      context: app
      target: builder.
    ports: 
      - '8000:8000'

The file looks fine, but running docker compose config will throw a misleading error:

yaml: while scanning a simple key at line 5, column 7: line 6, column 5: could not find expected ':'

The real problem is the space after target:, because it is a unicode non-breaking space instead of a normal ASCII space. They look identical, but to YAML they are different characters entirely. To make matters worse, the docker error message does not hint at a unicode problem at all.

Finding and fixing these issues is not really possible in an automated way, you will have to deal with them manually.

Manually debugging unicode problems

As a last resort, manually checking and editing unicode characters is the only way to really find and fix every possible issue. For simplicity, we only cover text that has already been converted to UTF-8 and does not contain broken byte sequences.

As a first check, you can list all suspicious characters with grep:

grep -nP '[^\x09\x0A\x20-\x7E]' file.txt

We use the word "suspicious" broadly here, flagging every character that is not printable ASCII, a space, tab or newline. This check will help you narrow down the lines to look at in the normal printable format and exclude the ones that contain unicode characters you allow, for example in strings.

To catch invisible or lookalike characters, you can use sed to display them as escape sequences instead:

grep -nP '[^\x09\x0A\x20-\x7E]' file.txt --color=never | sed -n 'l' | grep --color=always '\\'

Remember to force disable colored output on grep here or the result will be littered with escape sequence bytes. The output will contain escape sequences and mark the starting backslash in color for easier scanning. Escape sequences can either be of human-readable form where supported, for example \r or \a, falling back to three-digit octal representation like \304 for anything else. Multiple consecutive escape sequences often belong to a single unicode character.

If you are more familiar with hexadecimal, xxd can be a decent alternative:

xxd -g1 --color=always sample.txt

Given a sample file sample.txt:

Let's visit a café.

The output may look something like this:

00000000: 4c 65 74 27 73 20 76 69 73 69 74 c2 a0 61 20 63  Let's visit..a c
00000010: 61 66 c3 a9 2e 0a                                af....

Enabling coloring support is vital for xxd, because it is the only way to distinguish between real dots and dots replacing unprintable/unicode characters in the right column:

Characters in green are literal/printable ASCII characters, orange are ASCII control sequences like newlines or line feeds, and red are unicode sequences.

The screenshot clearly marks not only the é in Café, but also the non-breaking space after visit in red dots / red hex bytes.