Text Encoding Explained: UTF-8, ASCII, Latin-1, and Unicode
Demystify character encoding — why text appears as garbled symbols, how UTF-8 became the universal standard, and practical strategies for handling encoding issues in your workflow.
Key Takeaways
- Every text file is a sequence of bytes.
- UTF-8 uses variable-length encoding: ASCII characters (U+0000 to U+007F) use one byte, identical to ASCII.
- Double encoding**: UTF-8 bytes interpreted as Latin-1, then re-encoded to UTF-8 → `café` (two bytes for é become four)
- Use UTF-8 everywhere — source files, databases, APIs, and file storage.
Word Counter
Count words, characters, sentences, and paragraphs.
Why Encoding Matters
Every text file is a sequence of bytes. Character encoding defines the mapping between bytes and characters. When the wrong encoding is assumed, text appears as garbled symbols (mojibake) — café becomes café when UTF-8 is decoded as Latin-1. Understanding encoding prevents data corruption.
The Encoding Timeline
| Era | Encoding | Characters | Bytes/Char |
|---|---|---|---|
| 1960s | ASCII | 128 | 1 |
| 1980s | Latin-1 (ISO 8859-1) | 256 | 1 |
| 1990s | Windows-1252 | 256 (extended) | 1 |
| 2000s | UTF-8 | 1,114,112 (all Unicode) | 1-4 |
| Current | UTF-8 dominates (>98% of web) | — | — |
How UTF-8 Works
UTF-8 uses variable-length encoding: ASCII characters (U+0000 to U+007F) use one byte, identical to ASCII. Latin characters use two bytes. CJK characters and emoji use three to four bytes. This backward compatibility with ASCII is why UTF-8 won — existing ASCII text is valid UTF-8 without modification.
Common Encoding Problems
- Double encoding: UTF-8 bytes interpreted as Latin-1, then re-encoded to UTF-8 →
café(two bytes for é become four) - BOM insertion: Some tools prepend a Byte Order Mark (EF BB BF) that other tools do not expect
- Database charset mismatch: Connection charset differs from table charset
Best Practice
Use UTF-8 everywhere — source files, databases, APIs, and file storage. Specify encoding explicitly (charset=utf-8) rather than relying on platform defaults. Detect and convert encodings with the Peasy encoding tools.