Start here: how long is "café"?#
TL;DRthe 30-second version
- Two separate layers get mixed up. A character set assigns every character a number called a code point (Unicode's job). An encoding decides how that number is written as bytes (UTF-8's job). Character ≠ code point ≠ byte.
- UTF-8 is the encoding that won the web. It uses 1 to 4 bytes per character, and the first 128 characters are one byte each, identical to old ASCII — so plain English text is unchanged.
- "String length" has four honest answers: bytes, code units, code points, and grapheme clusters (what a human sees as one character). For plain English they're equal; for é or an emoji they diverge.
- The same visible text can be two different byte sequences (é as one code point, or as "e" plus a combining accent). You fix that with normalization before you compare, hash, or store text.
- You cannot sort names by byte value. Correct alphabetical order is locale-specific — "ä" sorts near "a" in German but after "z" in Swedish — so sorting uses a separate rule called collation.
Type "café" into a program and ask for its length. Depending on the language and how the text was entered, you might get 4, 5, or 6 — and every one of those answers is defensible. That is unsettling. A four-character word should have length four. The reason it doesn't is that "length" is really four different questions wearing one name, and until you can tell them apart, text will keep surprising you.
The confusion has a single root: we treat a character and the bytes that store it as the same thing. For plain English on an old system they nearly were, so a generation of code assumed one character equals one byte. That assumption is now wrong for most of the text on Earth — names, currencies, accents, emoji. This page takes the two apart and keeps them apart, and once they're separate the surprises stop being surprises.
Two layers people mix up#
Start with the layer most people never separate out. There is the character itself — the letter "A", the euro sign "€", the grinning face — and there is the number we agree to call it by. Unicode is a giant table that hands every character a unique number, called a code point. The letter "A" is code point 65. We write code points in a fixed style: U+ followed by the value in hexadecimal (base 16). So "A" is U+0041, the euro sign is U+20AC, and the grinning face 😀 is U+1F600. A code point is just a number with a name. It says nothing yet about bytes.
The second layer is the encoding: the rule that turns a code point into actual bytes on disk or on the wire, and back again. This is the step that has choices. Code point U+0041 could be stored as a single byte, or as two bytes, or as four, depending on which encoding you pick — and the reader has to use the same encoding to get "A" back out. The character set names the character; the encoding stores it. Keeping these two jobs in separate boxes is the whole trick of this topic.
It helps to know why two layers exist at all, because it wasn't planned — it was cleaned up after a mess. Here is the short history, and each step is forced by the last one breaking.
- ASCII (1963): 7 bits per character, so 128 possible characters — enough for the English alphabet, digits, and punctuation. "A" was 65, and that fit in one byte with a bit to spare. Simple, and for English, done.
- The code-page mess: 128 characters can't hold accents, Cyrillic, Greek, Arabic, or Chinese. So everyone used the spare 128 values of a byte for their own local characters — a French code page, a Russian one, dozens of them. The same byte 200 meant a different character in each. Open a Russian file with a French code page and you got garbage, because the number no longer agreed on what it named.
- Unicode (1991 onward): one universal table for every character in every writing system, each with a single unshared code point, so byte 200 stops being ambiguous. Unicode reserves room for about 1.1 million code points and has assigned well over 150,000 of them — Latin, Han characters, emoji, and more.
- The remaining question: a table of 1.1 million numbers can't fit each number in one byte. So how do you write a code point as bytes? That is exactly the job Unicode left to the encodings — and UTF-8 is the answer that won.
How a code point becomes bytes: UTF-8#
UTF-8 is a variable-width encoding: a character takes between 1 and 4 bytes, and the encoding picks the smallest that fits. Small code points get one byte, larger ones get more. That single decision is why UTF-8 won, and it buys two things at once.
- It's backward-compatible with ASCII. Code points U+0000 to U+007F — the original 128 ASCII characters — are stored as a single byte with the exact same value ASCII used. So every ASCII file ever written is already valid UTF-8, and English text costs one byte per letter. No migration, no penalty.
- It's compact for the common case. A page of English is one byte per character; you only pay the extra bytes on the characters that actually need them (accents at two, most other scripts at three, emoji at four). You never pay for range you don't use.
Here is what real characters look like once UTF-8 has turned them into bytes. Byte values are shown in hexadecimal, the way the numbers topic writes them.
| Character | Code point | UTF-8 bytes | Byte count |
|---|---|---|---|
| A | U+0041 | 41 | 1 |
| é | U+00E9 | C3 A9 | 2 |
| € | U+20AC | E2 82 AC | 3 |
| 😀 | U+1F600 | F0 9F 98 80 | 4 |
Notice "A" is a single byte 0x41 — that's 65, the same number ASCII gave it. The euro sign takes three bytes; the grinning face takes four. The number of bytes is not the number of characters, and this table is the first place that becomes concrete.
Go deeperUnder the hood: the exact bit layout, and why UTF-8 never loses its place
UTF-8 uses the top bits of each byte as a length tag. A byte starting with 0 is a lone ASCII character. A byte starting with 11 is the first byte of a multi-byte character, and the count of leading 1s tells you how many bytes the character has. Every following byte starts with 10, marking it as a continuation. The x's are the bits of the actual code point, filled in from the low end.
| Code point range | Bytes | Bit pattern |
|---|---|---|
| U+0000 – U+007F | 1 | 0xxxxxxx |
| U+0080 – U+07FF | 2 | 110xxxxx 10xxxxxx |
| U+0800 – U+FFFF | 3 | 1110xxxx 10xxxxxx 10xxxxxx |
| U+10000 – U+10FFFF | 4 | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx |
This tagging makes UTF-8 self-synchronizing: because a leading byte (0… or 11…) can never be confused with a continuation byte (10…), you can drop into a byte stream at any point and find the start of the next character by skipping forward past any 10… bytes. If one byte is corrupted or lost, you resynchronize at the very next character instead of garbling the whole rest of the file. That property is a big part of why UTF-8 is robust on real networks, and it's defined in RFC 3629.
UTF-8 vs UTF-16 vs UTF-32#
UTF-8 is one encoding of Unicode, not the only one. Two others show up constantly, and knowing how they differ explains a lot of otherwise-baffling behavior in real languages and databases. They all encode the exact same Unicode code points — they just lay them out in bytes differently.
- UTF-8: 1 to 4 bytes per character, ASCII-compatible. The dominant encoding of the web, files, and network protocols. When in doubt, this is the answer.
- UTF-16: 2 bytes for most characters, 4 bytes for the rest. Not ASCII-compatible. It's the in-memory string format of JavaScript, Java, C#, and Windows — a historical choice from when 2 bytes seemed like enough for everything.
- UTF-32: a flat 4 bytes for every character, no exceptions. Simple to index — character number N is at byte 4N — but it wastes space (four bytes to store the letter "A"), so it's used inside programs for convenience, rarely for storage or transfer.
| UTF-8 | UTF-16 | UTF-32 | |
|---|---|---|---|
| Bytes per character | 1–4 | 2 or 4 | always 4 |
| ASCII-compatible | Yes | No | No |
| Size for English text | Smallest (1 byte each) | 2× (2 bytes each) | 4× (4 bytes each) |
| Fixed-width indexing | No | No (surrogate pairs) | Yes |
| Where you meet it | Web, files, APIs, everywhere | JS/Java/C#/Windows in memory | Some internal processing |
Go deeperUnder the hood: UTF-16 and surrogate pairs
UTF-16's unit is 16 bits, which covers code points up to U+FFFF directly. But Unicode goes up to U+10FFFF, so the characters above U+FFFF — including every emoji — don't fit in one 16-bit unit. UTF-16 encodes them as a surrogate pair: two 16-bit units drawn from a reserved block (U+D800–U+DFFF) that exists for exactly this purpose and holds no real characters.
The math: take the code point, subtract 0x10000, and you have a 20-bit number. The high 10 bits, added to 0xD800, form the first unit; the low 10 bits, added to 0xDC00, form the second. So 😀 (U+1F600) becomes the pair 0xD83D 0xDE00. This is why the grinning face is two code units in UTF-16 but four bytes in UTF-8 — and why a JavaScript string reports the emoji's length as 2. That surprising "2" is the surrogate pair leaking through.
So how long is a string? Four honest answers#
Now the opening puzzle resolves. "Length" is four different measurements, and they only agree for plain ASCII. Here they are, from the machine's view up to the human's.
- Bytes: how much storage the text takes in its encoding. Depends on the encoding — "café" is 5 bytes in UTF-8.
- Code units: the fixed-size pieces of the chosen encoding (8-bit for UTF-8, 16-bit for UTF-16). This is what most languages' built-in length returns, which is why a JS string counts an emoji as 2.
- Code points: the count of Unicode characters, ignoring how they're stored. "café" is 4 code points if é is stored as one, but 5 if é is stored as "e" plus a separate accent mark.
- Grapheme clusters: what a human perceives as a single character. This is almost always the answer a person means by "length." "café" is 4 grapheme clusters no matter how the é is stored.
The gap between code points and grapheme clusters is where text gets genuinely strange. A grapheme cluster can be built from several code points that combine visually. The clearest example is the family emoji 👨👩👧👦: it looks like one character, but it's actually four people-emoji joined by an invisible "zero-width joiner" code point between each. That's 7 code points, 25 bytes in UTF-8, 11 code units in UTF-16 — and exactly 1 grapheme cluster. One character to a human; a small paragraph to a machine.
| Text | Bytes (UTF-8) | Code points | Grapheme clusters |
|---|---|---|---|
| A | 1 | 1 | 1 |
| café (é as one code point) | 5 | 4 | 4 |
| café (é as e + accent) | 6 | 5 | 4 |
| 😀 | 4 | 1 | 1 |
| 👨👩👧👦 (family) | 25 | 7 | 1 |
PredictA form limits a "display name" to 10 characters, and enforces it by checking the string's built-in .length (which counts UTF-16 code units) ≤ 10. A user pastes a name made of 6 emoji. What happens, and what should the check have counted?
Hint: How many UTF-16 code units does one emoji above U+FFFF take?
The check rejects it even though it's only 6 characters a human sees. Each emoji above U+FFFF is a surrogate pair — 2 UTF-16 code units — so 6 emoji count as 12 code units, over the limit of 10. The user typed 6 characters and got told they typed 12. The check counted the wrong unit: a "10 characters" rule that a person will judge by eye should count grapheme clusters, the thing a human perceives as one character. Counting code units (the built-in length) is measuring storage pieces, not characters. This exact mismatch is why naive length limits misbehave on names with accents and emoji, and why serious input validation uses a grapheme-aware library instead of the built-in length.
Which encoding should you use?
For almost everything you store, send, or serve, the answer is UTF-8, and picking anything else needs a reason. But the trade is real, so here's when each earns its place.
- Use UTF-8 for files, databases, JSON, HTML, and every network protocol. It's ASCII-compatible, it's the smallest for English-heavy and markup-heavy text, and it's what the rest of the world expects — an HTTP response says so with a charset like "Content-Type: text/html; charset=utf-8". Choosing anything else on the wire is asking for interoperability pain.
- You'll meet UTF-16 whether you choose it or not, because it's the in-memory string format of JavaScript, Java, C#, and the Windows API. That's a historical choice you inherit, not one you'd make today; the friction is the surrogate-pair length surprises above. Many systems keep strings in UTF-16 in memory but still read and write UTF-8 on disk and on the network.
- UTF-32 trades space for simplicity: fixed 4-byte width means character N is at a known offset, handy for some text-processing internals. But it's 4× the size of UTF-8 for English, so it almost never leaves the program that uses it.
- One caveat for CJK text (Chinese, Japanese, Korean): those characters are 3 bytes in UTF-8 but 2 in UTF-16, so for text that's overwhelmingly CJK, UTF-16 can be smaller. That's the one place the "UTF-8 is always smallest" rule of thumb bends.
Same text, different bytes: normalization
Here's a problem that bites in production. The character é can be written two ways in Unicode: as the single code point U+00E9, or as a plain "e" (U+0065) followed by a "combining acute accent" (U+0301) that renders on top of it. Both look identical on screen — é — but they are different code points and therefore different bytes. A byte-for-byte comparison says "café" ≠ "café", which is nonsense to the human who typed the same word both times.
Normalization is the fix: a defined procedure that rewrites text into one canonical form so equal-looking text becomes equal bytes. The two forms you'll see named are NFC (composed — prefer the single é, U+00E9) and NFD (decomposed — prefer e + accent). NFC is the usual choice because it's more compact and is what most input already uses. The rules are specified in the Unicode standard's normalization annex (UAX #15).
Sorting names: you can't sort by byte value
Sorting is where the character-versus-byte gap does the most damage, because the wrong answer looks plausible. The obvious way to sort strings is to compare their bytes (or their code points) in order — it's built into every language and it's fast. For plain ASCII it even works. But it puts every uppercase letter before every lowercase one (because "Z" is code point 90 and "a" is 97), so "Zebra" sorts before "apple". And it scatters accented letters far from their plain cousins, because "é" sits at a code point nowhere near "e". A byte sort is not alphabetical order; it just resembles it for the easy cases.
Worse, there is no single correct order to hard-code, because correct alphabetical order depends on the language. The letter "ä" is a good example: in German it sorts right next to "a" (in the phonebook convention, as if it were "ae"), but in Swedish it's a distinct letter that comes after "z", at the very end of the alphabet. The same list of names has two different correct orders depending on whether a German or a Swedish reader is looking at it. No byte comparison can capture that, because the answer isn't in the bytes — it's in the locale.
The real tool is collation: a locale-aware ordering that knows each language's rules. The Unicode Collation Algorithm (defined in UTS #10) turns each string into a sort key — a sequence of weights that handle case, accents, and letter identity in the right priority — and you compare those keys instead of raw bytes. Databases expose this as a collation setting on a column, and it's why the same query can return a different order on two servers configured for different locales. The rule to remember: user-facing sorting is a locale decision, never a byte comparison.
Where this shows up
- The web runs on UTF-8. It's the default for HTML, and the overwhelming majority of web pages declare it — a page or API states it with a charset ("charset=utf-8") so the browser decodes the bytes correctly. Get that header wrong and accented text turns to garbage on screen.
- JavaScript, Java, and C# hold strings as UTF-16 in memory, which is why their built-in string length counts UTF-16 code units and reports an emoji as length 2. Every "why is my emoji length 2" bug traces back to this.
- Databases carry a character set and a collation. PostgreSQL and MySQL let you set them per column; the collation decides sort order and equality for that text, so a name column sorted "correctly" in one locale is sorted differently in another by design.
- Filenames are a quiet trap: macOS has historically stored filenames in a decomposed (NFD-like) form while Linux stores whatever bytes you give it, so a file named "café" created on one can fail to match the same name typed on the other until both are normalized.
- Usernames, emails, and identifiers get normalized before comparison specifically to stop two visually-identical values from being treated as different accounts — the same normalization step that protects password hashing.
Pitfalls & gotchas
Is "Unicode" an encoding?
No — this is the single most common mix-up. Unicode is the character set: the table that assigns every character a code point (a number). UTF-8, UTF-16, and UTF-32 are the encodings: the rules for turning those code points into bytes. "Save as Unicode" in some old software actually meant "save as UTF-16," which is exactly the sloppiness that keeps the confusion alive. Say which encoding you mean.
What is mojibake — the garbled é‚ text I sometimes see?
It's what you get when text is written with one encoding and read with another. Bytes that were valid UTF-8 for "é" get misread by a program assuming an old single-byte code page, so one character turns into two or three wrong ones. The bytes are fine; the reader used the wrong decoding rule. The fix is to make sure the encoding is declared and agreed on both ends — which is the whole point of a charset header.
Why did my string reverse turn 😀 into broken characters?
Because a naive reverse flips code units or bytes, not characters. Reverse a UTF-16 string by code units and you swap the two halves of an emoji's surrogate pair, producing an invalid sequence that renders as a replacement box. Reverse UTF-8 by bytes and you scramble the multi-byte characters the same way. Correct text reversal has to operate on grapheme clusters, not the storage pieces underneath them.
How can two names that look identical be a security problem?
Two ways, both from the layers in this page. First, homoglyphs: different code points that look the same, like the Latin "a" (U+0061) and the Cyrillic "а" (U+0430) — an attacker registers a lookalike domain or username the eye can't distinguish. Second, normalization attacks: if one part of a system normalizes text and another doesn't, an attacker can craft input that passes a check in one form and does damage in another. Both are why identity-critical text is normalized to one form and often restricted to a safe set of characters before it's trusted.
QuizYou're building login. A user sets their passphrase on their phone and it hashes fine; on their laptop the same typed passphrase is rejected. The characters look identical on both. What's the most likely cause?
- The two devices use different encodings, so UTF-8 vs UTF-16 changed the password.
- An accented or composed character was entered in different Unicode forms (composed vs decomposed), so the un-normalized bytes differ and hash differently.
- The hash function is non-deterministic.
- The laptop's byte order is reversed.
Show answer
An accented or composed character was entered in different Unicode forms (composed vs decomposed), so the un-normalized bytes differ and hash differently. — The passphrase contains a character (say é, or a non-Latin script) that Unicode can represent two ways — one code point, or a base letter plus a combining mark. The phone and laptop keyboards produced different forms, so the byte sequences differ even though the text looks identical, and a hash of different bytes is a different hash. The fix is to normalize the passphrase to a single canonical form (NFC) before hashing, on every device — the exact reason password handling normalizes before it hashes (/hashing-passwords/sim). It's not the encoding (both would decode to the same code points) and the hash is deterministic; the difference is upstream, in which code points were entered.
In an interview
Text encoding shows up as a follow-up ('how would you store user names from any country?') or as a bug-hunt ('why is this emoji breaking?'). The signal you're being tested for is whether you separate the layers instead of hand-waving 'just use Unicode.'
- Lead with the two layers: Unicode is the character set (characters to code points), an encoding like UTF-8 is code points to bytes. Naming that split cleanly is most of the battle.
- Default to UTF-8 for storage and transport, and say why: ASCII-compatible, compact, universal. Note that UTF-16 is an in-memory format you inherit in JS/Java, not something you'd choose for the wire.
- When asked about string length, ask back 'length in what — bytes, code points, or characters a human sees?' and mention grapheme clusters for the emoji case. That single question signals you know the trap.
- For any compare/sort/dedup on text, name the two rules: normalize (NFC) before comparing or hashing so equal-looking text is equal, and collate (locale-aware) for sorting, because you can't sort names by byte value.
- The trap to avoid: claiming one character is one byte, or that reversing/truncating a string is safe on bytes. Both break the moment text leaves plain English, and interviewers reach for exactly that case.
References
- The Absolute Minimum Every Software Developer Must Know About Unicode (Joel Spolsky) — The classic essay that first made the character-set-vs-encoding split click for a generation of programmers.
- The Unicode Standard — The source of truth: code points, planes, and the character database.
- RFC 3629 — UTF-8, a transformation format of ISO 10646 — The formal definition of UTF-8's byte layout.
- UAX #15 — Unicode Normalization Forms — NFC / NFD / NFKC / NFKD, and when each is canonical.
- UTS #10 — Unicode Collation Algorithm — How locale-aware sorting turns strings into comparable sort keys.