🔤 Python Information Lab — Unicode and UTF-8 🐍
Posted on: November 19, 2026
Description:
Why does len("café") return 4, while its UTF-8 encoding takes 5 bytes? The two expressions count different things.
Unicode text and encoded bytes
Unicode assigns code points to characters. UTF-8 represents Unicode scalar values using one to four bytes. Python str is a sequence of Unicode code points, not simply a UTF-8 byte array. Python Unicode HOWTO
| Text | Code point | Python length | UTF-8 bytes | Hex bytes |
|---|---|---|---|---|
| A | U+0041 | 1 | 1 | 41 |
| é | U+00E9 | 1 | 2 | C3 A9 |
| 中 | U+4E2D | 1 | 3 | E4 B8 AD |
| 🐍 | U+1F40D | 1 | 4 | F0 9F 90 8D |
The notebook executes these examples and checks that decoding restores the original text.
Count the word, then its bytes
text = "café"
payload = text.encode("utf-8")
print(len(text)) # 4 code points
print(len(payload)) # 5 bytes
print(payload.hex(" ")) # 63 61 66 c3 a9
print(payload.decode("utf-8")) # café
c, a and f use one byte each. Precomposed é uses two. This measures the encoded payload, not Python's total object-memory usage.
One visible letter can contain two code points
composed = "\u00e9" # é
combined = "e\u0301" # e + combining acute accent
print(len(composed), len(combined)) # 1 2
print(composed == combined) # False
The strings normally look alike. Their UTF-8 encodings take two and three bytes respectively. Python counts their code points, rather than visible letters or grapheme clusters. Emoji sequences can also contain multiple code points.
Normalize when the application requires it
import unicodedata
normalized = unicodedata.normalize("NFC", combined)
print(composed == normalized) # True
NFC converts these canonically equivalent spellings to the same form. It is not a general “one visible character equals one code point” fix, nor does it make all look-alike text equal. Python normalization documentation
Byte slicing can break decoding
partial = "é".encode("utf-8")[:1] # only C3 remains
try:
partial.decode("utf-8")
except UnicodeDecodeError:
print("Incomplete UTF-8 sequence")
The original C3 A9 sequence needs both bytes. Arbitrary truncation may leave invalid UTF-8. Slicing a string first respects code-point boundaries, but may still split a user-perceived character.
Choose the right measurement
- Code points:
len(text). - UTF-8 payload bytes:
len(text.encode("utf-8")). - User-perceived characters: a suitable grapheme-segmentation method for the application.
Specify the encoding at input/output boundaries and define what your length limit means. A 100-byte limit is not a 100-character limit.
No comments yet. Be the first to comment!