AW Dev Rethought

✂️ Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away - Antoine de Saint-Exupéry

🔤 Python Information Lab — Unicode and UTF-8 🐍


Description:

Why does len("café") return 4, while its UTF-8 encoding takes 5 bytes? The two expressions count different things.


Unicode text and encoded bytes

Unicode assigns code points to characters. UTF-8 represents Unicode scalar values using one to four bytes. Python str is a sequence of Unicode code points, not simply a UTF-8 byte array. Python Unicode HOWTO

Text Code point Python length UTF-8 bytes Hex bytes
A U+0041 1 1 41
é U+00E9 1 2 C3 A9
中 U+4E2D 1 3 E4 B8 AD
🐍 U+1F40D 1 4 F0 9F 90 8D

The notebook executes these examples and checks that decoding restores the original text.


Count the word, then its bytes

text = "café"
payload = text.encode("utf-8")
print(len(text))                 # 4 code points
print(len(payload))              # 5 bytes
print(payload.hex(" "))          # 63 61 66 c3 a9
print(payload.decode("utf-8"))    # café

c, a and f use one byte each. Precomposed é uses two. This measures the encoded payload, not Python's total object-memory usage.


One visible letter can contain two code points

composed = "\u00e9"       # é
combined = "e\u0301"      # e + combining acute accent
print(len(composed), len(combined))  # 1 2
print(composed == combined)          # False

The strings normally look alike. Their UTF-8 encodings take two and three bytes respectively. Python counts their code points, rather than visible letters or grapheme clusters. Emoji sequences can also contain multiple code points.


Normalize when the application requires it

import unicodedata
normalized = unicodedata.normalize("NFC", combined)
print(composed == normalized)  # True

NFC converts these canonically equivalent spellings to the same form. It is not a general “one visible character equals one code point” fix, nor does it make all look-alike text equal. Python normalization documentation


Byte slicing can break decoding

partial = "é".encode("utf-8")[:1]  # only C3 remains
try:
    partial.decode("utf-8")
except UnicodeDecodeError:
    print("Incomplete UTF-8 sequence")

The original C3 A9 sequence needs both bytes. Arbitrary truncation may leave invalid UTF-8. Slicing a string first respects code-point boundaries, but may still split a user-perceived character.


Choose the right measurement

  • Code points: len(text).
  • UTF-8 payload bytes: len(text.encode("utf-8")).
  • User-perceived characters: a suitable grapheme-segmentation method for the application.

Specify the encoding at input/output boundaries and define what your length limit means. A 100-byte limit is not a 100-character limit.


Link copied!

Comments

Add Your Comment

Comment Added!