charset-normalizer: Detect Text Encoding in Python (With Real Tests)

Here's a German sentence that a Windows program saved as cp1252 (Windows-1252). Watch what the usual one-liners do with it:

>>> text = "Grüße aus München — schöne Größe für die Straße."
>>> raw  = text.encode("cp1252")        # how a Windows app stored it

>>> raw.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 2

>>> raw.decode("utf-8", errors="replace")
'Gr\ufffd\ufffde aus M\ufffdnchen \ufffd sch\ufffdne ...'   # mojibake, data lost

>>> raw.decode("latin-1")                # "works" — but it's WRONG
'Grüße aus München \x97 schöne Größe für die Straße.'
>>> raw.decode("latin-1") == text
False                                    # the — turned into an invisible \x97

1. `decode('utf-8')` — raises UnicodeDecodeError and stops your program. At least it's honest.
2. `errors='replace'` — never crashes, but every non-ASCII byte becomes \ufffd (the � replacement char). The data is gone.
3. `decode('latin-1')` — the sneaky one. latin-1 maps *every* byte to *some* character, so it always succeeds and never raises. But it's not the right encoding: the em-dash byte 0x97 decoded to an invisible C1 control character, so the string silently differs from the original.

charset-normalizer looks at the bytes, ranks the encodings that could have produced them, and gives you back the best match already decoded. Same raw bytes as above:

>>> from charset_normalizer import from_bytes
>>> result = from_bytes(raw).best()
>>> result.encoding
'cp1250'
>>> result.language
'German'
>>> str(result) == text
True                       # exact original text, no data loss

Two things worth noticing. First, str(result) is the decoded text — you don't decode again yourself. Second, it guessed cp1250, not the cp1252 I encoded with — yet the text came back identical. That's the key mental model for encoding detection:

For files there's a matching helper, from_path, so you never have to read bytes and guess yourself:

from charset_normalizer import from_path

result = from_path("mystery.csv").best()
print(result.encoding)   # e.g. 'utf_8', 'cp1252', 'shift_jis'
text = str(result)       # decoded contents, ready to use

if result is None:
    print("couldn't decode as any known encoding")

Here's the part most people miss. You often don't need to install anything, because charset-normalizer is a mandatory dependency of requests — the most-installed package in the Python world. If requests is in your environment, so is charset-normalizer:

>>> from importlib.metadata import requires
>>> [r for r in requires("requests") if "charset" in r]
['charset_normalizer<4,>=2']

That's not an accident. requests uses it internally for response.apparent_encoding — when a web server doesn't declare a charset, requests calls charset-normalizer to figure out how to decode the page. So import charset_normalizer works out of the box in a huge number of projects, no pip install required.

And because the library is pure Python with zero C extensions (I checked: Requires: is empty, License is MIT), it runs literally anywhere Python does — including in the browser. You can paste this straight into our Online Python Compiler, micropip install charset-normalizer, and detect encodings client-side with nothing uploaded.

If you're in a locked-down environment where you genuinely can't add or import the package, you can approximate detection with the standard library: sniff for a BOM, then try a short, ordered list of candidate encodings and take the first that decodes cleanly. The trick is ordering — latin-1/cp1252 decode anything, so they must come last.

import codecs

BOMS = [(codecs.BOM_UTF8, "utf-8-sig"),
        (codecs.BOM_UTF16_LE, "utf-16"),
        (codecs.BOM_UTF16_BE, "utf-16")]
# strict encodings first; latin-1/cp1252 accept ANY byte, so keep them last
CANDIDATES = ["utf-8", "shift_jis", "gbk", "cp1252", "latin-1"]

def detect(raw: bytes):
    for bom, enc in BOMS:
        if raw.startswith(bom):
            return enc, raw.decode(enc)
    for enc in CANDIDATES:
        try:
            return enc, raw.decode(enc)   # strict: raises on bad bytes
        except UnicodeDecodeError:
            continue
    return None, None

Does it work? I ran both approaches against the same 33 samples — 7 encodings (UTF-8, UTF-8-BOM, latin-1, cp1252, UTF-16, Shift-JIS, GBK) across English, French, German, Russian, Japanese, Chinese, and emoji text. Real results:

That table looks like the stdlib loop *wins* — and I want to be honest about why it doesn't in the ways that matter:

• \* The "label" column is misleading. charset-normalizer's mismatches were valid supersets that decode identically (pure-English samples detected as ascii instead of utf-8; Shift-JIS as cp932). Its only two *real* text failures were 52-byte French snippets where it picked cp1250 over cp1252 — short input is genuinely the hardest case for statistical detection.
• \*\* The stdlib 100% is on easy mode. My samples are short, clean, and — crucially — I *knew the candidate encodings in advance* and hand-ordered them. Feed the loop an encoding you didn't list, and it silently falls through to latin-1 and returns confident garbage.
• You lose the metadata. The stdlib loop gives you a decoded string and nothing else. charset-normalizer also told me the language (German, Japanese…) and a confidence ranking of alternatives — which is exactly what you need when the guess is wrong and you want to know how much to trust it.

The classic detector was chardet, and you'll still see it in older tutorials. requests switched to charset-normalizer in 2021 (v2.26) mainly over licensing — chardet is LGPL, and requests wanted an MIT/BSD-friendly, permissively-licensed detector it could bundle. charset-normalizer is MIT, actively maintained, pure-Python, and its API (from_bytes(...).best()) hands you the decoded text directly instead of just a {encoding, confidence} dict. For new code, prefer charset-normalizer.

It also ships a command-line tool, normalizer, so you can inspect a file without writing any code. It prints structured JSON — here's the real output on my cp1252 German file (trimmed):

$ normalizer mystery.txt
{
    "path": ".../mystery.txt",
    "encoding": "cp1250",
    "alternative_encodings": ["cp1252", "cp1254", "cp1257", "cp1258"],
    "language": "German",
    "alphabets": ["Basic Latin", "General Punctuation", "Latin-1 Supplement"],
    "has_sig_or_bom": false
}

Notice it lists alternative_encodings — cp1252 is right there as a candidate. That's the honest way to present detection: a ranked set, because on ambiguous bytes there's often no single "true" answer, only encodings that happen to produce the same characters.