From Python 3.15, open() and other text I/O use UTF-8 by default on every operating system, whatever the computer’s locale settings. Before 3.15, Python used the locale encoding when you didn’t pass encoding=, which on many Windows machines meant cp1252. That’s the source of countless UnicodeDecodeError: 'charmap' codec can't decode byte errors. The change only affects code that doesn’t specify an encoding, and the best fix, on any version, is still to write encoding="utf-8" yourself.
This is PEP 686, and it shipped in Python 3.15.0 on 9 October 2026. The What’s New in Python 3.15 page says I/O without an explicit encoding, “for example, open('flying-circus.txt'), will use UTF-8.” I tested the examples below on Python 3.15.0rc3 and Python 3.14.8 on Linux, and pasted the real output. Where Windows behaves differently, I say so and reproduce the Windows error by passing cp1252 explicitly.
Python has a feature called UTF-8 Mode, added back in Python 3.7. When it’s on, Python ignores the locale and uses UTF-8 for files, standard input and output, and the file system encoding. Until now it was off by default (except under the minimal C/POSIX locale on Unix). Python 3.15 turns it on by default.
I checked on 3.15:
import sys, locale
print("utf8_mode:", sys.flags.utf8_mode)
print("preferred:", locale.getpreferredencoding(False))
with open("cafe.txt", "w") as f:
f.write("café £5 — ✓\n")
print("default open encoding:", open("cafe.txt").encoding)
print(open("cafe.txt").read().strip())
utf8_mode: 1
preferred: utf-8
default open encoding: utf-8
café £5 — ✓
No encoding= anywhere, and accents, the pound sign, an em dash and a tick mark all round-trip correctly. Running the same script with LC_ALL=C, a minimal locale, gave identical output on 3.15.
If the words in error messages are confusing, here’s a quick glossary:
é or £.é is one byte (0xE9) in cp1252 but two bytes (0xC3 0xA9) in UTF-8.cp1252, which only covers around 256 characters.café, produced when bytes are decoded with the wrong encoding.The key idea: a file doesn’t record which encoding it uses. The program reading it has to know or guess, and before 3.15, Python’s guess depended on your computer.
cp1252. Code that opened files without an encoding read them as cp1252 before 3.15 and as UTF-8 from 3.15.The practical result: a script that reads a UTF-8 file without an encoding now works the same on your Windows laptop and your Linux server. A script that relied on reading cp1252 files without saying so can break.
Most people find this topic by googling an error message. There are two you’re likely to see, and they’re mirror images of each other.
This is the classic Windows error before 3.15. It happens when a UTF-8 file is read with a Windows code page. I reproduced it by passing cp1252 explicitly:
open("names.txt", "w", encoding="utf-8").write("José, Zoë, Ágnes\n")
try:
print(open("names.txt", encoding="cp1252").read())
except UnicodeDecodeError as e:
print("UnicodeDecodeError:", e)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 14: character maps to <undefined>
Á is stored in UTF-8 as two bytes, 0xC3 0x81, and 0x81 doesn’t map to anything in cp1252, so decoding fails. Even when it doesn’t fail, you get mojibake, garbled text like this:
open("cafe2.txt", "w", encoding="utf-8").write("café\n")
print(open("cafe2.txt", encoding="cp1252").read().strip())
café
On Python 3.15, open("names.txt") without an encoding reads UTF-8 by default, so this error disappears for UTF-8 files.
The opposite problem is what 3.15 can introduce. Here’s a file saved in cp1252, for example by old Windows software or an Excel “CSV” export, read with the new default:
with open("latin.txt", "wb") as f:
f.write("café £5\n".encode("cp1252"))
try:
open("latin.txt").read()
except UnicodeDecodeError as e:
print("UnicodeDecodeError:", e)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte
In cp1252, é is the single byte 0xE9, which isn’t valid UTF-8 on its own. The fix is to say what the file really is:
print(open("latin.txt", encoding="cp1252").read().strip())
café £5
The 3.15 documentation itself says: “For best compatibility between versions of Python, ensure that an explicit encoding argument is always provided.” For files you create, use UTF-8:
with open("data.txt", "w", encoding="utf-8") as f:
f.write("café\n")
This works identically on 3.8 and 3.15, on Windows and Linux. The same applies to pathlib: Path("data.txt").read_text(encoding="utf-8").
If a file came from an old system, find out its encoding (the program that exported it, or a tool such as file -i on Linux and macOS) and pass it: encoding="cp1252", encoding="latin-1" and so on.
When a file has a few bad bytes and you can live with replacements, use the errors argument:
print(open("latin.txt", errors="replace").read().strip())
caf� �5
The � replacement character shows where data was lost. Use this for logs or quick inspections, never for data you’ll save back, because the original characters are gone.
Files saved by Excel and some Windows editors start with a byte order mark (BOM). Read as plain UTF-8, it shows up as an invisible \ufeff at the start of your first line, which breaks CSV headers:
with open("bom.csv", "wb") as f:
f.write("\ufeffname,city\nMahir,Dhaka\n".encode("utf-8"))
print(repr(open("bom.csv").readline()), repr(open("bom.csv", encoding="utf-8-sig").readline()))
'\ufeffname,city\n' 'name,city\n'
With encoding="utf-8-sig", Python removes the BOM if it’s there and reads normally if it isn’t. That’s my default for CSV files from spreadsheets; it pairs well with the techniques in my post on reading a CSV file in Python without pandas.
A few one-liners tell you what your interpreter will do. io.text_encoding(None) is the most direct, because it returns the encoding open() uses when you don’t pass one:
import sys, io
print(sys.getfilesystemencoding(), sys.stdout.encoding, io.text_encoding(None))
utf-8 utf-8 utf-8
That was Python 3.15. On Python 3.14, io.text_encoding(None) returned locale in my test, meaning “use the locale encoding”, which is exactly the default that 3.15 changes. sys.flags.utf8_mode (shown earlier) is the other quick check: 1 means UTF-8 Mode is on.
If you keep receiving cp1252 files, convert them to UTF-8 once and stop worrying about it. With pathlib that’s two lines:
import pathlib
text = pathlib.Path("latin.txt").read_text(encoding="cp1252")
pathlib.Path("latin-utf8.txt").write_text(text, encoding="utf-8")
print(pathlib.Path("latin-utf8.txt").read_bytes())
print(open("latin-utf8.txt").read().strip())
b'caf\xc3\xa9 \xc2\xa35\n'
café £5
The raw bytes show é is now two bytes (\xc3\xa9) and £ is \xc2\xa3: proper UTF-8. Reading it with the 3.15 default works with no encoding= at all. If you want to check that a path exists before converting it, my guide to checking if a file exists in Python covers the pathlib way.
When files arrive in a mix of encodings, try UTF-8 first and fall back to cp1252:
def read_text_any(path, encodings=("utf-8-sig", "cp1252")):
data = pathlib.Path(path).read_bytes()
for enc in encodings:
try:
return data.decode(enc), enc
except UnicodeDecodeError:
pass
raise ValueError(f"could not decode {path}")
print(read_text_any("latin.txt"), read_text_any("bom.csv"))
('café £5\n', 'cp1252') ('name,city\nMahir,Dhaka\n', 'utf-8-sig')
Order matters: UTF-8 is strict, so if it decodes without error the file is almost certainly UTF-8. cp1252 accepts nearly any bytes, so it must come last, or it would “succeed” on UTF-8 files and give you mojibake.
If your code genuinely needs the computer’s locale encoding, say so explicitly with the special value encoding="locale", which has been supported since Python 3.10:
print(open("cafe.txt", encoding="locale").encoding)
On my Linux test box this printed UTF-8, because the locale was UTF-8. Under LC_ALL=C it printed ANSI_X3.4-1968, which is the formal name for ASCII. On a typical Windows PC it would be a code page such as cp1252. The point is that the choice is now visible in the code instead of hidden in a default.
According to the 3.15 notes, you can restore the old behaviour for a whole program with the PYTHONUTF8=0 environment variable or the -X utf8=0 command-line option:
$ python3.15 -X utf8=0 -c "import sys; print(sys.flags.utf8_mode)"
0
With UTF-8 Mode off under an ASCII-only C locale, even writing a file failed in my test, because ASCII can’t represent é:
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 3: ordinal not in range(128)
That’s exactly the kind of fragile behaviour the change was designed to remove. Treat PYTHONUTF8=0 as a temporary workaround while you add explicit encodings, not a permanent setting.
Python 3.10 added an opt-in warning that flags every place that relies on the default encoding. Turn it on with -X warn_default_encoding (or PYTHONWARNDEFAULTENCODING=1):
# w.py
open("cafe.txt").read()
$ python3.15 -X warn_default_encoding w.py
w.py:1: EncodingWarning: 'encoding' argument not specified
open("cafe.txt").read()
It worked the same on 3.14. Run your test suite with this option, fix each warning by adding encoding=, and your code will behave identically before and after the upgrade.
"rb", "wb") is untouched. Bytes are bytes.json.load() reads from whatever file object you pass, so open the file with encoding="utf-8". The JSON standard requires UTF-8 for data exchanged between systems anyway. My guide to writing JSON to a file with pretty print shows the ensure_ascii=False option that keeps accented characters readable.utf8mb4). Python’s default file encoding doesn’t change those.subprocess with text=True uses the default encoding to decode a command’s output, so it now defaults to UTF-8 too. Pass encoding= if a Windows command prints in another code page.The 3.15 notes describe UTF-8 as “a de facto standard for representing text, including nearly every webpage on the internet, many common file formats, programming languages, and more.” Most files beginners deal with, from JSON and CSV to Markdown and source code, are already UTF-8. Defaulting to anything else meant the same script could work on one computer and fail on another, purely because of regional settings. Making UTF-8 the default removes that surprise.
-X warn_default_encoding on your current Python.encoding="utf-8" (or "utf-8-sig" for spreadsheet CSVs) to every warned open() and read_text().cp1252 on Windows.PYTHONUTF8=0 briefly while you fix it.Python 3.15 also brings new tools such as frozendict and lazy imports, so upgrading has plenty of upside once your encodings are explicit.
From Python 3.15, yes: text I/O without an explicit encoding uses UTF-8 on all platforms (PEP 686). Before 3.15, Python used the locale encoding, which is often cp1252 on Windows.
It means a UTF-8 file was read with a Windows code page. Open the file with encoding="utf-8", for example open("file.txt", encoding="utf-8"). On Python 3.15 that’s also the default.
The file isn’t UTF-8; it’s probably in a legacy encoding such as cp1252 or Latin-1. Open it with its real encoding, for example encoding="cp1252", or convert it to UTF-8 once and keep it that way.
Set the environment variable PYTHONUTF8=0 or run Python with -X utf8=0. That restores the locale-based default. It’s best used temporarily while you add explicit encodings.
It tells open() to use the current locale encoding, the pre-3.15 default. It’s available since Python 3.10 and makes the choice explicit in your code.
Yes. The Python documentation recommends always passing an explicit encoding for compatibility between versions. It also makes your code’s intent obvious to readers.
It’s UTF-8 that automatically removes a byte order mark (BOM) at the start of a file if one is present. Use it for CSV files saved by Excel to avoid a stray \ufeff in your first column name.
// note
This is a learning note from studying the web. It is one small topic, written so I can remember it. It is not a course and not a claim that I have finished the subject.
If a sentence is wrong, say so from the contact page and name this title. Drafts never appear here. Related notes, when they exist, are other published posts, and the same sample rule applies to each of them.
Python 3.15 adds frozendict, a built-in immutable, hashable dictionary. Tested examples: creating one, why you can’t change it, using it as a dict key or in a set, merging with |, isinstance checks and a fallback for older Python.
enumerate(items, start=1) gives you a counter that starts at 1 alongside each item. Tested examples for lists, dicts, zip, reversed order, file line numbers and comprehensions, plus why it beats range(len()) and the list.index() trap.
Python 3.15 adds the lazy soft keyword: lazy import json loads the module only when you first use it. Tested examples of the startup saving, where lazy isn’t allowed, how errors appear, -X lazy_imports=all, filters and __lazy_modules__.
msconfig opens Windows System Configuration, a built-in troubleshooting tool for controlling how Windows starts. What each tab does, how to do a clean boot, diagnostic startup and safe boot, how to escape a safe mode loop, and why it matters for security.