Fix: Fix HTTPDigestAuth UTF-8 username/password encoding

The Bug

Repo: psf/requests Issue: #6102 Status: Fixed PR: https://github.com/psf/requests/pull/7571

Description: HTTPDigestAuth produces malformed Authorization headers when credentials contain non-ASCII characters passed as UTF-8 bytes.

Fix scope: 4 lines added in src/requests/auth.py

Root Cause

The bug lives in HTTPDigestAuth.build_digest_header() at src/requests/auth.py, line 166. When users pass credentials as bytes — a common workaround for non-ASCII usernames and passwords — the code stores them directly and then interpolates them into f-strings during digest hash computation and header construction.

Python’s f-string interpolation of a bytes object produces its repr, so b'Ond\xc5\x99ej' becomes the literal string "b'Ond\\xc5\\x99ej'" in the header. The resulting Authorization header looks like:

Digest username="b'Ond\xc5\x99ej'", realm="test", ...

This is doubly broken: the wire format is wrong (the server sees a literal b'...' prefix), and the HA1 hash is computed over the repr string rather than the actual UTF-8 decoded username, so even a lenient server would reject the response.

The deeper issue is an encoding assumption at the library boundary. The HTTPDigestAuth type overload accepts bytes | str for both username and password, but build_digest_header never decodes bytes to str before using them in string operations. This gap exists because the class hierarchy — HTTPDigestAuth inherits from HTTPBasicAuth — historically assumed credentials would always be strings. The introduction of bytes in the type union widened the contract without updating the internal handling.

Digest authentication is especially sensitive to encoding because RFC 7616 requires the server and client to agree on the charset for the username. When bytes arrive already encoded as UTF-8, the correct behavior is to decode them immediately so that subsequent string operations (hashing, header formatting) work on the logical characters, not the wire representation. The original HTTPBasicAuth code used latin1 encoding for str-to-bytes conversion, which fails for any codepoint outside the Latin-1 range — precisely the characters that motivated users to pass pre-encoded bytes in the first place.

Code Analysis

Before the fix (simplified from src/requests/auth.py:build_digest_header):

A1 = f"{self.username}:{realm}:{self.password}"
HA1 = hash_utf8(A1)

base = (
    f'username="{self.username}", realm="{realm}", nonce="{nonce}", '
    f'uri="{path}", response="{respdig}"'
)

When self.username is b'Ondřej', f-string interpolation yields "b'Ond\\xc5\\x99ej'" — the bytes repr, not the decoded text. The HA1 hash computes over the wrong string, and the header carries a syntactically invalid username.

After the fix (PR #7571, commit 1e8a4f3):

username = self.username
password = self.password
if isinstance(username, bytes):
    username = username.decode("utf-8")
if isinstance(password, bytes):
    password = password.decode("utf-8")

A1 = f"{username}:{realm}:{password}"

The fix decodes bytes to str at the top of the method, before any string operations. This is a minimal, targeted change — four lines that ensure the rest of the method operates on decoded strings without altering the constructor, the stored state, or any other code path.

The accompanying regression test constructs both str-based and bytes-based credentials with identical non-ASCII characters (Ondřej / heslíčko) and asserts they produce the same username="Ondřej" in the header output, confirming that bytes are transparently decoded as UTF-8.

The Fix

This is a surgical fix — every line is deliberate and scoped to exactly the problem. The decode happens early in build_digest_header so that all downstream consumers (HA1 hash computation, header template strings, and the final Authorization value) consistently work with str. No constructor changes, no type coercions elsewhere, no refactoring of the class hierarchy.

Pattern & Takeaways

Pattern: Bytes-vs-str confusion at I/O boundaries — the codebase accepted bytes in the type signature but treated the value as str internally, relying on f-string behavior that silently produced wrong output instead of raising an error.

Key insights:

1. Decode at the boundary. When a library accepts bytes | str for a textual field, bytes should be decoded to str immediately — ideally in the constructor or at the very first use site. Delaying decoding lets bytes leak into string operations where Python’s implicit repr conversion becomes a silent corruption vector.

2. f-string interpolation of bytes is a trap. Unlike str(a_bytes) which also produces the repr, an f-string looks natural and doesn’t signal that a conversion is happening. Any codebase that mixes bytes and str in f-strings should be audited for this pattern.

3. Type annotations widen the contract. Adding bytes to a type overload without updating the implementation creates a false sense of safety. The type checker says “bytes is fine” but the runtime produces garbage. Static analysis cannot catch silent repr corruption — only tests with actual non-ASCII bytes input will.

4. Digest auth encoding matters. RFC 7616 mandates charset handling for digest authentication. Libraries that ship HTTP auth implementations should encode credentials as UTF-8 by default, not latin1, to support the full range of Unicode usernames and passwords used globally.

Code review checklist: When reviewing polymorphic parameters, ask (1) Is every type variant actually exercised in tests? (2) Does the implementation handle each variant before any string formatting or hashing? (3) Are bytes decoded with the correct encoding (UTF-8, not latin1) at the earliest reasonable point?

Transfer Potential

Varies — the specific bytes-repr-in-fstring trap is Python-specific, but the underlying pattern (accepting multiple types at an interface but handling only one internally) is universal. Every codebase with polymorphic parameters should test every variant at the boundary. The minimal-change principle and boundary-condition thinking from this fix transfer to any language or library.

---

Auto-generated from PR #6102. View all patches on GitHub.

References

[1] psf/requests [2] #6102 [3] https://github.com/psf/requests/pull/7571