Regex Surprises: When gitleaks's Pattern Matching Breaks

The bottom line: Regular expressions with optional groups and alternative branches produce unexpected matches on edge case inputs, causing downstream failures. Always test regex against empty and minimal inputs.

---

The Problem

gitleaks/gitleaks issue #2121 exposes a subtle edge case in how regex handles boundary conditions with optional groups. The fix is only 1 line, but the pattern behind it applies across projects.

PR: https://github.com/gitleaks/gitleaks/pull/2163

Status: Submitted (awaiting review)

What Goes Wrong

Regular expressions with optional groups and alternatives behave unexpectedly when input matches multiple branches. The regex engine backtracks and may match unintended branches.

import re

# Edge case: optional group matches differently than expected
pattern = r'(foo)?(bar)?'
# Matches 'foobar', 'foo', 'bar', AND empty string!
result = re.match(pattern, '')  # Matches: both groups omitted

Common failure patterns to watch for:

• Optional groups — A group like (foo)? makes its content optional, but the overall match can succeed on empty input when all groups are optional
• Alternative branches — Patterns with | may match the wrong branch when backtracking across optional groups
• Anchor assumptions — Without explicit ^...$ anchors, partial matches on optional groups produce surprising results

Concrete example: a secrets scanner with the pattern (api.?key)?(token)? matches empty log lines in CI output. Every pipeline run that produces empty or whitespace-only log lines fires a false positive, desensitizing the team to real leaks. This exact class of bug — optional-group regex matching empty input — was the root of gitleaks#2121.

How to Detect This in Your Code

1. Identify optional groups — Search your regex patterns for (...)? or (...)* constructs
2. Test edge inputs — Run each pattern against empty string, single-character, and boundary inputs
3. Add explicit anchors — Wrap patterns in ^(?:...)$ when you need a full-string match
4. Use verbose mode — Add re.VERBOSE with comments to document group semantics

The Fix Pattern

When you hit this bug in the wild, the fix pattern is straightforward:

# Before: optional group can match empty
pattern = r'(prefix)?(suffix)?'

# After: require at least one group to match
pattern = r'(?:(prefix)(suffix)?|(suffix))'

Key Takeaways

• Always test regex against empty and minimal inputs. Optional groups make the seemingly-mandatory actually optional.
• Add unit tests for boundary cases — empty strings, whitespace-only, and partial matches
• Document group semantics — Use comments or named groups to make optionality explicit for future maintainers
• Consider a regex linter — Tools like regexlint or pyre2 can catch ambiguous optional groups before they reach production

---

Discovered while fixing gitleaks/gitleaks#2121. View the fix post for the specific diff.