David Lord

Regex performance is an issue, but don't call it a vulnerability

Regex performance should be treated as a regular performance issue, not a DoS vulnerability. Yes, they should be fixed. No, they should not be reason for everyone to panic.

As a popular web application framework, Flask gets occasional private security reports through various channels. Security reports, whether valid or not, are always much more urgent than regular issues, both because they should be fixed but also because reporters often declare they will release the report publicly if we don't respond quickly. This puts a strain on volunteer maintainer time and attention, which is already stretched thin. We have to drop what we're working on, figure out if the report is valid, then either convince the reporter it's not valid or come up with a fix. There's additional work to write a good description and accurate CVSS score as well.

The expressiveness of regular expressions (regex) makes it somewhat easy to write sub-optimal patterns that will take much longer to run than expected when given weird input. It's becoming very common to see security reports about "ReDoS" vulnerabilities in web applications, the idea being that an attacker will craft an HTTP request that they know will be parsed by a slow regular expression, in order to tie up resources.

This sounds bad, but in practice it's mitigated by two things. First, it's way, way easier for an attacker to set up a traditional (D)DoS. Second, you should always deploy your application with resource limits such as timeouts, max CPU, max memory, etc, as any number of things could cause unexpected resource usage. The operating system, Docker, proxies, and platforms all have ways to set these limits, sometimes with fine control over how they're applied. Many platform hosting services already set default limits for you, as well as provide other DoS protections.

William Woodruff, who among other things works on the security of the Python Package Index (PyPI), writes about this in much more detail. In his December 2022 post ReDoS "vulnerabilities" and misaligned incentives, he lays out why ReDoS reports are typically more of a DoS on developer time, and are the product of misaligned incentives in the security reporting community. His article really resonated with me, it put into words what I'd already been feeling about certain categories of reports I'd had to deal with over the years.

Regular expressions can be complex. It's often not trivial to modify them without introducing other bugs. Quoting from William's post:

... it’s easy to submit a ReDoS vulnerability report to an open source project. Remediation is another matter: even simple patches ... require serious maintainer attention in ecosystems where even tiny changes can produce breakages for millions of downstream users. ... I’ll sum this point up with a thought experiment: let’s say someone submits a patch for a “high” severity ReDoS in a popular tool and, in the process, accidentally introduces a bug that causes failures on millions of machines until a fixed patch can be rolled out. Which was worse: the ReDoS or the actual DoS?

If you discover a regular expression performance issue, it's a valid bug to report. We're always interested in making parsing faster or more correct. Please report it to us. We're an open source community project, so please consider contributing a fix as well. A good strategy I use is to break up complex expressions into separate smaller steps instead. This usually makes the code easier to understand and maintain, a great bonus.

But a slow regex isn't a security vulnerability. Don't make framework maintainers drop everything to address it as fast as possible. Don't make application developers update their dependencies when they're already deploying with limits. We're happy to fix regex performance, but we'll treat it like any other performance improvement.

#flask #security