State of Pallets 2024
This year at PyCon US, members of the Pallets team organized an in person mini FlaskCon. I presented the first talk of the day, State of Pallets 2024. Here's the text version of the talk, with the slides as images. A PDF of the slides is available here.
Hello and welcome to FlaskCon! Iāll be presenting about the current state of the Pallets organization and projects.
My name is David Lord. Iām the lead maintainer of Flask and the Pallets projects, but not the original author. You can find me pretty much everywhere @davidism. Iāve been a member of the Python community for 15 years, and am a PSF fellow. I have a full time job as a software engineer, then devote plenty of my free time to maintaining Flask and other libraries and helping out other Python programmers and projects. I also enjoy scuba diving, hiking, and brewing beer with my friend in my garage, and Iām currently learning Japanese. I just got to āJinjaā ē„ē¤¾.
Welcome to FlaskCon! This is a mini, one day conference focused on Flask. Weāve held a few conferences online in the past, starting in 2020, but this is our first in person event. You can find past talk recordings of those talks on pyvideo.org. Today, weāll have four technical talks about working with Flask, separated by lunch in the middle. After the talks, members of the Pallets team will be hanging around for office hours. Please stop by to say hi, to tell us about what you do using our libraries, or to ask us questions. Weāre excited to meet more of our community in person. We will also be running two days of sprints. Regardless of your familiarity with the libraries, everyone can be a successful contributor, please consider joining us.
What is Pallets? Pallets is the open source community organization that develops and supports popular Python frameworks and libraries: Flask, Jinja, Click, Quart, Werkzeug, MarkupSafe, and ItsDangerous. It was created 8 years ago, right around when I became a maintainer as well, in order to move ownership from the original author, Armin Ronacher, to a group of maintainers. These libraries power applications of all sizes around the world. They are some of the most popular Python libraries, downloaded hundreds of millions of times each month. The goal of Pallets is to grow the community around these projects to create a sustainable group of maintainers, contributors, and users. You can join us on our Discord chat or by watching the repositories and discussions on GitHub.
Who is Pallets? There are three active core maintainers, myself @davidism, Phil @pgjones, and Adrian @ThiefMaster. Iām focused on pretty much everything at once, Phil maintains Quart and is applying his async expertise to Werkzeug and Flask as well. Adrian is always around on Discord, and also helps out with issue triage and evaluating security issues. Pallets is also its community. We have an active Discord server where people can ask and answer questions, as well as discuss contributing to the projects. We have four great moderators on the team, Kara, Carson, Guillaume, and Radoslov, who keep everything on track. There are also other people on the team helping organize events like these, coordinate documentation translations, and supporting us in other ways. Adam, Abdur, and David C all helped organize FlaskCon today.
The Pallets organization is part of the Python Software Foundation through their Fiscal Sponsor program. The PSF provides 501Ā©(3) tax-exempt status, and manages administrative issues like donations, accounting, budgeting, insurance, and legal support, as well as access to nonprofit discounts. This has been a huge help for me over the years, giving us access to the knowledge and resources of the PSF staff when we need it.
We are fairly lucky in the open source world to be a project that receives a steady stream of donations and financial support. Itās not enough to hire someone full time, but it does allow us more opportunities. As part of the PSF, we can take individual or recurring donations directly through them. We are also part of GitHub sponsors, so anyone or their company who is already on GitHub has an easy way to donate to us. We are partnered with Read the Docs and their Ethical Ads platform to receive some of the monthly ad revenue from our documentation. We are also part of TideLift, a platform that divides up enterprise subscriptions from companies to many of their dependencies, in return for maintainers keeping up with a few basic tasks. Through these Iām able to be paid a bit for my time each month, as well as attend more conferences where I can give talks and run sprints. Iāve used some funds to hire a designer to refresh and unify the projectsā logo design. Iām currently working on hiring a manager to help me organize the next six months of my time and responsibilities, so I can better get work done for Pallets, and to produce documentation about maintaining our projects. We want to find other ways to use our funds as well, such as organizing a conference outside of PyCon in the future, hiring developers for specific long term tasks, or adding bounties to certain types of issues similar to what the urllib3 team has done.
On July 1, 2022, I managed to bring Flaskās inbox to zero. A year later, I did the same for Werkzeug, which is the underlying HTTP and WSGI library. Every pull request was merged, every issue responded to and decided on. This is something Iād been working towards for years. When I first became a maintainer, every project had multiple pages of open issues and unreviewed PRs. Probably around a thousand open issues combined. This made it incredibly difficult to make decisions or know where to focus effort, as there were just an overwhelming number of possibilities. Was this issue actually an issue? Did this feature request make sense? Did one request depend on another (or more) or conflict? PRs were severely out of date, or incomplete, or need refactoring. Day by day I tackled issues, and slowly built a giant graph in my head of all the interrelated parts. I gained a better understanding of all of the libraries, and a better sense for what I thought should and should not be done, and how to do it. With the inbox at a manageable size now, itās much easier to evaluate each new request.
Depending on how you use Flask and Werkzeug, you may or may not have noticed a few waves of deprecations and removals over the past two years, in addition to the bug fixes and features. Iāve been applying what Iāve learned about the code base to identify inconsistent, outdated, or duplicated code and refactoring to improve that. As much as possible we try to remain compatible while issuing deprecation warnings, before removing things. This is not done lightly, as weāre acutely aware of how many people use our libraries, but I think it does represent an improvement to the libraries and a win for maintainability. The latest refactor was to Werkzeug, with two large removals. The first is that we no longer accept bytes in most places, only strings. This was a holdover from our transition from Python 2 only, to 2 and 3, to 3 only. The second is that we deprecated our very old slightly customized copy of Pythonās urllib in favor of using urllib directly. Both of these changes were to code that was called everywhere during each request, and produced significant speedups to request handling, around 50% faster in benchmarks at the time. Because of these refactors, the code is both faster and easier to maintain. The other major focus has been on async support and more code sharing between Flask and its async counterpart, Quart.
Phil Jones wrote Quart to be a drop in async replacement for Flask. It has the same developer API, and many of Flaskās extensions work with Quart. We began talking a few years ago about how we could bring this accomplishment into Flask itself. The async programming paradigm is popular, and some people perceived Flask as outdated because it didnāt support it directly. Today, Quart is an official part of the Pallets projects. Phil has done amazing work to figure out how to enable some async in Flask, as well as unify the interfaces Flask and Quart both use behind the scenes. Flask still has a ways to go before it is truly async the way Quart is, and there are some very tricky questions to answer before it will be possible, but we are making progress. Our goal is to some day have a single package that offers both Flaskās WSGI and Quartās ASGI interfaces.
MarkupSafe and ItsDangerous are two smaller libraries we maintain. Both of them are essentially ādoneā and stable. Both have been at inbox zero for a few years. Notably, MarkupSafe recently simplified and improved the speed of its C speedup code. Weāve also got an experimental PR reimplementing the speedups in Rust. Look for a 3.0 release soon.
The final two core projects are the Jinja templating library and the Click command line framework. Both of these are also extremely popular libraries. Unfortunately, with all the other things Iāve been focused on, I have not yet been able to focus on either. I currently have a 90% done rewrite of Clickās command argument parser, replacing our modified copy of optparse. I have a similar desire to refactor Jinjaās parser and compiler to make it easier to maintain and extend. As part of that, I also want to create a more formal grammar definition of the Jinja language, which would make understanding the current capabilities and how feature requests fit in a lot easier.
Weāve made a lot of other changes big and small to the libraries over the past few years. Weāre now using pyproject.toml to specify package metadata, and use flit_core instead of setuptools as the build backend. We pin our development environment dependencies to make sure every contributor has the same setup that we test with. A significant amount of work has gone into fully type annotating all of the libraries, although none of us are experts with the typing system and it can definitely still be improved. All the libraries pass with Mypy in strict mode, and many also pass with Pyright. Iāve also done a lot of work to fully automate our release process and improve its security. PyPI is configured to require 2FA, and we take advantage of PyPIās new trusted publishing feature to upload releases directly from GitHub CI. SLSA provenance files are generated to attest that the package you download is the same that was built in CI. Maintainers no longer need to build or upload manually, which has made making releases, especially quick bug fixes, much much easier to make and secure. As part of security, weāve also started using GitHubās private security advisory system to take in and work on security issues privately, and control the final CVE message that goes out.
Thereās still more work to be done on the core projects, but we now are also focusing on the ecosystem of extensions. Flaskās extensions make it great, but many of them have even fewer resources than we do. Libraries have become less maintained as their authors move on to other things and no longer have time available. This makes more susceptible to breaking as changes are made to the core libraries, as well as creating forks that can cause confusion or uncertainty over what to use. Iāve jumped in a few times on extension projects to quickly fix them after a release caused an issue. To address this long term, Iāve been working on a new Pallets-Eco organization. The goal of this is to enable community maintenance of the extension ecosystem for our libraries. Mostly Flask, but Click and Jinja also have plenty. Itās a similar idea to the Jazzband organization which maintains many Django extensions. Authors can transfer their libraries to the pallets-eco org on GitHub and PyPI. They retain access, but the core maintainers do as well, so they can step in to make a quick release to keep things working if needed. And we can also help add maintainers from the community. Weāve had great success with Flask-SQLAlchemy, Flask-Session, Flask-Classful, Blinker, and Flask-DebugToolbar, to name a few. Weāve had others transferred as well. Iāve been working to standardize and document Pallets project layout, tools, and configurations, and apply that to pallets-eco projects as well. The more project management is standardized, the easier it should be for contributors to jump on any project in the ecosystem. Getting involved with a pallets-eco project is a great new way to contribute to Pallets.
Iāll stop being humble for a minute. I donāt have an exact measure for hours or lines of code, but Iād guess Iām handling ~85% of maintaining our projects. All the stuff Iāve talked about here, and that I do to maintain the projects, I had to discover and learn from scratch when I first started being a maintainer. This is not to downplay everyone elseās work, but to emphasize how much of the projects currently depend on only one person. I donāt like this situation, the pressure of this much responsibility has brought with it a lot of stress, and Iāve suffered severe burnout multiple times each of the past four years. On a happier note, I recently got married, and my wife and I are planning on raising children. My available free time is only likely to go down in the future. My goal by the end of this year is to make it possible to change this situation. Iāve been writing a new website for Pallets with up to date explanations about the organization and our policies. Iāve been creating a standard project layout and a template project that demonstrates that, with all the tools and configurations we use. And Iāll be writing about all those tools and processes, as well as the various other maintainer knowledge Iāve accumulated. I want to create maintainer documentation, so anyone can maintain our projects, regardless of if Iām available. Iāve been making progress, and though I still have a ways to go, Iām feeling good about it.
Pallets is a community organization. We will do well if we support each other and contribute together. My goal is to see Pallets grow, and you can help with that. Please stick around to listen to the talks, but also come by and introduce yourselves to us any time. Itās easier than you think to get involved in some way, not only writing code, but reproducing issues, researching standards, improving documentation, translating, automating, adding coverage, and more. We greatly appreciate all the help you give us. Weāll be having office hours later today if you want to chat about anything related to Flask and the Pallets projects, as well as sprints starting Monday which are a great way to learn how to contribute. Outside of PyCon itself, we would love to have more people participating in Q&A on our Discord server and on each projectās GitHub discussion page.
Thank you.
