There's a couple of reasons for the current situation in terms of the plethora of tools.
One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).
Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.
Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.
And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).
And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.
Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.
It was on purpose because TOML has not reached 1.0 yet and breaking people using a module in the stdlib for parsing TOML because TOML itself changed would be an absolute mess. Plus not a single packaging project said it was an issue to either depend on or vendor a TOML parser.
Once TOML hits 1.0 I plan to work on getting a TOML parser into Python's stdlib.
(disclaimer: I am one of the co-authors of PEP 518 that apparently handled things in way that was "a little absurd" .)
Do note that Python the language is developed by a separate group as Python the packaging ecosystem. So even when Guido was a BDFL he completely stayed out of the packaging situation.
And there is a packaging BDFL, but there's a severe lack of time and effort in the form of volunteers to tackle a lot of the projects, big or small. Plus rolling these things out takes years and it takes even longer to gain traction as people are often reluctant to change their build tools once things work.
As an example of timelines, take what my blog post covers. PEP 518 and the concept of pyproject.toml was conceived in 2016, four years ago. Getting PEP 517 finalized and it all worked into pip didn't happen until pip 19 in January 2019, over a year ago (https://pip.pypa.io/en/stable/news/#id203). And a year later there's still enough of a need to educate folks that I wrote this blog post to help get the word out.
IOW the timelines are huge and the pay-off is way down the line. And that's if you manage to be motivated enough to do the work. Burn-out is a huge problem in the packaging world. I mean reading that you think "it is a fucking mess and it's unacceptable" doesn't motivate me and others to keep trying to improve things if that's how us volunteers will be treated for years to come while we try to rectify the situation. Add on to the fact that we aren't getting paid for this and it makes the idea of pulling out my laptop tonight to work on some packaging stuff feel rather pointless.
So please be understanding that people are working on things as best they can and that overt negativity about it all will work against you getting what you want.
Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis. No one owes us anything. No one should feel entitled. I am sorry if my complains come out as such. Python runs the world and it is because of the contributions of many amazing people.
Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that. The community is orders of magnitude bigger than ruby by now, and anything that requires coordination is much harder to accomplish than in a lot of other ecosystems.
Python is now on par with ecosystems like C/C++ (where different build and packaging conventions are legions) or Java (where ever 10 years they write a new build tool on top of the previous one(s) to hide the complexity, and still, like python, they struggle to make simple executables 30 years on...). It’s just a hard problem to cater for so many different needs.
That’s the “oh we can’t use methods from others, we are too special” argument, which is always BS. Coupling the misapprehension to a snooty misconception is doubly unfortunate.
> Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that.
Them having different conventions and standard tools is the failure, not a constraint.
What are the different needs that these ecosystems have which mean they couldn't use a common package manager? I don't believe there are any.
For instance, in the scientific computing world, compiled extensions are hugely important, whereas they're relatively rare in web development.
There's probably no logical reason why a single common package manager for these domains is impossible. But they have different priorities, and people write tools that solve their own problems. Who is going to decide which package managers are unnecessary, and convince all of the users who are happy with them to switch to something else?
I think something people are forgetting when comparing Python's packaging system to other languages and runtimes is how old Python and its ecosystem is. For instance, Python's public release predates Linux's public release(Feb 1991 versus Aug 1991). The first release of setuptools was 2004 and it isn't like people weren't sharing Python code even before then (https://github.com/pypa/setuptools/commit/8423e1ed14ac1691c2...). That's over 5 years before npm was released (https://en.wikipedia.org/wiki/Npm_(software)). That means npm got to learn from Python and any other project that predated it. It also means they didn't have Python's momentum to contend with (I certainly never expected Python to become _this_ popular, and that makes it very hard to change as you literally have tens of millions of people to re-educate when you change something).
So while it's fine to say you wished Python's packaging approach was as good as another languages, please do so know that it is not fair to say that if one community got something to work for their needs it means Python should have been able to as well; it's not an apples-to-apples comparison considering the trees were quite possibly planted at different times and are different varietals.
I'm not sure how old you are, but if you don't know the name "Borland" you might not realize how novel it is to have a compiler on every major OS. While Python's age means it has been around to see a lot of changes occur, that also means it had pre-existing practices and code to also consider whenever a shift in the industry occurred. Plus in hindsight things look obvious, but during the transition it isn't obvious what will stick and what won't.
It's pretty remarkable that things have continued to function since my 17 years with Python alone have seen things like:
- Compilers becoming available on all OSs
- Linux becoming a thing
- Laptops becoming common
- Smartphones
- Web-based email clients
As I have said a few times in comments on this thread, it's fine to feel like there's room for improvement, but trying to "stick it" to Python and its packaging ecosystem in some way for not being perfect is pointless for you and demotivating for those who are actually in a position to try to make things better for you (and even if you aren't a Python user you should care to some extent about it as Instagram runs on Python and thus it is helping to deliver you those cat photos while you're stuck at home ).
It sounds a bit more understandable if you rephrase it as "a small group of largely unpaid volunteers should find a solution for which millions of users and several billion-dollar businesses will happily abandon their current long-running and well-tested solutions".
First, programmers have choices, and Python's lack of a good user experience on this front will turn off people that have better options with other languages which have found a way to deliver a better experience given the same constraints as python.
Second, there is the python foundation, which funds people to work on areas of python. This should be a priority, and can be done by paid contributors.
People in the Python ecosystem have come up with plenty of new things over the years - eggs, wheels, conda, poetry... Coming up with something new is the easy bit. Getting everyone to agree that a new thing is better and stop using/recommending other things is much harder - almost impossible, if those other things serve some use cases fairly well.
The fragmented ecosystem is one of the main complaints about Python packaging today. And that's really tricky, because the main way the open source world knows to solve problems is to create new tools, and that doesn't help when the problem is 'too many tools'.
And it (and by "it" I mean the Python community) absolutely did! The main problem that it has come up with numerous different somethings that have been competing to become standards, across a wide range of related, but separate areas (packaging, versioning, environments...), and only a few of them (like pip) managed to become partial (but not at all universal) de facto standards.
not at all. from my previous experience with ruby i was using gems with compiled bindings to libraries for: mysql, memcached, json, xml, regexp and http. and probably some more i don't recall right now.
Python does all of that and more. There are still occasional issues with the thorniest setups, but binary wheels absolutely took care of the average case. You even get a massive desktop toolkit luke QT with a simple “pip install pyqt”.
Getting there might not be the nicest experience, but most packages nowadays can and do get there.
The issue is that the environment varies a lot and there's usually a lot of C++ code that needs to be compiled to interact with the python plugins based on different environments with 500+ libraries mixing & matching...
I would not use Node as an example of good packaging. It really hasn't been that long since left-pad. Then there's the fact that you still have multiple different tools to do the same thing (npm vs. yarn) and the end result node_modules/ is a fucking mess anyways.
I think you can certainly use Node as an example of much better developer experience. Even the difference between npm and yarn is fairly minimal, and there is a solid community 'default' in Yarn.
I think part of the reason we ended up with so many trivial libraries and dependencies in the Node ecosystem was precisely because it is so easy to create and publish packages.
I love Python as a language, but working with the packaging even just to install dependencies is like pulling teeth every time you start a new project - nevermind publishing packages yourself.
No it's not... 99.9% of the time, installing dependencies is a breeze. Packaging is where the issue lies, but pulling those dependencies down is painless. You run one command and everything works. Unlike say npm/node/yarn where 50% of the time after coming back to work on a project your node_modules is broken and you have to either reinstall npm or node or node + npm.
The problem we're talking about isn't "installing a dependency", the problem is creating reproducible builds across people and between dev and production. The secondary problem is managing dependencies over the long run (updating, transitive dependencies, etc). Ruby/Bundler does this so much better than requirements.txt by a long shot.
Node's packaging story sucks in many ways, but IMO python's is worse. I spent quite some time working with python when there was no first-party dependency sandboxing (i.e. there was virtualenv but no venv) and even today there are tools that work with one but not the other.
That said, PEPs 517 and 518 sound like they're going in the right direction from the article.
ied ( http://gugel.io/ied/ ) had a great idea for reducing redundant installations in node_modules/, I wish the project got some traction. It worked as advertised when I used it 4 years ago.
Maven is “so good” that people wrote another tool on top (Gradle) so they don’t have to touch the Mavenstrosity. More or less like Maven was built to get rid of that other nightmare, Ant.
Yes, Python’s packaging is partly bad because it’s based on C extensions, which are intrinsically bad, but it’s also just plain bad compared to its peers, like Ruby and NPM.
NPM also has support for C extensions, and they mostly work reliably (the only problems I've had are forgetting to `npm rebuild` when switching between node versions, or when switching between native mac builds and linux-in-docker builds.
In my personal experience, extensions written in C don't necessarily compile at first attempt because the dependency management there is even worse. May be it fails with a missing openssl or readline library and I have to go find whatever OS specific thingy I need to do to recover.
C/C++ have no equivalent of npm/bundler/pip. Instead there are incompatible flavors of Makefiles for incompatible C compilers. It's a hard problem space to work in.
Thanks for the link. About the lack of resources problem though, I have a few points to make.
I think one of the unfortunate things that happened is indeed the arrival of new languages. Go, Rust, Swift, whatever is taking away a bunch of senior developers to work on those. Even though Python is still "most wanted" and "most popular", imho that does not reflect the reality among senior developers who are capable of taking on large tasks like this.
Sometimes the frustration expressed by package management has more to do with build dependencies rather than the distribution itself. Like a library that has a C dependency. In this context, the world of Go is equally messy, when CGO is involved. Luckily in their case, most people do rewrite libraries in Pure Go.
Thanks for the links - pyoxidizer looks promising with some good docs and being built on rust (but it won't work with QT's PySide2 likely until the next major release[0]), while briefcase looks to have been tailored to "beeware" usage and is lacking any standalone usage information. Hoping both these work out, PyInstaller works but other options would be welcomed.
It's all about capacity, not will. The PyPA as a whole knows what is lacking, but when you're all just a bunch of volunteers it's hard to tackle massive projects that take potentially years to complete and then even longer to see gain traction in the community when you are doing it all in your spare time. And that's if you don't burn out before you finish (e.g. calling projects "garbage" is not motivating to the few people volunteering their time to try and fix the situation while keeping what is there afloat and running).
Yes. I am not familiar with the python development process, but atleast have they an announced roadmap to move to something better that the present suboptimal methods.
> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded.
You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies.
This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies.
Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3.
First of all, as an aside, thank you for all the work you do on Python! I actually got started pretty early on in my professional career with Python (3.5+ too, which I am thankful for in hindsight). I was using VS Code and the Python Extension before MS even acquired its developer (is Don still there? I hope he is. Such a genuinely down to earth guy). Without the work you guys do I don't know if I would have taken the programming the same way, or at that the time I did in my life.
I will say, I didn't realize the funding situation was _that_ dire, especially since on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so. Its not billions of dollars but it sounded like the situation had a positive uptick substantially, so, in short, I was wrong!
Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented? (A loose example of this would be Igalia[0], which makes surprisingly good amounts of money to implement features in web browsers on behalf of the major browser vendors)
I've always thought this would be a pretty natural way to have a steady source of revenue and the profits could all feed back into the PSF.
> First of all, as an aside, thank you for all the work you do on Python!
Welcome!
> is Don still there?
Yep, working on the data science features of the extension.
> on Talk Python I remember specifically Michael and some others talking about the massive influx of money into Python and its ecosystem in the last year and half or so.
As I said there have been donations for PyPI and pip just got its big donation. The CZI donation was also much bigger than pip and covered a lot of the science ecosystem.
> Has the PSF ever considered having an actual business? Like say, consulting and/or having developers work on features that businesses pay to have implemented?
There isn't a need for the PSF to have a business subsidiary to accomplish that as people can donate money for such things if they truly wanted to. It's also really hard to manage due to tax laws in the US around non-profits and such. Plus there would be start-up capital costs, etc. just like any other business. I just don't think the risk of trying to get it set up would lead to any benefit when people have always had the option to donate directly to see if something could be done.
If people want to help out financially they can go to https://donate.pypi.org for packaging donations and https://www.python.org/psf/donations/python-dev/ for the language and CPython. And if someone wanted to fund a specific thing I'm sure the PSF is happy to discuss such an idea to see how feasible it would be.
That makes sense, on its face, as to why they don't have a business arm, at least currently.
One thing I would suggest, on its face, is that any time someone has to reach you (as in, the foundation, not you specifically), that introduces friction, even the smallest amount of it, can leave lost donations on the table. If there was a simple page even, that said something like
Do you have a feature you need implemented in Python? Are there features you would like to fast track? For a simple donation, we can prioritize the work you need to grow your business
This with a nice little contact form, would eliminate that friction, rather than being in a situation where I might be reaching out to the wrong wing of the foundation, or not feeling comfortable sending a general inquiry etc.
you could also put a feature page that tracks how much money needed to implement x too, to solicit donations. I've seen this work in other communities. I think Django has gotten funding using this technique before.
Just a thought, I know you guys are doing everything you can and then some, and Python is unique in that its one of the few language communities of its size without any formal corporate benefactor.
> For a simple donation, we can prioritize the work you need...
I'm not an expert, but I'm pretty sure a non-profit can't do things like this. If the person giving the money expects a tangible benefit in return, it's not a donation.
The specific restriction from my understanding is that a US non-profit that accepts money for a specific thing must spend every penny of that money only on that thing. So if the PSF was given $1,000,000 to solve a specific problem and while trying to do it they go bankrupt they still couldn't touch that money to stay afloat. So non-profits avoid that situation since they typically have enough to survive a year or so, but not enough to be able to restrict what they might need to do with their cash.
It would probably go a long way if they advertised a roadmap of things they’d like to work on given donations. Maybe they have this out there and I just haven’t seen it, but I imagine it would be easy enough for me to persuade my small employer to throw a few thousand USD their way each year until some sane packaging solution is developed.
If there's a specific pain point your employer would like to see fixed and is willing to donate money to help resolve that pain point then please go to https://donate.pypi.org/.
Poetry tries to help you manage your whole project while flit is just for building your library (and optionally uploading; you can always use twine to do the upload). It's all-in-one versus one-job-one-tool.
If you use a grepping tool like ripgrep it will automatically ignore everything in your `.gitignore`, so at least for that specific case you're covered.
One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).
Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.
Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.
And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).
And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.
Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.