If anyone's interested I've implemented a fairly user friendly lazy import mechanism in the form of context managers (auto_proxy_import/init) at https://pypi.org/project/lazyimp/ that I use fairly heavily. Syntactically it's just wrapping otherwise unmodified import statements in a with block, so tools 'just work' and it can be easily disabled or told to import eagerly for debugging. It's powered primarily by swapping out the frame's f_builtins in a cext (as it needs more power than importlib hooks provide), but has a lame attempt at a threadsafe pure python version, and a super dumb global hook version.
I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.
AIUI, as the official way setuptools is used is a setup.py calling setuptools.setup(), doing this will remain supported in perpetuity and will really still be powering everything under the hood, it's just no longer what people are told to do when they ask 'how do I make a new python library?' - newcomers and people with simple requirements will be directed to poetry or whatever and avoid that whole mess. For obvious historical reasons python's pretty averse to large breaking changes, and its support for all kinds of weird native interops and obscure platforms is one of its major assets (but also why setuptools is such a mess), so that's almost certainly not going anywhere, even if it's now kept even further back in the shadows. Regarding pyproject.toml if you already have a setup.py doing what you need then it's really only for specifying suported python versions and setup_requires (solving the problem of your setup.py itself needing Cython/wheel/etc).
What throws me for a loop is that support for C extensions isn't documented in https://setuptools.readthedocs.io/en/latest/ , nor is there any mention of how to do something equivalent to the current "python setup.py build", so I can't tell what is "official" regarding continued setuptools support for my current use of distutils.
Here's how my setup.py starts:
from setuptools import setup
from distutils.core import Extension
from distutils.command.build_ext import build_ext
I use Extension() to define the C extension.
I use build_ext to create a subclass which figures out the appropriate OpenMP compile and link flags, which depend on the compiler being used.
I also use that build_ext subclass to handle code generation. This Python code replaced a horribly complex set of preprocessor #defines and made it much easier to debug and profile the code.
Oh, and macOS's default C compiler doesn't support OpenMP so I added support for disabling the OpenMP flags, for example, with:
CHEMFP_OPENMP=0 pip install chemfp-3.5.1.tar.gz
I looked at poetry. It says "only pure python wheels are supported", so that's not (yet?) a replacement for distutils, yes?
And it looks like pyproject.toml is supposed to replace MANIFEST/MANIFEST.in ?
Lastly, I have a commercial product. I prefer to distribute source code to my customers. I grudgingly provide pre-compiled wheels for Linux, which people use for evaluation. My main source repository has the license check code ("src/chemfp_lm.h"), which is used to make the wheels. The MANIFEST.in excludes src/chemfp_lm.h during the sdist that goes to my paying customers. To handle source compilation for both cases, setup.py changes the Extension() configuration depending on if src/chemfp_lm.h is present.
This was simple using distutils. I have no clue how to replace distutils in the first place, much less be able to handle my special case in this future Python world.
Or as you say, "all kinds of weird native interops".
FWIW, my code still supports 2.7 and 3.6+, and has no required dependencies on other Python packages, so I didn't realize there was a problem. ;)
To be clear, I have optional dependencies on "zstandard" (in PyPI), on three computational chemistry packages with Python APIs but which aren't available through PyPI, and on one Java computational chemistry JAR file, connected via jpype (in PyPI) and itself with an optional dependency on a ZStandard JAR file.
Which means pip-resolvable dependencies hasn't been something I've had to care about over the last 11 years of development.
If you're building anything past a hello world c ext you're already off the path of shit like poetry, in which case AIUI we're basically on our own as we've always been lol, for better and worse.
Again I don't think setup.py is going away. Given that distsetuputiltools is a jenga tower of decades old monkeypatches documentation is understandably sparse, but they're at least passively open about how gross it is to work within. Subclassing or otherwise manually overriding the behavior of distutils classes like build_* is just how stuff is done at this level and I don't expect that to change, beyond just updating your stuff to subclass or hook direct setuptools equivs of existing distutils classes. distutils is already vendored by setuptools anyway, so you're already not really hitting 'distutils' as much as 'setuptools._distutils' aliased to distutils upon importing setuptools (and specifically importing it first ( https://github.com/pypa/setuptools/blob/c121d289da5d19cf6df2... )), so all this really does is collapse a level of hack.
As long as py2 remains supported I expect whatever changes happen here to be py2 compat, but I do expect new setuptools to break compat with old setup.py's. But setup.py being what it is it wouldn't be out of character to make your setup.py manually support both old pre-distutils-removal setuptools and new post-distutils-removal setuptools, but that's just how setup.py maintenance goes lol.
ed: ok well not so much py2 being 'supported' as 'no showstopping incompatibilities for anyone still straddling that ever widening gap' :)
To reuse a phrase from the native-Spanish-speaking Americans of the Southwest, whose family moved there in the 1700s or earlier, "I didn't cross the border, the border crossed me."
In the late 1990s, when I started with Python, writing Python C extensions was definitely not off the beaten path. I remember reading the design discussion, and I think I contributed some input too. I probably knew the original developers from going to Python conferences.
I know that at 20+ years old it's also long in the tooth. There's certainly features I would like that just aren't going to happen with it, like being able to emit a Makefile that I can use during development for incremental rebuilds, instead of taking 10 seconds to compile.
And distutils isn't going away for another couple of years, so I can plan time to migrate.
But I don't have an inkling of what that migration path might be, and can find no mention that the current set of developers are even aware of this issue. (Not that I've looked for more than a few hours.)
> distutils isn't going away for another couple of years, so I can plan time to migrate
I'd say it's not going away for as long as setuptools will be based on it.
> BTW, is "setup.cfg" dead and forgotten now?
Basically, the same answer as above: setup.cfg was (as I understand) an internal solution to declarative build configuration (originally introduced in now-discontinued distutils2). It is basically precursor to pyproject.toml, and it was actually one of the alternatives proposed in PEP 518 [0] (but was declined because it depends on Python's configparser, making it not much more universal than setup.py).
The setuptools documentation says: "All python package must provide a pyproject.toml and specify the backend (build system) it wants to use."
What is the build system which indicates the heritage build system?
Oh, I see. That documentation conflicts with PEP 517, which states:
> If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py (either directly, or by implicitly invoking the setuptools.build_meta:__legacy__ backend).
I find this whole thing frustrating. Eg, the setuptools documentation mentions 'setuptools.Extension' in passing, while talking about Cython, but does not document how Extension() is used??
So far I haven't found a PEP 517-compatible build system which handles C-extensions. The closest is "bento", mentioned in the PEP, but it hasn't been updated since August 2014.
Significant but as it's freethreaded you only run one large process. That alone allows for all kinds of additional optimizations that would be pointless with a lot of little singlethreaded processes in which sharing anything mutable between them has massive overhead.
Past some critical mass local dev indeed simply doesn't work, but the majority of codebases aren't ~that~ large. And until that point I feel there's an analogy between having a light laptop and remotely doing your heavy lifting with the whole working remote thing in general. In theory, on paper, and assuming reliable decent internet connectivity, with a little elbow grease there should be no discernable difference between thin client remote dev and an in-your-lap xeon, just like with high quality low latency video teleconferencing with a team that knows how to do it there should be nothing lost with physical distance. In practice though the overhead of doing any little thing, no matter how small the overhead may be optimized down to, is still real. Non-verbal communication is lost on zoom and the distance between you and your real UI grew by hundreds of miles and dozens of links, and the tradeoffs your environment makes when there's no such thing as a 'quick check' add up fast. But it's obviously not one dimensional, and there's probably irony in the weight shedding that offers you 20 hrs of battery life coming with a price tag of requiring reliable wifi to get any real work done.
I'm a fellow fan of antlr but it's not really an option for the base interpreter. They're (rightfully) very stingy about deps. cpython supports a wide range of platforms and has its own very permissive license - it's a pretty self-contained codebase and they prefer to keep it that way.
The whole point of python is to eschew 'magic' in favor of dumb explicitness, and this is about as big of a conflict as you can get to that. Python lacks statement lambdas not because they've never considered the possibility but explicitly because "if it's that important it deserves a name (as an inner function)". Type annotations work so well in python as compared to ruby or clojure exactly because of this philosophy. The PEP seems to make no mention of things like reproducibility, IO in macro expansion, static analysis like in mypy, or any of the other realities of things like this. If macro expansion is limited to something carefully statically interpretable like constexpr then I'm cautiously receptive to it, OR as it positions itself to be 'for experimenting with language features' and that's genuine then I'd be fine with it if it's only enabled in debug builds, but in lieu of those the instant it's released generally it's a python 4 flag day.
> You could use this to do little stunts like adding an "unless x:" statement equivalent to "if not x:"
You say that like it's a good thing. Python is a language empowered by its constraints. You have indeed always been able to pull stunts like this but it's strongly frowned upon - python is not ruby, lambdas intentionally do not have statements, and terseness is not a virtue.
I tried to prove you wrong and run multiple tests that all shown you were right. Then I tried to run them on different python versions, down to Python 2.7, expecting that my knowledge was just dated.
No, I was just wrong.
Don't know why I was so convinced of this.
> Regarding docstrings you can access the underlying function via the __wrapped__ attribute,
This assumes one knows about that. A lot of people never even heard of partial(), and won't know about _ _wrapped_ _. Not to mention I never seen a tool using help() fetching _ _wrapped_ _._ _doc_ _ if it exists.
> and signature forwarding is too brittle and magical for stdlib.
@functools.wraps() does signature forwarding to great success, and is in the stdlib.
Nevertheless, I don't think it would be the right strategy for partial(), because you don't have the exact same signature. But at least having something like:
This function is the result of x() being wrapped with partial(x, 1, foo='bar').
Here is the original docstring of x():
....
I always liken Pypy to HotSpot in that to this day the numerical performance of the latter isn't spectacular and nobody really cares - it's built to handle the harder job of making vast tangled codebases of non-numerical application code run fast, not just tight math loops which are already handled perfectly well by other more specialized tools.
I was skeptical and cautious with it at first but I've since moved large chunks of my codebase to it - it's caused surprisingly few problems (honestly none besides forgetting to handle some import-time registration in some modules) and the speed boost is addictive.