"overengineered" is not the term I would use to describe Python packaging. I would say it is "under-engineered". As in, "Why engineer a configuration file when you can just do it in code?".

This tendency towards what initially seems like the "simple" solution pervades the Python ecosystem and often requires complex engineering to work around later.

Summary: They fixed Python packaging by not having to run Python to resolve dependencies. Also, they used Rust.

Moral of the story: Use less Python. Use declarative configuration and other langauges instead.

>> you can literally write python wrappers of Julia compiled libraries like you would c++ ones. > Yes, please. What do I google? Why can't julia compile down to a module easily?

Try JuliaC for compiling shared libraries if that is what you mean by a "module": https://github.com/JuliaLang/JuliaC.jl

That said Julia's original design focused on just-in-time compilation rather than ahead-of-time compilation, so the AOT process is still rough.

> I don't understand why there's so much friction between julia and python. You should be able to trivially throw a numpy array at julia and get a result back.

You can throw a numpy array at Julia and get a result back. See https://juliapy.github.io/PythonCall.jl/stable/juliacall/

> I wish there was a way to specify the offset base with something like a C #define or compiler directive.

Julia has OffsetArrays.jl implementing arbitrary-base indexing: https://juliaarrays.github.io/OffsetArrays.jl/stable/

The experience with this has been quite mixed, creating a new surface for bugs to appear. Used well, it can be very convenient for the reasons you state.

  julia> A = collect(1:5)
  5-element Vector{Int64}:
   1
   2
   3
   4
   5

  julia> B = OffsetArray(A, -1)
  5-element OffsetArray(::Vector{Int64}, 0:4) with eltype Int64 with indices 0:4:
   1
   2
   3
   4
   5

  julia> A[1]
  1

  julia> B[0]
  1

I tried this in Julia with TidierData.jl, and it looks quite similar to the R version.

  using TidierData, DataFrames
  using PalmerPenguins: load

  penguins = load()

  @chain penguins begin
    DataFrame
    @drop_missing(body_mass_g)
    @group_by(species, island)
    @summarize(
      body_weight_mean =
        mean(body_mass_g),
      body_weight_std =
        std(body_mass_g)
    )
    show(_, allrows=true)
  end

> Unfortunately, there's no alternative, and I don't see R as much easier, there are plenty of ugly things as well there.

Have you tried Polars? It really discourages the inefficient creation of intermediate boolean arrays such as in the code that you are showing.

> There's Julia -- it has serious drawbacks, like slow cold start if you launch a Julia script from the shell, which makes it unsuitable for CLI workflows.

Julia has gotten significantly better over time with regard to startup, especially with regard to plotting. There is definitely a preference for REPL or notebook based development to spread the costs of compilation over many executions. Compilation is increasingly modular with package based precompilation as well as ahead-of-time compilation modes. I do appreciate that typical compilation is an implicit step making the workflow much more similar to a scripting language than a traditionally compiled language.

I also do appreciate that traditional ahead-of-time static compilation to binary executable is also available now for deployment.

After a day of development in R or Python, I usually start regretting that I am not using Julia because I know yesterday's code could be executing much faster if I did. The question really becomes do I want to pay with time today or over the lifetime of the project.

Yes, I tried Polars both in Python and Rust. In Python it's more verbose, as the other commenter points out. In Rust it's very painful.

> Have you tried Polars? It really discourages the inefficient creation of intermediate boolean arrays such as in the code that you are showing.

The problem is not usually inefficiency, but syntactic noise. Polars does remove that in some cases, but in general gets even more verbose (apparently by design), which gets annoying fast when doing explorative data analysis.

It sounds like there are many Python users who have acclimated to the situation of needing three or more tools to work with Python and do not see the benefit or value of being able to do this all with one potentially faster tool.

While I understand that some have acclimated well to the prior situation and see no need to change their methods, is there really no objective self-awareness that perhaps having one fast tool over many tools may be objectively better?

What tooling do you use?

The creators got burned on Swift for TensorFlow, their first MLIR project. One of the problems with that first venture under Google was that the language was not Python.

The Python superset concept was always a gimmick. The goal was always to juxtapose Python with a language that superficially looks like Python in order for you to completely migrate from Python to Mojo. It is just providing a smooth ramp for you to do so in the same Apple migrated folks from Objective-C to Swift.

Embrace. Extend. Extinguish.