It's not really machine learning, it's just optimization (optimize the space of shapes to create the desired filter). The paper doesn't call it ML either

Cool, but I'd rather see a nicer shape, like a human cochlea, which also splits sounds into frequencies.

Perhaps if they added some more constraints (e.g. on smoothness) they would end up with a shape like that.

It seems plausible that Mother Nature had to make some compromises when designing the cochlea, like how much easier it is to grow a (rather symmetric) cochlea than the weird shape from the article and maybe optimizing more for certain sounds than for others.

3d printers and search algorithms don't have that restriction and can directly optimize for optimal splitting and minimal acoustic losses.

But is allowing an irregular shape not more likely to get you stranded in a local optimum?

Why? And why wouldn't you expect an irregular shape to be the global optimum?

A cochlea doesn't really spatially disperse the frequencies, though, right? Rather, the isolated frequencies are extremely localized within the coil, and the hearing receptors are each located at the one point where they need to be.

This object is sort of an eversion of a cochlea. Perhaps it could be made with a "nice" shape, but I wouldn't assume so.

IIRC it is an array of cilia of different sizes that filters sound into different frequencies in the human ear.

What sort of ML method would do shape optimization?

Quite a lot could probably do it. Have it adjust positions/shapes of blocks placed in a virtual setting, have its fitness function be based off how split apart the frequencies are, press train and make a coffee. Sort of like this https://rednuht.org/genetic_cars_2/