> your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science

But all science works like this. All scientific models can have errors, and presumably they all do. Even really basic empirical science is only reliable to the extent that our models of how photons and our visual systems work, and those models have errors we know about and probably others we don’t know about.

Fallibility does not mean that something is not science, on the contrary, denying that some theory or model is fallible is profoundly unscientific.

But of course, that doesn’t mean we should accept “black box” algorithms as the end of the story. We should strive to develop explanations for those things just like for all other things.

> So uhhh, isn't this like not science?

Very little of technology has anything to do with validating hypotheses.

> meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model.

This is fundamentally a cost saving mechanism to avoid hiring engineers to code heuristics useful to business. There is nothing related to science here at all. A "black box" model is fashionable to those who prefer to observe and not create meaning, even if the observed meaning is deeply flawed from a human perspective. After all, people spend money based on less all the time.

At our company, we are working with "mechanistic" or mathmatical models rather than stastical approches. I have observed :

1. It is hard than it should be to explain the concept to people (particularly VCs) 2 . people struggle to understand that a mechanistic model could have more utlity than a machine learning black box 3. people think you are doing something wrong if you are not using a neural network 4. The less people understand about neural networks, the more they seem to believe they are appropriate for all predictive / modelling problems 5. There is generally quite a low understanding of scientific method in the startup / VC space (speaking as someone who has worked in and around academia for years) vs how "scientific" people believe they are because it sounds good to be data driven and scientific about running startups and funding them.

Do you mean like symbolic regression or algorithmic information theory based stuff?

If so, I'd love to get in touch, shoot me an email

We are using biophysics based approaches. Will send an email over :)

Send me one, too, if you don't mind. I like collecting these techniques to give to researchers or practitioners wanting to try new things.

>> I guess I can sum this up with, I wish people looked at AI more as a tool to help guide our intuition helping us solve problems we already have well defined knowledge (and data) of, and not as an means to an end itself.

Problem is, most machine learning algorithms cannot incoprorate background knowledge except by hard-coding inductive biases (as, e.g. the convolutional filters in convolutional neural nets). Unfortunately, this is a very limited way to incorporate existing knowledge.

This is actually why most machine learning work tries to learn concepts end-to-end, i.e. without any attempt to make use of previously learned or known concepts: because it doesn't have a choice.

Imagine trying to learn all of physics from scratch- no recourse to knowledge about mechanics, electomagnetism, any kind of dynamics, anything. That's how a machine learning algorithm would try to solve a physics problem. Or any other problem for which "we already have well defined knowledge (and data) of". We might as well be starting at around before the stone age.

Machine Learning is about solving a task by automatically learning patterns based on examples and trying to generalize to new data. Granted, it's not necessarily the best approach to understand deeply how a phenomenon occurs.

You could use a black box model if you're more interested in predicting correctly images of handwritten digits than in understanding how the pixels relate to each other.

Of course, usually people want both accuracy and interpretability. It boils down to understanding what's more important for the problem at hand and making the compromises accordingly.

The goal of science is to produce a testable hypothesis. The world is full of things that people can 'explain', or even prove with mathematics, but don't hold up to basic experimental testing or reproducibility.

Every new theory has a plain english explanation that's easy to understand. Few of them have the raw accuracy or reliability of top ML models

https://en.wikipedia.org/wiki/Replication_crisis

> could pen an equation (model) on how to spot a fucking cat or a hotdog without AI? Of course someone could

I had a philosophy lecturer who would vehemently disagree that it is even possible to construct an algorithmic AI to decide what is and isn’t a cat.

For a start, do you mean domestic cats, or the cat family? What about photos, sculptures and other representations?

I mean, is Garfield a cat?

People use statistical machine learning over algorithmic AI because trying to model the real world with algorithms is an endless and often pointless exercise.

I think you are missing the key part of the appeal here (or framing it as a negative).

Let's look at your question. Writing an equation for a cat is hard, actually really hard. Humans cannot reliably explain their decisions here. If I ask a person to tell me how they classify between cat and not cat, the answer will invariably be something along the lines of "well, it has the general shape of a cat". Which is actually just a huge combination of heuristics it took about 10 years to work out. There is quite a lot of work in neuroscience suggesting that the actual decision you make when you classify a cat happens before a rational is developed.

We could encode a function for that, but it relies on us knowing a lot about cats, which takes time and only works for toy examples.

If you use a convolution neural network, you can get close to human level performance on much more complex topics with little domain specific insight. There is no universal law for classifying hand written letters - they are an individual's interpretation of some symbols we made up. This task will always be 'non-rigorous' because the very underlying thing is not actually well defined. When does a 3 become an 8?

So we could have a person toil away and come up with a bunch of heuristics that we encode in a regression, but why is this better than having a machine learn those heuristics? Most problems are not life or death. What is the real added value in having people hand crafting features for predicting traffic jam delays or customer retention, when the end use is probably just to have a rough indication.

As somebody who does research using a huge range of models, I object that we should be guided by our intuition- our intuition is mostly wrong about non trivial problems.

Basically any "equation" somebody has discovered for what happens in a neutron star is "simple". There is a large amount of observational data, it is a consequence of some already well proven theorem, or relies on something well established to narrow the range of possible descriptions immensely, or (most commonly in my experience) the equation is basically a human version of deep learning, where grad students toil away making tweaks and heuristics until a point that the description fits the data somewhat well, and then there is some attempt to ascribe meaning after the fact.

For example, we can describe the trajectory of a comet using a "few" lines of high school level math. This means it is actually feasible for a person to have a reasonable intuition about what is happening, as the problem is actually dominated by a handful of important variables. Good luck getting anywhere near simple to describe cats (again, in a domain where the line of what is and isn't a cat is actually not even a property of it's physical attributes, so the problem is not properly defined under your requirement). To tell if something is or is not a cat, would require a DNA sequence. That is how we define the cat. So by your own definition, we do not have sufficient data in our dataset to properly do this classification.

I'm not sure you really understand the point you make about "statistical tests for bullshit". Most statistical tests are themselves ivory towers of theory and assumption which nobody ever verifies in practice (which is as unsciency as anything you accuse machine learning of). And people do actually use well grounded ways of evaluating machine learning models. Cross validation is very common and predates most machine learning, and has various "correctness" results.

For any model we build, if we do not have data that encodes some pathological behaviour we can test it out on, there is no test, no statistical procedure to tell us that model is flawed. If we have that data, we can run the exact same test on a black box model.

You should not conflate science with formalism or complexity. Running a statistical test is pseudoscientific unless you do it correctly and appropriately.

Saying something is not scientific because the data may not contain enough information to fully answer the question is flat out wrong.