Getting expert witnesses (cheaply/pro-bono) is the missing catalyst here. What does Joe User know about hidden weights and logical evaluation orders?

It's definitely the next step, but right now even with the experts you can't review how the conclusion was arrived at which is a big problem.

Expert without access to algorithm may tell only so much.

They may analyze statistics behind answers, which is often costly.

But they may find it hard to prove that the bias is due to negligence.

What does Joe user know about the legal system? That’s why Joe needs a lawyer.

Either the algorithm or the code can be wrong.

We see frequent updates (aka error corrections) to our phone operating system and apps on a near daily basis.

To be admissible evidence, a computed result must show the entire decision tree and the code at each decision point. This should be in the initial disclosure to the defendant.

Id ask how was the system tested? For the system complete test records to be produced for examination. Id ask to see the test data for the specific software version This includes test cases, test case data and test case results The test results should also include any needed setup information and configuration, such that test results can be reproduced, exactly.

If its not been tested and/or there is no record of it, how can it be known that the system operates correctly?

Interesting that a bill is even necessary. Doesn’t the defense already have a right to preview the evidence used against them?

A black box algorithm passes all tests that it has accuracy and precision both above 95%. Nobody understands how it works but all tests support the performance stats. Should this mysterious black box algorithm be allowed for use in courts to improve decisions?

Crazy that the victim now has to be debugging the accusers systems. Still, better to have the option than not.

Why a "victim"? Honest question.

I was imagining the scenario where a person is being unfairly accused. If that's not the case my comment doesn't apply.

Because this is counter-accusation.

If judge has all reasons to believe in fairness of the algorithm, then counter-accusation is the only way through.

How can the “fairness” be determined if the specifics of the algorithm are not known to the judge, defense or jury?

We can’t allow people to be prosecuted by evidence that relies on black boxes.

The burden should be on the prosecution to prove that the algorithm is fair and reliable; the defendant is presumed innocent, ie falsely accused.