I'm using A|B as a blanket term for things that make a company think it's improving by using the opinions of others in lieu of having an opinion themselves. Focus groups also fall into this category.

It works for hardware, because that's the only way to do it. There's no 2 populations for hardware. All decisions are one way doors.

Software must use feedback, otherwise you'll end up with duds like Camera Control, 3D touch, and .. Apple Maps (1.0).

Apple doesnt have that culture, and it shows in their software.

I don’t dislike them inherently, but as people tend to believe they’re a one-stop solution it’s worth pointing out what they can’t do:

- On their own they don’t provide a hypothesis. You need to do user research to discover those.

- On their own they don’t tell you the value in what you’re testing. You need further analysis to understand that.

- They’re typically a short-term solution to longer-term problems. It’s easy to use them to achieve local maxima without understanding the wider picture.

- They’ll only tell you what the majority of your users find effective, without highlighting what hurts the journey for a minority of users. For example you could A/B test something that improves the journey for every group apart from visually impaired users, but block them completely, and still implement it because it’s the winning bucket.

Yes, there are guardrails you can put in place for all of these, and good teams do, but that requires a degree of maturity that is hard for a lot of organisations.

It's a good tool for optimizing around local maxima and is (generally) incapable of propelling you through a major product gap to the next (and hopefully greater) maxima.

But IMO the industry dogma is to use it for everything, particularly around greenfield development and new product areas that are pre-PMF.

Importantly also is that in many organizations A/B testing has become a crutch to avoid understanding the underlying system being measured.

Conversion rate rises by 5% if the button is green. Why? But rather than using experimentation as a tool for structured understanding many organizations devolve to "just test every change".

The practical outcome is that product teams commit elementary errors because they fail to understand why their products are successful, and product velocity slows as teams prove unable/unwilling to make any decisions without pushing something to prod.

Instagram is a product of a/b testing. Go figure