These are AI bots. Computational capacity is not a limiting factor. I'd argue that my desktop consumer PC is less capable of efficiently solving a PoW than a multi-GPU cluster in a data center.
Even if, as you say, crawlers will hit the PoW thousands of times more, the only way to make it a barrier is if the cost is higher than the profit to be gained. Otherwise it's merely an expense to be passed on to the customer.
Even if, as you say, crawlers will hit the PoW thousands of times more, the only way to make it a barrier is if the cost is higher than the profit to be gained. Otherwise it's merely an expense to be passed on to the customer.