Yeah that’s fair, and I do agree with Cloudflare stamping out that behaviour.

What I’m trying to say is there are cases where AI agents act for the user in what the traditional user agent role of browsers would be.

ETA: That doesn’t excuse things like not having a search index to prevent mass scale access, this would be near 1-1 access patterns per user, which would be infrequent/spaced out