It clearly makes a difference for me. My servers which have SSH listening on port 22 have countless failed login attempts. Those running it on another random port have almost none.
Oh, I'm aware it's "boxes". But, I got into the "boxen" habit in the middle 90s and now that I'm "old" I've chosen to become stubborn and stuck in my ways. Also, I think it's more fun. I've got other things to be pedantic about.
OpenSSH on port 22 gets at least one attempt a minute for me (and that's with fail2ban blocking IPs on a single failure.) Dropbear on port 2222 gets at most one an hour.