It doesn’t provide any serious defense against a targetted attack. If your server is being targetted then, as you say, they will port scan you and find out where your doors are.
However, moving SSH off the default port of 22 will deter some of the non-targetted and amateur script kiddie type attacks. These are relatively unsophisticated users who are using scripts to port scan large blocks of IP addresses at a time specifically to see if port 22 is open and when they find one, they will launch some sort of attack on it (brute force, dictionary attack, etc). If your machine is in that block of IPs being scanned and it is not running SSH on port 22 then it will not respond and therefore will not show up in the list of machines for this script kiddie to attack. Ergo, there is some low-level security provided but only for this type of opportunistic attack.
By way of example, if you have the time – log dive on your server (assuming SSH is on port 22) and pull out all the unique failed SSH attempts that you can. Then move SSH off that port, wait some time, and go log diving again. You will undoubtedly find less attacks.
I used to run Fail2Ban on a public webserver and it was really, really obvious when I moved SSH off port 22. It cut the opportunistic attacks by orders of magnitude.