If this happens always, it literally means that the machine exists but that it has no services listening on the specified port, or there is a firewall stopping you.
If it happens occasionally – you used the word “sometimes” – and retrying succeeds, it is likely because the server has a full ‘backlog’.
When you are waiting to be accepted on a listening socket, you are placed in a backlog. This backlog is finite and quite short – values of 1, 2 or 3 are not unusual – and so the OS might be unable to queue your request for the ‘accept’ to consume.
The backlog is a parameter on the listen function – all languages and platforms have basically the same API in this regard, even the C# one. This parameter is often configurable if you control the server, and is likely read from some settings file or the registry. Investigate how to configure your server.
If you wrote the server, you might have heavy processing in the accept of your socket, and this can be better moved to a separate worker-thread so your accept is always ready to receive connections. There are various architecture choices you can explore that mitigate queuing up clients and processing them sequentially.
Regardless of whether you can increase the server backlog, you do need retry logic in your client code to cope with this issue – as even with a long backlog the server might be receiving lots of other requests on that port at that time.
There is a rare possibility where a NAT router would give this error should its ports for mappings be exhausted. I think we can discard this possibility as too much of a long shot though, since the router has 64K simultaneous connections to the same destination address/port before exhaustion.