Que raro
no sé si esto lo has encontrado ya en internet...
Question/Problem: WSAECONNREFUSED (10061) Connection refused
Answer/Solution: Connection refused. No connection could be made because the target machine actively refused it. This usually results from trying to connect to a service that is inactive on the foreign host.
User suggestions: Either you went to the wrong host, or the server application you're trying to contact isn't executing. Check the destination address you are using. If you used a hostname, did it resolve to the correct address? If the hostname resolution uses a local hosttable, it's possible you resolved to an old obsolete address. It's also possible that the local services file has an incorrect port number (although it's unlikely).
Pero parece que va bien..Si quieres postea el código haber si hay algun error..
Un saludo
PD:Nunca me ha pasado nada de eso con los sockets, mi problema es con los puertos del ruter, pero si los tienes abiertos...