Yes, that's how TCP/IP works. You can't connect to a TCP/IP connection unless something is listening on that combination of IP Address and TCP Port Number.

As I said before what exactly are you trying to do ?

If, for instance, you're trying to connect to your external IP address from another computer you'll probably have to enable Port Forwarding for the Port(s) in question on your Router. Normally, in most Routers the not 'well known' ports are not forwarded by default.

EDIT: If you're trying to connect to Computer A from Computer B on your local network then Computer A will listen on a Port and Computer B will connect to Computer A's IP address and the same Port number.