Where is the code that closes the listener located?

I can't say that I have ever had this problem, though that might be that I don't use TCP much, but I have used UDP a fair amount, and that listens to ports, and has a similar issue. In my case, the easiest solution was to change a setting on the socket such that there can be multiple shared users of the same port. I had to do that because I had multiple modules communicating via UDP on the same port on the same computer. That solution might work for you (it can be found by searching the networking forum for threads started by me, and having the word UDP, it is the thread with 7-8 replies).

However, that's not a good solution in this case, because you have an actual problem, and getting around the problem by sharing a port is nothing but a hack (and not the super moderator here, either). A better solution would be to solve the problem, which is why I asked that first question. If your TCP is in a class, then does it implement IDisposable, and is the socket closing code in the Dispose method? If not, then I think it should be, or at least the Dispose method should ensure that the socket is closed.