TCP Retransmission Situation
I started a thread similar to this in the .NET forum, but new information has come to light that prompted me to ask this question:
I have a program that has sent requests to a service (which I don't have any control over). In general, one instance of the program might send out a dozen, or so, requests to the service, each for a different tag. This has been going fine for a few years. Just recently, the folks who maintain the service told me that they were getting spammed by requests, which I tracked down to the program in question (there are two programs using the request code, but one is used much less frequently).
After a bit of discussion, I got the logs from the service for the last several months. Upon studying the data I can see that most of the instances of the program do exactly what they are supposed to. I see a series of requests for different tags, but few repeats. Some repeats are understandable, so long as it is 2-4, so those are fine. However, there were two exceptions, and there was a pattern to them. Two different computers called repeatedly with the same tags. Each computer had their own set of tags they were calling about, but one called 50-70 times per tag, while the other called 400+ times per tag.
There is no reason, or means, by which the program could be doing this. I've ruled that out both by studying the code at length, as well as by experimentation with the same dataset used by the computer that had called 400+ times for each tag. The program doesn't ever repeat a call without the user pressing a button. There are no loop structures around the service call, so there is no means for looping by accident. Also, the program lives with whatever the response is, even if it is 404, because it's not essential that the call go through, so the program has no means to repeat a call, and testing shows that it does not. It behaves exactly as I intended it to.
However, there's all that repeating, and there's a pattern to it. On both computers, the number of repeat calls started really high, then faded off over the next couple hours. So, the one program started at 400 calls per tag and over the next hour, or so, it dropped down to calling less than 100 times per tag. It wasn't a totally smooth decline, but seemed to drop in steps. The same pattern was seen for the other computer, though the repeats dropped faster and didn't last as long.
A couple people have suggested malware might be a cause for this, but that decline doesn't seem consistent with malware. What kind of malware would ping a lot, then ping less and less, though still a goodly number of times?
My main suspicion is a network issue. This happened during a time of network changes, so there was plenty of disruption, both that which I know about and that which I do not know about. So, I'm wondering if this is possible.
My understanding of the details of TCP protocol is somewhat superficial. A packet is sent out, then an ack is returned. In this case, the requests are tiny, so I would expect that each request was nothing more than a single packet (a tag is only 7 bytes in length, with possibly a further check byte or two). I'm wondering if it would be possible that the ack responses were being blocked/dropped such that the connection was established, then the packet was sent, but when no ack was received, the sender repeated the request...at great length, until an ack finally got through.
Is that something that could happen? If so, I'm looking for more information around how this kind of thing could happen.
Re: TCP Retransmission Situation
Re: TCP Retransmission Situation
Yeah, that kind of re-confirms my suspicions. Unfortunately, I believe the event was transitory and over with, so there is nothing to gather any longer. I'm thinking that the connection was established with the three-way handshake....and then ack responses started getting lost, or something else was going on (possibly due to time stamps) that caused excessive retransmissions.
