Concurrent fle access

[Bulldog]

I have a text file which is being constantly written to by an app. That file is shared with a second app which reads/processes the data. I do the reading part using a filestream/streamreader and it all works fine.

The file will however get larger and larger over time. So I really want the second app to read the newest line(s) of data, then delete what it's just read. Since the file is shared however, I cannot delete the file and/or recreate it (since that will cause the 'writing app' to crash). Is there anyway I can delete lines from a shared file but keep it intact?

[jmcilhinney]

In situations like this, you really should provide a single point of contact with the file, which means a service. Your other applications can talk to the service and the service can then manage access to the file. If that's not possible then I would suggest that any write operation on the file should require exclusive access and all operations should include fault tolerance, e.g. try to access the file and, if that fails, wait for a period of time before trying again.

[Bulldog]

I've just tried messaging but that was disappointingly slow (and the write/read rates were not well matched). I was just about to try switching between two files, but I think the latency would be too high.

I will take a look at making a service, to act as an arbiter.

[Shaggy Hiker]

Why does there have to be a file at all in this case? Well...I guess I can think of a good reason, such as storage of the final data. But as far as the applications transferring data between each other, it would be faster to just do that directly using something like TCP, UDP, or some other such thing. Then you wouldn't have to worry about more than one application dealing with the file. Of course, there could be plenty of extenuating circumstances, so that suggestion may well be a non-starter.

[Bulldog]

The file(s) are used by more than one app, so they'll always be present. The 'writing' app incidentally is a sizable C++ project over which I have full control, so I can play around with it.

I did try messaging but it was actually very slow, not sure why. I think it's because the writer is already messaging a server (using a custom protocol) and sending a lot of data. I wrote a client to tap into that protocol but it was very sluggish and occasionally stopped completely.

I did get one thing working at full speed, not an arbitration service as such but just a nutty idea I had earlier;

(a) Writer overwrites FileA and then appends lines of data to FileA
(b) When writer has written 10k lines, it writes a SWOP line to FileA.
(c) Writer then resets count
(d) Writer overwrites FileB and then appends lines of data to FileB

In parallel,

(a) Reader reads from FileA (it's ok if the file happens to be empty)
(b) If Reader encounters a SWOP instruction, it checks FileB exists (if not it waits)
(c) Reader starts reading from FileB

This then swops back from FileB to FileA after 10k lines.

This works quite well because the Read interval is faster than the Write interval, the files never exceed 10k lines of data.

I'll make something better tomorrow...