[RESOLVED] Why is reading a File in chunks quicker than in one go?

[some1uk03]

Hello,

I stumbled upon something that's making me think now..

Previously, the program was Loading chunks from the files by
opening them as Binary, and reading chunks off:

Get #fFile, xPos + 1, RAMBuffer

This was pretty fast. 90 files took 5 secs to load.

But the problem was, since its using the Native OPEN statement it doesn't seem to handle Unicode paths.
So if a user is using German, Chinese Windows etc... the Path to the File would contain Unicode, hence error!

So I thought, why don't I instead Load the File all at Once [via CreateFile] in to a String Buffer and then using mid$() to split chunks out to strings. Now this takes almost 3x the time 16 seconds to complete :/


I thought doing things in Memory should be quicker than reading off of the HD in a loop ?

[techgnome]

Hmmm... I'd be willing to bet it's the use of mid$() that's actually the problem, not the reading chunks vs whole file. String manipulation in general is always slower (your first snip insinuates binary access, the second indicates string manipulations). And it has more to do with how string manipulation happens - they're immutable, which means in order to change a string, a new memory area needs to be allocated, and the string copied. A Byte-based buffer on the other hand doesn't have that limitation.

-tg

[some1uk03]

OMG...

Having done further tests, I can now confirm that the delay lays not with the MID function, but I had a Single instr check which seemed to be taking ages as it used to do a binary comparison. Having changed it to a textcomparison it's all super quick now even faster than the previous Binary version From 5 secs down to less than a second

[LaVolpe]

That doesn't sound quite right either. Binary comparison should be far faster than text comparison. With binary comparison, either byte equality exist or not. For text comparison, byte can bet this or that or maybe this other thing. I'd guess that looping and/or loading bytes into strings (vice versa) may be the bottleneck.

[some1uk03]

Yes, binary comparison should be faster than text, but if the compared input is of Binary ?

Well, I literally analysed it line by line and even skipping WHOLE chunk so it goes alll the way to the end of the loop to check speeds with Gettickcount and skipping all mid() functions literally had no effect on speed, until the instr() call.

As soon as changing

FPos = InStr(1, strGetNames, "RFG", vbBinaryCompare)
to
FPos = InStr(1, strGetNames, "RFG")

Everything is super fast

[DataMiser]

vbBinaryCompare is the default so you did not really change anything

[dilettante]

For all we know he managed to get something called "vbBinaryCompare" defined with a non-zero value. Probably as a side-effect of not using Option Explicit.