[RESOLVED] Huge Files/Memory problem...

[TupacShakur]

Hey all,
this is my problem:

I have a number (about a 100) of 5 MB's files, which are slices of one huge file. I need to put those slices together to get the original file, of around 500 MB's. Now in theory, i can do this, but when it comes to execution, it's causing a whole load of problems - system wise, crashes, freezing - due to the fact that at some point, i have the whole file loaded in memory (either a string variable or a string array). One additional thing is that i am using binary read/write access cause the file is not a text file (it may be a video, a data pack...).

So what would you advise me to do?? And i am not til now fully understanding the binary access usage. I mean how can i write-append each slice after reading it into the target file, instead of building the whole file in memory??

Thanks in avance to all u experts .

[Jacob Roman]

If you are using a loop, could it be because you are not using DoEvents to prevent it from freezing?

[CVMichael]

Why are you reading the whole file in the first place ?

Just read and write with a ~64K buffer....

[BodwadUK]

Open your large file

start and loop on your smaller files and copy it in 1MB or less at a time. Dont store it all in strings just your bits you are copying over

[Joacim Andersson]

Also use byte arrays instead of strings since VB internally use Unicode strings which use 2 bytes for each character, meaning it will take up twice as much memory.

[BodwadUK]

Are you sure? You can only chr() upto 255 cant you hence 1 byte per character

[Joacim Andersson]

The Chr() function is an Ansi function, that returns a Unicode string. All Input(), Line Input #, Print #, statements also use Ansi strings. VB also always pass Ansi strings to API functions, so a lot of conversion is done. Nevertheless internally a string is stored in Unicode format and it has done so since VB4-32bit.

If you don't believe me just try the following to see how many bytes a string represents:

MsgBox LenB("Hello") '5 characters, but 10 bytes of memory

[BodwadUK]

I never knew that. How strange

[TupacShakur]

Many thanx to all who posted here, it was really helpful.
This is my 1st experience with binary/random file access and the problem was that i wasnt able to write an "append" function (i figured it out almost after i started this topic, it's about the 2nd param in the Put function).

Anyway, now that i got deep into the subject, 3 questions arose:


1) Which is better/faster of these possibilities (in terms of system resources usage and speed):

- To Get and Put a 100 times a 1 MB chunk?

- To Get and Put 20 times a 5 MB chunk?

- To Get and Put 1 time a 100 MB chunk?


2) Is it a bad practice to open and close the file i'm writing the data to every time i want to write a chunk? I mean in case i opt to write 100 times in 1 MB chunks, is it bad to open and close the file each time, or is it the same as opening it once and closing it once (again, in terms of systems resources and speed)??


3) I learned from this thread that Strings variables are 2 bytes of size, cause they hold Uniocde data, so other than memory usage, there's also the problem of the conversions that occur. I tried to use byte arrays before using strings, but the problem is that my array isnt of fixed size. I am always rediming it due to some considerations. When i write to disk a non fixed-size byte array with the Put call, 2 bytes are added to the file by the Put function itself. Thus, my data is not being written as it is. What can i do?

Thanks againg !!

[BodwadUK]

safer to do 1MB at a time to prevent memory issues.


Put doesnt add anything. remember unless to specify Array(1 to whatever) the array starts at 0

[Joacim Andersson]

1) Which is better/faster of these possibilities (in terms of system resources usage and speed):

This is actually a harder question then it might seem to answer. When it comes to speed one could say that writing/reading the whole enchelada in one large chunk would be the fastest. This is not true in all cases, simply because of memory issues. Let's say that a regular desktop computer today have 128MB of RAM. When that is filled Windows starts disk swapping, and when that occurs things might take longer then if you yourself had used smaller chunks. In general, with the examples you've provided I would go with 5MB chunks for a file that is >=100 MB in size when it comes to writing the data. When it comes to reading it it depends on what you're doing with the data input. But the general rule would still apply.

2) Is it a bad practice to open and close the file i'm writing the data to every time i want to write a chunk? I mean in case i opt to write 100 times in 1 MB chunks, is it bad to open and close the file each time, or is it the same as opening it once and closing it once (again, in terms of systems resources and speed)??

Opening and closing a file always takes time and resources. I would definitly not suggest you closing the file until you're done.

3) I learned from this thread that Strings variables are 2 bytes of size, cause they hold Uniocde data, so other than memory usage, there's also the problem of the conversions that occur. I tried to use byte arrays before using strings, but the problem is that my array isnt of fixed size. I am always rediming it due to some considerations. When i write to disk a non fixed-size byte array with the Put call, 2 bytes are added to the file by the Put function itself. Thus, my data is not being written as it is. What can i do?

The Put # statement only pads the byte array if you open it as Random. Don't do that open it as Binary instead!

[TupacShakur]

This is actually a harder question then it might seem to answer. When it comes to speed one could say that writing/reading the whole enchelada in one large chunk would be the fastest. This is not true in all cases, simply because of memory issues. Let's say that a regular desktop computer today have 128MB of RAM. When that is filled Windows starts disk swapping, and when that occurs things might take longer then if you yourself had used smaller chunks. In general, with the examples you've provided I would go with 5MB chunks for a file that is >=100 MB in size when it comes to writing the data. When it comes to reading it it depends on what you're doing with the data input. But the general rule would still apply.

This the answer i was looking for ! I mean this is exactly the issue, the memory swap issue.. I am using Winram Turbo, and i am testing my program in a very stressful simulated system. So what would u suggest for the following ranges (saying 3 or 4 ranges between 100 MB's and 1 GB):
100 MB -> ??? : using 5 MB's buffer.
??? MB -> ???
....

Opening and closing a file always takes time and resources. I would definitly not suggest you closing the file until you're done.

Thanks for the advice, i will readapt my code to do so.

[Joacim Andersson]

This the answer i was looking for ! I mean this is exactly the issue, the memory swap issue.. I am using Winram Turbo, and i am testing my program in a very stressful simulated system. So what would u suggest for the following ranges (saying 3 or 4 ranges between 100 MB's and 1 GB):
100 MB -> ??? : using 5 MB's buffer.
??? MB -> ???

It's not a question of having much larger buffers for larger files. When you've reach the roof for when a system starts the disk swapping you have. That doesn't change because the file is larger. This is of course different on different systems. You and your users must of course understand that a 1 GB file is a huge amount of data and to process that takes some time. On most systems you can probably use larger buffers then 5MB. I don't know what exactly you are doing in your program but maybe one option would be to use a standard size buffer and have like an Option dialog box in witch the user can change the default size if he/she likes to.

[BodwadUK]

its always best to keep it low so that you know it will work on other pcs

thats what i find anyway

[TupacShakur]

Thanks to all who replied to this thread, and to Joacim in particular. It really helped me optimise my code for best peroformance, specially on the matter of the smaller buffers, byte arrays vs. strings, and other issues.

After many tests, monitoring memory and clocking execution, i decided on 3 kind of buffers (1 MB, 5 MB, and 10 MB) for 3 file ranges.