Hi,


I have an application. It loads data from several folders. IN each folder, it has many csv files. Bascially, they are for indexing like:

index, value
index, value
....


the program loads data from all files (all data are in the variables, list(of integer)), then do some calculations. during that, it will need search the index to get some parameters.

it works fine. however, as time goes, we have more and more indexing files. sometimes, the data under ONE folder could have several millions indexes.

Now, the program runs slowly even on a 8G PC (although result is still correct), and we are having more data coming in.

my question is: if I change this to use a sql expression database, it sure will use much much less memory, but how about the performance? will it run much faster? or it really depends, and I have work a working version to compare?

is there a general rule like when the data is such such size, use database will be much better?

(most cases, it will use SELECT to query the table only).

any suggestions?



thanks bear