I am developing a database program and for some reason it cannot be based on SQL.
Rather, it needs to be based on flat files.
For each entity for which I have to keep track of data, I declare a record type:
Code:
public type Something
...
...
End Type

Dim MyRec     as Something
And then I create a random access file like this:
Code:
      MyFilePath = "C:\somefolder ... \MyRecFile.DAT"
      FNr = FreeFile
      Open MyFilePath For Random As FNr Len = RecLen
         Put #FNr, 1, MyRec
      Close #FNr
Now, the issue is that, potentially, there could be millions of records, maybe tens of millions, or even more.
With a system like the above, all tens of millions of records will be stored in a single file (in the above example "MyRecFile.DAT").
Is this SINGLE FILE approach a good way of dealing with this situation or maybe it is better to break it down into multiple files?
What I am thinking is that I can for example set a limit of 100,000 records (or whatever number that is best) per file, and produce numbered files like:
Code:
MyRecFile1.DAT
MyRecFile2.DAT
MyRecFile3.DAT
...
and so on and so forth
...
so that if a new record is to be inserted into the N'th file and that N'th file already has 100,000 record, the program would instead create an N+1'th file, and insert the new record into that new file.
I am not sure which approach is better, and that is why I am asking.
Please note that there will be a lot of activities on this database, for example updating some records, searching, etc.
If a single file grows to tens of millions of records or even more, won't it be a problem to handle all kinds of updates and other activities?
Which approach is better; a single file, or a growing number of multiple files?
And if multiple files is better, what should the number of records per file be?
Please advise.
Thanks.