Error 14 Out of String Space *RESOLVED*

I am using the split function to parse data out of text files. I am now having a problem with one of the text files that I am reading. The problem is that the file is 114 megs, and this is where the "out of string space" error is coming from. Below is a sample of the code that I am using. Once the lines() array is populated I can iterate through the array and parse the data I need. Is there another way to read this information into an array like I am doing with the split function using the lines() array? If not does any one have any other suggestions?

As always, thanks in advance.

Michael


lines() = Split(ReadFile(TextFile), vbCrLf)


Public Function ReadFile(filename As String) As String

Dim isopen As Boolean

'Open file and assign a filenumber
fnum = FreeFile()
Open filename For Binary As #fnum
isopen = True
ReadFile = Input(LOF(fnum), fnum)

End Function

wow ... ive never seen anyone hit the limit on string variables ..

you could loop thru to EOF populateing array one line at a time with "Line Input #"

Is there a way that the file can be broken into sections using some sort of buffer?

Yeah, you're going to have to chunk it. That's some serious file you've got, I sure hope it was created automatically, and isn't somebodies magnum opus...cause if it is, that opus is mega magnum.

The basic problem that I see is that you are reading the whole file and trying to parse it as a giant string. Can you read it line by line? Wouldn't each line be separated by vbCRLF that way?

What is the best way to chunk the file? Is there a simple way to do this?

This is the line of code that is giving me the problem.

ReadFile = Input(LOF(fnum), fnum)

I think the file is just too big.

Quote:

---

Originally posted by MichaelC2468
What is the best way to chunk the file? Is there a simple way to do this?

This is the line of code that is giving me the problem.

ReadFile = Input(LOF(fnum), fnum)

I think the file is just too big.

---

try this:

VB Code:

Private Sub Form_Load()
Dim arrTextFile() As String
Call ReadFileByChunks("C:\test.txt", 100, arrTextFile())
Stop
End Sub
Sub ReadFileByChunks(sFileName As String, ChunkSize As Long, arrText() As String)
    Dim Data As String
    Open sFileName For Binary As #1
    i = 0
    ReDim arrText(1 To (LOF(1) / ChunkSize) + 1)
    Do Until LOF(1) = Loc(1) Or EOF(1)
        Data = ""
        i = i + 1
        If LOF(1) - Loc(1) < ChunkSize Then
            Data = String(LOF(1) - Loc(1), 0)
        Else
            Data = String(ChunkSize, 0)
        End If
        Get #1, , Data
        arrText(i) = Data
    Loop
End Sub

How is the data in your text file seperated? By vbCrLf's, spaces, ???

Quote:

---

Originally posted by Eras3r
How is the data in your text file seperated? By vbCrLf's, spaces, ???

---

It is seperated by VbCrLf's.

Quote:

---

Originally posted by MichaelC2468
It is seperated by VbCrLf's.

---

Then reading by line should work

VB Code:

Dim arrText() As String
Dim i As Integer
i = 0
Open "c:\test.txt" For Input As #1
Do Until EOF(1)
    i = i + 1
    ReDim Preserve arrText(i)
    Line Input #1, arrText(i)
Loop

Quote:

---

Originally posted by Muddy
Then reading by line should work

VB Code:

Dim arrText() As String
Dim i As Integer
i = 0
Open "c:\test.txt" For Input As #1
Do Until EOF(1)
    i = i + 1
    ReDim Preserve arrText(i)
    Line Input #1, arrText(i)
Loop

---

I've tried this, and once it starts running the "Line Input" code it hangs. I left it for like 30 minutes and nothing. The code that was posted earlier that breaks the file into chunks works really well. The only thing is that I never know what the chunk size is going to be, because the location of the vbCrLf, or line length can vary from month to month.

I am really stuck on this one. I can't seem to come up with a solution to what appears to be a simple problem.

Quote:

---

Originally posted by MichaelC2468
I've tried this, and once it starts running the "Line Input" code it hangs. I left it for like 30 minutes and nothing. The code that was posted earlier that breaks the file into chunks works really well. The only thing is that I never know what the chunk size is going to be, because the location of the vbCrLf, or line length can vary from month to month.

I am really stuck on this one. I can't seem to come up with a solution to what appears to be a simple problem.

---

could it be that there are no carriage returns (sounds kinda freaky to use that old term from the typwriter days doesnt it? :-) ) maybe they are in fact just vbtab or vblf. (line input reads till chr(13) (carrriage return) )

you definitly want to be using "Line Input" if your data is broken up by vbCrLf's.....

I found the problem...
The carrier that we get the file from changed something in it. I now have a fixed length delimited file that appears to be dumped as a single line with vbcrlfs embedded. I resorted to "Chunking" it by the record length and stripping away the vbcrlfs. The processing is now going as smoothly as ever.

Hi, Michael

I got the same issue with your. But I do not understand your solution. Can you help to give me more details about it? It will be better that post your code here.

To be honest, this was so long ago I forgot about it. However, it seems like I used the solution mentioned by muddy a few posts up and read the file in chuncks.

Using byte arrays is a more efficient way to read a bigger file:

Code:

---

Dim bytFile() As Byte, FF As Integer

FF = FreeFile

Open "test.txt" For Binary Access Read As #FF
    ReDim bytFile(0 to LOF(FF) - 1)
    Get #FF, , bytFile
Close #FF

---

If you only need to get a partial position from the byte array as a string, you can use the following API function to do it:

Declare Function ByteString Lib "oleaut32" Alias "SysAllocStringByteLen" (Ptr As Byte, ByVal Length As Long) As String

So if your file contains the text "ABCDE" and you use a line like this:

MsgBox ByteString(bytFile(1), 3)

You will see a message box with letters "BCD". Remember, index 0 = first byte.


Making the entire byte array a string can be done like this:

strFile = StrConv(bytFile, vbUnicode)
Erase bytFile

Remember though that this is only good for textual data and with big files you may get the string space error again. Also, using this for binary files can be harmful, because bytes (0 – 255) will result in different character codes (especially often in the 128 – 255 range) and the results depend on which locale the code is executed.