Sugestion about reading huge text file

[RS_Arm]

Hello guys.
This is my problem:
I have to read a log file which can be really big (more than 40Mb).
This file is located on a remote server so, it is out of question copying this file to a local computer.. What I need to do is only reading the new log-entries, so I can put that information in a database.
So, this is my question: can I read the file bottom-up using vb6. If so, how?
If not, what kind of access do you recommend me?

Thank you.

[RhinoBull]

You can try splitting entire text into array and then read array backwords.
It would be memory intence tough but also this could be true for any approach you choose.

[aikidokid]

Don't know if this is an option, or even do-able but here's an idea.
(I'm sure if it can or can't be done, then one of the gurus will say )

Get the line count of the file and save the total number of lines, so each time you search the file you only start at the last line before something new has been added.

[RS_Arm]

RhinoBull, doing that, will I have to "push" the entire file to the server that will run my app? Because my problem is not in memory but in network/Internet traffic.

aikidokid, I don't know if it is possible but I shouldn't do that because the log file that I will try to read may be deleted by the software that created it and then, the order wont correspond to the truth.

Thank you

[Comintern]

What about getting a mapped view of the file (see the link in my sig) and reading through it backwards as a byte array? It may be tricky to figure out when to stop, but then the index of the array will correspond to a byte offset in the mapped file, and you can just use that to copy the "memory" into a local string variable. I'm not sure how well memory maps to files work over a network, but it's worth a shot.

[zeezee]

How about using seek function?
Keep the last read file location and seek to that.

Option Explicit
Dim fileNum As Integer

Private Sub Command1_Click()
    Dim str As String
    fileNum = FreeFile
    Open Text2.Text For Input As #fileNum
    
        If Val(Text1.Text) <= LOF(1) Then
            Seek #fileNum, Val(Text1.Text)
        Else
            Seek #fileNum, LOF(1)
        End If
        Line Input #fileNum, str
        MsgBox str
        MsgBox Seek(fileNum)
        Text1.Text = Seek(fileNum)
    Close #fileNum
    
    
End Sub

Private Sub Command2_Click()
    fileNum = FreeFile
    Open Text2.Text For Input As #fileNum
    Text1.Text = LOF(fileNum)
    Close #fileNum
End Sub

Private Sub Form_Load()
    Text2.Text = "c:\temp\temp.txt"
    Text1.Text = 1
End Sub

[Code Doc]

Years ago before memory became cheap, I used to wrestle with big files by embedding unique pointers in them as they were created or changed. Then I built a separate key or index file that was packed with long integers that I used to reach the pointer locations in the monster file.

To reach the information corresponding to the pointers, I used a technique similar to ZeeZee's, using Seek() to find what I needed. As the huge file was updated, the index file was updated with it. During retrieval, I read the index file first, which was small, and then leapfrogged to the huge file using Seek().

That way, I never had to read in much more than 1 Kb at a time to find and retrieve whatever I wanted. On occasion, I still use this procedure with files that exceed 10 Mb.

[RhinoBull]

RhinoBull, doing that, will I have to "push" the entire file to the server ...

Not sure what that means...

... my problem is not in memory but in network/Internet traffic.

You can't get away from network traffic but memory will still be affected when you assign text to a some variable and also when looping through array (this also affects cpu).

[zeezee]

What I need to do is only reading the new log-entries,

Contradicting

the log file that I will try to read may be deleted by the software that created it and then, the order wont correspond to the truth.

This will be a big problem. If this situation is there, how would you make your program find out where to start other than reading the whole file line by line ????

Even if you start from the bottom, if the log is changed from previous log, how would you find it out?

BTW , what other options would you have?

Is this log a fixed length record type file? If so, you may seek to given line by giving one record length * no of records. You may even use Get statement.

You can use ADO Text Stream or FSO.
FSO has a skipline method so you can loop from start to given line using that.

You can handle the file in Client Side , then get only the results to your server.

[westconn1]

i have not done this, but i would try reading the files size, divide the file size into workable sized segments (maybe 1 mb), depending how much size you expect the new entries to take, then use open for random or binary and get from the last segment to first, if the last segment does not include all the new entries, then read the segment previous to it etc. checking the start of the segment should be able to determine if all the new entries are included

[leinad31]

No way to read from the end of the file backwards, only forward even by selecting an arbitrary position in file.

If network traffic is the issue then implment application that reads the file at the remote server. If manual intervention is required then request remote desktop access. If you can modify the application that creates the log then have it insert a log entry directly to the database.

[rnd me]

No way to read from the end of the file backwards, only forward even by selecting an arbitrary position in file.
.

i am sorry, but that's just simply wrong.


mshttp will allow you to do so, or any other http provider for that manner, you dont need any server software installed, just a remotely decent server installation.

read up about the range header:

W3C HTTP specification : Range Headers


do a head request, notice the "Content-Length", and subtract, oh, say 1000000 from that to download the last meg.

once the sctring is in vb, Reverse() it if need be, and you should good to go!

[RhinoBull]

once the sctring is in vb, Reverse() it if need be, and you should good to go!

Do you fully understand what you just suggested?
We are talking here about huge amount of characters to be reversed.
That means they will be doubled (at least) before you know it and therfore it may have significant impact on your entire system (performance wise of course).
Besides, what's http got to do with this if file is local or reachable within lan?

[rnd me]

Besides, what's http got to do with this if file is local or reachable within lan?

http servers allow you to fetch partial files.
Drives, shares, etc typically don't provide that ability.
also i don't think it needs reversed.
if you grab the last 100kb for instance, you can split that by vbcrlf (or whatever).

something like:

dim lines() as string
dim out as string
lines= split(request, vbcrlf)
lines(0)=""  'throw away the first line, it might be partial.
out=join(lines)
msgbox out

log files are usually pretty easy to parse, being based upon lines. You cannot do this with an xml doc for instance. You could get the ubound of lines above, and get just the last 100 or so. maybe you want to saveSetting the last log event, and automatically download further behind if needed. i leave that up to you. The hard part is getting it...

when i think about files "located on a remote server", i think server. why download 39 megs of redundant data and, as rhino points out, process locally?

working on just the data you need allows maximum efficiency from both a computational and practical standpoint. i believe that what the OP was asking about.

[RhinoBull]

... why download 39 megs of redundant data and, as rhino points out, process locally?..

To run program directly on the server you must install it first - far not every company allows that.
Therefore the only option left is mapping folder and on some workstation and process file "locally".
And again, how is http going to help you here I have not a slightest idea.

[rnd me]

To run program directly on the server you must install it first - far not every company allows that.
Therefore the only option left is mapping folder and on some workstation and process file "locally".
And again, how is http going to help you here I have not a slightest idea.

My idea can best be expressed as an example:

1. open new standard exe
2. add a reference to Microsoft XML, v3.0
3. add a big textbox to the form, make it multi-line
4. paste the code below and press <F5>

Private Sub Form_Load()

Dim url2 As String      ' the address of the page
Dim resp As String      ' a string to hold the resulting response
Dim lines() As String   ' an array of the results, partitioned by line
Dim out As String       ' the final output

url2 = "http://tde.ornl.gov/Wateruse.csv"  'a free data set
resp = getEnd(url2, "5000")   'get the last 5kb

lines = Split(resp, vbLf)
lines(0) = "" 'throw away the first line, it might be partial...

out = Join(lines, vbCrLf)
Text1 = out

End Sub


Function getEnd(url As String, numberOfBytes As String) As String
  Dim IO As New MSXML2.XMLHTTP30
    Call IO.open("GET", url, False)
    Call IO.setRequestHeader("Content-Type", "text/html")
    Call IO.setRequestHeader("Range", "bytes=-" & numberOfBytes)
    Call IO.send("")
 getEnd = IO.responseText
 
 End Function   'getEnd

you should see the last 30-40 lines of a "log-like file"

This is my offer as a response to the original post:

So, this is my question: can I read the file bottom-up using vb6. If so, how?
If not, what kind of access do you recommend me?

If the server that holds your data is not http-addresable, or not a compliant http implementation, then answer to the original question is No, it is not possible. This is not because of a limit of VB6, but of the underlying infrastructure.

There may also be a vendor-proprietary approach to end of file reading, depending upon the type of network access you have to the "remote location". Might be worth checking into if above doesn't work...


Cheers, and good luck.

[leinad31]

Its not different from what was already posted wherein you do a binary Get starting at an arbitrary position in the file. The details of the implementation was simply hidden. But sure why not, its good for the thread starter to learn web related stuff. You might as well include permission related samples since its required per request unlike a mapped drive.