Printable View
Hello guys.
This is my problem:
I have to read a log file which can be really big (more than 40Mb).
This file is located on a remote server so, it is out of question copying this file to a local computer.. What I need to do is only reading the new log-entries, so I can put that information in a database.
So, this is my question: can I read the file bottom-up using vb6. If so, how?
If not, what kind of access do you recommend me?
Thank you.
You can try splitting entire text into array and then read array backwords.
It would be memory intence tough but also this could be true for any approach you choose.
Don't know if this is an option, or even do-able but here's an idea.
(I'm sure if it can or can't be done, then one of the gurus will say ;))
Get the line count of the file and save the total number of lines, so each time you search the file you only start at the last line before something new has been added.
RhinoBull, doing that, will I have to "push" the entire file to the server that will run my app? Because my problem is not in memory but in network/Internet traffic.
aikidokid, I don't know if it is possible but I shouldn't do that because the log file that I will try to read may be deleted by the software that created it and then, the order wont correspond to the truth.
Thank you
What about getting a mapped view of the file (see the link in my sig) and reading through it backwards as a byte array? It may be tricky to figure out when to stop, but then the index of the array will correspond to a byte offset in the mapped file, and you can just use that to copy the "memory" into a local string variable. I'm not sure how well memory maps to files work over a network, but it's worth a shot.
How about using seek function?
Keep the last read file location and seek to that. :confused:
:wave:Code:Option Explicit
Dim fileNum As Integer
Private Sub Command1_Click()
Dim str As String
fileNum = FreeFile
Open Text2.Text For Input As #fileNum
If Val(Text1.Text) <= LOF(1) Then
Seek #fileNum, Val(Text1.Text)
Else
Seek #fileNum, LOF(1)
End If
Line Input #fileNum, str
MsgBox str
MsgBox Seek(fileNum)
Text1.Text = Seek(fileNum)
Close #fileNum
End Sub
Private Sub Command2_Click()
fileNum = FreeFile
Open Text2.Text For Input As #fileNum
Text1.Text = LOF(fileNum)
Close #fileNum
End Sub
Private Sub Form_Load()
Text2.Text = "c:\temp\temp.txt"
Text1.Text = 1
End Sub
Years ago before memory became cheap, I used to wrestle with big files by embedding unique pointers in them as they were created or changed. Then I built a separate key or index file that was packed with long integers that I used to reach the pointer locations in the monster file.
To reach the information corresponding to the pointers, I used a technique similar to ZeeZee's, using Seek() to find what I needed. As the huge file was updated, the index file was updated with it. During retrieval, I read the index file first, which was small, and then leapfrogged to the huge file using Seek().
That way, I never had to read in much more than 1 Kb at a time to find and retrieve whatever I wanted. On occasion, I still use this procedure with files that exceed 10 Mb.
Not sure what that means... :confused:Quote:
Originally Posted by RS_Arm
You can't get away from network traffic but memory will still be affected when you assign text to a some variable and also when looping through array (this also affects cpu).Quote:
Originally Posted by RS_Arm
Contradicting :confused:Quote:
What I need to do is only reading the new log-entries,
This will be a big problem. If this situation is there, how would you make your program find out where to start other than reading the whole file line by line ????Quote:
the log file that I will try to read may be deleted by the software that created it and then, the order wont correspond to the truth.
Even if you start from the bottom, if the log is changed from previous log, how would you find it out?
BTW , what other options would you have?
Is this log a fixed length record type file? If so, you may seek to given line by giving one record length * no of records. You may even use Get statement.
You can use ADO Text Stream or FSO.
FSO has a skipline method so you can loop from start to given line using that.
You can handle the file in Client Side , then get only the results to your server.
:wave:
i have not done this, but i would try reading the files size, divide the file size into workable sized segments (maybe 1 mb), depending how much size you expect the new entries to take, then use open for random or binary and get from the last segment to first, if the last segment does not include all the new entries, then read the segment previous to it etc. checking the start of the segment should be able to determine if all the new entries are included
No way to read from the end of the file backwards, only forward even by selecting an arbitrary position in file.
If network traffic is the issue then implment application that reads the file at the remote server. If manual intervention is required then request remote desktop access. If you can modify the application that creates the log then have it insert a log entry directly to the database.
i am sorry, but that's just simply wrong.Quote:
Originally Posted by leinad31
mshttp will allow you to do so, or any other http provider for that manner, you dont need any server software installed, just a remotely decent server installation.
read up about the range header:
W3C HTTP specification : Range Headers
do a head request, notice the "Content-Length", and subtract, oh, say 1000000 from that to download the last meg.
once the sctring is in vb, Reverse() it if need be, and you should good to go!
Do you fully understand what you just suggested?Quote:
Originally Posted by rnd me
We are talking here about huge amount of characters to be reversed.
That means they will be doubled (at least) before you know it and therfore it may have significant impact on your entire system (performance wise of course).
Besides, what's http got to do with this if file is local or reachable within lan?
Quote:
Originally Posted by RhinoBull
http servers allow you to fetch partial files.
Drives, shares, etc typically don't provide that ability.
also i don't think it needs reversed.
if you grab the last 100kb for instance, you can split that by vbcrlf (or whatever).
something like:log files are usually pretty easy to parse, being based upon lines. You cannot do this with an xml doc for instance. You could get the ubound of lines above, and get just the last 100 or so. maybe you want to saveSetting the last log event, and automatically download further behind if needed. i leave that up to you. The hard part is getting it...Code:
dim lines() as string
dim out as string
lines= split(request, vbcrlf)
lines(0)="" 'throw away the first line, it might be partial.
out=join(lines)
msgbox out
when i think about files "located on a remote server", i think server. why download 39 megs of redundant data and, as rhino points out, process locally?
working on just the data you need allows maximum efficiency from both a computational and practical standpoint. i believe that what the OP was asking about.
To run program directly on the server you must install it first - far not every company allows that.Quote:
Originally Posted by rnd me
Therefore the only option left is mapping folder and on some workstation and process file "locally".
And again, how is http going to help you here I have not a slightest idea.
Quote:
Originally Posted by RhinoBull
My idea can best be expressed as an example:
1. open new standard exe
2. add a reference to Microsoft XML, v3.0
3. add a big textbox to the form, make it multi-line
4. paste the code below and press <F5>
you should see the last 30-40 lines of a "log-like file"Code:
Private Sub Form_Load()
Dim url2 As String ' the address of the page
Dim resp As String ' a string to hold the resulting response
Dim lines() As String ' an array of the results, partitioned by line
Dim out As String ' the final output
url2 = "http://tde.ornl.gov/Wateruse.csv" 'a free data set
resp = getEnd(url2, "5000") 'get the last 5kb
lines = Split(resp, vbLf)
lines(0) = "" 'throw away the first line, it might be partial...
out = Join(lines, vbCrLf)
Text1 = out
End Sub
Function getEnd(url As String, numberOfBytes As String) As String
Dim IO As New MSXML2.XMLHTTP30
Call IO.open("GET", url, False)
Call IO.setRequestHeader("Content-Type", "text/html")
Call IO.setRequestHeader("Range", "bytes=-" & numberOfBytes)
Call IO.send("")
getEnd = IO.responseText
End Function 'getEnd
This is my offer as a response to the original post:
If the server that holds your data is not http-addresable, or not a compliant http implementation, then answer to the original question is No, it is not possible. This is not because of a limit of VB6, but of the underlying infrastructure.Quote:
Originally Posted by RS_Arm
There may also be a vendor-proprietary approach to end of file reading, depending upon the type of network access you have to the "remote location". Might be worth checking into if above doesn't work...
Cheers, and good luck.
Its not different from what was already posted wherein you do a binary Get starting at an arbitrary position in the file. The details of the implementation was simply hidden. But sure why not, its good for the thread starter to learn web related stuff. You might as well include permission related samples since its required per request unlike a mapped drive.