-
Sep 15th, 2017, 06:22 PM
#1
Thread Starter
Hyperactive Member
Log File Size
I need to keep a log file of connected ip's every 5 minutes or so, it could be from 20 to 1000's of ips every five minutes. I am doing a raw log/file print append, but the files are getting pretty large fast, about 2mb a day for 400 ips all day.
Does anyone have a suggestion on making this more efficient?
The log is a auditing record to confirm my apps results and is required by legal authority
Thanks
-
Sep 15th, 2017, 07:49 PM
#2
Re: Log File Size
Originally Posted by axisdj
I need to keep a log file of connected ip's every 5 minutes or so, it could be from 20 to 1000's of ips every five minutes. I am doing a raw log/file print append, but the files are getting pretty large fast, about 2mb a day for 400 ips all day.
Does anyone have a suggestion on making this more efficient?
The log is a auditing record to confirm my apps results and is required by legal authority
Thanks
If the IP's repeat much, you can store then in a database table, and in a linked table just store the date/time in a field..
-
Sep 15th, 2017, 07:51 PM
#3
Thread Starter
Hyperactive Member
Re: Log File Size
That was the only thing I could think of, thanks Eduardo
-
Sep 15th, 2017, 09:18 PM
#4
Re: Log File Size
You could also periodically ZIP the log files...for example, before logging check the file size of the current log file. If it is > some arbitrary size then ZIP it up under a different file name and start a new log.
-
Sep 16th, 2017, 01:40 AM
#5
Re: Log File Size
Originally Posted by axisdj
That was the only thing I could think of, thanks Eduardo
I agree with Eduardo, I think that the more efficient way is to use a database because it make more simple to use in multi-user environment and create searches, reports, and so on...
But if the above mentioned characteristics do not need it, then the jpbro solution is also practicable, I use it in some scenarios.
-
Sep 16th, 2017, 03:38 PM
#6
Re: Log File Size
axisdj,
Why don't you go binary? It sounds like you're staying ANSI with your log file. For any IP, that's going to range from 8 to 15 bytes, depending on the IP (6.6.6.6 or 192.168.100.200).
If you went binary, each/every entry would take four bytes. Every entry of an IP ranges from 0 to 255, which is a byte. Just do CByte() on each piece and then write those into a binary file.
You'd need a bit of a header that possibly kept track of the number of entries in the file, but that's no biggie. You also couldn't read the file with Notepad, but I don't know if that's important or not.
Just my idea,
Elroy
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
-
Sep 16th, 2017, 04:29 PM
#7
Re: Log File Size
I think it is a good idea to convert IPs to Longs, but that won't support IPV6.
Partially based on this function:
Code:
Private Function IpToLong(nIP As String) As Long
Dim iOcts() As String
iOcts = Split(nIP, ".")
If Val(iOcts(0)) > 127 Then
IpToLong = ((Val(iOcts(0)) - &H80&) * &H1000000 Or &H80000000)
Else
IpToLong = Val(iOcts(0)) * &H1000000
End If
IpToLong = IpToLong Or (Val(iOcts(1)) * &H10000) Or (Val(iOcts(2)) * &H100&) Or Val(iOcts(3))
End Function
Private Function LongToIP(nLong As Long) As String
Dim iOcts(3) As String
iOcts(3) = CStr(nLong And &HFF)
iOcts(2) = CStr((nLong And &HFF00&) / &H100&)
iOcts(1) = CStr((nLong And &HFF0000) / &H10000)
If nLong < 0 Then
iOcts(0) = CStr(((nLong And &HFF000000 And Not &H80000000)) / &H1000000) + &H80
Else
iOcts(0) = CStr((nLong And &HFF000000) / &H1000000)
End If
LongToIP = Join(iOcts, ".")
End Function
-
Sep 16th, 2017, 07:38 PM
#8
Re: Log File Size
Hmmm, I didn't exactly say Longs, but that could work with IPV4. But Bytes, with the number of bytes changing depending on whether it's IPV4 or IPV6 could work. Possibly use two separate files, one that's IPV4 and the other that's IPV6. Also, is a timestamp being stored? That'd take more, but storing it as binary is always going to be much more efficient than ANSI.
Best Regards,
Elroy
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
-
Sep 16th, 2017, 10:23 PM
#9
Re: Log File Size
Originally Posted by Elroy
Hmmm, I didn't exactly say Longs, but that could work with IPV4. But Bytes, with the number of bytes changing depending on whether it's IPV4 or IPV6 could work. Possibly use two separate files, one that's IPV4 and the other that's IPV6. Also, is a timestamp being stored? That'd take more, but storing it as binary is always going to be much more efficient than ANSI.
Best Regards,
Elroy
I was thinking on a Long database field.
May be it could be converted to another data type that can hold it.
The idea is to have it in a field that can be queried.
Last edited by Eduardo-; Sep 17th, 2017 at 04:20 PM.
-
Sep 17th, 2017, 11:20 AM
#10
Re: Log File Size
Hmmmm, I'm not sure what the overhead is for adding a record to a database, but that seems to be going in the wrong direction to some degree. Especially if the main objective is to create a smaller file.
Also, if we're talking about very simple log entries, it wouldn't be much work to write a query procedure, if that's needed.
But hey, axisdj seems to have gotten quiet, so who knows if we're talking about things he needs or not.
Take Care,
Elroy
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
-
Sep 17th, 2017, 01:37 PM
#11
Re: Log File Size
Originally Posted by Elroy
Hmmmm, I'm not sure what the overhead is for adding a record to a database, but that seems to be going in the wrong direction to some degree. Especially if the main objective is to create a smaller file.
Also, if we're talking about very simple log entries, it wouldn't be much work to write a query procedure, if that's needed.
But hey, axisdj seems to have gotten quiet, so who knows if we're talking about things he needs or not.
Take Care,
Elroy
The "direction" of using a database is not to repeat IPs. (Read post #2 above.)
-
Sep 17th, 2017, 04:09 PM
#12
Re: Log File Size
If this is a log, it seems unlikely that write concurrency is required. If it is then you might be better off having the "writers" call a separate singleton process that does the actual writing.
Readers are easier, unless they need to read the current log as well as past released logs.
It sounds like you periodically (every 5 minutes?) log all of the current connections' remote end IP addresses. So you have a timestamp and a bag of IP addresses for each log record.
IPv4 addresses can indeed by stored as a DWORD, which fits into a VB Long just fine. If you have IPv6 as well then expand on this and you'd maybe need a "type" field as well as 128 bits per address.
In any case there is little advantage in having a table of IP addresses and using foreign keys. For IPv4 that will take more space and for for IPv6 the extra overhead will make it take more space as well.
The easiest way to do this is to have two files per log.
File 1 would have records containing a timestamp value, a file 2 offset value, and a file 2 record length value.
File 2 would just have blobs of DWORD IP addresses. Or if you have IPv6 instead then blobs of 8-byte IP addresses. Or if both then deal with that, probably adding an address type for each address.
This can all be wrapped together into a class for doing the actual writing and reading.
It isn't going to get much more compact than that, nor much faster. A database is a silly direction to be looking at for something like this.
-
Sep 18th, 2017, 10:26 AM
#13
Thread Starter
Hyperactive Member
Re: Log File Size
I agree, Thanks to all that gave advice, again, I cannot say it enough, a great group of very talented people here.
Originally Posted by dilettante
If this is a log, it seems unlikely that write concurrency is required. If it is then you might be better off having the "writers" call a separate singleton process that does the actual writing.
Readers are easier, unless they need to read the current log as well as past released logs.
It sounds like you periodically (every 5 minutes?) log all of the current connections' remote end IP addresses. So you have a timestamp and a bag of IP addresses for each log record.
IPv4 addresses can indeed by stored as a DWORD, which fits into a VB Long just fine. If you have IPv6 as well then expand on this and you'd maybe need a "type" field as well as 128 bits per address.
In any case there is little advantage in having a table of IP addresses and using foreign keys. For IPv4 that will take more space and for for IPv6 the extra overhead will make it take more space as well.
The easiest way to do this is to have two files per log.
File 1 would have records containing a timestamp value, a file 2 offset value, and a file 2 record length value.
File 2 would just have blobs of DWORD IP addresses. Or if you have IPv6 instead then blobs of 8-byte IP addresses. Or if both then deal with that, probably adding an address type for each address.
This can all be wrapped together into a class for doing the actual writing and reading.
It isn't going to get much more compact than that, nor much faster. A database is a silly direction to be looking at for something like this.
-
Sep 18th, 2017, 10:31 AM
#14
Re: Log File Size
Originally Posted by dilettante
A database is a silly direction to be looking at for something like this.
I actually completely agree with that statement, but I'm glad you said it first. A database just seems like way overkill for this job.
EDIT1: However, I'd have 3 (or maybe even 4) files. If 4:
1. IPV4 data.
2. IPV4 timestamps.
3. IPV6 data.
4. IPV6 timestamps.
Make them all binary to save space, and have a small header in the files that tracks how many entries are in each file, and just update that header-count each time an entry is made.
Best Regards,
Elroy
Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.
-
Sep 18th, 2017, 02:33 PM
#15
Re: Log File Size
I got the idea that for each interval event there would 1 timestamp and 0 to many IP addresses. No need to record one timestamp per IP address per interval.
Of course another issue that hasn't been thought out is for 1 Remote IP address there might be multiple connections. IP address isn't good enough, you also need the remote port number. NAT is very common, so expecting not to have duplicates is naive. For all we know multiple clients might be running on any given client machine too.
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|