Page 1 of 2 12 LastLast
Results 1 to 40 of 59

Thread: [RESOLVED] multiple range checking algorithm vb6

  1. #1

    Thread Starter
    Hyperactive Member
    Join Date
    Jul 2002
    Posts
    505

    Resolved [RESOLVED] multiple range checking algorithm vb6

    Hello,

    Working on a new project that needs to identify ony USA based IP addesses.

    I have a list of USA based IP's which I will convert to ranges of Longs.. example (3.0.0.0 - 3.255.255.255 = 218103808 - 22229811)

    So I will place this list into memory somehow.

    Would like suggestions on how best to find when in ip is in range from a list of about 7000 ranges?

    thanks

  2. #2
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by axisdj View Post
    Would like suggestions on how best to find when in ip is in range from a list of about 7000 ranges?

    thanks
    Are you familiar with using AND OR XOR bit masking. That is the answer I'd think.
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  3. #3

    Thread Starter
    Hyperactive Member
    Join Date
    Jul 2002
    Posts
    505

    Re: multiple range checking algorithm vb6

    very weak when it comes to that, any help would be appreciated

  4. #4
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    10,909

    Re: multiple range checking algorithm vb6

    IMHO, this seems to be another case where the Decimal data type might come in handy.

    When looking at this page (which is what you're trying to filter for, I believe), we see that there isn't a great deal of rhyme-or-reason for the ranges.

    Also, regardless of whether you go little-endian or big-endian on the bytes as they're placed into Longs. You're going to run into a byte that's >127. What that means is that you're going to have negative values for your Longs. To solve this problem, I'd tend to employ the Decimal type, which can hold much larger integers.

    Then, it just seems that you have a lot of less-than and greater-than testing to do.

    Good Luck,
    Elroy
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  5. #5
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    10,909

    Re: multiple range checking algorithm vb6

    It seems that something like the following (along with all the ranges on the Nirsoft page) would get it done:

    Code:
    
    Option Explicit
    
    Public Function MakeDecimalFromIP(ip As String) As Variant
        ' Input must be a correctly formatted IP address.
        '
        Dim a() As String
        Dim s As String
        '
        a = Split(ip, ".")
        s = Right$("00" & a(0), 3) & Right$("00" & a(1), 3) & Right$("00" & a(2), 3) & Right$("00" & a(3), 3)
        MakeDecimalFromIP = CDec(s)
    End Function
    
    Public Function IP_IsInRange(IP_ToTest, IP_LowRange, IP_HighRange) As Boolean
        Dim lo As Variant
        Dim hi As Variant
        Dim ip As Variant
        '
        lo = MakeDecimalFromIP(IP_LowRange)
        hi = MakeDecimalFromIP(IP_HighRange)
        ip = MakeDecimalFromIP(IP_ToTest)
        '
        IP_IsInRange = ip >= lo And ip <= hi
    End Function
    
    
    Enjoy,
    Elroy

    EDIT1: Also, as you scan that Nirsoft page, you should see that you can do a tremendous amount of consolidating of those ranges, as they're contiguously connected. Also, I probably should have converted to Hex in my MakeDecimalFromIP, but it's going to work fine just as it is.
    Last edited by Elroy; Aug 17th, 2017 at 01:03 PM.
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  6. #6

    Thread Starter
    Hyperactive Member
    Join Date
    Jul 2002
    Posts
    505

    Re: multiple range checking algorithm vb6

    Ok.. the above is exactly what I was thinking, But I have 7000 ranges to check...

    what would be the fastest way to do that?

  7. #7
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by axisdj View Post
    Ok.. the above is exactly what I was thinking, But I have 7000 ranges to check...

    what would be the fastest way to do that?
    make an array with the base value and the top value

    then run the array and figure if => than and < than

  8. #8

    Thread Starter
    Hyperactive Member
    Join Date
    Jul 2002
    Posts
    505

    Re: multiple range checking algorithm vb6

    sorry guys, my fault for not giving more detail.

    That was what I was going to do, traverse the array until I find a match. But I would need to do 100's of these per second the first time th ip visits ( would save it in a list for quicker access later). So I guess what I am asking is there a fast way to set up the array so that it would be searched as fast as possible to find a range match.


    Thinking about it now, I could make the range Minimum number the index of the array??

    WP

    Quote Originally Posted by flyguille View Post
    make an array with the base value and the top value

    then run the array and figure if => than and < than
    Last edited by axisdj; Aug 17th, 2017 at 02:28 PM. Reason: later thought

  9. #9
    Fanatic Member Spooman's Avatar
    Join Date
    Mar 2017
    Posts
    868

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by axisdj View Post
    Hello,

    Working on a new project that needs to identify ony USA based IP addesses.

    I have a list of USA based IP's which I will convert to ranges of Longs.. example (3.0.0.0 - 3.255.255.255 = 218103808 - 22229811)

    So I will place this list into memory somehow.

    Would like suggestions on how best to find when in ip is in range from a list of about 7000 ranges?

    thanks
    I'm not quite clear as to how this translates into this, which BTW, equals 195,873,997.

    Be that as it may, would it help to deal with the 4 "parts" individually?
    ie ..
    3 to 3
    0 to 255
    0 to 255
    0 to 255

    EDIT-1:

    For example .. "to be checked" IP is 004.011.212.003

    Your list of 7,000 has:

    003.001.001.002 .. 1st in list
    :
    003.101.201.004
    :
    003.255.000.255 .. last in list

    You'd know at "pass 1" that there is no 004 as the 1st "part", so, you're done
    Please correct me if I've missed the (ahem) point.

    Spoo
    Last edited by Spooman; Aug 17th, 2017 at 02:46 PM.

  10. #10
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by axisdj View Post
    sorry guys, my fault for not giving more detail.

    That was what I was going to do, traverse the array until I find a match. But I would need to do 100's of these per second the first time th ip visits ( would save it in a list for quicker access later). So I guess what I am asking is there a fast way to set up the array so that it would be searched as fast as possible to find a range match.


    Thinking about it now, I could make the range Minimum number the index of the array??

    WP
    VB is very fast you can check against an array dozen of thousands of times per second.

    Just take your time to measure.... it is simple numericals.

    oh, and another thing.

    you really don't needs to convert it to decimals, normally the Internet is build up in pre-assigned ranges to ISPs and are all BASED on the first and second number.

    Don't use decimals.

    uses ranges this way (first*256)+second, so it still Integer value.

    so

    private const Total = 1000
    private LastInUse as integer
    private Base(0 to Total-1) as long
    private Top(0 to Total-1) as long

    then

    _ConnectionRequest
    dim x as integer, y as integer
    dim STRD as string
    dim num as long

    strd=.remoteIP

    x=instr(strd,".")
    y=instr(x+1,strd,".")
    num=val(mid$(strd,1,x-1)) * 256 + val(mid$(strd,x+1,y-(x+1)))

    if LastInUse>=0 then
    for x=0 to LastInUse

    if num>= base(x) and num <top(x) then ' Ok, is in USA.

    next x
    endif




    EDIT: But if you want it FASTER, I recommend you to use LOOK UP TABLES.

    IT is easy.



    private IsInUSA(0 to 255, 0 to 255) as boolean


    Configure the table one time.... TRUE if in USA, FALSE if not.


    then...

    _ConnectionRequest
    dim x as integer, y as integer
    dim STRD as string


    strd=.remoteIP

    x=instr(strd,".")
    y=instr(x+1,strd,".")

    if IsInUSA(val(mid$(strd,1,x-1)), val(mid$(strd,x+1,y-(x+1)))) then ' Is in USA.
    Last edited by flyguille; Aug 17th, 2017 at 02:56 PM.

  11. #11

    Thread Starter
    Hyperactive Member
    Join Date
    Jul 2002
    Posts
    505

    Re: multiple range checking algorithm vb6

    Brilliant Answers guys, I was going in that direction

    Thank you to all for your insight, will let you know how it turns out.

    WP

    Quote Originally Posted by flyguille View Post
    VB is very fast you can check against an array dozen of thousands of times per second.

    Just take your time to measure.... it is simple numericals.

    oh, and another thing.

    you really don't needs to convert it to decimals, normally the Internet is build up in pre-assigned ranges to ISPs and are all BASED on the first and second number.

    Don't use decimals.

    uses ranges this way (first*256)+second, so it still Integer value.

    so

    private const Total = 1000
    private LastInUse as integer
    private Base(0 to Total-1) as long
    private Top(0 to Total-1) as long

    then

    _ConnectionRequest
    dim x as integer, y as integer
    dim STRD as string
    dim num as long

    strd=.remoteIP

    x=instr(strd,".")
    y=instr(x+1,strd,".")
    num=val(mid$(strd,1,x-1)) * 256 + val(mid$(strd,x+1,y-(x+1)))

    if LastInUse>=0 then
    for x=0 to LastInUse

    if num>= base(x) and num <top(x) then ' Ok, is in USA.

    next x
    endif




    EDIT: But if you want it FASTER, I recommend you to use LOOK UP TABLES.

    IT is easy.



    private IsInUSA(0 to 255, 0 to 255) as boolean


    Configure the table one time.... TRUE if in USA, FALSE if not.


    then...

    _ConnectionRequest
    dim x as integer, y as integer
    dim STRD as string


    strd=.remoteIP

    x=instr(strd,".")
    y=instr(x+1,strd,".")

    if IsInUSA(val(mid$(strd,1,x-1)), val(mid$(strd,x+1,y-(x+1)))) then ' Is in USA.

  12. #12
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: multiple range checking algorithm vb6

    FYI: However you are creating your ranges, if you sort them in a way that the ranges that contain the most possible IPs are at the top of the array/list, then you are likely to match those first. A more advanced option would be a modified binary search. A binary search can determine if a value fits within a list of 1 million items, in 20 iterations or less. If interested, may want to search the forum on the key words: binary search
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  13. #13
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    10,909

    Re: multiple range checking algorithm vb6

    Unless I'm confused, which I seem to be this evening, the attached does what you're looking for.

    Enjoy,
    Elroy

    EDIT1: And I'm not saying it couldn't be done better and/or faster, but that certainly seems to get it done. Truth be told, if it were for myself, there are quite a few areas I'd tend to clean up before I'd actually use it. But again, it works.

    EDIT2: Also, the more I think about it, this is a place where you could probably use Currency and still not have to deal with negative numbers. You could just ignore the four decimal places and still have enough precision, especially if you went to hex before putting into Currency. Also, Currency comparisons are almost certainly faster than Decimal comparisons. But I'll let you (or others) sort that out.

    EDIT3: Actually, no need to convert the IP to hex. Currency has plenty of precision without needing to do this. Therefore, even faster. But one thing you could do is to convert the arrays to Currency arrays when they're loaded. That would certainly speed up things, especially when checking multiple IPs to see if they're in the USA.
    Attached Files Attached Files
    Last edited by Elroy; Aug 17th, 2017 at 06:29 PM.
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  14. #14
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    The single line solution I gave him, is insuperable, it stand up against DoS attacks. Other way it would be floodable from foreign sources.

  15. #15
    Fanatic Member Spooman's Avatar
    Join Date
    Mar 2017
    Posts
    868

    Re: multiple range checking algorithm vb6

    axis

    Picking up on LaVolpe's search suggestion, I found these 2 by CVMichael.

    Sort the array first: http://www.vbforums.com/showthread.p...-algorithm)&s=
    Then do binary search: http://www.vbforums.com/showthread.p...ghlight=binary

    Spoo

  16. #16
    PowerPoster
    Join Date
    Jun 2015
    Posts
    2,229

    Re: multiple range checking algorithm vb6

    Can you give an example of how you're receiving the IP Ranges? is it an actual range or something like CIDR notation?.


    Here's some routines you can use, but please note The IP's and Masks are in network byte order (Big Endian) Which is different than what people are used to dealing with, and what you find typically on the net.


    Code:
    Private Const STATUS_SUCCESS As Long = 0
    Private Type LARGE_INTEGER
        LowPart     As Long
        HighPart    As Long
    End Type
    Private Declare Function RtlLargeIntegerShiftLeft Lib "ntdll" (ByVal LowPart As Long, Optional ByVal HighPart As Long, Optional ByVal ShiftCount As Long = 1) As LARGE_INTEGER
    Private Declare Function RtlLargeIntegerShiftRight Lib "ntdll" (ByVal LowPart As Long, Optional ByVal HighPart As Long, Optional ByVal ShiftCount As Long = 1) As LARGE_INTEGER
    Private Declare Function RtlIpv4StringToAddressW Lib "ntdll" (ByVal S As Long, ByVal Strict As Byte, ByRef Terminator As Long, ByRef Addr As Long) As Long
    Private Declare Function inet_addr Lib "ws2_32" (ByVal cp As String) As Long
    Private Declare Function ntohl Lib "ws2_32" (ByVal netlong As Long) As Long
    Private Declare Function htonl Lib "ws2_32" (ByVal hostlong As Long) As Long
    
    Private Function Shl(ByVal Value As Long, Optional ByVal ShiftCount As Long = 1) As Long
        Shl = RtlLargeIntegerShiftLeft(Value, 0, ShiftCount).LowPart
    End Function
    Private Function Shr(ByVal Value As Long, Optional ByVal ShiftCount As Long = 1, Optional Arithmetic As Boolean) As Long
        Shr = RtlLargeIntegerShiftRight(Value, Arithmetic And Value < 0&, ShiftCount).LowPart
    End Function
    
    Public Function IPInCIDR(IP As String, CIDR As String) As Boolean
        Dim Network() As String: Network = Split(CIDR, "/")
        IPInCIDR = InNetwork(Str2IP(IP), Str2IP(Network(0)), Network(1))
    End Function
    
    Public Function InNetwork(ByVal IP As Long, ByVal RangeStart As Long, ByVal CIDR As Long) As Boolean
        Dim Netmask As Long: Netmask = CIDR2Netmask(CIDR)
        InNetwork = (IP And Netmask) = (RangeStart And Netmask)
    End Function
    
    Public Function Str2IP(ByRef DottedNotation As String) As Long
        RtlIpv4StringToAddressW StrPtr(DottedNotation), 1, 0, Str2IP    ' Network Order
    End Function
    
    Public Function CIDR2Netmask(ByVal Bits As Long) As Long
        Static LookupMask(1 To 32) As Long
        If LookupMask(1) = 0 Then
            Dim i As Long
            For i = 1 To 32
                LookupMask(i) = htonl(Shl(-1, 32 - i))  ' Network Order!
            Next
        End If
        CIDR2Netmask = LookupMask(Bits)
    End Function
    VB Code:
    1. IPInCIDR("192.168.3.1","192.168.3.0/29")


    As LaVolpe was alluding to in the beginning, using netmask(s) the actual range check reduces to a couple mask operations (And) and a single compare (=).

    So speed should not be an issue. I don't have a range to netmask convertor but this site has some.
    https://www.ip2location.com/tutorial...e-into-netmask

    the issue with using CIDR notation is that if your range spans networks, then you will need multiple CIDR/netmasks to describe the range.
    Last edited by DEXWERX; Aug 18th, 2017 at 10:47 AM.

  17. #17

    Thread Starter
    Hyperactive Member
    Join Date
    Jul 2002
    Posts
    505

    Re: multiple range checking algorithm vb6

    hello,

    I used flyguille solution. It is SUPER fast...

    Got it all working.

    Thanks for all the suggestions. What a great group of Developers we have here!

    WP

  18. #18
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by axisdj View Post
    hello,

    I used flyguille solution. It is SUPER fast...

    Got it all working.

    Thanks for all the suggestions. What a great group of Developers we have here!

    WP
    Yup!., lookup tables, and old trick from old asm days, when you didn't has CLOCK power, maybe there can be more ROMs in the board, lookup tables all the way!. for COSINE, SINE, for MULTIPLY in z80 didn't sopport it, and characters behaviour (overall in shoot'em up games).. it is amazing what you can do with just lookup tables, creating your own processing power without using the CPU to process it!

  19. #19
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    10,909

    Re: multiple range checking algorithm vb6

    Ohhh, I loved the old Z80 CPU. A chip you could easily get the entire instruction set in your head.

    And, of course, my most noted Z80 assembly contribution to the world:
    http://www.99-bottles-of-beer.net/la...z80)-2158.html

    Smiles,
    Elroy
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  20. #20
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by Elroy View Post
    Ohhh, I loved the old Z80 CPU. A chip you could easily get the entire instruction set in your head.

    And, of course, my most noted Z80 assembly contribution to the world:
    http://www.99-bottles-of-beer.net/la...z80)-2158.html

    Smiles,
    Elroy
    Include it was shipped with instructions which the engineers who design it didn't know the instructions even exists.

    like


    LD ixH, register
    LD ixL, register

    (8 bits of any instruction using the IX e IY 16bit indexers register)

    The only one problem abusing of that, is the extra 4 clocks it uses, because you ADDed the $FD or $DD instruction table prefix.

  21. #21
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    10,909

    Re: multiple range checking algorithm vb6

    Well, truth be told, when I resorted to assembler, I typically wrote for the 8080, as it was machine code compatible with the Z80, whereas vice-versa wasn't true. Once upon a time, I had quite a few 8080 routines that I'd assemble into .OBJ files and then link them in with a larger program.

    And, of course, here's my infamous contribution to 8080 assembler code:
    http://www.99-bottles-of-beer.net/la...080)-2157.html
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  22. #22
    PowerPoster dilettante's Avatar
    Join Date
    Feb 2006
    Posts
    24,487

    Re: multiple range checking algorithm vb6

    That solution might work great... as long as it actually does what you need.

    But usually IP address whitelists and blacklists are not quite so... black and white. There may even be cases where you have just a few IPs to restrict within some given subnet.


    Did you consider an ADO solution?

    Here there is a MakeList.vbp for a program to create a text file of IP ranges. Run that first. Then there is an InList.vbp for a program to test IP addresses against those ranges.

    "InList" starts by loading the text file. This could have been a persisted Recordset instead for faster loading AND could have omitted the "RangeID" column that was included here just for demo purposes.

    After loading it kicks off a check of a bunch of IP addresses (here 1000 of them) and times the process.

    Then after that you can manually enter and test IP addresses.


    Name:  sshot1.png
Views: 1090
Size:  2.5 KB


    Name:  sshot2.png
Views: 1161
Size:  2.3 KB


    You may have exactly what you need already. This can be a useful technique for more demanding applications though.
    Attached Files Attached Files

  23. #23
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by dilettante View Post
    That solution might work great... as long as it actually does what you need.

    But usually IP address whitelists and blacklists are not quite so... black and white. There may even be cases where you have just a few IPs to restrict within some given subnet.


    Did you consider an ADO solution?

    Here there is a MakeList.vbp for a program to create a text file of IP ranges. Run that first. Then there is an InList.vbp for a program to test IP addresses against those ranges.

    "InList" starts by loading the text file. This could have been a persisted Recordset instead for faster loading AND could have omitted the "RangeID" column that was included here just for demo purposes.

    After loading it kicks off a check of a bunch of IP addresses (here 1000 of them) and times the process.

    Then after that you can manually enter and test IP addresses.


    Name:  sshot1.png
Views: 1090
Size:  2.5 KB


    Name:  sshot2.png
Views: 1161
Size:  2.3 KB


    You may have exactly what you need already. This can be a useful technique for more demanding applications though.

    that is no problem with the single line solution look up table, including controlling the third level of the IP, it is 256*256*256 = 4MB / 8 (as boolean) if using a single bit per bool value, if not itis 4MB anyway.

    The lookup table is pre-filled one time at load time, and reworked if a change is done in the set of ranges to accept.

    Any other solution would be floodeable, so DoSable.

  24. #24
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by Elroy View Post
    Well, truth be told, when I resorted to assembler, I typically wrote for the 8080, as it was machine code compatible with the Z80, whereas vice-versa wasn't true. Once upon a time, I had quite a few 8080 routines that I'd assemble into .OBJ files and then link them in with a larger program.

    And, of course, here's my infamous contribution to 8080 assembler code:
    http://www.99-bottles-of-beer.net/la...080)-2157.html
    yes, I notices de similarities with the regular registers, it is a shame z80 didn't has well supported the C language, as it is very expensive the handling of the STACK, and C language require a massive usage of the stack, like MOV AX , [SP + nn]

    In z80 those luxuries didn't exists, it can be emulated like, ex hl, sp, IIRC, then work with HL ld de, offset , add hl, de, then ex hl,sp, pop de (for reading the actual value parameter), then undo with SBC the added value discounting the PÔP, it is all a pain in the ass, and sloooooooooow.

    or near method, INC SP INC SP.... DEC SP DEC SP and 8 bits ops.

    So in z80 programmers passed all by registers, and forget about passing through stack, so, freedom about parameters count is not possible. Unless ofcourse, static addressing for parameters.

    And that is how 8086 won the race.... of course, also was y monopoly practices form Intel, but that is another topic.

  25. #25
    PowerPoster dilettante's Avatar
    Join Date
    Feb 2006
    Posts
    24,487

    Re: multiple range checking algorithm vb6

    256 * 256 * 256 * 256 / 8 = 536,870,912 is a pretty big table.

    You can probably drop the 0 and 255 values of course, and there is a smaller set of subnets than this to consider... but the table will still get darned big.

    Trying to store as Boolean instead of packing bits it becomes massive, potentially too large to use at all.

    IP lists that do not have IP address level resolution are not IP lists. What you have is at best a subnet list unless you consider all 32 bits.

  26. #26
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by dilettante View Post
    256 * 256 * 256 * 256 / 8 = 536,870,912 is a pretty big table.

    You can probably drop the 0 and 255 values of course, and there is a smaller set of subnets than this to consider... but the table will still get darned big.

    Trying to store as Boolean instead of packing bits it becomes massive, potentially too large to use at all.

    IP lists that do not have IP address level resolution are not IP lists. What you have is at best a subnet list unless you consider all 32 bits.

    He is trying to point if inside USA or outside of USA..

    None company in the world, woud have a "part" of the fourth element in the IP. NONE!

    Companies are given whole 256 ranges.

    If not whole 65536 ranges. (ISPs). As it is like the routing works in Internet. Other way tables at routing would be huge, and slow.

  27. #27
    PowerPoster dilettante's Avatar
    Join Date
    Feb 2006
    Posts
    24,487

    Re: multiple range checking algorithm vb6

    That's why I wrote:

    Quote Originally Posted by dilettante View Post
    That solution might work great... as long as it actually does what you need.
    :
    :
    You may have exactly what you need already. This can be a useful technique for more demanding applications though.
    I was merely presenting an alternative that seems to be fast enough (1000 in less than 1/10 of a second) and far less of a fragile hack than yours.

  28. #28
    PowerPoster
    Join Date
    Jun 2015
    Posts
    2,229

    Re: multiple range checking algorithm vb6

    Why don't we just turn this into a contest
    With actual IP range data. Then we can get Schmidt in here.

    @dil should your DottedIPAddrToVarUI8() be renamed to DottedIPAddrToVarUI4()

  29. #29
    PowerPoster dilettante's Avatar
    Join Date
    Feb 2006
    Posts
    24,487

    Re: multiple range checking algorithm vb6

    Yep, you are right. Typo when I did a find/replace. Thanks, wrong is wrong and even wrong naming is wrong.

  30. #30
    PowerPoster Elroy's Avatar
    Join Date
    Jun 2014
    Location
    Near Nashville TN
    Posts
    10,909

    Re: multiple range checking algorithm vb6

    What are we trying to do here? Are we trying to figure out the fastest way to see if an IP address is in the USA range?

    If that's the case, then the attached project has procedures in it for doing it about as fast as it's going to get.

    Enjoy,
    Elroy

    EDIT1: Actually, it could be sped up somewhat by consolidating many of those ranges, because they're often contiguous. That would make the binary search a bit faster. They only reason they're not already consolidated is because they were assigned on different dates.
    Attached Files Attached Files
    Last edited by Elroy; Aug 18th, 2017 at 02:28 PM.
    Any software I post in these forums written by me is provided "AS IS" without warranty of any kind, expressed or implied, and permission is hereby granted, free of charge and without restriction, to any person obtaining a copy. To all, peace and happiness.

  31. #31
    Fanatic Member Spooman's Avatar
    Join Date
    Mar 2017
    Posts
    868

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by DEXWERX View Post
    Why don't we just turn this into a contest
    Someone should give dday9 a phone call

  32. #32
    PowerPoster
    Join Date
    Jun 2015
    Posts
    2,229

    Re: multiple range checking algorithm vb6

    Here's some actual data ready for your benchmarks.
    Attached Files Attached Files

  33. #33
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by flyguille View Post
    None company in the world, woud have a "part" of the fourth element in the IP. NONE!
    Companies are given whole 256 ranges.
    That's actually not true - there is hundreds if not thousands of ranges which differ in the
    last part (even across countries, not only companies).

    Example from RIPE: (all ranges share the first part: 37.59.223.___)

    IT -> https://apps.db.ripe.net/search/quer...t=37.59.223.39
    NL -> https://apps.db.ripe.net/search/quer...t=37.59.223.47
    PL -> https://apps.db.ripe.net/search/quer...t=37.59.223.63
    CZ -> https://apps.db.ripe.net/search/quer...t=37.59.223.79
    GB -> https://apps.db.ripe.net/search/quer...t=37.59.223.95
    DE -> https://apps.db.ripe.net/search/quer...=37.59.223.127
    FR -> https://apps.db.ripe.net/search/quer...=37.59.223.191

    Or to give another example from ARIN:
    The US-RangeSite here: http://lite.ip2location.com/united-s...address-ranges
    lists e.g. the range: 23.239.183.32 to 23.247.128.255 ...
    But in reality there is a whole lot of Sub-Ranges in there, which were re-assigned,
    as e.g. the range 23.246.198.192 to 23.246.198.223 belonging to CL (Chile) now.
    https://whois.arin.net/rest/customer/C05314851.html

    So these more detailed infos can be found over the registrars (of the appropriate region) directly,
    or by using more recent (and more "fine-granular") DataBases...

    The one I've found among the free available ones, is this one here (CC-licensed):
    https://db-ip.com/db/download/country
    It contains nearly half a Million range-records - and was the most detailed one
    (there's other free, liberally licensed DBs, but the next one which came close had only about 250,000 records).

    The advantage with those larger DBs is, that they cover all countries *and*:
    - that they leave no gaps!
    That means, the format within the CSV above (IPfrom, IPto, ...) is not really
    needed, because any "IPto-Value" is equal to the "IPfrom-Value minus one" from the next line.

    That makes it quite easy for much more efficient storage - and for a fast binary search.

    So I've re-formatted the above CSV in a Binary-Format, which takes this "gapelessness" into account,
    storing a single range-record in only 6 Bytes, which reduces the Raw-Binary-File to about 2.5MB
    (432616 Records * 6 Bytes-per-Record = 2,595,696).

    Further LZMA-compressing reduces these ~2.5MB to about 500kByte (something one can ship-along easily with ones App).

    Quote Originally Posted by DEXWERX View Post
    Why don't we just turn this into a contest
    With actual IP range data. Then we can get Schmidt in here.
    ... since you called ...
    Though the US-DB is a bit small (any algo would probably feel "fast enough").

    So I'd suggest to use the Data from the link I've posted above (the one with the 432616 Records)
    and use that...

    I've done that in the following Demo (which does not include the original CSV, but the 512KB LZMA-Raw-Binary-Data):
    http://vbRichClient.com/Downloads/IPRangesByCountry.zip

    Which needs some time to load (about 300msec, mostly due to the LZMA-decompression):


    But after the 0.5Mio Records sit in the Dictionary, the (Binary) search goes quite fast:


    My 0.02$...

    Olaf
    Last edited by Schmidt; Aug 18th, 2017 at 09:47 PM.

  34. #34
    Fanatic Member Spooman's Avatar
    Join Date
    Mar 2017
    Posts
    868

    Re: multiple range checking algorithm vb6

    olaf

    The advantage with those larger DBs is, that they cover all countries *and*:
    - that they leave no gaps!
    That kills my theory that the 1st "part" of the IP address was regionally based.
    Good stuff

    Spoo

  35. #35
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by Schmidt View Post
    That's actually not true - there is hundreds if not thousands of ranges which differ in the
    last part (even across countries, not only companies).

    Example from RIPE: (all ranges share the first part: 37.59.223.___)

    IT -> https://apps.db.ripe.net/search/quer...t=37.59.223.39
    NL -> https://apps.db.ripe.net/search/quer...t=37.59.223.47
    PL -> https://apps.db.ripe.net/search/quer...t=37.59.223.63
    CZ -> https://apps.db.ripe.net/search/quer...t=37.59.223.79
    GB -> https://apps.db.ripe.net/search/quer...t=37.59.223.95
    DE -> https://apps.db.ripe.net/search/quer...=37.59.223.127
    FR -> https://apps.db.ripe.net/search/quer...=37.59.223.191

    Or to give another example from ARIN:
    The US-RangeSite here: http://lite.ip2location.com/united-s...address-ranges
    lists e.g. the range: 23.239.183.32 to 23.247.128.255 ...
    But in reality there is a whole lot of Sub-Ranges in there, which were re-assigned,
    as e.g. the range 23.246.198.192 to 23.246.198.223 belonging to CL (Chile) now.
    https://whois.arin.net/rest/customer/C05314851.html

    So these more detailed infos can be found over the registrars (of the appropriate region) directly,
    or by using more recent (and more "fine-granular") DataBases...

    The one I've found among the free available ones, is this one here (CC-licensed):
    https://db-ip.com/db/download/country
    It contains nearly half a Million range-records - and was the most detailed one
    (there's other free, liberally licensed DBs, but the next one which came close had only about 250,000 records).

    The advantage with those larger DBs is, that they cover all countries *and*:
    - that they leave no gaps!
    That means, the format within the CSV above (IPfrom, IPto, ...) is not really
    needed, because any "IPto-Value" is equal to the "IPfrom-Value minus one" from the next line.

    That makes it quite easy for much more efficient storage - and for a fast binary search.

    So I've re-formatted the above CSV in a Binary-Format, which takes this "gapelessness" into account,
    storing a single range-record in only 6 Bytes, which reduces the Raw-Binary-File to about 2.5MB
    (432616 Records * 6 Bytes-per-Record = 2,595,696).

    Further LZMA-compressing reduces these ~2.5MB to about 500kByte (something one can ship-along easily with ones App).


    ... since you called ...
    Though the US-DB is a bit small (any algo would probably feel "fast enough").

    So I'd suggest to use the Data from the link I've posted above (the one with the 432616 Records)
    and use that...

    I've done that in the following Demo (which does not include the original CSV, but the 512KB LZMA-Raw-Binary-Data):
    http://vbRichClient.com/Downloads/IPRangesByCountry.zip

    Which needs some time to load (about 300msec, mostly due to the LZMA-decompression):


    But after the 0.5Mio Records sit in the Dictionary, the (Binary) search goes quite fast:


    My 0.02$...

    Olaf
    Well, with more reason, my method wins, because if it is that granulated, over the world it will be like having thousands of entries pointing to USA locations.

    so it will end to be an lookup table of 4MB*256 / 8, and a nice time at the beggining filling it once on startup time.

    Note if VB6 waste the 8 bits per boolean, you can really handle an array of bytes, and calling it like

    if (var(first, second, third , int(fourth/8)) and 2^(fourth mod 8) then ' Is in USA.

    and the declaration is:

    private Var(0-255,0-255,0-255,0-31) as byte

    the setup is



    var(first,second,third, int(fourth /8)) =var(first,second,third, int(fourth /8)) or (2^(fourth mod 8)) ' It mark that ip as IN USA.


    and the loop runnig it is a bit trickery.

    but easy


    first=init first
    second=init second
    third =init third
    fourth =init fourth

    while true
    var(first,second,third, int(fourth /8)) =var(first,second,third, int(fourth /8)) or (2^(fourth mod 8)) ' It mark that ip as IN USA.
    fourth =(fourth+1) mod 256: if fourth=0 then third = (third + 1) mod 256: if third=0 then second=(second+1) mod 256 : if second=0 then first=(first+1) mod 256

    if fourth=endfourth then if third = endthird then if second=endsecond then if first= endfirst then EXIT LOOP

    Wend

    That is the faster filling the array that I can think.

    TOTAL DATA USAGUE = 4MB*32 = 128MB, not that bad for modern PCs.

    Time requre to check on _ConnectionRequest = 1 line of code!!!! you can do hundreds of thousands per second.

    ADITIONAL FEATURE WHICH you can get from it, DISABLING temporarily individual IP when detecting DoS attacks!.

    PD: Now thinking in implement it in my DRM server, one never knows which competitor can get angry when growing my customers base in detriments of their sells. Better to be prepared!
    Last edited by flyguille; Aug 19th, 2017 at 09:54 AM.

  36. #36
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by flyguille View Post
    Well, with more reason, my method wins, ...
    No, not really... (because it's a memory-eater)

    Quote Originally Posted by flyguille View Post
    ...because if it is that granulated, over the world it will be like having thousands of entries pointing to USA locations.
    It depends on, which DB you choose...
    - about 7000 US-entries are in the NirSoft-DB (from the opener-post)
    - about 19000 US-entries in the CSV-File DEXWERX has posted
    - and about 88000 US-entries (of the 0.5Mio total) are in the "no-gaps, all countries" CSV-File which is the base for my approach

    BTW, there's something wrong with your math, when you write: TOTAL DATA USAGUE = 4MB*32 = 128MB

    Because a ByteArray dimensioned as: Dim LookupTable (0 to 255, 0 to 255, 0 to 255, 0 to 31) As Byte
    ...will take up half a Gigabyte (4Gig divided by 8).

    I would not want to use an App which takes up 512MB, for a simple IP-check...
    Not to mention, that such a large allocation of consecutive memory has a high chance to fail.

    The approach I've choosen:
    - is far less memory-intensive (takes up only about 16MB of Ram)
    - works for all IP-ranges (all countries, not only the US)
    - is quite fast with the lookups too (can achieve about 0.2Mio lookups per second)

    Olaf
    Last edited by Schmidt; Aug 19th, 2017 at 04:49 PM.

  37. #37
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by Schmidt View Post
    No, not really... (because it's a memory-eater)



    It depends on, which DB you choose...
    - about 7000 US-entries are in the NirSoft-DB (from the opener-post)
    - about 19000 US-entries in the CSV-File DEXWERX has posted
    - and about 88000 US-entries (of the 0.5Mio total) are in the "no-gaps, all countries" CSV-File which is the base for my approach

    BTW, there's something wrong with your math, when you write: TOTAL DATA USAGUE = 4MB*32 = 128MB

    Because a ByteArray dimensioned as: Dim LookupTable (0 to 255, 0 to 255, 0 to 255, 0 to 31) As Byte
    ...will take up half a Gigabyte (4Gig divided by 8).

    I would not want to use an App which takes up 512MB, for a simple IP-check...
    Not to mention, that such a large allocation of consecutive memory has a high chance to fail.

    The approach I've choosen:
    - is far less memory-intensive (takes up only about 16MB of Ram)
    - works for all IP-ranges (all countries, not only the US)
    - is quite fast with the lookups too (can achieve about 0.2Mio lookups per second)

    Olaf
    Yup, math is wrong, it is 512MB for the IP check. Depends on the availability, anyway it will be the choise about performance time. 32bits programs are up to 4GB? or VB6 has a silly limit of 2GB or something?

    anyway, include my biiiiiiigest server multipurpose loaded in ram is about 25MB....., so yup, no problem in using 512MB just for the ip check...., not to said that servers can have lots of rAM, tipically more than 16GB , being the usual 64GB.

  38. #38
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by flyguille View Post
    Yup, math is wrong, it is 512MB for the IP check.
    And that's why also the approach is wrong.
    Lookup-Table-Arrays don't make much sense, when they are as sparsely populated as in this case.

    Quote Originally Posted by flyguille View Post
    ... anyway it will be the choise about performance time.
    And also here you operate on a wrong assumption - because a fast (final) LookUp-Table access
    is the one thing - but before you can perform the access, there's other lines needed to prepare the input -
    and these extra-calls sum up in the end...

    I've now tried to simulate your approach in a little test-code-snippet -
    (with only a 256MByte-Array instead of your proposed 512MB) - but
    the principle is the one I've cobbled together from all your lines and comments.

    What I get here on my machine as "time needed per function call" is:
    - about 1.9 µsec when running in the IDE ... and
    - about 1.6 µsec when running native compiled

    That's nearly identical to the timings I achieve with the "full-coverage" approach
    (mine is a bit faster when native compiled, but a bit slower than yours when in the IDE)

    Code:
    Option Explicit
    
    Private LookupTable(0 To 255, 0 To 255, 0 To 255, 0 To 15) As Byte
    
    Private Sub Form_Click()
      Dim i As Long, Result As Boolean, T As Single
      T = Timer
        For i = 1 To 100000
          Result = IsInUSA("121.122.123.124")  
          If Result Then 
            'do something here
          End If
        Next
      Caption = Format$((Timer - T) * 10, "0.0") & " µSec per function-call"
    End Sub
    
    Function IsInUSA(RemoteIP As String) As Boolean
      Dim STRD As String
          STRD = RemoteIP
      
      Dim a As Integer, b As Integer, c As Integer
          a = InStr(STRD, ".")
          b = InStr(a + 1, STRD, ".")
          c = InStr(b + 1, STRD, ".")
     
      Dim first As Byte, second As Byte, third As Byte, fourth As Byte
          first = Val(Mid$(STRD, 1, a - 1))
          second = Val(Mid$(STRD, a + 1, b - (a + 1)))
          third = Val(Mid$(STRD, b + 1, c - (b + 1)))
          fourth = Val(Mid$(STRD, c + 1))
      
      IsInUSA = LookupTable(first, second, third, Int(fourth / 16)) And (fourth Mod 16)
    End Function
    HTH

    Olaf

  39. #39
    Fanatic Member
    Join Date
    Jan 2013
    Posts
    928

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by Schmidt View Post
    And that's why also the approach is wrong.
    Lookup-Table-Arrays don't make much sense, when they are as sparsely populated as in this case.



    And also here you operate on a wrong assumption - because a fast (final) LookUp-Table access
    is the one thing - but before you can perform the access, there's other lines needed to prepare the input -
    and these extra-calls sum up in the end...

    I've now tried to simulate your approach in a little test-code-snippet -
    (with only a 256MByte-Array instead of your proposed 512MB) - but
    the principle is the one I've cobbled together from all your lines and comments.

    What I get here on my machine as "time needed per function call" is:
    - about 1.9 µsec when running in the IDE ... and
    - about 1.6 µsec when running native compiled

    That's nearly identical to the timings I achieve with the "full-coverage" approach
    (mine is a bit faster when native compiled, but a bit slower than yours when in the IDE)

    Code:
    Option Explicit
    
    Private LookupTable(0 To 255, 0 To 255, 0 To 255, 0 To 15) As Byte
    
    Private Sub Form_Click()
      Dim i As Long, Result As Boolean, T As Single
      T = Timer
        For i = 1 To 100000
          Result = IsInUSA("121.122.123.124")  
          If Result Then 
            'do something here
          End If
        Next
      Caption = Format$((Timer - T) * 10, "0.0") & " µSec per function-call"
    End Sub
    
    Function IsInUSA(RemoteIP As String) As Boolean
      Dim STRD As String
          STRD = RemoteIP
      
      Dim a As Integer, b As Integer, c As Integer
          a = InStr(STRD, ".")
          b = InStr(a + 1, STRD, ".")
          c = InStr(b + 1, STRD, ".")
     
      Dim first As Byte, second As Byte, third As Byte, fourth As Byte
          first = Val(Mid$(STRD, 1, a - 1))
          second = Val(Mid$(STRD, a + 1, b - (a + 1)))
          third = Val(Mid$(STRD, b + 1, c - (b + 1)))
          fourth = Val(Mid$(STRD, c + 1))
      
      IsInUSA = LookupTable(first, second, third, Int(fourth / 16)) And (fourth Mod 16)
    End Function
    HTH

    Olaf
    what? that code is slow, you did it the worse way!. You don't use a set of string variables for the inspection, it hasn't sense, you just VAL it directly on the array dimensions.

    and you know there is SPLIT string right? so a simple string array as result of SPLIT, then val(st(1))...., val(st(2))...., and go on, . so not needs of three INSTRs plus the MID$().

    Result, it is just 3 simple lines (single instruction per line) for doing it ALL.
    Last edited by flyguille; Aug 19th, 2017 at 08:57 PM.

  40. #40
    PowerPoster
    Join Date
    Jun 2013
    Posts
    7,454

    Re: multiple range checking algorithm vb6

    Quote Originally Posted by flyguille View Post
    what? that code is slow, you did it the worse way!.
    Well, then let's see what your recommendations will bring, when I alter it to your wishes...

    Quote Originally Posted by flyguille View Post
    You don't use a set of string variables for the inspection, it hasn't sense, you just VAL it directly on the array dimensions.
    Ok, done...

    Quote Originally Posted by flyguille View Post
    and you know there is SPLIT string right? so a simple string array as result of SPLIT, ...
    ... as you wish, master...

    Quote Originally Posted by flyguille View Post
    ...then val(st(1))...., val(st(2))...., and go on, . so not needs of three INSTRs plus the MID$().
    Please forgive me, but I've found it this way in posting #10 (from a guy named "flyguille")...
    But Ok - I've adapted it now as commanded...

    Quote Originally Posted by flyguille View Post
    Result, it is just 3 simple lines (single instruction per line) for doing it ALL.
    I hope the new code below is to your satisfaction now - but I have to tell you that the changes you demanded,
    make it now run factor 2 slower than before... (about 3.3 µsec per call, compared to the former 1.6)
    What now?

    Wouldn't it be better, when you perhaps write "the ideal code" for the usage of your approach yourself -
    and post it here (preferrably within Code-Tags, if that's not too much to ask...).

    Code:
    Option Explicit
    
    Private LookupTable(0 To 255, 0 To 255, 0 To 255, 0 To 15) As Byte
    
    Private Sub Form_Click()
      Dim i As Long, Result As Boolean, T As Single
      T = Timer
        For i = 1 To 100000
          Result = IsInUSA("121.122.123.124")
          If Result Then
            'do something here
          End If
        Next
      Caption = Format$((Timer - T) * 10, "0.0") & " µSec per function-call"
    End Sub
    
    Function IsInUSA(RemoteIP As String) As Boolean
      Dim st() As String
          st = Split(RemoteIP, ".")
    
      IsInUSA = LookupTable(Val(st(0)), Val(st(1)), Val(st(2)), Int(Val(st(3)) / 16) And (Val(st(3)) Mod 16))
    End Function
    Olaf
    Last edited by Schmidt; Aug 19th, 2017 at 09:40 PM.

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width