[RESOLVED] How to make checksum from DGV cells?

[pourkascheff]

Suppose you have a huge table of numbers on DGV and you want to perform a sort of logical function on first cell to the end cell.

The goal
is to make a small hexadecimal code to compare two huge csv files or dgv tables are different or not. Very useful in electronics logic and microcontrollers programming. However CRCs are more reliable. I mean checksum is just summing right? Instead of their more complex xor polynomial functions. The possibility to have different files with similar checksum is extremely low.

Anyways,
The problem
is where a "string" cell comes in, unless [in total numeric cells] it works fine... Program_Flash of microcontrollers are full of ASCII and unicodes but checksums are automatically generated anyway.

What I've tried
1) I tried .GetHashCode method of the file itself (an object) and that was crazy. I'm still reading about it but I have no idea what is it. Too much bytes and file name (and probably mod/acc date affects hash code) If someone can describe a bit about it I would be delighted even more since it may be useful in further coding..
2) Is there another way (similar to checksum) to compare two files are same or not with a simpler built-in functions?
3) I came up with this approach for prior idea:

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        Dim CHECKSUM As Long = 0 'MAX SHOULD BE FFFF = 65535
        For row = 0 To DataGridView1.RowCount - 1
            For column = 0 To DataGridView1.ColumnCount - 1
                CHECKSUM = CHECKSUM + CLng(DataGridView1.Item(column, row).Value) 'REQUIRES STRING-FRIENDLY CONVERSION
            Next
        Next
    End Sub

- I also tried System.Text classes but I'm not sure I'm doing it right cause I didn't find .ASCIIdecoder and even I'm not sure I should use ASCII or UTF8 or something else...

CHECKSUM = CHECKSUM + CLng(System.Text.ASCIIEncoding.ASCII(DataGridView1.Item(column, row).Value)

Update:
1) Conversion code is false and has syntax error. Pasted it just for record.
2) Error message on try/catch exception is: Invalid cast exception was unhandled: Conversion from string "TEKST" to type 'Long' is not valid.

Note:
1) I guess performing it on an array or a collection of variable could be better but I'm watching the gridview.
2) I found this kind of thread making way more effective to discrete it in separated paragraphs.

[Niya]

A checksum is just hashing and hashing is easy in .Net. Here's a simple function that hashes a String:-

    Public Function ComputeMD5Hash(ByVal inputString As String) As String
        Return String.Join("", From b In MD5.Create().ComputeHash(Encoding.UTF8.GetBytes(inputString)) Select b.ToString("x2"))
    End Function

If you wanted to hash an array of Strings, you could simple join all the Strings in the array into one String and hash that using the above function:-

    Public Function HashArray(ByVal data As String()) As String
        Return ComputeMD5Hash(String.Join(",", data))
    End Function

Example of hashing an array of names:-

        Dim names = {"John", "Price", "Homer"}

        Debug.WriteLine(HashArray(names))

[pourkascheff]

Wow, you just saved me lots of time. Let me shake your hand Niya.

I was messing around with

Imports System.Security.Cryptography

and found MD5, wasn't sure I see them in compressed files/cds or what.

I only added

Imports System.Text

to your code. And make the x2 to X2 to be more readable. That's it

String.Join("", From b In MD5.Create().ComputeHash(Encoding.UTF8.GetBytes(inputString)) Select b.ToString("X2"))

By the way I expected a 4 digit C$ like the others but for your example it returns:

e82f5c37580630f309ef47a6cd088c3a

dang, no space in UI for this dragon txt =)))
*Edit:
Well, it turned out MD5 is a 128-bit (32 digits) format by its nature and nothing I can do about it. Extracting a non-md5 standard code from it maybe (Split it in 8 parts and do things on it which only useable in my app and nowhere else)

[pourkascheff]

One last question:
In this scenario which type is the name??? It works but in "string" type "Value of type '1-dimensional array of String' cannot be converted to 'String'" and simultaneously "Value of type 'String' cannot be converted to '1-dimensional array of String'" errors occur.

        Dim names = {"John", "Price", "Homer"}
        Debug.WriteLine(HashArray(names))

How can I create long "name" collection from a dgv members/cells and feedthrough HashArray function?

[wqweto]

If you wanted to hash an array of Strings, you could simple join all the Strings in the array into one String and hash that using the above function:-

    Public Function HashArray(ByVal data As String()) As String
        Return ComputeMD5Hash(String.Join(",", data))
    End Function

Btw, that's not how you hash arrays as this allows easy collisions with

        Dim names = {"John,Price", "Homer"}

        Debug.WriteLine(HashArray(names))

i.e. different arrays -> same hash.

Basically you have to come up with better scheme for "canonicalization" than a simple join with commas to prevent obvious collisions.

cheers,
</wqw>

[Niya]

Btw, that's not how you hash arrays as this allows easy collisions with

        Dim names = {"John,Price", "Homer"}

        Debug.WriteLine(HashArray(names))

i.e. different arrays -> same hash.

Basically you have to come up with better scheme for "canonicalization" than a simple join with commas to prevent obvious collisions.

cheers,
</wqw>

Good point. This one might be better:-

    Public Function HashArray2(ByVal data As String()) As String

        Dim hash As IEnumerable(Of Byte) = Nothing
        Dim m = MD5.Create()
        Dim hf = Function(s As String) m.ComputeHash(Encoding.UTF8.GetBytes(s))

        For Each s In data.Select(Function(ss, i) ss & i.ToString)
            If hash Is Nothing Then
                hash = hf(s)
            Else
                hash = hash.Zip(hf(s), Function(h1, h2) h1 Xor h2)
            End If
        Next

        Return String.Join("", From b In hash Select b.ToString("X2"))
    End Function

The one hashes each String individually and simply XORs all the hashes. It also appends the index of each String in the array before hashing so that the final hash is order sensitive which is to say arrays with the same elements but different ordering would produces different hashes.

[Niya]

How can I create long "name" collection from a dgv members/cells and feedthrough HashArray function?

I've seen enough of you to know you're not a beginner. I really doubt you don't know how to do this. I mean you could just read the cells into something like a List(Of T) then call ToArray on that List to get an array that you could pass to the HashArray function. Really basic stuff.

Well, it turned out MD5 is a 128-bit (32 digits) format by its nature and nothing I can do about it. Extracting a non-md5 standard code from it maybe (Split it in 8 parts and do things on it which only useable in my app and nowhere else)

Well you could truncate the MD5 value. Here's a version of the improved HashArray function that returns a hash digest as a 32 bit unsigned integer:-

    Public Function HashArray3(ByVal data As String()) As UInteger

        Dim hash As IEnumerable(Of Byte) = Nothing
        Dim m = MD5.Create()
        Dim hf = Function(s As String) m.ComputeHash(Encoding.UTF8.GetBytes(s))

        For Each s In data.Select(Function(ss, i) ss & i.ToString)
            If hash Is Nothing Then
                hash = hf(s)
            Else
                hash = hash.Zip(hf(s), Function(h1, h2) h1 Xor h2)
            End If
        Next

        Return (From i In Enumerable.Range(0, 4)
                Select BitConverter.ToUInt32(hash.Skip(i * 4).Take(4).ToArray, 0)).ToArray.
                 Aggregate(Function(acc, v) acc Xor v)

    End Function

It just takes the original MD5 hash and divides into 4 byte chunks and XORs them together to produce a final 32 bit value.

[wqweto]

Yes, hashing individual elements is one option, though it might not be most performant.

Another option is work on the "canonicalization" e.g. convert each string to a byte-array, length-prefix each one with 4-byte size (in bits) and concat these. Finally hash this canonical representation.

This is what they do in TupleHash in SHA-3 like this (rust code)

impl Hasher for TupleHash {
    fn update(&mut self, input: &[u8]) {
        self.state.update(left_encode(input.len() * 8).value());
        self.state.update(input)
    }

    fn finalize(mut self, output: &mut [u8]) {
        self.state.update(right_encode(output.len() * 8).value());
        self.state.finalize(output)
    }
}

i.e. each chunk is length-prefixed and finally size output hash is appended (because SHA-3 can produce output hash of arbitrary size so they prevent length-extension atack of some sort).

cheers,
</wqw>

[Niya]

Would this be a good approximation of it?

    Public Function HashArray4(ByVal data As String()) As UInteger

        Dim ToUInt32 = Function(hash As Byte()) (From i In Enumerable.Range(0, 4)
                                                 Select BitConverter.ToUInt32(hash, i * 4)).
                                                 Aggregate(Function(acc, v) acc Xor v)

        Using ms As New MemoryStream
            Using sr As New BinaryWriter(ms)

                For Each s In data

                    sr.Write(s.Length * 8)
                    sr.Write(Encoding.UTF8.GetBytes(s))

                Next

                'We want to output a hash digest that
                'is 32 bits in length
                sr.Write(32)

                ms.Position = 0
                Return ToUInt32(MD5.Create().ComputeHash(ms))
            End Using
        End Using
    End Function

Two things to note here is that I used MD5 because SHA-3 is not readily available in .Net. It might be available through a Nuget package in which case it would be pretty easy to substitute in place of MD5.

The second thing is that I left out the left and right encoding partly because I think for OP's purposes collision resistance might be more important than security but mainly because it's just a bit over my head. I couldn't find a single explanation of left/right encoding that is simple enough for me to understand. I don't know what it does, only that it might be related to security.

[Niya]

Also to answer this question specifically:-

2) Is there another way (similar to checksum) to compare two files are same or not with a simpler built-in functions?

This is actually pretty easy to do. Here's one using MD5 again:-

    Public Function CompareFiles(ByVal fileName1 As String, ByVal fileName2 As String) As Boolean

        Dim m = MD5.Create()

        Using fs1 = File.OpenRead(fileName1)
            Using fs2 = File.OpenRead(fileName2)
                Return m.ComputeHash(fs1).SequenceEqual(m.ComputeHash(fs2))
            End Using
        End Using

    End Function

This function takes two file names and compare the MD5 hashes of their contents and returns True if the hashes are the same and False is they are not. Example usage:-

Debug.WriteLine(CompareFiles("d:\0.jpg", "d:\a.jpg"))

[pourkascheff]

I love to be a part of this community. Thanks all.
So far so good. Niya's code in a practical way at once for enthusiasts, looks like this:

Imports System.Text
Imports System.Security.Cryptography
Public Class Form2
    Public Function ComputeMD5Hash(ByVal inputString As String) As String
        Return String.Join("", From b In MD5.Create().ComputeHash(Encoding.UTF8.GetBytes(inputString)) Select b.ToString("X2"))
    End Function
    Public Function HashArray(ByVal data As String()) As UInteger
        Dim hash As IEnumerable(Of Byte) = Nothing
        Dim m = MD5.Create()
        Dim hf = Function(s As String) m.ComputeHash(Encoding.UTF8.GetBytes(s))
        For Each s In data.Select(Function(ss, i) ss & i.ToString)
            If hash Is Nothing Then
                hash = hf(s)
            Else
                hash = hash.Zip(hf(s), Function(h1, h2) h1 Xor h2)
            End If
        Next
        Return (From i In Enumerable.Range(0, 4)
                Select BitConverter.ToUInt32(hash.Skip(i * 4).Take(4).ToArray, 0)).ToArray.
                 Aggregate(Function(acc, v) acc Xor v)
    End Function
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim names As String() = Split(TextBox1.Text, ", ") '(CAN BE VARY) WHICH ALREADY CONTAINS: John, Price, Homer
        MsgBox(Conversion.Hex(HashArray(names)).ToString)
    End Sub
End Class

...you're not a beginner.

That's a relief you didn't understimate me, but I'm afraid I am a beginner @ .NET.

My question was about "name"s type. You left it empty and only worked for your already-written collection. I tried "Object" and no errors happened (Not in run-time = Unhandled exceptions). But now I tried magical () to represent a collection and seperate its members by ", " so not only I can generate hash by any input (or even empty) string, but also there's no need for DGVs.

This function takes two file names and compare the MD5 hashes of their contents and returns True if the hashes are the same and False is they are not.

Sorry in advance, I bothered you. It's pleasant for user to actually see what's going on and they ARE different rather than a Yes or No, True or False. (However he or she has no idea what does those ABCDEF0~9s mean XD) But like Microsoft Windows Copy/Paste similar items dialog which asks you and shows you differences in date and size in bold. You get the point...

[Niya]

My question was about "name"s type. You left it empty and only worked for your already-written collection

My apologies. I use type inference out of habit. It's a nice feature for productivity but it can confuse people sometimes. This:-

Dim names = {"John", "Price", "Homer"}

Is the same as this:-

Dim names As String() = {"John", "Price", "Homer"}

It is a String array. This part:-

{"John", "Price", "Homer"}

Is evaluated as a String array by the compiler so when it's assigned to a variable declared without an explicit type, the compiler will assume you want the variable to be a String array and make it so.

Sorry in advance, I bothered you. It's pleasant for user to actually see what's going on and they ARE different rather than a Yes or No, True or False. (However he or she has no idea what does those ABCDEF0~9s mean XD) But like Microsoft Windows Copy/Paste similar items dialog which asks you and shows you differences in date and size in bold. You get the point...

So you want to know which parts of the file are different? I don't know about that one. That might be a project by itself. Hashing isn't meant to tell you where it's different, only that it is different.

[Niya]

Also, you might want to use HashArray4 from this post since it may be quite a bit faster as it only performs the hashing once.

[pourkascheff]

So you want to know which parts of the file are different?

Even reading it feels like a nightmare...

Hashing isn't meant to tell you where it's different, only that it is different.

You are absolutely right.
* I won't close the thread for a few more days until I test it.

[wqweto]

Btw, .Net hashers support online hashing (i.e. hashing in chunks when the source cannot fit in memory) which can be used for HashArray4 function like this

    Public Function HashArray4(ByVal data As String()) As UInteger

        Dim ToUInt32 = Function(hash As Byte()) (From i In Enumerable.Range(0, 4)
                                                 Select BitConverter.ToUInt32(hash, i * 4)).
                                                 Aggregate(Function(acc, v) acc Xor v)
        Using hasher = MD5.Create()
            Dim lengthBuffer() As Byte
            For Each s In data
                Dim inputBuffer = Encoding.UTF8.GetBytes(s)
                lengthBuffer = BitConverter.GetBytes(inputBuffer.Length * 8)
                hasher.TransformBlock(lengthBuffer, 0, lengthBuffer.Length, Nothing, 0)
                hasher.TransformBlock(inputBuffer, 0, inputBuffer.Length, Nothing, 0)
            Next
            lengthBuffer = BitConverter.GetBytes(4 * 8)
            hasher.TransformFinalBlock(lengthBuffer, 0, lengthBuffer.Length)
            Return ToUInt32(hasher.Hash)
        End Using
    End Function

cheers,
</wqw>

[Niya]

Sweet! That one is even better. I have never explored HashAlgorithm derived classes that much. I didn't even know it could be done like that. I also never noticed they implement IDisposable.