Make Hex File

I found a need to use data samples in Hex format that were created using "C" based programs. The problem I ran into was the fact that they used lower case characters (eg 5a) and VB used upper case characters (eg 5A). Manually editing this information proved to be time consuming and prone to errors, so I made the attached program to do the job.

As an example, the samples provided by https://tls13.xargs.org/ range from:

Code:

---

202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f

---

to

Code:

---

16 03 03 00 7a 02 00 00 76 03 03 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f 20 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff 13 02 00 00 2e 00 2b 00 02 03 04 00 33 00 24 00 1d 00 20 9f d7 ad 6d cf f4 29 8d d3 f9 6d 5b 1b 2a f9 10 a0 53 5b 14 88 d7 f8 fa bb 34 9a 98 28 80 b6 15

---

Samples using space separators are much easier to read, but using no spaces is much quicker to convert to byte format.

To use this simple program, paste the string information into the upper text box. If the sample contains no spaces, the program will automatically convert lower case characters to upper case, add space separators, and display the result without spaces in the lower text box.

If the sample already contains spaces, the case conversion is done in the upper text box, and clicking the "Remove Spaces" transfers that information to the lower text box without spaces in a more convenient width.

To add that information to a file, use the "Save as File" button to enter the file name and path. By default, a ".hex" extension is added to the file name, and the file path is the current path. Files are created using File API calls.

J.A. Coutts

In your second Hex example with newlines, did you intend for the newlines to be encoded as &H00 in the resultant file?

Attachment 194661

In case it's of use, here's a HexToBytes() function that handles newlines without converting them to &H00, and it also runs about 2x as fast when using your second Hex string example (with newlines). Non-newline Hex strings are process only a little bit faster than your existing HexToByte function.

Code:

---

Function HexToBytes(ByVal p_Hex As String) As Byte()
  ' Accepts a consistently format Hex string and converts it to a byte array
  ' Hex values must be 2-characters long (i.e. with leading Zero if necessary)
  ' Hex values can have a single space separator or no separator
  ' Hex strings can have a single CRLF, LF, or CR line ending character per line, or can be a single continuous line
  ' Hex string using a space separator and multiple lines can have a single space character before the end of the line
  '        or no space character before the end of the line.
 
  Dim l_HasSpaces As Boolean
  Dim l_Step As Long
  Dim l_Len As Long
  Dim l_Index As Long
  Dim l_ByteString As String
  Dim la_Bytes() As Byte
  Dim ii As Long
  Dim l_NewLinePos As Long
  Dim l_ReplaceChar As String
 
  l_HasSpaces = (Mid$(p_Hex, 3, 1) = " ")
 
  ' Check for CRLF, LF-only, and CR-only line breaks
  ' If any linebreak is found, check whether the hex lines are space terminated immediately before the line break
  ' So that we can replace line-breaks with either a " " or empty string
  ' (ensuring Hex string is consistently padded at 2 or 3 characters per value)
  l_NewLinePos = InStr(1, p_Hex, vbLf)
  If l_NewLinePos > 0 Then
      If l_HasSpaces Then
        If Mid$(p_Hex, l_NewLinePos - 1, 1) <> " " Then
            If Mid$(p_Hex, l_NewLinePos - 2, 1) <> " " Then
              l_ReplaceChar = " "
            End If
        End If
      End If
     
      p_Hex = Replace$(Replace$(p_Hex, vbLf, l_ReplaceChar), vbCr, vbNullString)
 
  Else
      If l_HasSpaces Then
        If Mid$(p_Hex, l_NewLinePos - 1, 1) <> " " Then
            l_ReplaceChar = " "
        End If
      End If
     
      If InStr(1, p_Hex, vbCr) > 0 Then
        p_Hex = Replace$(p_Hex, vbCr, l_ReplaceChar)
     
      End If
  End If
 
  ' Determine size of resultnat Byte array
  ' and the number of characters to step over when grabbing hex values from string
  If l_HasSpaces Then
      l_Step = 3
      l_Len = Len(p_Hex) / 3
  Else
      l_Step = 2
      l_Len = Len(p_Hex) / 2
  End If
 
  ReDim la_Bytes(l_Len - 1)
  l_ByteString = "&H--"  ' This will be mutated per loop use for building the byte array more quickly
 
  For ii = 1 To Len(p_Hex) Step l_Step
      Mid$(l_ByteString, 3) = Mid$(p_Hex, ii, 2)  ' Stuff the 2-digit hex value into the &H-- string
     
      la_Bytes(l_Index) = CByte(l_ByteString)
      l_Index = l_Index + 1
  Next
 
  HexToBytes = la_Bytes
End Function

---

Note that the above has only been lightly tested, so if you're going to use it I recommend throwing a bunch of Hex at it to see if it chokes anywhere.

Removing the superfluous "Exit Function" from "HexToBytes = la_Bytes: Exit Function" will cause the compiler to optimize the code generation by moving the array pointer in the function result instead of copying the whole array all over again.

Also the CryptStringToBinaryW has a CRYPT_STRING_HEX flag that performs this conversion for you (likely a lot faster if it even matters).

Quote:

---

Originally Posted by VanGoghGaming

Removing the superfluous "Exit Function" from "HexToBytes = la_Bytes: Exit Function"

---

Good catch, thanks - accidentally left that behind after removing some errorhandling/testing code below it. Saves about 50-100ms per 100k runs compiled ;) (but surely more if you have large arrays).

It's been a long day and my brain isn't working right, but something about the "Exit Function"/"End Function" array assignment optimization was bugging me and I found this conversation between Dilettante and Bonnie West on the topic. My reading indicates that the Exit Function (despite being useless in my case because the End Function immediately followed it) should have resulted in the compiler optimizing by transferring ownership of the array and not making a copy. That said, I did notice a small performance boost when I removed the Exit Function. Am I misunderstanding?

LOL should have kept reading that thread, looks like I was a part of it :afrog:

Guess it is time for bed.

Quote:

---

Originally Posted by jpbro

In your second Hex example with newlines, did you intend for the newlines to be encoded as &H00 in the resultant file?

---

I am afraid that I don't understand your question. The whole idea was to highlight and copy directly from the Web page using Ctrl-c and paste to the first text box with a Ctrl-v. There should be no newlines. The only valid Hex characters are 0123456789ABCDEF..

J.A. Coutts

Quote:

---

Originally Posted by jpbro

In case it's of use, here's a HexToBytes() function that handles newlines without converting them to &H00, and it also runs about 2x as fast when using your second Hex string example (with newlines). Non-newline Hex strings are process only a little bit faster than your existing HexToByte function.

Note that the above has only been lightly tested, so if you're going to use it I recommend throwing a bunch of Hex at it to see if it chokes anywhere.

---

This is my own HexToByte routine.

Code:

---

Public Function HexToByte(HexStr As String) As Byte()
    Dim lLen As Long
    Dim lPntr As Long
    Dim bTmp() As Byte
    Dim bHex() As Byte
    If Len(HexStr) > 1 Then
        lLen = Len(HexStr) / 2
        ReDim bHex(lLen - 1)
        For lPntr = 0 To UBound(bHex)
            bHex(lPntr) = Val("&H" & Mid$(HexStr, lPntr * 2 + 1, 2))
        Next lPntr
        HexToByte = bHex
    Else
        bHex = bTmp
    End If
End Function

---

J.A. Coutts

I prefer shorter ones (though somewhat less performant):

Code:

---

Public Function ToHex(baData() As Byte) As String
    Dim lIdx            As Long
    Dim sByte          As String
   
    ToHex = String$(UBound(baData) * 2 + 2, 48)
    For lIdx = 0 To UBound(baData)
        sByte = LCase$(Hex$(baData(lIdx)))
        Mid$(ToHex, lIdx * 2 + 3 - Len(sByte)) = sByte
    Next
End Function

Public Function FromHex(sText As String) As Byte()
    Dim baRetVal()      As Byte
    Dim lIdx            As Long
   
    On Error GoTo QH
    '--- check for hexdump delimiter
    If sText Like "*[!0-9A-Fa-f]*" Then
        ReDim baRetVal(0 To Len(sText) \ 3) As Byte
        For lIdx = 1 To Len(sText) Step 3
            baRetVal(lIdx \ 3) = "&H" & Mid$(sText, lIdx, 2)
        Next
    ElseIf LenB(sText) <> 0 Then
        ReDim baRetVal(0 To Len(sText) \ 2 - 1) As Byte
        For lIdx = 1 To Len(sText) Step 2
            baRetVal(lIdx \ 2) = "&H" & Mid$(sText, lIdx, 2)
        Next
    Else
        baRetVal = vbNullString
    End If
    FromHex = baRetVal
QH:End Function

---

ToHex dumps in lower-case. FromHex works on input both with and w/o delimiter (space, dash or any)

cheers,
</wqw>

Quote:

---

Originally Posted by jpbro

It's been a long day and my brain isn't working right, but something about the "Exit Function"/"End Function" array assignment optimization was bugging me and I found this conversation between Dilettante and Bonnie West on the topic. My reading indicates that the Exit Function (despite being useless in my case because the End Function immediately followed it) should have resulted in the compiler optimizing by transferring ownership of the array and not making a copy. That said, I did notice a small performance boost when I removed the Exit Function. Am I misunderstanding?

---

I was wondering after the discussion in the thread came up with line numbers before Exit Sub/Function being detrimental to the optimization but did not discuss if there being reference local variables which need to be terminated (by calling Release) stops the optimization as well.

We've been putting line numbers on all our release builds and apparently squashed this retval optimization in the process. Bummer!

cheers,
</wqw>

Quote:

---

Originally Posted by couttsj

I am afraid that I don't understand your question. The whole idea was to highlight and copy directly from the Web page using Ctrl-c and paste to the first text box with a Ctrl-v. There should be no newlines. The only valid Hex characters are 0123456789ABCDEF

---

In your demo app, if you copy & paste multi-line Hex, click Remove Spaces and then save the file, the newlines aren't removed and are converted to &H00 in the resultant saved file:

Attachment 194664

Resultant Hex has 00 at each line break point:

Attachment 194665

I would expect the newlines characters to either be dropped/not encoded as &H00, or encoded as &H13 &H10.

Quote:

---

Originally Posted by couttsj

This is my own HexToByte routine.

Code:

---

Public Function HexToByte(HexStr As String) As Byte()
    Dim lLen As Long
    Dim lPntr As Long
    Dim bTmp() As Byte
    Dim bHex() As Byte
    If Len(HexStr) > 1 Then
        lLen = Len(HexStr) / 2
        ReDim bHex(lLen - 1)
        For lPntr = 0 To UBound(bHex)
            bHex(lPntr) = Val("&H" & Mid$(HexStr, lPntr * 2 + 1, 2))
        Next lPntr
        HexToByte = bHex
    Else
        bHex = bTmp
    End If
End Function

---

J.A. Coutts

---

Yup, I saw that in your demo code - I just provided an alternative if you are interested that is about 2x as fast. This probably only matters if you are doing a lot of HexToByte calls in a tight loop though, and in that case it might be worth looking at the CryptStringToBinaryW API that VanGoghGaming mentioned to see if it's even faster (I haven't tried it). I was mostly just having fun to see if I could write something that handled newlines and spaces better, and also be faster.

Quote:

---

Originally Posted by jpbro

In your demo app, if you copy & paste multi-line Hex, click Remove Spaces and then save the file, the newlines aren't removed and are converted to &H00 in the resultant saved file:

I would expect the newlines characters to either be dropped/not encoded as &H00, or encoded as &H13 &H10.

---

I see what happened now. You copied from the posting rather than the Web site I referred you to. I have not encountered Cr or CrLF characters as yet, but I suppose it would not take a lot of effort to remove any invalid Hex character pair.

J.A. Coutts

Ran into the need to edit out CrLf bytes, so the attachment has been updated.

J.A. Coutts