[RESOLVED] Conversion Problem

[couttsj]

I need some help with a Crypto call. According to MSDN, the following C++ call sets the random string from the client:

Data.pbData = pClientRandom;
Data.cbData = cbClientRandom;
CryptSetKeyParam(
	hMasterKey, 
	KP_CLIENT_RANDOM, 
	(PBYTE)&Data, 
	0);

Converting to VB, the function is declared as:

Private Declare Function CryptSetKeyParam Lib "advapi32.dll" (ByVal hKey As Long, ByVal dwParam As Long, ByVal pbData As Long, ByVal dwFlags As Long) As Long

The call:

CryptSetKeyParam(hMasterKey, KP_CLIENT_RANDOM, hClientRandom, 0)

returns an error. However, the call works when using KP_SCHANNEL_ALG (20) instead of KP_CLIENT_RANDOM (21) or KP_CLIENT_SERVER (22). According to Microsoft, the structure of "pbData" will vary, depending on the value of "dwParam". In the case of KP_CLIENT_RANDOM, "pbData" is a handle to a byte string containing 32 bytes of random data from the client followed by whatever cbClientRandom is. Some sources suggest that this is a BLOB structure, which means that cbClientRandom should be a 4 byte long integer (DWORD) containing the length of pClientRandom.

I have tried every combination I can think of, but the call always returns an error. Any help would be appreciated.

J.A. Coutts

[Jacob Roman]

Well I researched it, and it looks like your declarations are correct according to this site: http://forums.asp.net/t/988210.aspx/1

They have a vb6 example using quite a few APIS from the advapi32.dll, including CryptSetKeyParam. So in that site I linked go to the ***The follwing is my VB 6.0 method*** section and hopefully thatll lead you in the right direction on utilizing this API correctly.

[Bonnie West]

According to MSDN, the following C++ call sets the random string from the client:

Data.pbData = pClientRandom;
Data.cbData = cbClientRandom;
CryptSetKeyParam(
hMasterKey,
KP_CLIENT_RANDOM,
(PBYTE)&Data,
0);

Converting to VB, the function is declared as:

Private Declare Function CryptSetKeyParam Lib "advapi32.dll" (ByVal hKey As Long, ByVal dwParam As Long, ByVal pbData As Long, ByVal dwFlags As Long) As Long

The pbData parameter would be best declared as:

ByRef pbData As Any

That declaration allows the greatest flexibility when passing any kind of data.

The call:
CryptSetKeyParam(hMasterKey, KP_CLIENT_RANDOM, hClientRandom, 0)
returns an error. However, the call works when using KP_SCHANNEL_ALG (20) instead of KP_CLIENT_RANDOM (21) or KP_CLIENT_SERVER (22). According to Microsoft, the structure of "pbData" will vary, depending on the value of "dwParam". In the case of KP_CLIENT_RANDOM, "pbData" is a handle to a byte string containing 32 bytes of random data from the client followed by whatever cbClientRandom is. Some sources suggest that this is a BLOB structure, which means that cbClientRandom should be a 4 byte long integer (DWORD) containing the length of pClientRandom.

The following attempt assumes a lot about your code:

Private Type DataType
    cbData As Long      'Length, in bytes, of the data pointed by pbData
    pbData As Long      'Pointer to the data
End Type

Private Declare Function CryptSetKeyParam Lib "advapi32.dll" (ByVal hKey As Long, ByVal dwParam As Long, ByRef pbData As Any, ByVal dwFlags As Long) As Long

Private Sub Routine(ByVal hMasterKey As Long)
    Dim cbClientRandom As Long, pClientRandom As Long, Data As DataType, ByteString(0 To 31) As Byte

    cbClientRandom = 32& 'bytes
    pClientRandom = VarPtr(ByteString(0&))

    Data.cbData = cbClientRandom
    Data.pbData = pClientRandom

    Debug.Print "CryptSetKeyParam = " & CryptSetKeyParam(hMasterKey, KP_CLIENT_RANDOM, Data, 0&)
    Debug.Print "Err.LastDllError = " & Err.LastDllError
End Sub

I have tried every combination I can think of, but the call always returns an error.

You can get more info about the error by passing Err.LastDllError to the FormatMessage function.

[couttsj]

Well I researched it, and it looks like your declarations are correct according to this site: http://forums.asp.net/t/988210.aspx/1

They have a vb6 example using quite a few APIS from the advapi32.dll, including CryptSetKeyParam. So in that site I linked go to the ***The follwing is my VB 6.0 method*** section and hopefully thatll lead you in the right direction on utilizing this API correctly.

Thanks for the feedback. Unfortunately, that link does not help since it does not include Secure Channel. The CSPs provided on my system are:
Provider Type Provider Name
------------------------------
1 Microsoft Base Cryptographic Provider v1.0
13 Microsoft Base DSS and Diffie-Hellman Cryptographic Provider
3 Microsoft Base DSS Cryptographic Provider
1 Microsoft Base Smart Card Crypto Provider
18 Microsoft DH SChannel Cryptographic Provider
1 Microsoft Enhanced Cryptographic Provider v1.0
13 Microsoft Enhanced DSS and Diffie-Hellman Cryptographic Provider
24 Microsoft Enhanced RSA and AES Cryptographic Provider
12 Microsoft RSA SChannel Cryptographic Provider
1 Microsoft Strong Cryptographic Provider
Default Provider Name : Microsoft Strong Cryptographic Provider

SCHANNEL (type 12 or 18) must be used in order to support SSL3 or TLS1, and utilizes the library file schannel.dll.

I am still working on Bonnie's suggestions.

J.A. Coutts

[couttsj]

Reply to Bonnie West

Your code as supplied does not error out. Unfortunately, the same call using KP_SCHANNEL_ALG errors out now. I need to look at it further. Why did you reverse the order of cbData & pbData? If I put them back in the original order, your code gives me the same error that I was getting before (-2146893819).

J.A. Coutts

Addendum: error translates as "System Error 0x80090005 Bad Data". Doesn't really tell us anything we don't already know.

[Bonnie West]

Why did you reverse the order of cbData & pbData? If I put them back in the original order, your code gives me the same error that I was getting before (-2146893819).

Well, most Windows API structures put the cb* member first. I just assumed this was also the case for your structure. BTW, I didn't see the KP_CLIENT_RANDOM constant being mentioned anywhere in the CryptSetKeyParam documentation.


EDIT

After a bit more digging in MSDN, I saw that KP_CLIENT_RANDOM (and the C++ snippet you posted in the OP) appeared in Deriving Bulk Encryption and MAC Keys. It looks like the Data structure is of type CRYPT_DATA_BLOB, which is an alias for the CRYPTOAPI_BLOB/CRYPT_INTEGER_BLOB structure.

Here are the amended declarations:

Private Const KP_CLIENT_RANDOM As Long = 21 'For setting the Secure Channel client random data

Private Type CRYPT_DATA_BLOB
    cbData As Long          'The count of bytes in the buffer pointed to by pbData.
    pbData As Long          'A pointer to a block of data bytes.
End Type

Dim Data As CRYPT_DATA_BLOB

[couttsj]

Reply to Bonnie:

If I change my original code to an empty byte string (32 VbNull) instead of the actual random bytes, it does not error out. I suspect that is why your original code worked. Experimenting a little, if I try to change the MAC Key Hash protocol to CALG_SHA1 instead of CALG_MD5 using KP_SCHANNEL_ALG as the parameter, it now errors out. It almost seems that any change from the default settings causes an error. I seem to remember somewhere that before using SCHANNEL you are supposed to enumerate the protocols. It never said why, but I will give that a try and let you know.

J.A. Coutts

Edit:
After acquiring the context for RSA SCHANNEL, enumerating algorythms shows:
---------------------------------------------------------
ALGID dwBits algType namelen Name
------------------------------
26126 128 Data_Encrypt 8 AES 128
26128 256 Data_Encrypt 8 AES 256
26114 128 Data_Encrypt 4 RC2
26625 128 Data_Encrypt 4 RC4
26113 56 Data_Encrypt 4 DES
26121 112 Data_Encrypt 13 3DES TWO KEY
26115 168 Data_Encrypt 5 3DES
32772 160 Hash 6 SHA-1
32771 128 Hash 4 MD5
32776 288 Hash 12 SSL3 SHAMD5
32773 0 Hash 4 MAC
41984 1024 Exchange 9 RSA_KEYX
32777 0 Hash 5 HMAC
19461 40 Msg_Encrypt 12 SSL2 MASTER
19457 384 Msg_Encrypt 12 SSL3 MASTER
19462 384 Msg_Encrypt 12 TLS1 MASTER
19458 0 Msg_Encrypt 16 SCH MASTER HASH
19459 0 Msg_Encrypt 12 SCH MAC KEY
19463 0 Msg_Encrypt 12 SCH ENC KEY
------------------------------------------------------------------
Unfortunately, the error still remains.

[couttsj]

Thanks Bonnie;

The code below now works. I can't sat that I understand, but it appears that the order has something to do with it.

Private Type CRYPT_DATA_BLOB
    cbData As Long          'The count of bytes in the buffer pointed to by pbData.
    pbData As Long          'A pointer to a block of data bytes.
End Type

    Dim Data As CRYPT_DATA_BLOB
    bClientRandom = StrConv(CLIENT_RANDOM, vbFromUnicode)
    Data.cbData = UBound(bClientRandom) + 1
    Data.pbData = VarPtr(bClientRandom(0))
    If CryptSetKeyParam(hMasterKey, KP_CLIENT_RANDOM, Data, 0) = 0 Then _
        Err.Raise Err.LastDllError, , "Could not set Client Key Paramters (CryptSetKeyParam API)"

I changed the KP_SCHANNEL_ALG calls to use the its Data Type instead of a VarPtr to it, and it now works as well. By the way, the Hash code change to SHA1 works as well if I use cbits = 160 instead of 128. This Crypto stuff is sure convoluted.

J.A. Coutts

[Bonnie West]

The code below now works. I can't sat that I understand, but it appears that the order has something to do with it.

Great! I myself don't know much about Crypto API too. I just followed the documentation.

This Crypto stuff is sure convoluted.

Definitely!



BTW, don't forget to mark this thread Resolved!

[couttsj]

I think I know why the the order is reversed. In C++ code Microsoft lists it as:
Data.pbData = pClientRandom 'Handle to byte string
Data.cbData = cbClientRandom 'handle to length of byte string
In order to get the call to work, we have to reverse the order:
Data.cbData = cbClientRandom 'handle to length of byte string
Data.pbData = pClientRandom 'Handle to byte string
When I export a BLOB, I have to reverse the order of the resulting byte string to get the correct value, and I assume that the Microsoft compliler automatically handles the reversal by declaring it as a BLOB structure. Does this mean that I should reverse the string itself before sending it to the call? At this point in time I have no idea if Secure Channel is getting the correct information or not, as I have not tested the complete code yet.

J.A. Coutts

[Bonnie West]

I think I know why the the order is reversed. In C++ code Microsoft lists it as:
Data.pbData = pClientRandom 'Handle to byte string
Data.cbData = cbClientRandom 'handle to length of byte string
In order to get the call to work, we have to reverse the order:
Data.cbData = cbClientRandom 'handle to length of byte string
Data.pbData = pClientRandom 'Handle to byte string

I doubt the order of assignment has any effect. However, the CRYPT_DATA_BLOB Type declaration must conform to the documentation.

Does this mean that I should reverse the string itself before sending it to the call?

Sorry, but I don't know the answer to that. I haven't messed that much with the Crypto API. Testing should clear things up though.

[couttsj]

Of course you are correct Bonnie. I assumed that the order they were listed was the order in which they were defined (one should never assume). When I checked the Wincrypt.h file, I find the type definition has them in the correct order, along with many alias names.

typedef struct _CRYPTOAPI_BLOB {
	DWORD cbData;
	BYTE* pbData;
} CRYPT_INTEGER_BLOB, *PCRYPT_INTEGER_BLOB,
  CRYPT_UINT_BLOB,    *PCRYPT_UINT_BLOB,
  CRYPT_OBJID_BLOB,   *PCRYPT_OBJID_BLOB,
  CERT_NAME_BLOB,     *PCERT_NAME_BLOB,
  CERT_RDN_VALUE_BLOB,*PCERT_RDN_VALUE_BLOB,
  CERT_BLOB,          *PCERT_BLOB,
  CRL_BLOB,           *PCRL_BLOB,
  DATA_BLOB,          *PDATA_BLOB,
  CRYPT_DATA_BLOB,    *PCRYPT_DATA_BLOB,
  CRYPT_HASH_BLOB,    *PCRYPT_HASH_BLOB,
  CRYPT_DIGEST_BLOB,  *PCRYPT_DIGEST_BLOB,
  CRYPT_DER_BLOB,     *PCRYPT_DER_BLOB,
  CRYPT_ATTR_BLOB,    *PCRYPT_ATTR_BLOB;

I don't know of any easy way of testing TLS handshakes. The very nature of the beast means that each session is different. I have used a packet sniffer to capture various sessions, but I cannot even duplicate these because I do not know the random string that the client used. It was sent encrypted using the Server Certificate Public key, and only the Private key can be used to decrypt it.

J.A. Coutts

[Bonnie West]

I don't know of any easy way of testing TLS handshakes. The very nature of the beast means that each session is different. I have used a packet sniffer to capture various sessions, but I cannot even duplicate these because I do not know the random string that the client used. It was sent encrypted using the Server Certificate Public key, and only the Private key can be used to decrypt it.

I understand now what we're dealing with! Previously, I had no idea of what you were talking about, so I apologize for my earlier statement. Well now, it looks like we can only rely on whatever documentation is available. Unfortunately, I have only scratched the surface of the Crypto API; again I apologize if I can't be of much help!