Troubleshooting a TLS 1.3 connection is not easy. Because I was having difficulty getting "curve25519" to work properly in Win 10/11, I decided to implement a simulation program that I could step through.
Even though "ECDH_P256" and "curve25519" are both theoretically ECDH algorithms, there are slight differences. This is a result of the way that MS handles X22519 in relation to the other ECDH algorithms. ECDH_P256, ECDH_P384, and ECDH_P512 fit well into the MS nomenclature, but X25519 does not. There was originally a distrust in CISA as to the provision of back doors in the protocols, so X25519 was adopted as an alternative. X25519 is not supported in Windows versions prior to part way through Win 10. In theory, X25519 is faster than the others, but I cannot substantiate that, at least not from a client perspective using VB6.
In the attached program, I have retained the networking class and module for a couple of reasons. There are a few routines in them that are still used, and it shows the similarity between the network code and the command buttons that I have used in the simulation. The network code is found in the sub TCP.
The ECDH_P256 routine simulates an actual connection to the Gmail SMTP server. The network traffic was recorded using PacketVB, while DebugPrintByte statements recorded the various calculation results in the Client.
The X25519 routine parallels the Simple 1-RTT Handshake routine in RFC 8448. To simplify the comparison, I have included a text file (RTT_HS.txt), that restructures the text to better coincide with our program.
In naming the variables for the simulation, "X" is used for X25519 related items, "P" is used for ECDH_P256, "C" is used for Client, and "S" is used to represent Server related items. It is hopeful that this simulation will help VB6 programmers better understand TLS 1.3.
J.A. Coutts
Updated: 06/18/2025 - See post #2 for details
Updated: 09/04/2025 - See post #20 for details
Last edited by couttsj; Sep 4th, 2025 at 04:14 PM.
Verification of GMail POP3 and SMTP accounts has been added to TLS_Test. The Account Name and Password are fictitious, so you will have to change those to get the proper results. The password is a 16 byte password supplied by Google as part of GMails 2 factor authentication. It has been tested using both "ECDH_P256" and "X25519".
The last obstacle encountered to get X25519 to work with MS was the order in the Client Hello extension "Supported Groups". I had assumed (wrongly) that the Public Key being identified and provided in the "Keyshare" extension only had to be included in the "Supported Groups" extension. By trial and error I discovered that it had to be at the top of the list to avoid a non-explanatory Fatal Alert. That doesn't make a lot of sense to me.
Once connected to the server, POP3 on Port 995 and SMTP on Port 465 pretty much follow the standards. The one exception I am aware of with GMail is the lack of support for the "STAT" and LIST" commands. GMail will not return information on messages that it considers to have been read. For this reason, I use the Web based program to delete messages.
The noticeable differences between X25519 and the other ECDH protocols is:
1. The Public and Private key blobs for X25519 use the full 64 bytes with the last 32 bytes being zero.
2. The first and last bits of the Private key portion must be adjusted. Wqweto calls this clamping, and can be seen in the GetXKeys sub.
3. For X25519, BCryptOpenAlgorithmProvider requires a pointer to "ECDH" rather than the Key Algorithm name.
4. It also requires the use of BCryptSetProperty, which fails if you try to use it with the other ECC keys.
5. When calculating the KeyShare, the other keys require an extra byte (&H04) in front of the buffer. X25519 does not. See function GetKeyShare for details. TLS 1.3 does not support compression, but for some obscure reason the &H04 is used to indicate no compression.
Even though I have finally got X25519 to work with GMail, Fastmail still returns a non-explanatory Fatal Alert 40 and Fastmail Help has not been very helpful. They are unwilling or unable to tell me what it doesn't like about the Client Hello.
Your ClientHello procedure is in a shoddy state with multiple HexToByte blobs which remain black magic. You do realize that TLS packets are nested chunks which are type and length prefixed, yet there are no helper procedures to aid you generate these correctly while the source code remaining readable.
Compare with this code:
Code:
'--- Record Header
pvBufferWriteRecordStart uOutput, TLS_CONTENT_TYPE_HANDSHAKE, uCtx
'--- Handshake Header
lMessagePos = uOutput.Size
pvBufferWriteLong uOutput, TLS_HANDSHAKE_CLIENT_HELLO
pvBufferWriteBlockStart uOutput, Size:=3
pvBufferWriteLong uOutput, TLS_LOCAL_LEGACY_VERSION, Size:=2
If pvArraySize(.LocalExchRandom) = 0 Then
pvTlsGetRandom .LocalExchRandom, TLS_HELLO_RANDOM_SIZE
End If
pvBufferWriteArray uOutput, .LocalExchRandom
'--- Legacy Session ID
pvBufferWriteBlockStart uOutput
If .HelloRetryRequest Then
pvBufferWriteArray uOutput, .RemoteSessionID
ElseIf pvArraySize(.LocalSessionID) = 0 And (.LocalFeatures And ucsTlsSupportTls12) <> 0 Then
'--- non-empty for TLS 1.2 compatibility
pvTlsGetRandom baTemp, TLS_HELLO_RANDOM_SIZE
pvBufferWriteArray uOutput, baTemp
Else
pvBufferWriteArray uOutput, .LocalSessionID
End If
pvBufferWriteBlockEnd uOutput
'--- Cipher Suites
pvBufferWriteBlockStart uOutput, Size:=2
For Each vElem In pvTlsGetSortedCipherSuites(.LocalFeatures)
pvBufferWriteLong uOutput, vElem, Size:=2
Next
pvBufferWriteBlockEnd uOutput
'--- Legacy Compression Methods
pvBufferWriteBlockStart uOutput
pvBufferWriteLong uOutput, TLS_COMPRESS_NULL
pvBufferWriteBlockEnd uOutput
'--- Extensions
pvBufferWriteBlockStart uOutput, Size:=2
If LenB(.RemoteHostName) <> 0 Then
'--- Extension - Server Name
pvBufferWriteLong uOutput, TLS_EXTENSION_SERVER_NAME, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteLong uOutput, TLS_SERVER_NAME_TYPE_HOSTNAME '--- FQDN
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteString uOutput, .RemoteHostName
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
End If
If LenB(.AlpnProtocols) <> 0 Then
'--- Extension - ALPN
pvBufferWriteLong uOutput, TLS_EXTENSION_ALPN, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
For Each vElem In Split(.AlpnProtocols, "|")
pvBufferWriteBlockStart uOutput
pvBufferWriteString uOutput, Left$(vElem, 255)
pvBufferWriteBlockEnd uOutput
Next
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
End If
'--- Extension - Supported Groups
pvBufferWriteLong uOutput, TLS_EXTENSION_SUPPORTED_GROUPS, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
If pvCryptoIsSupported(ucsTlsAlgoExchX25519) Then
pvBufferWriteLong uOutput, TLS_GROUP_X25519, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoExchSecp256r1) Then
pvBufferWriteLong uOutput, TLS_GROUP_SECP256R1, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoExchSecp384r1) Then
pvBufferWriteLong uOutput, TLS_GROUP_SECP384R1, Size:=2
End If
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
'--- Extension - OCSP Status Request
pvArrayByte baTemp, 0, TLS_EXTENSION_STATUS_REQUEST, 0, 5, 1, 0, 0, 0, 0
pvBufferWriteArray uOutput, baTemp
If (.LocalFeatures And ucsTlsSupportTls12) <> 0 Then
'--- Extension - EC Point Formats
pvArrayByte baTemp, 0, TLS_EXTENSION_EC_POINT_FORMAT, 0, 2, 1, 0
pvBufferWriteArray uOutput, baTemp '--- uncompressed only
'--- Extension - Extended Master Secret
pvArrayByte baTemp, 0, TLS_EXTENSION_EXTENDED_MASTER_SECRET, 0, 0
pvBufferWriteArray uOutput, baTemp '--- supported
'--- Extension - Encrypt-then-MAC
pvArrayByte baTemp, 0, TLS_EXTENSION_ENCRYPT_THEN_MAC, 0, 0
pvBufferWriteArray uOutput, baTemp '--- supported
'--- Extension - Renegotiation Info
pvBufferWriteLong uOutput, TLS_EXTENSION_RENEGOTIATION_INFO, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput
pvBufferWriteArray uOutput, .LocalLegacyVerifyData
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
'--- Extension - Session Ticket
pvBufferWriteLong uOutput, TLS_EXTENSION_SESSION_TICKET, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteLong uOutput, 0, Size:=4
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteArray uOutput, .LocalSessionTicket
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
End If
'--- Extension - Signature Algorithms
pvBufferWriteLong uOutput, TLS_EXTENSION_SIGNATURE_ALGORITHMS, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
If pvCryptoIsSupported(ucsTlsAlgoExchSecp256r1) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_ECDSA_SECP256R1_SHA256, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoExchSecp384r1) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_ECDSA_SECP384R1_SHA384, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoExchSecp521r1) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_ECDSA_SECP521R1_SHA512, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoPaddingPss) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PSS_RSAE_SHA256, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PSS_RSAE_SHA384, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PSS_RSAE_SHA512, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PSS_PSS_SHA256, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PSS_PSS_SHA384, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PSS_PSS_SHA512, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoPaddingPkcs) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PKCS1_SHA224, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PKCS1_SHA256, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PKCS1_SHA384, Size:=2
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PKCS1_SHA512, Size:=2
End If
'--- legacy SHA-1 based signatures
If pvCryptoIsSupported(ucsTlsAlgoDigestSha1) Then
If pvCryptoIsSupported(ucsTlsAlgoPaddingPkcs) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_RSA_PKCS1_SHA1, Size:=2
End If
If pvCryptoIsSupported(ucsTlsAlgoExchSecp256r1) Or pvCryptoIsSupported(ucsTlsAlgoExchSecp384r1) Then
pvBufferWriteLong uOutput, TLS_SIGNATURE_ECDSA_SHA1, Size:=2
End If
End If
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
If (.LocalFeatures And ucsTlsSupportTls13) <> 0 Then
'--- Extension - Post Handshake Auth
pvArrayByte baTemp, 0, TLS_EXTENSION_POST_HANDSHAKE_AUTH, 0, 0
pvBufferWriteArray uOutput, baTemp '--- supported
'--- Extension - Key Share
pvBufferWriteLong uOutput, TLS_EXTENSION_KEY_SHARE, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteLong uOutput, .ExchGroup, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteArray uOutput, .LocalExchPublic
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
'--- Extension - Supported Versions
pvBufferWriteLong uOutput, TLS_EXTENSION_SUPPORTED_VERSIONS, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput
pvBufferWriteLong uOutput, TLS_PROTOCOL_VERSION_TLS13, Size:=2
If (.LocalFeatures And ucsTlsSupportTls12) <> 0 Then
pvBufferWriteLong uOutput, TLS_PROTOCOL_VERSION_TLS12, Size:=2
End If
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
If .HelloRetryRequest And SearchCollection(.RemoteExtensions, "#" & TLS_EXTENSION_COOKIE) Then
'--- Extension - Cookie
pvBufferWriteLong uOutput, TLS_EXTENSION_COOKIE, Size:=2
pvBufferWriteBlockStart uOutput, Size:=2
pvBufferWriteBlockStart uOutput
pvBufferWriteArray uOutput, .HelloRetryCookie
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
End If
End If
pvBufferWriteBlockEnd uOutput
pvBufferWriteBlockEnd uOutput
pvTlsAppendHandshakeHash uCtx, uOutput.Data, lMessagePos, uOutput.Size - lMessagePos
pvBufferWriteRecordEnd uOutput, uCtx
The nested structure is apparent with each pvBufferWriteXxxStart is matched by pvBufferWriteXxxEnd. The latter calculates number of bytes emitted since matching pvBufferWriteXxxStart and puts this in the header. This prevents whole class of errors with wrong chunk sizes.
Everything in TLS in specified in RFCs. There is no undocumented 04 byte for uncompressed point format for NIST Curves. It's documented enum with 02/03 meaning compressed encoding with positive/negative Y coordinate of the point.
RFC 8422 documents that staring with TLS 1.2 the protocol only supports uncompressed encoding for points on NIST Curves.
I agree that the ClientHello function is not very streamlined, but it was put together under pressure. In my opinion, The committee that put together TLS 1.3 did an excellent job considering what they had to work with. But the result is very fractured and fragmented.
When my Internet supplier (Telus) suddenly farmed out their email service to GMail after a significant outage, I had to move quickly to put together something that would interface with GMail. I did not trust cloud storage (and still don't), so that meant maintaining my own database and interfacing directly with GMail POP3 and SMTP servers. The Client Hello function was simply a way of recreating what already worked. That was all fine and dandy until Fastmail decided to move the functional POBox system over to their own system. They receive email addressed to POBox, forward it to Telus (GMail), and flag it as deleted. After the move, I was suddenly swamped with spam/scam because GMail used the Sender address instead of the return address I provided when someone with a GMail address replied to one of my emails. That allowed hackers to learn my real address, so I decided to move the sending of email over to Fastmail and change the Telus (GMail) account name.
Since then, both Fastmail and GMail have improved their spam detection, but I was already well into trying to get Fastmail to send emails. With GMail, there were lots of third parties who could provide advice on how to access the GMail servers. Such was not the case with Fastmail, and their Help Desk has been less than helpful. From the very beginning I decided to use perfect forward secrecy (PFS) only to improve security and to try and reduce the complex fragmented nature of TLS. It has not been easy and I sincerely appreciate the help you have provided.
You realistically cannot expect any mail provider helpdesk to answer questions about TLS errors their hosts return. Even their admins don't understand the intricacies of CH packets as implemented in TLS 1.3 as they use ready-made distro packages which ship with their own TLS libraries (e.g. openssl) so admins mostly configure the distro POP3/IMAP/SMTP service and have no internal knowledge about TLS support of the particular postfix version they are using.
Anyway, for SMTP clients Windows ships with SChannel so that one will not have to reimplement TLS with each new version of the protocol which gets standardized.
By trial and error I was able to reduce the GMail Client Hello to 196 bytes. So I tried the same thing with Fastmail using the Client Hello produced by OpenSSL. It contained everything but the kitchen sink (329 bytes), but it got me past the Fatal Alerts. (original shown with lower case hex)
Whenever I tried to remove some of the unused parts such as the supported versions, the Fatal Alert was all I got back. How did OpenSSL know to use all that unnecessary stuff, and why does Fastmail not accept anything less?
I'm sending a 281 bytes CH and this works fine with smtp.fastmail.com:465. Don't see anything extraordinary with their TLS server support like bogus Client Hello Retries etc.
The size of the handshake packet is determined more by the number of ciphersuites supported and the number of signatures supported -- these lists take up most of the space.
cheers,
</wqw>
Last edited by wqweto; Jun 23rd, 2025 at 12:40 PM.
Yes, these are mostly TLS 1.2 extensions I'm including for backcompat with 1.2 servers. You can reduce ciphersuites and signature types supported but keep the ones they choose to use initially.
Well, I got it down to 146 bytes. SessionID was trimmed to zero length, 14 cipher suites were eliminated, EC point formats was eliminated, 11 signature algorithms were eliminated, and TLS 1.2 support was deleted.
The interesting part of the signature algorithms is that I has to delete each algorithm one at a time before testing for it to work, and when I got to "rsa_pss_rsae_sha256", I received a Fatal Alert. I have no explanation for this behaviour, and I presume that "rsa_pss_rsae_sha256" is a required algorithm.
The next problem I have to work on is the fact that upon receiving the certificate, my code returns "Handshake Authentication Error".
The signature algorithm advertised by the client must be compatible with the server's certificate. Rarely the servers are deciding whether to use RSA vs ECDSA certificate based on client's supported signature algorithms.
So in this case rsa_pss_rsae_sha256 works because fastmail.com is using RSA based certificate and openssl supports RSA-PSS signature scheme.
FYI, older Crypto API does not support PSS (newer CNG does) so I had to reimplement PSS signature scheme in pure VB6 based on RFC 8017 to support these signatures in VbAsyncSocket. I can safely say that signatures support consumed well over half the development time and it is the most trickier part of the TLS spec.
It's coming as HandshakeType = 8 i.e. encrypted_extensions. Seems to be an empty list here which is redundand traffic but mostly harmless.
cheers,
</wqw>
How can you tell that it is a type 8? I can't even get it to Decrypt properly. I recorded the encrypted record, the hsReadKey, and the hsReadIV. Then I created a simulation program and tried several different ways to Decrypt it with no success.
Code:
hsReadKey:
6F F5 E4 07 BB 0A 09 84 DE 39 AD 48 47 8F 12 12
hsReadIV:
A5 51 1E 23 B7 88 A4 DE FB FC 98 E5
Encrypted:
80 22 24 77 6D 7A 59 0B 8A CF 85 83 DA 23 30 FE
D6 BA C7 33 73 FA 7B
Decrypted:
5D C8 CA 8C 96 BA
Record Authenticated:
0B 8A CF 85 83 DA 23 30 FE D6 BA C7 33 73 FA 7B
J.A. Coutts
Addendum: 07/16/2025
By ignoring the error, the above results were achieved.
Last edited by couttsj; Jul 15th, 2025 at 05:56 PM.
I am still struggling with the Encrypted Extensions sent by Fastmail. Regardless of what I do, the handshake decryption returns Error 0XC000A002 (STATUS_AUTH_TAG_MISMATCH). The code I am using works great with GMail. The difference between the two is that GMail sends the Encrypted Extensions, Certificate, and Finished records in one compacted record, and Fastmail sends them in separate records.
I suspect that the AuthData is not what BCryptDecrypt is expecting, but my attempts to figure out how the AuthTag is calculated were very frustrating because of all the different options available.
Code:
2A 2D FD 5F E6 5D 51 77 0C 86 01 0E F8 B9 2D 44 - hsWriteKey
2B D5 5A F6 46 FF 68 2A D0 6F 01 EA - hsWriteIV
1D AF 3F 07 E9 B8 74 42 DA 7D 2D 88 15 B0 4C 8D - hsReadKey
56 9C E2 58 1B 9A E2 C8 0D 8E 51 28 - hsReadIV
17 03 03 00 17 EF 6B 48 29 E7 AE 0D B4 50 24 F2 - Complete Record
27 F3 5A A7 21 4E 8E 6E 38 9F 4F 3E
17 03 03 00 17 - Header (not Encrypted)
EF 6B 48 29 E7 AE - Encrypted Data
0D - Data type (Encrypted)
B4 50 24 F2 27 F3 5A A7 21 4E 8E 6E 38 9F 4F 3E - AuthTag (not Encrypted)
My best guess as to what the decrypted 6 bytes should look like is:
Code:
08
00
00 02
00 00
Does anyone have any idea what BCrypt is looking for in the way of Authentication Data?
Using the values provided here (https://tls13.xargs.org/), I was able to locate one of the problems with Encrypted Extensions. To provide for the ability to add extra characters to an encrypted record, TLS 1.3 adds a non zero character (Record Type) to the record to be encrypted. This is because the encryption itself can contain zeros, and a non zero character is needed to act as a separator between the data and the zero buffering.
When GMail sends a combined record, the entire record is sent using the character 0x17 in the Record Header, and this is the character that must be used as the Record Type to verify the Encrypted Extensions as well as the Certificate, Certificate Verify, and Finished records.
When a separate Encrypted Extensions record is sent by the server, 0x17 is used in the header to identify it as an encrypted record, but 0x16 is used as the extra Record Type (handshake record). This is the character that must be used to verify the record.
Handshake records (0x16) are not encrypted while Application Data records (0x17) are always encrypted which is what probably made the difference in your case.
It turns out that using 0x16 as the added non zero character was not the problem at all. I had requested 1301 (AES GCM 128) as my preferred cryptographic suite. What I failed to notice was that Fastmail responded with 1302 (AES GCM 256) instead. My routines were not yet capable of adjusting to other suites, and removing 1302 as an option, Fastmail simply went to 1303. The only way to get it to use 1301 was to make that the only available option.
I do intend to support other options in the future, but in this case I was after the fastest method. It takes less then a second to check for new mail, and the extra security provided by 1302 or 1303 is not worth the time and effort. Every connection to the server provides a new set of keys.
The problem I have now is that SMTP is returning "500 5.5.2 Error: bad syntax". Since this has nothing to do with TLS 1.3, I will post this problem separately if necessary.
The TLS_Test program has been updated to reflect the addition of Account Verification on Fastmail. It works, but I am not completely happy with the code I had to use to get Fastmail to function properly, as the "QUIT" command produces a Syntax Error that I have not been able to resolve.
On both sites (GMail & Fastmail) I have provided fictitious credentials which will not verify. To get it to produce positive results, you must change strSMTPAcct, strSender, & strPW to real values. In both cases the Password is a 16 byte variable provided by each site as part of 2 Factor Authentication.
Reply With Quote
Originally Posted by wqweto
