Page 1 of 3 123 LastLast
Results 1 to 40 of 108

Thread: Simulate TLS 1.3

  1. #1

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Simulate TLS 1.3

    To understand TLS 1.3, https://tls13.ulfheim.net/ is useful, but unfortunately it contains several discrepancies if you want to follow it in detail (eg. labels are not complete). For the detail, https://tools.ietf.org/html/rfc8448 is better. Unfortunately, Win 8.1 does not support x25519, so the best I could come up with was a simulation without generating the Agreed Secret.

    Like previous cryptographic protocols, TLS 1.3 uses a Session Hash. Unlike previous protocols, it uses 2 sets of keys and encrypts part of the handshake. The Session Hash uses the decrypted data, and Write keys on the Server are Read keys on the Client (and visa versa).

    The attached program attempts to duplicate the steps in the IETF trace example for the Simple 1-RTT Handshake, separating the Client steps from the Server. In the Client and Server portions, the Hash is not calculated or shown, as it is included in the Info. Clicking "Client" or "Server" takes you to the first step of calculating the "Early Secret". Using the "Enter" key advances through each step until the keys are summarized at the end.

    The Key options on the other hand don't show all the information used, the Session Hash is calculated, and calculations are made as soon as the information is available.

    The next step will be to add the actual encryption/decryption as well as the application data.

    J.A. Coutts
    Attached Images Attached Images  
    Attached Files Attached Files

  2. #2

  3. #3

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Btw, the Illustrated TLS site source code is available on github: https://github.com/syncsynchalt/illustrated-tls13

    Which labels are missing? I can send a PR w/ the fixes.

    cheers,
    </wqw>
    label = "derived" instead of label = "tls13 derived". The "tls13 " seemed to be missing on all the labels. There were other problems, but that was the most obvious. Other than missing details, the site was very informative.

    J.A. Coutts

  4. #4
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Yes, they always prefix the label with "tls13 " in hkdf.sh on line 22. On line 18 they have let labellen=labellen+6 which adds the size of "tls13 " prefix.

    Btw, in my effort I never needed pure Expand so from HKDF primitives the module finished with implemented Extract method (which is a glorified HMAC) and ExpandLabel combo as they did in hkdf.sh. That would be like to combine CreateInfo and Expand methods on your clsHKDF in a single ExpandLabel method with more parameters.

    cheers,
    </wqw>

  5. #5

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    This version adds the actual packet formulation and transmission to the simulation plus a sample encrypted data packet. To accomplish this, separate Client and Server programs are utilized. Because Win 8.1 does not support x25519, the Client Hello and Server Hello had to be imported. Normally, errors 0 to 12 are returned instead of the Agreed Secret to indicate where the error occurred, but this error is not currently being intercepted. Instead, the Agreed Secret used by RFC 8448 is imported.

    I vaguely remember something about the Session Hash being reset after the Handshake is established, but I could not find any information to verify that. If anyone can point me towards such information, it would be greatly appreciated.

    J.A. Coutts
    Attached Images Attached Images   
    Attached Files Attached Files

  6. #6
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    > I vaguely remember something about the Session Hash being reset after the Handshake is established, but I could not find any information to verify that. If anyone can point me towards such information, it would be greatly appreciated.

    Take a look at Section 4.4.1. The Transcript Hash of RFC 8446 where they talk about "synthetic handshake message".

    Resetting the handshake messages hash and starting w/ a simple 32-byte Hash(ClientHello1) in the current buffer for handshake messages (i.e. storing only a hash of the handshake messages before reset) is done no matter if the server uses cookie extension on HRR or not i.e. doesn't matter if the server is implementing HRR in stateless fashion or keeps state in some connection bound object as is the case with cTlsSocket class -- its instances don't use cookies for HRR as the TLS connection's previous state (before reset) is available in the instance itself so no need to implement HRR in a stateless fashion.

    cheers,
    </wqw>

  7. #7

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Take a look at Section 4.4.1. The Transcript Hash of RFC 8446 where they talk about "synthetic handshake message".

    Resetting the handshake messages hash and starting w/ a simple 32-byte Hash(ClientHello1) in the current buffer for handshake messages (i.e. storing only a hash of the handshake messages before reset) is done no matter if the server uses cookie extension on HRR or not i.e. doesn't matter if the server is implementing HRR in stateless fashion or keeps state in some connection bound object as is the case with cTlsSocket class -- its instances don't use cookies for HRR as the TLS connection's previous state (before reset) is available in the instance itself so no need to implement HRR in a stateless fashion.

    cheers,
    </wqw>
    I have always found that RFC's are extremely difficult to follow, but I think what it is saying is that the Session Hash (Transcript Hash) is only used during the Handshake. So maintaining the Session Hash post Handshake is rather pointless in TLS 1.3.

    One other thing I noticed is that only the Client and Server Hello record headers are classified as RT_HANDSHAKE (&H16). The rest of the handshake records are classified as RT_APPLICATION_DATA (&H17). That seems a little odd to me.

    J.A. Coutts

  8. #8
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Quote Originally Posted by couttsj View Post
    The rest of the handshake records are classified as RT_APPLICATION_DATA (&H17). That seems a little odd to me.
    The only type of records that can be protected (encrypted) are RT_APPLICATION_DATA so encrypted handshake, alerts and actual data can pass through hardware middle-boxes designed for TLS 1.2 without traffic being expected. These firewalls cannot decrypt data traffic even in TLS 1.2 but in TLS 1.2 the handshake was not encrypted and h/w inspected it as it passed in plain-text (so called Stateful Packet Inspection).

    It is the last byte of the decrypted data (just before the zero padding) that we discussed above that holds the actual record type. If the wrapper record type is *not* RT_APPLICATION_DATA on encrypted records then this should raise unexpected_message (10) alert in both server and client.

    There is tlsfuzzer project that is very useful for testing TLS server implementation adherence. I'm currently running those tests on my TLS 1.3 server and a lot of flaws, deviations from the expected behavior and pure crashes are uncovered. This same fuzzer is used by GnuTLS and openssl (probably) too.

    So yes, alert records can be unencrypted if happening before bulk traffic secrets are established and tunneled encrypted through app data records past this point.

    cheers,
    </wqw>

  9. #9
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Here is a custom build of openssl that I used to develop my implementation: openssl-dev.zip (includes VS2015 debug runtime and config file)

    The main caveat is that this custom build dumps randoms, secrets and derived keys to console as these are generated, derived and used during handshake and bulk operations.

    Here are some sample commands with openssl.exe you can use to start a test web server that your client implementation can negoatiate TLS parameters to.

    Generate RSA certificate

    Code:
    c:> openssl req -x509 -newkey rsa:2048 -keyout privkey.pem -out cert.pem -days 365 -nodes -config openssl.cnf
    Start server with this RSA certificate

    Code:
    c:> openssl s_server -key privkey.pem -cert cert.pem -accept 10443 -www
    Navigate to https://localhost:10443 in chrome or curl to inspect TLS 1.3 handshake

    Generate ECC certificate

    Code:
    c:> openssl ecparam -name prime256v1 -genkey -noout -out ecprivkey.pem
    c:> openssl req -new -sha256 -key ecprivkey.pem -out csr.csr -config openssl.cnf
    c:> openssl req -x509 -sha256 -days 365 -key ecprivkey.pem -in csr.csr -out eccert.pem -config openssl.cnf
    Start server with this ECC certificate + debug dump traffic too

    Code:
    c:> openssl s_server -key ecprivkey.pem -cert eccert.pem -accept 10443 -www -debug
    Generate PFX exchange file for ECC certificate + private key

    Code:
    c:> openssl pkcs12 -export -inkey ecprivkey.pem -in eccert.pem -out eccert.pfx
    cheers,
    </wqw>

  10. #10

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    I ran into an issue with calculating the Server Finished message in the last simulation. It didn't show up because I was not yet verifying that message on the Client. According to RFC 8448, the Key and Finished message should be:
    Code:
    tls13 finished expanded (32 octets):
    00 8d 3b 66 f8 16 ea 55 9f 96 b5 37 e8 85 c3 1f 
    c0 68 bf 49 2c 65 2f 01 f2 88 a1 d8 cd c1 9f c8
    
    finished extracted (32 octets):
    9b 9b 14 1d 90 63 37 fb d2 cb dc e7 1d f4 de da 
    4a b4 2c 30 95 72 cb 7f ff ee 54 54 b7 8f 07 18
    I had no trouble with the Client Finished record, but the Server Finished uses the same Key as RFC 8448 and gets a different Finished message:
    Code:
    tls13 finished:
    00 8D 3B 66 F8 16 EA 55 9F 96 B5 37 E8 85 C3 1F 
    C0 68 BF 49 2C 65 2F 01 F2 88 A1 D8 CD C1 9F C8 
    
    Finished:
    2B 47 34 88 B0 E9 D7 08 5D 0B FF 61 AC DD 4E FE 
    3B 04 05 4B 30 6C 39 96 35 21 55 BA 24 B5 03 87 
    
    Session Hash after Server Hello:
    86 0C 06 ED C0 78 58 EE 8E 78 F0 E7 42 8C 58 ED 
    D6 B4 3F 2C A3 E6 E9 5F 02 ED 06 3C F0 E1 CA D8
    Unfortunately the RFC does not give any information on what it used to calculate the Finished message. Is it possible that they calculate the Session Hash for the EncryptedExtensions separately, even though the EncryptedExtensions, the Finished handshake message, the Certificate, and the CertificateVerify are all sent as one record?

    J.A. Coutts

    In answer to my own question, the record itself must be broken down into the individual messages before processing. The EncryptedExtensions, Certificate, and CertificateVerify must be individually or collectively added to the Session Hash before calculating the Server Finish. Then the Server Finish is added to the Session Hash, and all four are then assembled into a record, encrypted, and sent to the client.

    The client must decrypt the record, separate the individual messages, calculate the Session Hash using the first three messages, verify the Server Finish, and then add the server Finish to the Session Hash.

    J.A. Coutts
    Last edited by couttsj; Aug 16th, 2020 at 08:44 PM.

  11. #11
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Yes, Handshake messages can be chunked and multiplexed over the records layer. For instance a server certificates might not fit in a single record (max size of 16KB) so it has to be chunked. The multiplexing you experienced yourself when several handshake messages are sent together over a single record. The handshake hash is calculated over data in messages layer, so record headers are not included.

  12. #12

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    This version is supplied as a working Client/Server pair, and no longer uses RFC 8448 data. The Server is configured to act as a Web Server, and the Client does not act as a full browser. HTML code is displayed as delivered.

    The Server should deliver HTML content as well as binary files. The default is the Kitty page, and you can add files (jpeg, png, etc) to suit. Just add the file names to the Form_Load routine and copy the files to the working directory. As long as your browser supports TLS 1.3, you should be able to access those files in your browser using "https://localhost/(filename)" as the URL.

    J.A. Coutts

    Updated: 10/10/2020
    Updated: 10/12/2020
    Updated: 03/24/2021 (see post #95 for details)
    Attached Images Attached Images   
    Attached Files Attached Files
    Last edited by couttsj; Mar 24th, 2021 at 04:11 PM.

  13. #13
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Quote Originally Posted by couttsj View Post
    The client program however receives the ServerHello, followed immediately by the Encrypted Extensions, the Certifcate, the Certificate Verify, and the Server Finish in a single record. If you halt the program in the mClient_EncrDataArrival routine to examine the ServerHello, you will find the data in the incoming buffer is not what you would expect.
    Ouch!

    I told you before that sound foundations include proper winsock helper. E.g. cAsyncSocket class has a proper notification queue maintained by the ASM thunk which raises all receive notification sequentially and does not re-enter event handlers (esp. in the IDE). Otherwise it must be very difficult (next to impossible) to properly debug any client/server network code, when tracing one piece of the protocol might lead to receive event being fired for next chunks queued by the peer preemptively.

    cheers,
    </wqw>

  14. #14

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Ouch!

    I told you before that sound foundations include proper winsock helper. E.g. cAsyncSocket class has a proper notification queue maintained by the ASM thunk which raises all receive notification sequentially and does not re-enter event handlers (esp. in the IDE). Otherwise it must be very difficult (next to impossible) to properly debug any client/server network code, when tracing one piece of the protocol might lead to receive event being fired for next chunks queued by the peer preemptively.

    cheers,
    </wqw>
    It has worked flawlessly for me in multiple applications over many years. It works well even in streaming applications. The Winsock2 buffers handle the flow of data, and the only time it presents a problem is in the IDE, which is easy enough to work around. What I like about it is the speed and efficiency, and the ability to make subtle adjustments when necessary. For example, I can make it reuse a port without having to wait for the port to time out. That is the main reason I have never made it into a DLL or ActiveX. The downside is the fact that it uses non-blocking calls, which I believe does not make it suitable for high throughput server applications.

    J.A. Coutts

  15. #15
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    > It works well even in streaming applications

    No idea why these are any different.

    > The Winsock2 buffers handle the flow of data, and the only time it presents a problem is in the IDE, which is easy enough to work around.

    Does Winsock control re-enter its DataArrival event while stepping in the debugger?

    > The downside is the fact that it uses non-blocking calls, which I believe does not make it suitable for high throughput server applications.

    It's not non-blocking vs blocking calls because blocking calls are even worse for server applications. It's always non-blocking calls but whether message based WSAAsyncSelect vs multi-threaded completion ports makes the difference in server implementations.

    The problem with SimpleSock is that it's based on CSocketMaster's subclassing which is using decent implementation with EbMode for IDE protection but still it's not enough to prevent re-entrancy in the IDE. The queue with async notifications has to be implemented in the ASM thunk to allow stepping in the debugger while receiving following chunks of data does not re-enter DataArrival event.

    cheers,
    </wqw>

  16. #16

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    It's not non-blocking vs blocking calls because blocking calls are even worse for server applications. It's always non-blocking calls but whether message based WSAAsyncSelect vs multi-threaded completion ports makes the difference in server implementations.

    cheers,
    </wqw>
    I have to intercept and react to Error 10035 (WSAEWOULDBLOCK) because I am using nonblocking sockets. Multithreaded sockets are handled differently. From that I assumed (rightly or wrongly) that MS expected or preferred multithreaded sockets, as they have this to say about the error:

    "This error is returned from operations on nonblocking sockets that cannot be completed immediately, for example recv when no data is queued to be read from the socket. It is a nonfatal error, and the operation should be retried later. It is normal for WSAEWOULDBLOCK to be reported as the result from calling connect on a nonblocking SOCK_STREAM socket, since some time must elapse for the connection to be established."

    Was I wrong in making this assumption? Are socket operations via the Windows messaging system as efficient as multithreaded operations?

    By the way, I am sending you a PM about a different matter.

    J.A. Coutts

  17. #17
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    AsyncSelect based nonblocking sockets are both less efficient and less stable i.e. under load client connections fail to establish. Speaking about internet scale servers noone uses Windows ones for front-facing servers (the so called load-balancers or reverse proxies) anyway.

  18. #18

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    AsyncSelect based nonblocking sockets are both less efficient and less stable i.e. under load client connections fail to establish. Speaking about internet scale servers noone uses Windows ones for front-facing servers (the so called load-balancers or reverse proxies) anyway.
    That pretty much parallels my own understanding.

    J.A. Coutts

  19. #19

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Here is a custom build of openssl that I used to develop my implementation: openssl-dev.zip (includes VS2015 debug runtime and config file)

    The main caveat is that this custom build dumps randoms, secrets and derived keys to console as these are generated, derived and used during handshake and bulk operations.

    Here are some sample commands with openssl.exe you can use to start a test web server that your client implementation can negoatiate TLS parameters to.

    Generate RSA certificate

    Code:
    c:> openssl req -x509 -newkey rsa:2048 -keyout privkey.pem -out cert.pem -days 365 -nodes -config openssl.cnf
    Start server with this RSA certificate

    Code:
    c:> openssl s_server -key privkey.pem -cert cert.pem -accept 10443 -www
    Navigate to https://localhost:10443 in chrome or curl to inspect TLS 1.3 handshake

    Generate ECC certificate

    Code:
    c:> openssl ecparam -name prime256v1 -genkey -noout -out ecprivkey.pem
    c:> openssl req -new -sha256 -key ecprivkey.pem -out csr.csr -config openssl.cnf
    c:> openssl req -x509 -sha256 -days 365 -key ecprivkey.pem -in csr.csr -out eccert.pem -config openssl.cnf
    Start server with this ECC certificate + debug dump traffic too

    Code:
    c:> openssl s_server -key ecprivkey.pem -cert eccert.pem -accept 10443 -www -debug
    Generate PFX exchange file for ECC certificate + private key

    Code:
    c:> openssl pkcs12 -export -inkey ecprivkey.pem -in eccert.pem -out eccert.pfx
    cheers,
    </wqw>
    I finally got around to utilizing the info you supplied to generate an ECC Certificate. The very first command produced:
    Code:
    openssl ecparam -name prime256v1 -genkey -noout -out ecprivkey.pem
    The command ecparam is deprecated. Use 'pkeyparam' instead.
    so I utilized this command instead:
    Code:
    openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256
    -----BEGIN PRIVATE KEY-----
    MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgHTUX8tt9AZ9Kdmba
    /B4BKauCoiCzq1FB4Udxd9xrxz+hRANCAATo2kUMwSvbEQrVJ9Rblu/pEEYot81I
    a+QohTXWXsYxDD4rxsN3mxBlvmJ14koj0xNAO8zbTAQ7Z6PsbmTCJ2+C
    -----END PRIVATE KEY-----
    plus the other two commands.

    When I started the server and attempted to connect to it, I received a Fatal Alert 50.
    Code:
    openssl s_server -key ecprivkey.pem -cert eccert.pem -accept 443 -www -debug
    Using default temp DH parameters
    ACCEPT
    read from 0x1428d80 [0x145a95b] (5 bytes => 5 (0x5))
    0000 - 16 03 03 00 c4                                    .....
    read from 0x1428d80 [0x145a960] (196 bytes => 196 (0xC4))
    0000 - 01 00 00 00 bf 03 d8 00-bb 24 2a 71 8b f4 95 25   .........$*q...%
    0010 - 6d fa ee 76 22 91 05 6d-ef 20 9b 3d c1 91 97 05   m..v"..m. .=....
    0020 - ca f1 51 6a db 32 00 00-06 13 01 13 02 13 03 01   ..Qj.2..........
    0030 - 00 00 00 00 00 00 0e 8c-0c 00 00 09 6c 6f 63 61   ............loca
    0040 - 6c 68 6f 73 74 ff 01 00-01 00 00 0a 00 08 00 06   lhost...........
    0050 - 00 17 00 18 00 19 00 23-00 00 00 33 00 47 00 45   .......#...3.G.E
    0060 - 00 17 00 41 04 fc 1a 97-1b 55 16 a9 f9 38 d6 32   ...A.....U...8.2
    0070 - d0 e6 b0 1f d6 3e 45 f9-f8 9f 3f bc d8 8c ca 99   .....>E...?.....
    0080 - 39 c0 fb 46 a9 44 ca 3a-5b bf a4 ab 2a 44 ed e3   9..F.D.:[...*D..
    0090 - 2d 7a c3 c2 66 1f 4e d9-b9 9d 13 59 5c f9 56 07   -z..f.N....Y\.V.
    00a0 - c1 b7 f2 82 cd 00 2b 00-03 02 03 04 00 0d 00 08   ......+.........
    00b0 - 00 06 04 03 05 03 06 03-00 2d 00 02 01 01 00 1c   .........-......
    00c0 - 00 02 40 01                                       ..@.
    write to 0x1428d80 [0x145fb00] (7 bytes => 7 (0x7))
    0000 - 15 03 03 00 02 02 32                              ......2
    E8:0F:00:00:error:SSL routines::length too short:ssl\statem\statem_srvr.c:1462:
    I have no idea what "length too short" means. Can you offer any suggestions?

    J.A. Coutts

  20. #20

  21. #21

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Here

    0000 - 01 00 00 00 bf 03 d8 00-bb 24 2a 71 8b f4 95 25

    the message size in ClientHello is zero for some reason.

    cheers,
    </wqw>
    Nice catch! That wasn't the only problem with the Client Hello. That's what I get for copying code for the record header (5 byte header vs 4 bytes required). It did not make any difference with my server code because I cherry picked the data that I needed. The only length checked was the entire record length.

    J.A. Coutts

  22. #22
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Well, testing against a real server always straightens up the implementation. On the other hand dedicated fuzzing is merciless :-))

    cheers,
    </wqw>

  23. #23

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    wqweto;

    After the Server Hello, what is the 1 byte Type 20 record for?
    Code:
    14 03 03 00 01 
    01
    My client program was expecting encrypted data directly after the Server Hello. Type 20 (0x14) is either CHANGE_CIPHER_SPEC (which is deprecated), or MT_FINISHED (which usually comes encrypted). I can work around it, I would just like to know what the purpose of it is.

    J.A. Coutts

  24. #24
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    It must be CHANGE_CIPHER_SPEC type of record which is sent for TLS 1.2 compatibility reasons (those hardware middleboxes). The &H0303 version here is TLS_PROTOCOL_VERSION_TLS12 too for the same reason and the same all over the record layer.

    CHANGE_CIPHER_SPEC is like a trigger point for the client (or server) to use the new bulk secrets that have been negotiated in the handshake messages right before it -- at least for TLS 1.2 it's like this, for TLS 1.3 it's completely obsolete and new keys/IVs start being used at completely independent points.

    cheers,
    </wqw>

  25. #25

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    After ignoring the 1 byte record, I ran into an authentication error after receiving the encrypted Extensions, Certificate, and Certificate Verify records. Through a process of elimination, I found the 1 byte record apparently did not qualify as a Handshake record. After removing it from the Session Hash, I then ran into yet another authentication error after the OpenSSL server sent 2 New Session Ticket records (optional). I had not implemented Resumption on the Client, along with "tls13 res master" and "tls13 resumption". After implementing those 2 routines, and verifying the AP Read and Write Keys/IV, I still get an authentication error. I presume that the New Session Ticket uses a different Key/IV than the Application, and that somehow "tls13 resumption" is used in that process. Why else would we need to create it?

    After extensive searching, I have not been able to come up with the process used. Hopefully someone can shed some light on this.

    J.A. Coutts

  26. #26
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    You can safely ignore NewSessionTicket, no need to implement resumption at all for a fully-functional client. You'll need resumption when implementing a browser that needs multiple connection for simultaneous download of images etc.

    > I then ran into yet another authentication error after the OpenSSL server sent 2 New Session Ticket records

    What alert do you get?

    > After implementing those 2 routines, and verifying the AP Read and Write Keys/IV

    Did you verify your bulk IVs/Keys vs the ones hexdumped by the server? Each party comes up with two pairs of IV+Key -- one local and one for the remote host. All these must match between client and host or bulk encryption will fail.

    > I presume that the New Session Ticket uses a different Key/IV than the Application, and that somehow "tls13 resumption" is used in that process. Why else would we need to create it?

    No, these tickets are used only for new simultaneous connections to the remote host by something like a browser (so called 0-RTT).

    Two not very clear points come to mind:

    1. Notice that after CHANGE_CIPHER_SPEC legacy record you have to send Client Handshake Finished message still encrypted with handshake secrets and after it you derive the application secrets for bulk encryption.

    2. Notice that verify_data in Client Handshake Finished message is based on handshake (transcript) hash *including* Client Certificate and (optional) Client Certificate Verify messages while application secrets derived after it use handshake (transcript) hash as *before* Client Certificate being sent. Go figure!

    This last point did not become obvious to me by just reading the RFC and I had to understand the sources of picotls to figure it out.

    cheers,
    </wqw>

  27. #27

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    wqweto;

    Without decrypting the record, we do not know the record type. Without knowing what kind of record it is, it is difficult to ignore. I went so far as to use the data supplied by RFC 8448. The AP Keys/IV (Bulk) verify, but the encrypted value produced using those keys does not match the value shown in the RFC.
    RFC 8448:
    Code:
    {server}  derive write traffic keys for application data:
       key expanded (16 octets):  9f 02 28 3b 6c 9c 07 ef c2 6b b9 f2 ac
             92 e3 56
       iv expanded (12 octets):  cf 78 2b 88 dd 83 54 9a ad f1 e9 84
    
    {server}  derive read traffic keys for application data (same as
          client application data write traffic keys)
       key expanded (16 octets):  17 42 2d da 59 6e d5 d9 ac d8 90 e3 c6
             3f 50 51
       iv expanded (12 octets):  5b 78 92 3d ee 08 57 90 33 e5 23 d9
    
    NewSessionTicket (205 octets):  04 00 00 c9 00 00 00 1e fa d6 aa
       c5 02 00 00 00 b2 2c 03 5d 82 93 59 ee 5f f7 af 4e c9 00 00 00
       00 26 2a 64 94 dc 48 6d 2c 8a 34 cb 33 fa 90 bf 1b 00 70 ad 3c
       49 88 83 c9 36 7c 09 a2 be 78 5a bc 55 cd 22 60 97 a3 a9 82 11
       72 83 f8 2a 03 a1 43 ef d3 ff 5d d3 6d 64 e8 61 be 7f d6 1d 28
       27 db 27 9c ce 14 50 77 d4 54 a3 66 4d 4e 6d a4 d2 9e e0 37 25
       a6 a4 da fc d0 fc 67 d2 ae a7 05 29 51 3e 3d a2 67 7f a5 90 6c
       5b 3f 7d 8f 92 f2 28 bd a4 0d da 72 14 70 f9 fb f2 97 b5 ae a6
       17 64 6f ac 5c 03 27 2e 97 07 27 c6 21 a7 91 41 ef 5f 7d e6 50
       5e 5b fb c3 88 e9 33 43 69 40 93 93 4a e4 d3 57 00 08 00 2a 00
       04 00 00 04 00
    complete record (227 octets):  17 03 03 00 de 3a 6b 8f 90 41 4a 97
       d6 95 9c 34 87 68 0d e5 13 4a 2b 24 0e 6c ff ac 11 6e 95 d4 1d
       6a f8 f6 b5 80 dc f3 d1 1d 63 c7 58 db 28 9a 01 59 40 25 2f 55
       71 3e 06 1d c1 3e 07 88 91 a3 8e fb cf 57 53 ad 8e f1 70 ad 3c
       73 53 d1 6d 9d a7 73 b9 ca 7f 2b 9f a1 b6 c0 d4 a3 d0 3f 75 e0
       9c 30 ba 1e 62 97 2a c4 6f 75 f7 b9 81 be 63 43 9b 29 99 ce 13
       06 46 15 13 98 91 d5 e4 c5 b4 06 f1 6e 3f c1 81 a7 7c a4 75 84
       00 25 db 2f 0a 77 f8 1b 5a b0 5b 94 c0 13 46 75 5f 69 23 2c 86
       51 9d 86 cb ee ac 87 aa c3 47 d1 43 f9 60 5d 64 f6 50 db 4d 02
       3e 70 e9 52 ca 49 fe 51 37 12 1c 74 bc 26 97 68 7e 24 87 46 d6
       df 35 30 05 f3 bc e1 86 96 12 9c 81 53 55 6b 3b 6c 67 79 b3 7b
       f1 59 85 68 4f
    Simulation:
    Code:
    AP Write Key:
    9F 02 28 3B 6C 9C 07 EF C2 6B B9 F2 AC 92 E3 56 
    AP Write IV:
    CF 78 2B 88 DD 83 54 9A AD F1 E9 84 
    
    AP Read Key:
    17 42 2D DA 59 6E D5 D9 AC D8 90 E3 C6 3F 50 51 
    AP Read IV:
    5B 78 92 3D EE 08 57 90 33 E5 23 D9 
    
    New Session Ticket:
    04 00 00 C9 00 00 00 1E FA D6 AA C5 02 00 00 00 
    B2 2C 03 5D 82 93 59 EE 5F F7 AF 4E C9 00 00 00 
    00 26 2A 64 94 DC 48 6D 2C 8A 34 CB 33 FA 90 BF 
    1B 00 70 AD 3C 49 88 83 C9 36 7C 09 A2 BE 78 5A 
    BC 55 CD 22 60 97 A3 A9 82 11 72 83 F8 2A 03 A1 
    43 EF D3 FF 5D D3 6D 64 E8 61 BE 7F D6 1D 28 27 
    DB 27 9C CE 14 50 77 D4 54 A3 66 4D 4E 6D A4 D2 
    9E E0 37 25 A6 A4 DA FC D0 FC 67 D2 AE A7 05 29 
    51 3E 3D A2 67 7F A5 90 6C 5B 3F 7D 8F 92 F2 28 
    BD A4 0D DA 72 14 70 F9 FB F2 97 B5 AE A6 17 64 
    6F AC 5C 03 27 2E 97 07 27 C6 21 A7 91 41 EF 5F 
    7D E6 50 5E 5B FB C3 88 E9 33 43 69 40 93 93 4A 
    E4 D3 57 00 08 00 2A 00 04 00 00 04 00          
    
    Session Ticket Encrypted:
    17 03 03 00 DE 2A 92 7C DB EB 4F C1 59 17 E7 0D 
    F8 0E 52 CA AB CB 0F ED 7C B3 56 10 7B ED F5 83 
    B6 DB E7 E7 62 B0 E1 D7 BB D0 D4 A5 1A 1E FA B6 
    1C DB 4B F5 62 CE C1 C5 A5 52 AE 8E 36 1F 05 29 
    7A E3 D2 24 CF 34 7B 57 D9 F7 03 07 AE DF 0A 59 
    99 BB 06 AA 7A 06 14 30 B1 26 F2 9C D8 BC D2 CC 
    FB 54 F6 07 AD E6 21 5C 31 05 A9 C4 62 5C 12 7A 
    6C 64 B6 88 9F 46 0C F3 30 14 13 85 DD 8E 03 EA 
    09 E5 3A 57 06 E2 08 67 52 08 96 22 4E B0 80 85 
    2F 2F EA CD CE 89 F1 E2 DE 1B 22 36 8D 32 08 88 
    C2 CC 29 16 AC C1 59 C5 68 D6 10 DC 1F BB B5 EA 
    60 42 CE 75 9A 1E 0E 30 54 1D B0 B4 95 98 05 81 
    8D 44 11 E3 F4 63 F9 0C 09 05 4B EF 1D A5 DB 81 
    04 2B 9D F0 99 5E A6 53 83 19 B8 6D B8 4F 70 FA 
    9B A6 FF
    The reason I get the authentication error is because the decryption fails, and I presume the decryption fails because the wrong key/IV is being used. Several articles suggest that a special key/IV is being used, but none of them detail how it is accomplished.

    J.A. Coutts

  28. #28
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    > Without decrypting the record, we do not know the record type. Without knowing what kind of record it is, it is difficult to ignore.

    I mean ignoring it *after* decrypting it. (No need to generate resumption masters, secrets, etc. at all)

    > The AP Keys/IV (Bulk) verify, but the encrypted value produced using those keys does not match the value shown in the RFC.

    Is it AES-128 that they agree in ServerHello because you have to match it in your test?

    Also note that local IV is XOR'ed with local sequence number before feeding it to the AEAD cipher. This NewSessionTicket is probably the first message encrypted with the application keys so the SeqNo must be 1 so *last* byte of write IV must become &H84 Xor &H01 = &H85 before passing it to the ecrypt routine. (SeqNo is 8-bytes number in *network* order so big-endian encoded)

    Btw, what do you use for AAD? The message is 205 bytes long + 16 bytes for the built-in MAC = 221 total size so AAD must be 17 03 03 00 DD. (total size is 2-bytes number in *network* order so big-endian encoded)

    > Several articles suggest that a special key/IV is being used, but none of them detail how it is accomplished.

    Which articles?

    cheers,
    </wqw>

  29. #29

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Also note that local IV is XOR'ed with local sequence number before feeding it to the AEAD cipher. This NewSessionTicket is probably the first message encrypted with the application keys so the SeqNo must be 1 so *last* byte of write IV must become &H84 Xor &H01 = &H85 before passing it to the ecrypt routine. (SeqNo is 8-bytes number in *network* order so big-endian encoded)

    cheers,
    </wqw>
    Something you said above triggered me to question the value of the Sequence Numbers. When I reset those numbers to zero, the encrypted value of the New Session Ticket matched RFC 8448. Once I knew what I was looking for, I found this information in RFC 8446:

    5.3. Per-Record Nonce
    A 64-bit sequence number is maintained separately for reading and writing records. The appropriate sequence number is incremented by one after reading or writing each record. Each sequence number is set to zero at the beginning of a connection and whenever the key is changed; the first record transmitted under a particular traffic key MUST use sequence number 0.

    Thanks again;

    J.A. Coutts

  30. #30
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Quote Originally Posted by couttsj View Post
    the first record transmitted under a particular traffic key MUST use sequence number 0.
    Oops, my bad. I just realized the sequence number is post-incremented in my impl too. I don't care to use a 64-bit counter here but just treat the simple 32-bit VB native Long as unsigned value for XORing purposes.

    There is no cipher in TLS 1.3 that can be used for more that 2^32 records without key renegotiation anyway, so the sequence number must be reset before overflowing 32-bits.

    cheers,
    </wqw>

  31. #31

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Oops, my bad. I just realized the sequence number is post-incremented in my impl too. I don't care to use a 64-bit counter here but just treat the simple 32-bit VB native Long as unsigned value for XORing purposes.

    There is no cipher in TLS 1.3 that can be used for more that 2^32 records without key renegotiation anyway, so the sequence number must be reset before overflowing 32-bits.

    cheers,
    </wqw>
    I chose to use a 12 byte array as the Sequence Number. I used 12 bytes instead of 8 because it simplified the XOR operation. I have overcome the problem with the RFC 8448 New Session Ticket, but not the OpenSSL supplied ones. They still produce an authentication error and do not decrypt properly.

    Update to last download coming shortly.

    J.A. Coutts

  32. #32

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    The last download (Post #12) has been upgraded. Several bugs were discovered in the Client Hello. As well, a New Session Key is now sent from the server after the Handshake is complete.

    As in the previous download, this one simulates the Simple 1-RTT Handshake outlined in RFC 8448, but can be easily converted to connect with any server by commenting out the code identified as "Temporary Code". However, neither the Client nor the Server are built to handle Web traffic.

    The New Session Ticket provided by the server outside of the Temporary Code is mostly random and is not fully functional, as the code to handle a Pre Shared Key (PSK) does not exist in the server.

    J.A Coutts

  33. #33
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Quote Originally Posted by couttsj View Post
    . . . but not the OpenSSL supplied ones. They still produce an authentication error and do not decrypt properly.
    Do you use openssl-dev from above? Do its local IV/Key dumps match remote secrets you come up at your side of the communication?

    I reached the same point and this is *exactly* why I had to instrument the openssl server to dump its internal state -- to be able to check key negotiation results.

    cheers,
    </wqw>

  34. #34

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Do you use openssl-dev from above? Do its local IV/Key dumps match remote secrets you come up at your side of the communication?

    I reached the same point and this is *exactly* why I had to instrument the openssl server to dump its internal state -- to be able to check key negotiation results.

    cheers,
    </wqw>
    Used OpenSSL supplied in post #9.
    Code:
    openssl s_server -key ecprivkey.pem -cert eccert.pem -accept 443 -www -debug
    There is a code flow error in the current download when the Encrypted Extensions, Certificate, Certificate Verify, and Server Finish are sent separately, but once corrected, it returns this:
    Code:
    FD_READ 1016
    OK Bytes obtained from buffer: 239
    
    AP Read Key:
    D1 63 F4 83 0A B0 4F EB 3C A8 A8 3B D1 17 46 77 
    AP Read IV:
    4A 60 93 0F 2D FB 68 6B 51 1E 57 2A 
    AP Write Key:
    96 C6 2B C9 87 35 80 53 07 9D 5D 4A F0 D0 6E 72 
    AP Write IV:
    62 0E BF FF E5 27 B9 82 6A C9 26 D1 
    
    hkdf_expand, label=key, outlen=16, out=
    96 c6 2b c9 87 35 80 53 07 9d 5d 4a f0 d0 6e 72 
    hkdf_expand, label=iv, outlen=12, out=
    62 0e bf ff e5 27 b9 82 6a c9 26 d1 
    
    InBuffer:
    BE A0 8D 58 DC 68 A0 EE 09 9C 14 4E 6F AD 25 F8 
    2B ED 93 98 9B 95 87 4A 5F 2C 64 DC AE 61 17 C0 
    35 8F 6B 69 13 65 A0 60 85 A6 F4 5A 42 E4 BA EC 
    89 86 47 D1 44 26 84 3F 9A 48 90 40 FE 02 47 35 
    BD B3 09 FE FD 1F 68 EA D7 BA 09 82 AE 20 83 CF 
    0C E2 40 A4 28 62 24 9E 4E 98 E5 B8 EC 0A 0E 63 
    FB C8 18 F2 F8 EB 67 51 42 B3 88 2E 10 DB F0 69 
    07 FA A9 4C 08 05 53 19 C6 55 17 61 DC 0F 4D 2C 
    EB 96 CF 90 D6 CD 50 14 19 09 AF F1 69 C4 A0 39 
    16 66 81 F2 A9 7E 31 58 7B E7 A3 07 71 06 32 EB 
    76 C1 76 95 4C AA 3F 9E EA 9E 3A E9 DC 1F 57 4F 
    33 41 92 5D C8 E7 BB A5 7F EB 7C 32 35 7B 8B C6 
    60 CB A3 4F 62 49 73 B3 25 6B 2A 2E A4 FD F1 5A 
    2E 59 90 E7 E2 8A B7 CE FF AF 26 98 30 E5 FF F1 
    E2 BD 5B 0D CB FA 5A 06 E2 8D 
    
    Key Used:
    D1 63 F4 83 0A B0 4F EB 3C A8 A8 3B D1 17 46 77
    The OutBuffer is empty because of the Authentication error.

    J.A. Coutts

  35. #35
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    Upload a working sample that I can test against openssl-dev server and will try to debug what's wrong with bulk secrets post handshake.

    cheers,
    </wqw>

  36. #36

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Upload a working sample that I can test against openssl-dev server and will try to debug what's wrong with bulk secrets post handshake.

    cheers,
    </wqw>
    Getting it to work with everything in one record, or getting it to work with separate records was relatively easy. Getting it to work in an either/or situation is proving to be a lot more difficult. Give me a couple of days to work this out.

    J.A. Coutts

  37. #37

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    Quote Originally Posted by wqweto View Post
    Upload a working sample that I can test against openssl-dev server and will try to debug what's wrong with bulk secrets post handshake.

    cheers,
    </wqw>
    Updated again!

    It did not take as long as I thought it would.

    J.A. Coutts

  38. #38

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    It turns out that the problem was with the Sequence Number. It was getting reset when the Handshake data was sent in a single record, but not when it was sent in individual records. I still don't understand why, but I reset the numbers with a direct call instead of relying on re-dimensioning the arrays.

    I also tested OpenSSL using FireFox V82, and discovered a few idiosyncrasies. FireFox sends 2 ECC public keys; x25519 and secp256r1. That threw me for a loop, but when I checked the RFC, it is permitted. FireFox also uses x25519 by default, and I could not find a way to change that.

    I also discovered that it sends a 32 byte Session ID. The RFC says that even though it isn't used in TLS 1.3, it is optional and that when the server receives it in the Client Hello, it should send it back in the Server Hello. And, if the client sends a non-empty session ID, the server MUST send the change_cipher_spec.

    Another observation was that FireFox does not send an SNI (Sever Name Indication). The RFC is not very clear on this, but it appears that it is only necessary on resumption.

    The next thing I noticed was that OpenSSL does not send New Session Keys after the Handshake from FireFox. So I duplicated the Firefox Client Hello by reversing the x25519 and secp256r1 public keys, but OpenSSL still sends the New Session Keys. The only thing I am not doing yet is sending the change_cipher_spec. There sure are a lot of subtle exceptions to TLS 1.3.

    J.A. Coutts

  39. #39
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,092

    Re: Simulate TLS 1.3

    > . . . the server MUST send the change_cipher_spec

    To be compatible with TLS 1.2 middleboxes. This is when a TLS 1.3 client is running in compatibility (stealth) mode and the server must play the same game. When the session ID is empty this is a signal that the client does not care about being compatible with TLS 1.2 middleboxes anymore and everyone can stop smuggling traffic under disguise of the old version.

    Still the server CAN send change_cipher_spec and most always do even on empty session ID. I don't check if there is session ID and always send it back anyway as it's not against the RFC and is ignored by TLS 1.3 clients anyway.

    > FireFox sends 2 ECC public keys; x25519 and secp256r1. That threw me for a loop

    Chrome is even worse. It sends GREASE all the time just to excercise the protocol. In every collection where the server must ignore unexpected entry (extensions, ephemeral keys, etc.) chrome is sending "strange" IDs like &HAEAE or &HBABA just to make every shortcut in server implementation incompatible.

    > FireFox also uses x25519 by default, and I could not find a way to change that.

    The server changes the default curve in ClientHello by returning a HelloRetryRequest and *insisting* on using secp256r1 for instance.

    > FireFox does not send an SNI (Sever Name Indication)

    SNI is obsolete and replaced by ALPN. I'm supporting the latter only as SNI is viable only up to TLS 1.2

    > The RFC is not very clear on this, but it appears that it is only necessary on resumption.

    Doubt it but I'm not supporting all the resumption infrastructure yet (only collect New Session Keys as byte-arrays). 0-RTT is mostly browser specific, doubt anyone would use my repo to cobble a browser. It's impossible to write browsers in 2020 anymore (it's way more effort than writing an OS, bitmap and 3D graphics library, compiler and network stack combined).

    cheers,
    </wqw>

  40. #40

    Thread Starter
    Frenzied Member
    Join Date
    Dec 2012
    Posts
    1,468

    Re: Simulate TLS 1.3

    When I attempt to connect to my server software using FireFox V82, it returns a message saying that the Server Hello is malformed. As far as I can see the Hello is not malformed, but I recognize that the message returned may not divulge the real problem by design. There could be some other problem that the software does not want to divulge for security reasons.
    Code:
    SERVER HELLO:
    02 00 00 97 - Header (len=151)
    03 03 - TLS 1.2
    6E 1D A2 53 5A DF 1F 3E DA AD 1B 7B 88 41 38 F4 - Server
    C0 3B 80 34 5A 91 8B 3C DB C2 C7 BB 2B 2B 4B DE - Random
    20 - Session ID (len=32)
    01 FC 83 F2 6C 3A 2C 2E 3C 60 4A 8C B9 63 85 F0 - Session
    C4 6B A2 7B 1C AC 1C 52 26 4B 92 74 D4 7F 4C 01 - ID
    13 01 - Cipher Suite: TLS_AES_128_GCM_SHA256
    00 - Compression Method None
    00 4E - Extension Length (len=78)
    00 33 - key share
       00 45 - (len=69)
          00 17 - Elliptic curve: secp256r1
             00 41 - (len=65)
                04 
                96 DC 39 53 6D B0 12 D1 02 07 45 CA 8F 66 F9 CA 
                A8 94 19 75 D3 80 99 E5 C3 25 F9 C5 A0 32 58 F2 
                9E 87 64 3E 65 6E 13 A9 F0 70 1F 43 A6 18 84 B1 
                AF E8 B4 4E 42 17 37 ED 7C 63 22 E3 A0 53 48 05 
    00 2B - supported versions
       00 02 - (len=2)
          03 04 - TLS 1.3
    Any idea what that problem might be?

    J.A Coutts

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width