I recently spent some time looking at a very interesting issue, so I wanted to share it and save people some time troubleshooting similar issues. The problem was that a site requiring TLS client authentication was loading very slowly - about 30 seconds page loading time for the index page, which to say the least on today’s fast networks is incredibly slow. The setup was as follows:

Client Browser <—> ISA <—> Web Server

Naturally the first step was to look at what was happening on the wire. Firing off a sniffer and capturing the traffic, the strange thing was that there were multiple Change Cipher Spec (CCS) TLS messages. In general, when you negotiate a SSL/TLS session, you do Change Cipher Spec once and from there on the traffic is encrypted and flows between the client and the server. After spending some time looking at different traces one thing came out as a pattern - the number of encrypted CCS messages was roughly equal to the number of HTTP requests made from the client minus the number of TCP/IP connections used by the browser. So I fired up the debugger to help us confirm that this was indeed the case. Debugging the Microsoft SSL/TLS implementation is a bit tricky, since the handshake processing happens in the lsass.exe process (which hosts all security protocol providers) and the actual encryption/decryption of data happens in the process using TLS (the browser in this case). Some debugger tricks and a few sweat spots later, it was confirmed that for each HTTP request the server will send a HelloRequest message back to the client, which is the protocol message used by servers to force clients to reauthenticate. It was very strange that for each resource the brower fetches from the server a new renegotiation is required, since this is not the normal mode of operation.

What’s next in the network path after the browser - the ISA server. After digging around all the settings, what picked the attention was the setting below:

In the case of the slow config, the checkbox was unchecked and the timeout value was greyed out, though still showing 300. What the admin assumed is that 300 is the default value, so the unchecked box didn’t matter much. Well, what it really means is that the ISA server will *not* cache the client certificate at all, which in turn causes the client to authenticate for each request it sends to the ISA server. Needless to say, once the checkbox was checked, there were no extreneous CCS messages exchanged and the page was loading much much faster.

I never suspected that such an innocent looking checkbox can cause so much trouble : ).