netsekure rng

I recently spent some time looking at a very interesting issue, so I wanted to share it and save people some time troubleshooting similar issues. The problem was that a site requiring TLS client authentication was loading very slowly – about 30 seconds page loading time for the index page, which to say the least on today’s fast networks is incredibly slow. The setup was as follows:

Client Browser <—> ISA <—> Web Server

Naturally the first step was to look at what was happening on the wire. Firing off a sniffer and capturing the traffic, the strange thing was that there were multiple Change Cipher Spec (CCS) TLS messages. In general, when you negotiate a SSL/TLS session, you do Change Cipher Spec once and from there on the traffic is encrypted and flows between the client and the server. After spending some time looking at different traces one thing came out as a pattern – the number of encrypted CCS messages was roughly equal to the number of HTTP requests made from the client minus the number of TCP/IP connections used by the browser. So I fired up the debugger to help us confirm that this was indeed the case. Debugging the Microsoft SSL/TLS implementation is a bit tricky, since the handshake processing happens in the lsass.exe process (which hosts all security protocol providers) and the actual encryption/decryption of data happens in the process using TLS (the browser in this case). Some debugger tricks and a few sweat spots later, it was confirmed that for each HTTP request the server will send a HelloRequest message back to the client, which is the protocol message used by servers to force clients to reauthenticate. It was very strange that for each resource the brower fetches from the server a new renegotiation is required, since this is not the normal mode of operation.

What’s next in the network path after the browser – the ISA server. After digging around all the settings, what picked the attention was the setting below:

In the case of the slow config, the checkbox was unchecked and the timeout value was greyed out, though still showing 300. What the admin assumed is that 300 is the default value, so the unchecked box didn’t matter much. Well, what it really means is that the ISA server will *not* cache the client certificate at all, which in turn causes the client to authenticate for each request it sends to the ISA server. Needless to say, once the checkbox was checked, there were no extreneous CCS messages exchanged and the page was loading much much faster.

I never suspected that such an innocent looking checkbox can cause so much trouble : ).

One of the hard things to troubleshoot in Windows domains is NTLM authentication and the interesting MaxConcurrentAPI setting. When user account to be authenticated on the server does not belong to the local account database, the server must forward the authentication to a domain controller. It does so over the Netlogon secure channel. As such, it is governed by the MaxConcurrentApi setting as to how many outgoing authentication requests are allowed at a time. When you have a loaded server, this can create a problem, an example of which is described in an ISA server configuration article. Finding the root cause is very laborious process, so we’ve added performance counters to Netlogon to allow troubleshooting this issue. For Windows 2003, a hotfix is required, but for Windows 2008 and later it is present in the core install.
I was thinking of writing a very thorough guide on how authentication works in this type of scenario, but found out that the ISA documentation team has beaten me to it. Their description is targeted to ISA server, but you can replace it with any other application server doing user authentication using NTLM.

There are also another two articles (1, 2) that are useful and helpful material to read related to the throughput of Netlogon secure channel traffic. If you still think there is more that can be done to describe this setting and issues surrounding authentication, feel free to drop me a line or leave a comment.

P.S. If you are hitting this issue, you should seriously audit your network and applications as to why they are not using Kerberos for authentication. There are a few common pitfals as to why NTLM ends up being the auth protocol, but that might go in another post of its own.