netsekure rng

I recently spent some time looking at a very interesting issue, so I wanted to share it and save people some time troubleshooting similar issues. The problem was that a site requiring TLS client authentication was loading very slowly – about 30 seconds page loading time for the index page, which to say the least on today’s fast networks is incredibly slow. The setup was as follows:

Client Browser <—> ISA <—> Web Server

Naturally the first step was to look at what was happening on the wire. Firing off a sniffer and capturing the traffic, the strange thing was that there were multiple Change Cipher Spec (CCS) TLS messages. In general, when you negotiate a SSL/TLS session, you do Change Cipher Spec once and from there on the traffic is encrypted and flows between the client and the server. After spending some time looking at different traces one thing came out as a pattern – the number of encrypted CCS messages was roughly equal to the number of HTTP requests made from the client minus the number of TCP/IP connections used by the browser. So I fired up the debugger to help us confirm that this was indeed the case. Debugging the Microsoft SSL/TLS implementation is a bit tricky, since the handshake processing happens in the lsass.exe process (which hosts all security protocol providers) and the actual encryption/decryption of data happens in the process using TLS (the browser in this case). Some debugger tricks and a few sweat spots later, it was confirmed that for each HTTP request the server will send a HelloRequest message back to the client, which is the protocol message used by servers to force clients to reauthenticate. It was very strange that for each resource the brower fetches from the server a new renegotiation is required, since this is not the normal mode of operation.

What’s next in the network path after the browser – the ISA server. After digging around all the settings, what picked the attention was the setting below:

In the case of the slow config, the checkbox was unchecked and the timeout value was greyed out, though still showing 300. What the admin assumed is that 300 is the default value, so the unchecked box didn’t matter much. Well, what it really means is that the ISA server will *not* cache the client certificate at all, which in turn causes the client to authenticate for each request it sends to the ISA server. Needless to say, once the checkbox was checked, there were no extreneous CCS messages exchanged and the page was loading much much faster.

I never suspected that such an innocent looking checkbox can cause so much trouble : ).

One of the hard things to troubleshoot in Windows domains is NTLM authentication and the interesting MaxConcurrentAPI setting. When user account to be authenticated on the server does not belong to the local account database, the server must forward the authentication to a domain controller. It does so over the Netlogon secure channel. As such, it is governed by the MaxConcurrentApi setting as to how many outgoing authentication requests are allowed at a time. When you have a loaded server, this can create a problem, an example of which is described in an ISA server configuration article. Finding the root cause is very laborious process, so we’ve added performance counters to Netlogon to allow troubleshooting this issue. For Windows 2003, a hotfix is required, but for Windows 2008 and later it is present in the core install.
I was thinking of writing a very thorough guide on how authentication works in this type of scenario, but found out that the ISA documentation team has beaten me to it. Their description is targeted to ISA server, but you can replace it with any other application server doing user authentication using NTLM.

There are also another two articles (1, 2) that are useful and helpful material to read related to the throughput of Netlogon secure channel traffic. If you still think there is more that can be done to describe this setting and issues surrounding authentication, feel free to drop me a line or leave a comment.

P.S. If you are hitting this issue, you should seriously audit your network and applications as to why they are not using Kerberos for authentication. There are a few common pitfals as to why NTLM ends up being the auth protocol, but that might go in another post of its own.

After I upgraded netsekure.org to be a blog instead of the old page that I had to manually craft all the HTML for, I looked at the access log of apache to confirm it all works fine. There were some interesting requests to the site and I was curious where the traffic comes from and I found a few sites that provide you with IP-to-location mapping. Interestingly enough for some IPs, the results were quite different between different providers, so I checked each IP on multiple data providers.

Being the command line geek that I am, I didn’t want to go to multiple web sites for each IP I was interested, so I wrote a small python script to query them and display the results on the console. No more browser with multiple tabs and wasting precious milliseconds doing it manually. Feel free to grab the script and let me know if it works/doesn’t work for you. I’ve split the providers in two, since some of them have daily limit on how many queries are possible (dmw, infosniper) and the rest are unlimited, so the unlimited are always used and the rest (blogama, hostip, geoplugin) are optional.

A friend of mine passed along a very interesting link to Programmer Competency Matrix which I found very interesting. It is a very useful evaluation tool to figure out where one sits in the overall skillset a programmer/developer can have.

It is interesting to see how the attack methodologies change with time. As security protocols become more mature and secure, people have moved up the stack. Case in point is the Defating SSL presentation which outlines how an attacker can take advantage of MiTM techniques to bypass SSL. The attack is against the end-to-end scenario, not the protocol itself, even though the protocols themselves sometimes have (implementation) issues too. As time has proven already, the weakest link in most software systems is the user when it comes to security and we all know that the system is as secure as its weakest link. It is/will be interesting to watch the evolution of the Internet and tools in the future to account for the weakest link. Relying on just presentation is not enough as proved by the above presentation. We need to solve the problem in an end-to-end manner while minimizing the interaction with the user. By reducing user input, we reduce the attack surface against the weakest link and achieving a more robust system.

Putting up my screenrc file that some people have expressed interest in. It is nice and convenient as it sets up a few windows to start with and a “taskbar” on the bottom of the screen, so you can switch easily with Ctrl+Arrow and visually see which window you are at.

If you have any suggestions for improvement, let me know.

Well, after being few years late to the blogging world, I finally gave in and stopped caring about having ultimate control over the site I have. That is how RNG @ netsekure is starting now, as an attempt to be a collection of my random thinking put into writing.