Whenever I speak or write about WCF, I never forget to say how configurable it is and how much its long exception messages help when you are doing something wrong. However if you try to really push it to get really great performance under heavy request load, you will bump to a number of issues that need special care. In this post I will try to enumerate each one of them and hopefully at the end you will know how to get really fast activation of a service when hit by hundreds of concurrent requests.
The scenario
You have 300 clients hitting the same WCF service concurrently (or just one client that hits the service using 300 separate but concurrent threads). Your service runs using net.tcp binding and is hosted either under a standard windows process or IIS. We choose to use net.tcp because it is the recommended binding for the communication of two processes running under .net and on different machines. All issues bellow apply also to net.pipes, which can be used only if the two processes, client and server, rely on the same machine.
Warning!
The goal of this article is to guide through the process of achieving super fast service activation and not necessarily make your application run faster! This is a more complicated process which requires many other issues to be considered. Also I will not talk about implications of opening and running many concurrent threads.
Step 1 - Behavior configuration and throttling (server side only)
You need to start by configuring your service behavior in order to allow the many concurrent sessions, instances and calls. Here is one such configuration:
<behavior name="MyServiceBehavior">
<serviceThrottling maxConcurrentCalls="350" maxConcurrentSessions="350" maxConcurrentInstances="350" />
</behavior>
You can read more about service throttling
here and
here. In brief, if you are using single instance service (use of this attribute on your service class [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]) you do not care about maxConcurrentInstances, since you are using only one anyway. If you are using session-less service you do not care about maxConcurrentSessions, since you are not using sessions anyway. For sure you care about maxConcurrentCalls. Configure these values with numbers greater than the number of concurrent calls you are expecting.
Step 2 - Binding configuration and max connections (server and client side)
When you use net.tcp binding you need to configure the maximum connections as follows:
<netTcpBinding>
<binding name="Default" maxConnections="300" listenBacklog="300">
</netTcpBinding>
What do these settings mean? (pasted from MSDN documentation) maxConnections setting controls the maximum number of connections to be pooled for subsequent reuse on the client and the maximum number of connections allowed to be pending dispatch on the server. listenBacklog setting controls the maximum number of queued connection requests that can be pending.
Step 3 - Max connections, port sharing and IIS (server side only - Warning!!! Read even if you think that you are not using port sharing)
Although you will find dozens of articles, blog posts and samples regarding previous steps, you will not find easily information on how port sharing can affect your service. What is port sharing? .NET has a windows service called 'Net.Tcp Port Sharing Service' which allows sharing a single port among many processes. This way you can have two processes running and listening to the same port. Port sharing service will filter incoming requests and forward them to the correct service instance depending on the full url each one listens to. If you need this feature, you can enable it using the portSharingEnabled setting on your binding configuration.
However, even if you don't need it, you have to bear in mind that if you are hosting net.tcp services under IIS you are using port sharing even if you do not configure it! This is the only way IIS can host all your net.tcp services under the same port (by default this is 808).
So how does port sharing affects the performance of our service? Port sharing service has it's own configuration file which overrides some of the settings we mentioned above. To find this file go to your Local Services snap-in from your control panel and look for the "Net.Tcp Port Sharing Service". Right click on it, click Properties and look at the "Path to executable". Open that path using Windows Explorer and look for the SMSvcHost.exe.config. This is the file we are looking for. Don't mind if you find that this path of the file does not match the .NET Framework version you are using. This service is common for all framework versions since you may want to share one port in processes using different versions. Open this file and edit it. Here is one recommended configuration:
<net.tcp listenBacklog="300" maxPendingConnections="300" maxPendingAccepts="300"/>
Do not forget to restart the service.
Step 4 - Thread pool configuration
WCF utilizes thread pool in order to create and use threads for handling each incoming request. If you expect to receive 300 concurrent requests then the server will have to create 300 concurrent threads to handle them. Without proper configuration of the thread pool you will not get the responsiveness you expect. This is because thread pool creates only two threads per second. If you need 300 threads, this will take 150 seconds! This is not what we would call high performance! Right? You need to use ThreadPool.SetMinThreads and ThreadPool.SetMaxThreads to configure the thread pool. You can imagine what these methods are for and you can read more
here and
here. Both methods accept two parameters. First one is about worker threads and second one is about completion port threads. We care about the latter. Here is how you can set the completion port threads (second parameter) without affecting the worker threads (first parameter):
// get current values
int maxWorker, minWorker, maxCompletion, minCompletion;
ThreadPool.GetMaxThreads(out maxWorker, out maxCompletion);
ThreadPool.GetMinThreads(out minWorker, out minCompletion);
// set new values
bool resultMax = ThreadPool.SetMaxThreads(maxWorker, 500);
bool resultMin = ThreadPool.SetMinThreads(minWorker, 400);
In this code I set minimum threads to 400, although I expect 300 connections since thread pool may be used by other parts of my server and only by WCF.
Step 5 - ThreadPool bug
You think you are done? Think again! If you are using .NET 3.5 (with or without service pack 1) then you will notice that setting minimum number of threads simply does not work! This is because of a known bug of the thread pool described
here. In order to get around this you need to contact
Microsoft Support and get this fix. Unfortunately it is not available for direct download but most probably you will get this free of charge. After you install this fix you will notice that immediately after your client(s) create the 300 concurrent requests your server process will allocate 300 threads (w3wc.exe if you are hosting under IIS). Excellent! But bear with me for just one more second. We are almost but not exactly there yet!
Step 6 - Yet another bug?
Let your server at rest for 15 seconds. You will notice that the 300 threads are released and if you client(s) creates another 300 concurrent requests you are back to where you started from! One thread will be created every half second and the performance will drop! Now I'm not sure if this is a bug of the thread pool or WCF. Not sure exactly why or what, since I haven't found this one to be officially recorded in Microsoft Connect. And this one affects .NET 4.0 also! So what now? I tried to apply the solution mentioned in this
blog post and it worked for me. Briefly the solution proposed is to use the ThreadPool.RegisterWaitForSingleObject method to queue some dummy job there in intervals and keep the threads alive. Yes it's ugly, I know. But this is the only one I got so far!
We are done! Hopefully now you should have achieved to have really fast activation of your services on server side. Don't forget that it is your responsibility to harness this power after your services are activated.
Permalink
| Leave a comment »