Sitecore Performance Testing on Multi-Tenant Configurations

August 27, 2014

I ran 12 tests on a Sitecore multi-tenant installation with 26 existing public websites using ASP.Net web forms. The goal of the tests was to determine what configuration settings had the greatest impact on performance. In this installation, the most performant client and server configuration was using Sitecore 6.6 with fewer IIS sites after configuration tweaks.

My hypothesis was separate IIS Sites for each website with shared application pools would be the most performant if the configuration were tailored to each site. You can read more about this in my post on customizing multi-tenant configurations.

I found that Sitecore actually uses more server resources as the number of IIS Sites increased. The navigation timings were lowest when the websites were split into 5 IIS Sites. Site-specific configuration did have an impact on performance, but not enough to offset the penalty of more IIS Sites.

Additionally, I found that the web.config setting debug="true" and compiling in release mode had very limited impact on performance (and in some cases a negative impact). Finally, upgrading Sitecore to 7.0 SP1 and the application pools to .Net v4.5 increased memory usage by 6%.

How I Tested

Selenium is an open source browser automation project that allows you to easily control a browser from code. The Navigation Timing API is an API supported by most browsers that returns detailed statistics on the responsiveness of your websites. I followed this article on extracting navigation timings from Chrome since the Selenium Internet Explorer driver returned unexpected results.

I setup a Sitecore 6.6 installation on my local machine (Intel i5 3.1 GHz processor with 10 GB RAM running Windows 7 Professional 64-bit) and connected it to a SQL Server 2008 database on our development server. 18 existing, public websites were tested in the same sequential order. In between tests, the configuration and setup was adjusted and then the computer was restarted. 10 total tests were performed in 4 stages:

The homepage was requested from a cold start
The homepage was re-requested
A page that queries and displays many children was requested
The same category page was re-requested

During each test, I recorded the memory usage and the navigation timings reported by the browser.

Test Details

8/7/2014 - Sitecore 6.6 - Shared configuration
5 Application Pools / 5 Sites
Debug Mode / debug="true"
Peak Memory Usage: 7.01 GB
Average Initial Response Start: 6.43 seconds
Average Primed Response Start:1.61 seconds
Average Primed Page Load: 2.89 seconds
8/7/2014 - Sitecore 6.6 - 5 configurations
5 Application Pools / 5 Sites
Debug Mode / debug="true"
*used IIS virtual folders to split the configuration into 5 separate App_Config/Include folders and reduce settings
Peak Memory Usage: 6.99 GB
Average Initial Response Start: 5.27 seconds
Average Primed Response Start:1.49 seconds
Average Primed Page Load: 2.54 seconds
8/8/2014 - Sitecore 6.6 - 5 configurations
5 Application Pools / 5 Sites
Release Mode / debug="false"
*lowered cache levels, re-built in release mode, and set debug="false"
Peak Memory Usage: 6.96 GB
Average Initial Response Start: 10.54 seconds
Average Primed Response Start:1.77 seconds
Average Primed Page Load: 2.85 seconds
8/8/2014 - Sitecore 6.6 - 5 configurations
5 application pools / 5 sites
Release Mode / debug="false"
*increased cache limits for large sites
Peak Memory Usage: 6.71 GB
Average Initial Response Start: 6.22 seconds
Average Primed Response Start:1.67 seconds
Average Primed Page Load: 2.74 seconds
8/8/2014 - Sitecore 6.6 - 5 configurations
5 application pools / 5 sites
Release Mode / debug="false"
*increased prefetch limits and items
Peak Memory Usage: 6.98 GB
Average Initial Response Start: 6.12 seconds
Average Primed Response Start:1.44 second
Average Primed Page Load: 2.42 seconds
8/8/2014 - Sitecore 6.6 - 16 configurations
5 application pools / 16 sites
Release Mode / debug="false"
*tried splitting sites into separate sites but sharing application pool. Each IIS site had a customized App_Data/Include folder.
Peak Memory Usage: 7.5 GB
Average Initial Response Start: 15.69 seconds
Average Primed Response Start:1.83 seconds
Average Primed Page Load: 2.84 seconds
8/8/2014 - Sitecore 6.6 - 5 configurations
5 application pools / 5 sites
Debug Mode / debug="false"
*Reset configuration back to 5 sites and built in Debug Mode
Peak Memory Usage: 6.89 GB
Average Initial Response Start: 10.18 seconds
Average Primed Response Start:1.69 seconds
Average Primed Page Load: 2.82 seconds
8/26/2014 - Sitecore 7.0 SP1 - 5 configurations
5 application pools / 5 sites
Debug Mode / debug="false"
*Upgraded to Sitecore 7.0, built projects with a target framework of 4.5
Peak Memory Usage: 7.31 GB
Average Initial Response Start: 7.28 seconds
Average Primed Response Start:1.48 seconds
Average Primed Page Load: 2.52 seconds
8/26/2014 - Sitecore 7.0 SP1 - 5 configurations
5 application pools / 5 sites
Release Mode / debug="false"
*Rebuilt in release mode
Peak Memory Usage: 7.37 GB
Average Initial Response Start: 9.74 seconds
Average Primed Response Start: 1.56 seconds
Average Primed Page Load: 2.72 seconds
8/26/2014 - Sitecore 7.0 SP1 - 16 configurations
5 application pools / 16 sites
Release Mode / debug="false"
*Split out websites into 16 IIS sites
Peak Memory Usage: 8.87 GB
Average Initial Response Start: 9.74 seconds
Average Primed Response Start: 1.56 seconds
Average Primed Page Load: 2.72 seconds
8/27/2014 - Sitecore 7.0 SP1 - 1 configuration
1 application pool / 1 site
Release Mode / debug="false"
*Combined websites into 1 IIS Site
Peak Memory Usage: 4.76 GB
Average Initial Response Start: 5.07 seconds
Average Primed Response Start: 1.72 seconds
Average Primed Page Load: 3.7 seconds
8/27/2014 - Sitecore 7.0 SP1 - 1 configuration 1 application pool / 1 site Debug Mode / debug="false"
*Re-built in Debug Mode
Peak Memory Usage: 4.57 GB
Average Initial Response Start: 5.15 seconds
Average Primed Response Start: 1.78 seconds
Average Primed Page Load: 3.24 seconds

Potential Issues

My development machine is not a server and is not treated like one. It is on a corporate network with security and remote monitoring software that I am not able or willing to disable for these tests. While I took time to restart my machine between tests and only run Visual Studio and Windows Task Manager, there may have been background processes, network congestion, or other issues that could contribute to increase memory usage or delays.

Some of the webpages make calls to third party services before the page load event or even before the response is started. Delays in these services could affect the navigation timings.

Furthermore, my configuration changed before I finished testing. The 8-12 tests were performed on a Sitecore 7.0 SP1 installation. My computer was also in active development and new software and updates were installed in between tests.

Finally, this isn't an accurate test of real traffic. The tests executed 104 webpage requests (and any associated requests) but they were done sequentially and each test request recorded the navigation timings up to Load Event End before moving onto the next webpage.

Takeaway

The best server performance (lowest memory and processor usage) was seen in a configuration that had 1 IIS Site for 26 websites. However, the navigation timings increased ~20% from the 5 IIS Site configuration even with only 1 request at a time.

The best client performance (lowest navigation timings) was seen in a configuration with 5 IIS Sites for 26 websites. However, the sever memory usage increased 37% from the 1 IIS Site configuration.

We will probably reduce the number of IIS sites on our production servers to lower overall memory usage. This will allow us to better handle spikes in traffic while still customizing the configurations.