Magento & Zend Server Benchmarks

Magento & Zend Server Benchmarks

Magento is a PHP/Zend application which intensively uses the CPU. Since version 1.1.6, each new version includes some mechanisms aimed to improve the performances. The goal is to use fewer resources for a given e-shop, which mainly means less CPU, in order to host more users with the same hardware.

One key to achieve better performances is how to optimize PHP pages generation and service. “LAMP” servers are well known and usually run Apache server with mod-php, eventually in fast_cgi mod.

Zend, the PHP Company, made a specific server (Zend Server), which includes a web application stack that (among other things) improves application performances through page caching and opcode reorganization & acceleration.

Apache and Zend Server is an alternative to the usual Apache and mod-php to run Magento, the goal of theses studies & tests is to qualify and estimate the performances added by the use of this software.

Many thanks to Yoav Kutner (Varien’s CTO) for providing us with prefilled catalogs for 1.2 and 1.3 version of Magento. Thanks goes as well to Zend labs for providing help in configuration and tweaking of the Zend Server as well as explaining the in depth mechanism of the solution.

Methods & Tools Used

The benchmarks were done using siege (2.66-2), with different numbers of simultaneous threads (5, 10, 20 and 40). Each thread opens a connection to the web server, request a page, and start again as soon as the page is fetched.

Tests were run over 5 minutes each; average pages retrieved was counted for this benchmark.

Two kinds of tests were done: the first one is the simplest; we only load the main page in loop, as fast as possible.

The second test is based on logs produced by a visit, loading all the data the browser collected (including css, js, images …). As loading a single page would usually load multiple elements, the number displayed is much higher than the number of pages that would be actually seen by visitors.

(18 pages viewed by the user, for 91 items downloaded, a ratio around 5)

All results were obtained on the same hardware and operating system, for testing purposes, no reverse proxy was active during the benchmarks but APC code cache was running. This was a “standard”, default environment with no special performance tweaks installed.

Hardware:

2 processors: Quad-Core AMD Opteron(tm) Processor 2376 (2.3GHz), 8GB Ram

Software:

Operating system: Linux (2.6.27.10-grsec) on a Debian (lenny) 
Web Server: Apache2 (2.2.9-10+lenny2) 
PHP (mod-php): mod-php5 (5.2.6.dfsg.1-1+lenny3) / php-apc (3.0.19-2) 
PHP (Zend Server): zend-pe (1.0.0-1+b47) / mod-php5-zend-pe (5.2.9+b75)

Repositories:

deb http://ftp.fr.debian.org/debian/ lenny main 
deb http://security.debian.org/ lenny/updates main 
deb http://volatile.debian.org/debian-volatile lenny/volatile main 
deb http://repos.zend.com/deb/pe pe non-free

Magento Version Benchmarks

Those tests were realized on an 80 000 (later called 80 k) products catalog.

Graphs represent the average number of requests successfully loaded from the server per second during the 5 minutes test.

Loading the homepage

Magento 1.3 is much faster than 1.2 showing the main page. Flat Catalogs do not help much displaying this homepage, and seems to even slow down a little under heavy loads. The performance between 1.2 and 1.3 is doubled for low charges and is still 33% higher under heavy load.

For a standard value, under a “standard” load, we can consider that +40% is a reasonable value when running a 1.3 version instead of a 1.2, at least for the homepage.

Full visit cycle

The graph shows a slightly smaller increase with 1.3 compared to 1.2 and the flat catalog mechanism give an overall benefit which greatly increases the performances.

For the records, we can reasonably choose to keep theses values in mind:

Version 1.2 -> 1.3 : +66% 
Version 1.3 -> 1.3+Flat Catalog : +20%

Apache + mod-php VS Apache + Zend Server

Zend Server (sometimes refered as ZS later on) comes with several built-in technologies for enhancing application performance: 

• Zend Optimizer+ performs byte-code optimization and caching. This speeds up PHP applications by eliminating the process of reading scripts from disk and compiling them. Zend Optimizer+ runs automatically, and installing your application on top of Zend Server (ZS) is all you need to do in order to enjoy its benefits. During the test with ZS, APC Code cache is deactivated as Zend optimizer+ is doing the same job.
• Zend Page Cache allows caching of complete PHP pages. Page Caching greatly improves the performance of web applications while maintaining dynamic capabilities through an elaborate system of caching rules that could be based on request parameters and user session data. Page Caching also has the benefit of not requiring any code changes, and can be set up from the Zend Server UI. Only the “pro” version contains this precise piece of software which definitely makes a difference as we will see in a minute.
• Zend Data Cache is a set of API functions enabling a developer to store and manage data items (PHP strings, arrays and other data) and even output elements in either disk-based cache or shared memory cache. Zend Data Cache allows for precision-guided caching when Page Caching is not an option. The provided API is easy-to-use on existing code, and in many cases a developer can skip existing code sections by simply wrapping them with caching APIs. This precise piece of software would benefit from a little remastering of the code by Varien to really achieve a full support of this functionality. If done, we can imagine selectively flushing the cache when changing some pages on the servers and not destroying the whole Magento cache thus doing a “cold cache start” after a new functionality is put online.

Note: Zend Optimizer+ and Zend Data Cache are available in the free, community version of Zend Server, while Zend Page Cache requires a licensed Zend Server (full comparison of Zend Server and Zend Server Community Edition is located at http://www.zend.com/fr/products/server/editions).

Zend Server Configuration for Magento

In the benchmark, two caching rules have been defined:

The first rule caches the JavaScript files which are dynamically merged into one request by Magento. 
This simple rule results in a very substantial improvement to response times.

The second rule caches all web pages accessed by users who have nothing in their shopping carts or history (when the shopping cart is not empty or when the user history is saved, there is no point in page caching). As you can see in the screen capture below, this is accomplished by looking at $_SESSION variables and by splitting according to $_SERVER[’REQUEST_URI’] in addition to the QUERY_STRING.

Finally, Zend Optimizer+ has been enabled for optimizing and caching the PHP byte-code.

As you can see, Zend Page Cache seems a very powerful feature. It needs fine tuning and better configuration, but let’s see results :

Homepage / 80 000 products catalog

Load on main page doesn’t prove very constructive, although, on heavy load (40 concurrent requests), Zend Server (with or without page cage) is slightly better than basic mod-php, but almost no real improvements on this test. A 5% or less win is not to be taken seriously as it is the error margin of the tests.

Full visit / 80k products

Full visit graphic speaks for itself, whatever the load, Zend Server with Page Cache and Flat Catalog make a big difference.

Comparing a 1.3 with flat catalog with a the same configuration but with a Zend Community installed instead of just APC, our server yield up to 15% more performances, just using a free edition of ZS.

If we use a full Zend server (the licensed one including the page cache) on this 1.3 flat catalog, our server goes up to 30% more performances!

Mod PHP + APC -> ZS: +15% 
Mod PHP + APC -> ZS licensed edition: +30%

Response time

Homepage / 80k products

This graph show response time on the homepage measured during a load. No match on this test, under a good load, ZS, community or licensed edition, gives up a 35% boost in the load time, good to take.

Full visit / 80k products

Page Cache provides a better response time over a full visit of the site, even under very heavy load. As the difference remains thin, I would conclude it to be identical in that condition. However, the benefits of the page cache under a standard load remains a must have. Once again, a 35% win is to be considered as a reference value under a good load whether you are using a community or licensed version.

Under medium or light load, the page cache is giving a lot more power than the community edition can do. This strange result is probably mainly due to the page cache mechanism which prove to be not that efficient under heavy load because the system is using its resource in a different way. 
We didn’t had time to check if it was a False cache sharing effect on L2 or L3 processor cache or a Linux or ZS issue but the test has been runned several times yielding the same results.

So keep in mind that the benefits of the ZS Page cache mechanism is going to be less and less active as the server load gets high. Under a usual load, a reasonable load lets say, the Page Cache algorithm gives a nice boost to the loading time, driving it down for ~40%.

Additional technical details about the tests

Sample data

All tests were run with generated sample data (provided by Varien)

Description of the tests

Main page test

Each concurrent thread loads the main page, without storing the cookies to simulate simultaneous users.

Full visit test

We have a list requests based on a visit done with a real browser. Each concurrent thread loads each item one after another as fast as possible, and restarts after deleting all its cookies. This simulates simultaneous users, except that simultaneous.

List of requested pages for the 80k database (with POST information when it applies) can be found here.

Things to test in the next benchmark campaign

Mage Compiler

When running all those tests, 1.3.1.1 version including the Mage optimizer was about to be released and an immediate 1.3.1.2 version followed and we didn’t had time to include some tests around this precise point. Let’s just consider the fact that this mechanism is mainly made to lower I/O jobs by “compiling” the libraries files all in one or two includes only. If your servers were already using mainly their RAM to work, the performances increase will not be that valuable. But if your servers where low on RAM or using mainly their disks, you should feel confident with using this option which can do great good to your performances!

Nginx

We didn’t had the required time to consolidate the results involving Nginx + PHP tests. This solution ranked almost every time between Magento 1.3 + FC and Magento 1.3 + Zend server (without Page cache). So this combination yield average performances, not as sharp as a Magento 1.3 + Zend server or any more advanced combination.

More “custom” visits

We have used a scenario involving each time a research in the search bar of the website and some “standard” behaviors. A more precise test can be run replaying some real traffic pumped up from apache log on a real site, the customer’s behaviors being more realistic even if our scenario was as logical as possible.

Conclusion

First, let me say that no electrons were armed or injured during these benchmarks. Perhaps one or two CTO were put under constraint and continuous coffee perfusions but all of this was intend for the greater good of E-commerce!

If you wish a more professional conclusion, let’s say that using ZS community edition will only do you good and really can replace APC.

If using the licensed version, the Page cache is very efficient (under reasonable load of the servers) and can help using ~25% less machines to achieve similar hosting capacities. If your servers are billed in a “managed hosting” way, having one or two less servers billed per month can make a good difference in the budget.

About Zend Server : this software has many other great functionalities and this paper is only consider the “performances” issues, just pay a visit to Zend Website for a far more complete overview of the product.

Last but not least, creating a server with all optimizations and best practices, based on an a dual AMD 2376 with 8 Go of RAM and using ZS and page cache, you can try to reach up to ~40 000 unique visitors a day corresponding to ~2 500 Magento simultaneous session at maximum load. (if database is separated on a different server and you activate a reverse proxy like Squid or Varnish above your front web servers) 
This estimation is deeply linked to the website complexity and user standard behavior but these figures are given for a “standard” site and use, you can usually also expect a ~2 seconds loading time on the homepage.

If we had to sum up a quick dirty, average table of the performances, without any details, we would give that: