Benchmarking Zend Framework 1.8.4

About 2 months ago I sat in on a PHP performance session at work taught by non other than Rasmus Lerdof.  Aside from creating PHP he had some really neat stuff to talk about concerning performance.  Most of the presentation came from the php.net site, talks.php.net. Take a look at the talks when you have some time.

The question
Anywho…that’s when my benchmarking and optimizing curiosity kicked in.  It wasn’t until yet another talk at work concerning the benefits of an MVC framework (Symfony) that I started to think about Zend Framework and its performance out of the box (without caching).   So I sat down today and started to benchmark the framework with a simple question in mind, “How does a simple ZF application compare to an application not using it?”

The applications
To answer the question I created 3 web projects. The first web project was a static-HTML only application which displayed “hello world” on the screen, very simple.  (For those who just want to see the ab results click here as well as the Xdebug cachegrind output file.)

The second web project was a single PHP file with the below PHP code.

Listing 1.1 – Simple Hello world PHP
<?php echo "hello world"; ?>

The above 2 projects acted as my baselines. it would allow me to determine the overhead ZF introduced into the application.

Finally, the third web project was created using ZF’s Zend Tool and executing the below command.

Listing 1.2 – Zf Project Creation.
> zf create project helloworld_zf

Finally, I replaced the default index.phtml HTML with the simple PHP code shown in Listing 1.1. 

Yes some may say that these examples are far too simple, but I wanted to test the Framework’s process to load and what better way to do this than with a very simple example.  I also based it off the talk that was given comparing Symfony.

The Hardware & ab command
Here is the set-up.
1.  Windows machine running Apache 2.2
2.  PHP 5.2.9
3.  Zend Framework 1.8.4
4.  Pentium 4 3.20 GHz
5.  1Gbs of RAM

I used the Apache Benchmark, ‘ab’, tool which comes included with all Apache installation and ran the below command 3 times.

Listing 1.3
> ab -n 1000 -c 5 http://localhost/index.[html|php]

The results, for me, were very surprising. I have included all the ab outputs.

Observations
HTML ONLY – Reading 1
Concurrency Level:      5
Time taken for tests:   2.344 seconds
Complete requests:     1000
Failed requests:        0
Write errors:           0
Total transferred:      352000 bytes
HTML transferred:       83000 bytes
Requests per second:  426.67 [#/sec] (mean)
Time per request:       11.719 [ms] (mean)
Time per request:       2.344 [ms] (mean, across all concurrent requests)
Transfer rate:          146.67 [Kbytes/sec] received

HTML ONLY – Reading 2
Concurrency Level:      5
Time taken for tests:   2.375 seconds
Complete requests:     1000
Failed requests:        0
Write errors:           0
Total transferred:      352000 bytes
HTML transferred:       83000 bytes
Requests per second:  421.05 [#/sec] (mean)
Time per request:       11.875 [ms] (mean)
Time per request:       2.375 [ms] (mean, across all concurrent requests)
Transfer rate:          144.74 [Kbytes/sec] received

HTML ONLY – Reading 3
Concurrency Level:      5
Time taken for tests:   2.344 seconds
Complete requests:     1000
Failed requests:        0
Write errors:           0
Total transferred:      352000 bytes
HTML transferred:       83000 bytes
Requests per second:  426.67 [#/sec] (mean)
Time per request:       11.719 [ms] (mean)
Time per request:       2.344 [ms] (mean, across all concurrent requests)
Transfer rate:          146.67 [Kbytes/sec] received

Observations:
1.  Roughly 0.0117 seconds per request.
2.  Up to 424.79 requests satisfied per second.

Basic PHP – Reading 1
Concurrency Level:      5
Time taken for tests:   2.609 seconds
Complete requests:     1000
Failed requests:        0
Write errors:           0
Total transferred:      267000 bytes
HTML transferred:       81000 bytes
Requests per second:  383.23 [#/sec] (mean)
Time per request:       13.047 [ms] (mean)
Time per request:       2.609 [ms] (mean, across all concurrent requests)
Transfer rate:          99.93 [Kbytes/sec] received

Basic PHP – Reading 2
Concurrency Level:      5
Time taken for tests:   2.609 seconds
Complete requests:     1000
Failed requests:        0
Write errors:           0
Total transferred:      267000 bytes
HTML transferred:       81000 bytes
Requests per second:  383.23 [#/sec] (mean)
Time per request:       13.047 [ms] (mean)
Time per request:       2.609 [ms] (mean, across all concurrent requests)
Transfer rate:          99.93 [Kbytes/sec] received

Basic PHP – Reading 3
Concurrency Level: 5
Time taken for tests: 2.625 seconds
Complete requests: 1000
Failed requests:        0
Write errors:           0
Total transferred:      267000 bytes
HTML transferred:       81000 bytes
Requests per second:    380.95 [#/sec] (mean)
Time per request:       13.125 [ms] (mean)
Time per request:       2.625 [ms] (mean, across all concurrent requests)
Transfer rate:          99.33 [Kbytes/sec] received

Observations:
1. There seems to be a 100 request (roughly) drop when supporting PHP.
2. Requests per second satisfied =  382.47 (a drop of 42.32 requests or 9% drop)
3. Time per request = 13.073 or 0.0130 seconds per request (Increase of 11%)

Out of the box ZF 1.8.4 – Reading 1
Concurrency Level:      5
Time taken for tests:   68.297 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      197000 bytes
HTML transferred:       11000 bytes
Requests per second:    14.64 [#/sec] (mean)
Time per request:       341.484 [ms] (mean)
Time per request:       68.297 [ms] (mean, across a
Transfer rate:          2.82 [Kbytes/sec] received

Out of the box ZF 1.8.4 – Reading 2
Concurrency Level:      5
Time taken for tests:   70.281 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      197000 bytes
HTML transferred:       11000 bytes
Requests per second:    14.23 [#/sec] (mean)
Time per request:       351.406 [ms] (mean)
Time per request:       70.281 [ms] (mean, across all c
Transfer rate:          2.74 [Kbytes/sec] received

Out of the box ZF 1.8.4 – Reading 3
Concurrency Level:      5
Time taken for tests:   69.656 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      197000 bytes
HTML transferred:       11000 bytes
Requests per second:    14.36 [#/sec] (mean)
Time per request:       348.281 [ms] (mean)
Time per request:       69.656 [ms] (mean, across all c
Transfer rate:          2.76 [Kbytes/sec] received

Observations:
Framework not only has a slower response time but also satisfies less requests per second. Am I doing something wrong???? This cant be correct.

1. Request per second = 15.24 a change of 367.23 requests per second or a drop of 96% between trivial PHP and a ZF power app.
2. Time per request (time taken to satisfy 1 request) = 328.49 ms or 0.328 seconds a change of .315 seconds or an increase of 23%

Conclusion
Even though my benchmarking techniques might be flawed, taking a base line reading with no HTML as well as a very simple PHP script web project allowed me to at least compare. 

Based on these figures (if correct) Zend Framework has a HUGE overheard.  But this should not discourage any would be ZF coder because there are caching techniques which were not used such as using APC and the benefits of a good network admin 🙂 

Also the benefits of using any framework far outweigh the drawbacks in my personal opinion.  Where else can you get code which has been tested by the best and packed to contain methods for many of our typical coding requests.

Armando Padilla.