Benchmarks
Here you will find some benchmarks that should help you to make a decision whether to use FAPWS3 or to stay with your current webserver.
Nicolas (Benchmark of python web servers) has performed a quite complete comparison between different WSGI servers. Toegether with Gevent, Fapws3 is amongst the top perfomer. This is nice because you have now the choice between libevent (used by gevent) and libev (used by Fapws3).
Methodology to use before sending me your results
Benchmarking is not a so easy task. We have to be very careful with what we are doing and what could be the conclusions.
Thus before sending me results, thanks to provide the environment in which you are running your tests. For exmaple:
- Machine:
- Operating system
- Benchmark tool
- Python version
- Libev version
If you run Fapws3 (webserver) and the client (ApacheBench, "ab" in short) on the same host; thanks to re-nice ab: "nice -n20". Wait enough time between each run to assure all "waiting" connections (displayed by netstat) will be effectively closed (+- 1 minute on my machine). Execute each tests 3 times and take one "ab" report which is representative.
Fapws3 versus gevent
Environment
Freebsd -8.0 stable python 2.6 libev-2.8 Fapws-0.4-dev gevent-0.12 greenlet-0.2 ApacheBench-2.3
Test context
For gevent:
- For gevent, I've used the wsgiserver.py script provided on the sample directory hosted by google.
-
To avoid performance delta due to the printouts, I've redirected them to the /dev/null:
python test_gevent.py > /dev/null - I've used the "hello world" sample provided (debug mode deasabled)
-
The webservers and ApacheBenchmark tool were running on the same host by using the
nice -n10 ab .... Every measure was made 3x with a sufficiant delay between each of them to assure open sockets will be closed before the next run (netstat -a).
For Fapws3:
For testing:
Results
Fapws3 is about 56% quicker than Gevent.
I've got issues to run more than 5 concurrent request with Gevent (strange)
| Fapws3 | Gevent | |
| 1 concurrent request | 6065#/sec | 3843#/sec |
| 5 concurrent requests | 6168#/sec | 4807#/sec |
| 10 concurrent requests | 6290#/sec | error |
| 15 concurrent requests | 6274#/sec | error |
| 20 concurrent requests | 6401#/sec | error |
Fapws3 versus Cherokee + uwsgi
Cherokee is a very interesting webserver having lot of features which can, via uwsgi, offers wsgi features. Uwsgi is a C implementation of wsgi
Environment
Cherokee Web Server 1.0.4 uWSGI 0.9.6.6 Python 2.6.6 Fapws3-0.9 (using epoll) libev-4.1 sqlite-3.7.2 Django-1.2.3 Pound-2.5 Linux machine1 2.6.35-24-generic #42-Ubuntu SMP Thu Dec 2 01:41:57 UTC 2010 i686 GNU/Linux
Tests context
The goal is to compare cherokee + uwsgi with Fapws3 as server for a django application using sqlite3 as backend. The application has been developped as an internal website, we are thus unable to present it. In few words, this is a central reprository allowing company's members to share relevant informations. The website has to be light (running on an existing application server), but yet efficient
All tests have been made on my development machine by using the ApacheBenchmark tool:
nice -n20 ab -n10000 -c10 ...
For this test I've compared the most simple django application (the famous "it work" of django), a static file and my real (called django app) django application.
For the setup of Cherokee, I've followed the tuto of Hank Wang
Results
We have made several tests with different level of concurrency:
| concurrency | Fapws3 | cherokee + uwsgi | |
|---|---|---|---|
| Simple Django "It work" | 10 | 915 | 788 |
| 100 | 959 | 515 | |
| 200 | crash because uwsgi harakiri timeout was set to 10 seconds. 657 (with harakiri set to 60 seconds) | 737 | |
| Simple html file | 10 | 3521 | 6857 |
| 100 | 5088 | 5850 | |
| 200 | 5107 | 6439 | |
| django app (query + commit) | 10 | 24 | 23 |
| django app (just query) | 10 | 26 | 23 |
Results with load balancing amongst 10 instances
| concurrency | Fapws3 + pound | cherokee + uwsgi | |
|---|---|---|---|
| django app (query + commit) | 10 | 24 | 23 |
| django app (just query) | 10 | 44 | 41 |
On performance side, both solutions sounds very close to each other. On memory foot print, obviously, the cherokee + uwsgi solution is heavier.
For the real Django aplication, the memory foot print is as follow:
- Cherokee: 92MB
- uwsgi: 21MB per instances
- Fapws3: 19MB per instances
- Pound: 15MB
