64 vs 32 rather then SSD vs HDD

Introduction
Some time ago I promised to present results of the performance test of an Apache serving content from HDD and SSD. The tests confirmed that SSD gives Apache serving static content significant performance boost, however the other remedy for I/O problem has been found.

It's not so easy to test a system from inside. It also applies to the internet. The "image" you see is not exactly what your customers might see. There are some tools helping measure the "real image" of your website (i.e. KeyNote), but to test a difference of performance between two server visible as one entitle from outside (i.e. behind load-balancer) such test are not ideal. (Of course it is possible to prepare a A/B test when A means serving images from one server, B from another one). The web search hadn't brought any technique and parameters useful in such test (of course it might means that the search  was not very good). Therefore, the serving speed [byte/microsecond] parameter were define as ratio of file size [bytes] and response time [microseconds], respectively %B and %D in extended Apache log.


SSD vs HDD
In the first test the SSD box (Sun X4150 with 8 Xeon E5345 @ 2.33GHz cores and 8 GB of memory) was try out against HP ProLiant D380 G5 with the same amount of memory and 4 similar processors/cores (Xeon 5148 @ 2.33GHz). The test were split into two phase. Results for both ones are collected in Table 1. The first day both boxes serve the same amount of requests. The SSD machine sent files two time faster. Next day the weight of connection to machine with SSD was increase on the load balancer, but it doesn't change the results. The SSD box was much faster again. The data presented in Table 1 ensure that both boxes worked with similar set of files (similar size and number of requests).

Table 1. Comparison of server with SSD (SUN X4150) and HDD (HP D380 G5) drives.

SSD machine		HDD machine
Day I
Average speed 24.27563 Average time 13908.68485 Average size 7683.71685 Served files 14983388		Average speed 12.23371 Average time 24674.27763 Average size 7702.66678 Served files 14987023
Day II
Average speed 23.76447 Average time 13107.95822 Average size 7839.91839 Served files 21522835		Average speed 11.10400 Average time 26001.08215 Average size 7829.47865 Served files 10758116

During next phase of research the SSD machine was tested again wider set of servers with HDD servers (machines with Apache using different MPM (prefork and worker), with 8 or 16 GB of memory and with slower and faster processors). SSD machine was faster than any of the server with a classical hard drive. Adding memory as well as changing MPM didn't change the difference in the performance. On the other hand, the machine with more and newer processor sent files much faster than the old ones, however still 50% slower than the SSD test kit.


Table 1. Comparison of Apache performance on machine with a SSD against different machine using a HDD. SUN SSD means the test boxs, OLD HP a box with 4 L5430@2.33Ghz core and NEW HP a server with 8 E5345 cores.

		OLD HP		NEW HP	SUN SSD
speed	16GB (prefork)	8GB (prefork)	8GB (worker)	16GB (worker)	8GB, (worker)
Test 1	9.73669	10.99227	10.82629	16.81683	24.18491
Test 2	9.56062	10.57697	10.51381	16.32999	24.41852
Test 3	8.69836	9.83142	9.65313	16.16164	24.39481
Test 4	9.03057	10.12731	9.98168	15.60127	24.16711

32bit vs 64bit
In the mean time another two things occurred. On the one hand, another test machine, this time with 64bit OS, was built.The first results showed significant decrease of the load and the number of I/O (read) operation. Figure 1 and 2 are good indicator how drastic it was change, even if they were prepared on latter production boxes. Moreover, a change in some of the production server settings exhibited that the machine do not properly caching content served by Apache.




Figure 1. Load on a production system before and after changing OS from 32 to 64 flavour.



Figure 2. I/O operation on a production system before and after changing OS from 32 to 64 flavour.



Further investigation of both phenomena showed that the I/O problem mentioned in the first part of article was cause by inefficient caching, what was cause by memory wasting on a server with 32bit OS. By default Apache is using the sendfile() function rather than the combination of read() and write() functions to transfer files from a storage to a network interface (so called zero copy approach). It speeds up a data transfer by avoiding switching context from kernel to user space, but it also means that Apache cache is limited because on 32-bit OS it cannot allocate more than 3GB of memory.

To confirm that a 32bit OS was causing of performance issues another test on live system was conduct. When comparing two identical servers with 16GB of memory and 8 E5345 Xeon cores, one running a 32bit and second 64bit OS, following observation was made:

• The load decreases from around 2 to 0.5 (Figure 1).
• The read from disk decreases from 2.5 to 0.6 [megabytes/second]  (Figure 2).
• The memory usages increase from 2.5-3 to 16 (full memory) GB.
• The average Apache speed increase from around 12 to 30 [bytes/microsecond] (Figure 3).

Figure 3. Comparison of the Apache speed (speed of sending static content) on 32 and 64 bit OS.    
Summary
During the tests the SSD showed its superiority to the classical HDD storages. However in the case of serving a static content from a web server using a distributed memory caching technique might be faster, more scalable  and even cheaper solution.

Another conclusion from above research, probably less and less important as there are less and less of x86 32bit servers, that 64 bit OS might be really faster, especially in the case of system with big amount of memory.