Search This Blog

Thursday, January 28, 2010

64 vs 32 rather then SSD vs HDD

Some time ago I promised to present results of the performance test of an Apache serving content from HDD and SSD. The tests confirmed that SSD gives Apache serving static content significant performance boost, however the other remedy for I/O problem has been found.

It's not so easy to test a system from inside. It also applies to the internet. The "image" you see is not exactly what your customers might see. There are some tools helping measure the "real image" of your website (i.e. KeyNote), but to test a difference of performance between two server visible as one entitle from outside (i.e. behind load-balancer) such test are not ideal. (Of course it is possible to prepare a A/B test when A means serving images from one server, B from another one). The web search hadn't brought any technique and parameters useful in such test (of course it might means that the search  was not very good). Therefore, the serving speed [byte/microsecond] parameter were define as ratio of file size [bytes] and response time [microseconds], respectively %B and %D in extended Apache log.

In the first test the SSD box (Sun X4150 with 8 Xeon E5345 @ 2.33GHz cores and 8 GB of memory) was try out against HP ProLiant D380 G5 with the same amount of memory and 4 similar processors/cores (Xeon 5148 @ 2.33GHz). The test were split into two phase. Results for both ones are collected in Table 1. The first day both boxes serve the same amount of requests. The SSD machine sent files two time faster. Next day the weight of connection to machine with SSD was increase on the load balancer, but it doesn't change the results. The SSD box was much faster again. The data presented in Table 1 ensure that both boxes worked with similar set of files (similar size and number of requests).

Table 1. Comparison of server with SSD (SUN X4150) and HDD (HP D380 G5) drives.
SSD machine HDD machine
Day I
Average speed 24.27563
Average time 13908.68485
Average size 7683.71685
Served files 14983388
Average speed 12.23371
Average time 24674.27763
Average size 7702.66678
Served files 14987023
Day II
Average speed 23.76447
Average time 13107.95822
Average size 7839.91839
Served files 21522835
Average speed 11.10400
Average time 26001.08215
Average size 7829.47865
Served files 10758116

During next phase of research the SSD machine was tested again wider set of servers with HDD servers (machines with Apache using different MPM (prefork and worker), with 8 or 16 GB of memory and with slower and faster processors). SSD machine was faster than any of the server with a classical hard drive. Adding memory as well as changing MPM didn't change the difference in the performance. On the other hand, the machine with more and newer processor sent files much faster than the old ones, however still 50% slower than the SSD test kit.

Table 1. Comparison of Apache performance on machine with a SSD against different machine using a HDD. SUN SSD means the test boxs, OLD HP a box with 4 L5430@2.33Ghz core and NEW HP a server with 8 E5345 cores.
speed 16GB (prefork) 8GB (prefork) 8GB (worker) 16GB (worker) 8GB, (worker)
Test 1 9.73669 10.99227 10.82629 16.81683 24.18491
Test 2 9.56062 10.57697 10.51381 16.32999 24.41852
Test 3 8.69836 9.83142 9.65313 16.16164 24.39481
Test 4 9.03057 10.12731 9.98168 15.60127 24.16711

32bit vs 64bit
In the mean time another two things occurred. On the one hand, another test machine, this time with 64bit OS, was built.The first results showed significant decrease of the load and the number of I/O (read) operation. Figure 1 and 2 are good indicator how drastic it was change, even if they were prepared on latter production boxes. Moreover, a change in some of the production server settings exhibited that the machine do not properly caching content served by Apache.

Figure 1. Load on a production system before and after changing OS from 32 to 64 flavour.

Figure 2. I/O operation on a production system before and after changing OS from 32 to 64 flavour.

Further investigation of both phenomena showed that the I/O problem mentioned in the first part of article was cause by inefficient caching, what was cause by memory wasting on a server with 32bit OS. By default Apache is using the sendfile() function rather than the combination of read() and write() functions to transfer files from a storage to a network interface (so called zero copy approach). It speeds up a data transfer by avoiding switching context from kernel to user space, but it also means that Apache cache is limited because on 32-bit OS it cannot allocate more than 3GB of memory.

To confirm that a 32bit OS was causing of performance issues another test on live system was conduct. When comparing two identical servers with 16GB of memory and 8 E5345 Xeon cores, one running a 32bit and second 64bit OS, following observation was made:
  • The load decreases from around 2 to 0.5 (Figure 1).
  • The read from disk decreases from 2.5 to 0.6 [megabytes/second]  (Figure 2).
  • The memory usages increase from 2.5-3 to 16 (full memory) GB.
  • The average Apache speed increase from around 12 to 30 [bytes/microsecond] (Figure 3).

Figure 3. Comparison of the Apache speed (speed of sending static content) on 32 and 64 bit OS. 

During the tests the SSD showed its superiority to the classical HDD storages. However in the case of serving a static content from a web server using a distributed memory caching technique might be faster, more scalable  and even cheaper solution.

Another conclusion from above research, probably less and less important as there are less and less of x86 32bit servers, that 64 bit OS might be really faster, especially in the case of system with big amount of memory. 

Thursday, January 07, 2010

Short (mostly shell) fomulas

If you hit this website looking for a Linux/shell/UNIX related advice you might want to visit my account. I post "short, magic", Linux formula there now. You know something like:

How to split lines in ?
where [ctrl-V][Enter] - means type Ctrl+V and next Enter (you should see s/\,/,^M/g)