Escape, Escape

I couldn't understand why following command was working on a local machine, but not through ssh (following form):

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += $5;} END {print tsum}'" 

I asked my workmate and he also had problems for some time, but finally he suggested that we needed to "escape" something. After some try we found that ssh don't like $ character so following command works.

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += \$5;} END {print tsum}'"

Stone Redskin: comparision of Apache2 performance on HDD and SSD

Introduction
Recently, I had a chance to test the performance of a static content web servers. The initial analysis showed that the most important issue were the speed of a disks, which started to have problems with handling I/O operations. The numbers of files were huge what means that hard drives were engaged in many random access operation.

The latest tests has shown that the new Solid State Disk (SSD) mass storage beat the classic Hard Drive Disk (HDD) in such circumstances (in most others too). So it was quite natural to prepare a set of test helping to measure the effect of switch from a HDD to a SSD storage on the Apache performance.

Methodology
It should be keep in mind, that I wasn't interesting in a general comparison of SSD vs HDD, but concentrated my tests on the Apache performance. The Grinder 3.2 software was used to simulate a load on the web server. The list of requested URL based on the real Apache logs taken from the one of box serving the static content. To eliminate the influence of caching, before each test the memory cache was cleaned using following command echo 3 > /proc/sys/vm/drop_caches (suggested on Linux-MM).

Hardware
The test machine was the Sun X4150 server with a 8GB memory and 2 4-core Xeon E5345 @ 2.33GHz processors working under control of the 32 bit version of CentOS 5.2 and the standard version of Apache2 (2.2.3). Finally, all data were served from ext3 partitions with the noatime flag.
Disks
Following disks were used for tests.

• RAID 1 matrix consist of 2 classical rotating HDD with the root file system and the partition storing files for Apache (on LVM2 volume).

  Vendor: Sun       Model: root              Rev: V1.0
  Type:   Direct-Access                      ANSI SCSI revision: 02
  SCSI device sda: 286494720 512-byte hdwr sectors (146685 MB)
  

• Standard Intel SSD storage with the partition holding Apache data.

  Vendor: ATA       Model: INTEL SSDSA2MH16  Rev: 045C
  Type:   Direct-Access                      ANSI SCSI revision: 05
  SCSI device sdc: 312581808 512-byte hdwr sectors (160042 MB)
  

• 2 Intela SSD Extreme disks joined into the one LVM2 volume. It was necessary to create a partition big enough to keep all data for Apache.

  Vendor: ATA       Model: SSDSA2SH064G1GC   Rev: 045C
  Type:   Direct-Access                      ANSI SCSI revision: 05
  SCSI device sdd: 125045424 512-byte hdwr sectors (64023 MB)
  

Measured parameters
In the both table following acronyms has been used to describe measured parameters. (More info about them on Grinder web site.)

• Test - Test name
• MTT (ms) - Mean Test Time
• TTSD (ms) - Test Time Standard Deviation
• TPS -Transactions Per Second
• RBPS - Response Bytes Per Second
• MTTFB (ms) - Mean Time to First Byte

3oo - first test
In the first phase of tests I compared the Apache's performance serving 300 000 request using data stored on classic HDD as well as SSD. Kernels from the 2.6 tree allow to choose a I/O scheduler. In theory the best scheduler for  SSD devices is Noop, therefore in table below I compared results for the mentioned and default (CFQ) schedulers.

Test	MTT (ms)	TTSD (ms)	TPS	RBPS	MTTFB (s)
HDD CFQ	5.53	8.17	179.51	1231607.13	5.3
HDD Noop	5.53	8.09	179.30	1230119.51	5.29
SSD CFQ	0.77	3.06	1226.55	8415044.64	0.56
SSDn Noop	0.74	2.77	1280.17	8782969.21	0.56
SSDe CFQ	0.73	2.55	1280.23	8783381.50	0.52
SSDe Noop	0.71	3.05	1326.62	9101643.04	0.53

How we expected, the SSD disks (or rather Apache with content on them) proved to be much faster. The web server performance grown about 10 times when a HDD were substituted by a SSD. Another observation worth to note is that the results obtained using both sets of the SSD disks were very similar. Extreme Edition storages were few percent faster, but the different is probably too small to be the only reason to justify the higher cost. Additionally, it was clear that the Noop scheduler didn't dramatical change the Apache performance.

One hour data
It's obvious that 300k requests may not enough to show the full and true image, therefore I repeated test with a bigger set of data based on 1 hour worthy log. During that hour the original server had responded to 1 341 489 queries, but during creation of the file with input data for Grinder I saved the list of URL twice, therefore grinder was sending 2 682 978 queries during the test.
The results are presented in the next table. To the data collected from Grinder I added one more number, TT — the total time of the test, that is how long it took Grinder to send all the requests.

Test	MTT (ms)	TTSD (ms)	TPS	RBPS	MTTFB (s)	TT (h:m)
HDD CFQ	2.65	5.29	371.71	2145301.3	2.45	02:00
SSDn CFQ	0.63	3.19	1495.3	8630105.68	0.43	00:29
SSDn Noop	0.64	2.52	1478.77	8534692.28	0.43	00:30
SSDe CFQ	0.59	2.93	1594.06	9200064.95	0.42	00:28
SSDe Noop	0.61	2.62	1530.84	8835205.22	0.42	00:29

The increase of the queries number diminished the difference between the SSD and HDD disk performance, but also in second test the former storage was firm winner. I.e. the Total Time of test was 4 time shorter for any version of the SSD compare to the traditional disks. Another interesting observation is that difference in performance of Mainstream and Extreme disks decreased. Finally, the Noop scheduler didn't improve the results of that test too.


Summary
The results shown in the current study, as well as other not presented above, confirmed the hypothesis that SSD disks might be a good remedy for observed I/O problems. In the few weeks time you might expect some kind of appendix in which I will describe if baptism of fire on the battlefield of the web come off as well as the preliminary tests.

Linux Works in Cambridge

Some time ago I created the "Linux Jobs in Cambridge" map on Google Maps, but something was wrong. Recently, I decided that the title was not very propriety. It's not the map of Linux related opportunities, but the map showing how important is Linux and general Open Sources for Cambridge. So I changed the name to "Linux Work in Cambridge" and it seems to be the right idea. There are some new very interesting entries (even one pub). Cheek it out yourself, and maybe add or correct something.

View Linux Works in Cambridge in a larger map.

Expect and operation on many computers

Recently, I had to delete a directory on around 200 computers. The directory belonged to root, so using my account with public key authentication wasn't possible. I googled a bit, found the expect and wrote the following script.

#!/usr/bin/expect -f

set machine [lindex $argv 0]
set command [lindex $argv 1]
set timeout -1
spawn ssh -l root $machine $command
match_max 100000

expect "?*assword: $"
send "password\n"

expect eof

The script sets the name of a remote machine (set machine [lindex $argv 0]) and a command (set command [lindex $argv 1]) to execute from arguments it is started with. Next tries to connect to the remote machine (spawn ssh -l root $machine $command) and when it's asked for the password (expect "?*assword: $") send it (send "password\n"). Of course you have to change the password to the root password. Finally, it waits for the EOF from ssh (expect eof). I have confess that I don't remember what exactly set timeout -1 and match_max 100000 means ;) The script can be called with loop similar to one below.

for cell in 1{0..3}{0..9} ;\
do for box in {1..4} ;\
do echo  c${bc}-box0${app} ; \
./command.script bc${bc}app-0${app} "ls /var/log/httpd" ; \
done; \
done

One more thing. The script assumes that you has connected at least one to all machines or rather that the machines has been added to your .ssh/know_hosts file. If you plan to use script to initialize the first connection you should add following line

expect "Are you sure you want to continue connecting (yes/no)?"
send "yes\n"

before the line expect "?*assword: $", but in such case all machine haven't to be present in .ssh/know_hosts file.

How to find the not commented line using Vim

The significant part of BOFH's live consist of editing config files. It's not so uncommon that you need find not commented lines (i.e. to find it something is set). With vim it's very easy:

/^[^#]

The above line command the editor to: find a line which doesn't start with # or rather: find a string which is at the beginning of a line with the first character anything else then #. This advice will work not only for vim i.e. you can use it in grep as well:

[user@server]$ grep  "^[^#]" modprobe.conf 
alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptspi

I discussed the similar case some time ago in this note: How to find line not starting with X in Vim.

Reading from rather big files in Python

Recently I needed to open the big file (apache log - 14 GB or so) and cut some information from it. Of course use of file.read() and/or file.readlines() method wasn't possible. On the other hand, using file.readline() few (rather more than 20) million times doesn't sound right. Therefore, I looked for another resolution and found that you can limit the size of readlines().

f=open('filename','r')
opensize=2**27
longlist=[]
while 1:
   shortlist=[[l.split()[n] for n in [0,4,-2,-1]] for l in f.readlines(opensize)]
   if not list:
       break
   else:
       longlist.extend(shortlist)

The script open the 'filename' file and next in the loop:

• read from that file lines of size close to 128 Mb (2**27),
• cut first, fifth, next to last and last column from each line,
• add created (temporary) list to the output list.

It's worth to note that if

shortlist

is not created the script will leave the loop (lines 6 and 7). It not obligatory, but I like to work with 2 powers, therefore opensize=2**27.

one for AWK and one for SVN

Another two useful one liner. First awk. Sometimes you need to grab last "element" of lines in a file which has different numbers of spaces (or other separator). In such case use variable $NF (or $NF-1, $NF-2...). Good example for such situation might be apache log file, where user agent description is a string with various number of spaces, so it's hard to get columns after that. But you can use something similar to:

 tail bo-access_log.2009-06-22 | \
awk '{print "size:\t"$(NF-1) "\t time:\t" $NF}'

In the example log file the time is the last and size of file next to last field. Of course you can type it in one line. But Then you have to remove '\' character from end of first line. Second advice is related to SVN. I found reverting last submitted changes quite not clear there. Revert works only with no committed changes, so I used the command similar to below one.

 svn merge -r HEAD:{2009-06-21} .

The example reverts everything what has been submitted between 21st June 2009 and 'now'. However, today I found that PREV 'variable', so the following command should do I had wanted to achieve. Interesting how could I missed it?

 svn merge -r HEAD:PREV .

And one more update. In petke comments to this entry in Aral Balkan blog I found another one liner, which looks event easier:

svn update -r 2689

Vim substitution

Let say that you want to add string 'bprdp' and the end of each line beginning with string 'bc' and ending with comma you should use following command:

:% g/^bc/s/\,$/, bprdp/

% means the whole file g/ for each line with pattern after '/' in above case pattern is ^bc line beginning with bc s/\,$/, bprdb/ substitute comma (\,) followed by end of line character ($) with ', bprdb'.

I wrote this message based on Vim regular expression and Vim Command Cheat Sheet.

Control the Vim from the edited file

One of the very nice Vim feature I've learnt recently is the possibility of controlling the Vim from a edited file. Chosen Vim commands may to be put in one of the first (specially formatted) file lines. The line format is describe in 'modeline' help keyword (:help modeline). It worth to remember that text before and after main part has to be commenting out directive. Therefore, for example the line in HTML might looks similar to:

<-- vim: set tabstop=4 noexpandtab:-->

for python:

# vim: tabstop=4 noexpandtab:

If you like to learn more please check the modeline keyword in Vim help.

update: I forgot to add that you need to set modeline in .vimrc file.

My first Perl script

It's nothing big, but it's the first one and, as Perl is write only language, I'd better add the short description. The script takes a list of files passed as arguments to the command; reads all lines (http addresses) from them and creates the list of unique domains names.

#!/usr/bin/env perl

%seen = ();
foreach (@ARGV)
{
open (LFILE,"$_");

for $line ()
{
       @sline=split(/\//,$line);
       print ("@sline[2]\n") unless $seen{@sline[2]}++;
}

close LFILE;
}

Perl tutorial from tizag.com was helpful.

DarwinPorts via proxy

Recently, I needed a perl module not present on my MacOSX computer, which was behind a proxy. The friend suggested to use the Darwin ports rather the Perl from Apple. I downloaded and installed it to found I cannot install any port. The problem was due to the combination of using a proxy and the sudo rather then the root user. I guess such combination is rather common among MacOSX-perl users. So below I present the command which allows to use the Darwin ports from a normal MacOSX account. Generally note, you have to export both the RSYNC_PROXY as well as the http_proxy in the sudo environment.

sudo sh -c "export RSYNC_PROXY=proxy.server:port; \
export http_proxy=http://proxy.server:port; \
port install perl5.10 "

How to find the line not starting with "X" in Vim

I don't know why but most of Vim's search examples say nothing about how to find the line not starting with string "X". Finally, I found how to do this. I.e. following line find anything not started from del:

^[^d][^e][^l].*

For people not advance in regex. The consecutive signs means:

• ^ - a line starting with
• [^d] - character other than d;
• [^e] - character other than e;
• [^l] - character other than l;
• .* - any string (any character repeated any times).

Total size of quite new files

Following command count the size of all files (-type f) newer than 10 days (-mtime -10). The size is printed in megabytes. "%k" argument of printf returns size in kilobytes, but "a/1024" in awk change it to megabytes.

find -type f -mtime -10 -printf "%k\n"| \
 awk 'BEGIN {a=0} {a=a+$1} END {print a/1024}'

Remote diff

When you are working as a Linux SysAdmin quite often you have to compare files from two different machines. I found (here) the script which made my life easier, but after some time decided to customize and extent it a bit. Usually I compare file in the same location therefore the first argument of my script is file path. I also gave chance user to pass an argument for the diff command (4th argument, the default is '-b').

#!/bin/bash
#
# this acts as a remote diff program, accepting two files and displaying
# a diff for them.  Zero, one, or both files can be remote.  File paths
# must be in a format `scp` understands: [[user@]host:]file

[ -n "$1" ] || [ -n "$2" ] || [ -n "$3" ] || \
{ echo "Usage: `basename $0` file1 server1 server2" && exit 1;}

if test -e $4 
then
       opt="-b"
else
       opt=$4
fi

scp "$2:$1" rdiff.1 >& /dev/null
scp "$3:$1" rdiff.2 >& /dev/null
diff $opt rdiff.1 rdiff.2
rm -f rdiff.1 rdiff.2

How to create pictures thums in the one line

The following command can be paste as a one line

for i in `find /path/to/directory/with/pictures  -iname "*.JPG"`;\
do\
convert $i -resize 800x600 `dirname $i`/thumb-`basename $i`;\
done

The find command return a whole path to a file. But we want to add thumb- before the actual name of a file. Therefore `dirname $i` ensure that convert get the proper path and `basename $i` the actual file name (preceded by thumb-). It is also worth to note the iname option of find command. It is case insensitive version of the name.

Another Awk one-liner

This one-liner printing third word from the lines beginning with "Mem:" (precisely the lines which first word is "Mem:") but adding the serial number before the word.

awk 'BEGIN {a=1} \
{if ( $1 == "Mem:" ) \
  {printf "%4d  %s\n",  a, $3; a++}}' \
free-prefork.log >mem-prefork.log

BTW. This script is to help me to plot a gnuplot graph based on "used memory" number from free command. plot "mem-prefork.log" will do the rest of job. UPDATE Mike Hommey post force me to rethink my scripts and I found easy way to eliminate if clause:

awk 'BEGIN {a=1} \
/^Mem:/  {printf "%4d  %s\n",  a, $3; a++}' \
free-prefork.log >mem-prefork.log

Gnuplot with readline on MacOSX

Recently, I tried to use gnuplot on a Mac, and, of course, it wasn't working properly. Apple prepared and shipped with MacOS its own (broken) version of readline library with didn't work with gnuplot (known bug). So I grabbed the sources of readline, applied all of patches and built mine version of readline. Next, I added the proper option to gnuplot configure file, but it wasn't pass to makefile. I looked into the Makefiles and found the the TERMLIBS option had to be change. I.e. using such command:

find . -name Makefile \
-exec sed -i.old "s/TERMLIBS\ =/TERMLIBS = -L\/usr\/local\/lib/" {} \; 

Two more things about gnuplot and MacOSX.

1. I started to think to make a gnuplot.app for MacOSX, but sure how it should work.
2. I found that X11 term is much better then Aqua, in particular, it's allow to rotate 3D (s)plots.

Header for bonnie++ csv file

Recently I started to use bonnie++ to perform some disk tests. One of the thing annoy me is that bonnie++ output has no the 'csv header' describing what is in which column. It makes overview of results in a spreadsheet rather hard. Therefore I created my own one.

,,Sequential,Output,,,,,Sequential,Input,,,Random,,Sequential,Create,,,,,Random,Create,
,,Per Chr, ,Block, ,Rewrite, ,Per Chr, ,Block, ,Seeks, ,Create, ,Read, ,Delete, ,Create, ,Read, ,Delete
Machine,Size,K/sec,%CP,K/sec,%CP,K/sec,%CP,K/sec,%CP,K/sec,%CP,/sec,%CP,files,/sec,%CP,/sec,%CP,/sec,%CP,/sec,%CP,/sec,%CP,/sec,%CP

You can save above lines in the file called i.e. bonnie-header.csv and then cat it before csv part of bonnie.out file (of course it can have different name), by:

cat bonnie-header.csv `tail -1 bonnie.out` >bonnie.csv

After that the output should looks similar to this one: BTW. I found that default number of files creating for metadata benchmarks is low, so I increased it to 128.

BigPicture

I don't know how I might not to add this earlier but I didn't. The Big Picture it's a on-line addition to Boston Globe (I guess it only on-line, but I don't reader Boston Globe, as I'm leaving in proper Cambridge, at least at the moment). I saw it first time on Bad Astronomy Blog. That story was about picture of our sky but made from above. The pictures were amazing, but what more important that Big Picture brings new great pictures month after month. It's quite interesting how many of them are space/astronomy related, i.e. this one:

Polish UK X11 keybord layout

The UK and US keyboard layout are different. Polish one, as probably most other ones based on American. It's fine in Poland, but not so fine in UK. Therefore, I decided to create Polish layout, but based on UK keyboard. I based on Mikoła Kosmulski article describing how to prepare mixture of Polish and German layouts. My kebord setting looks this way:

partial default alphanumeric_keys
xkb_symbols "basic" {

    include "latin"

    name[Group1]="Poland based on GB";

    key   { [         q,          Q ] };
    key   { [         w,          W ] };
    key   { [         e,          E,      eogonek,      Eogonek ] };
    key   { [         o,          O,       oacute,       Oacute ] };

    key   { [         a,          A,      aogonek,      Aogonek ] };
    key   { [         s,          S,       sacute,       Sacute ] };
    key   { [         f,          F ] };

    key   { [         z,          Z,    zabovedot,    Zabovedot ] };
    key   { [         x,          X,       zacute,       Zacute ] };
    key   { [         c,          C,       cacute,       Cacute ] };
    key   { [         n,          N,       nacute,       Nacute ] };

    key   { [         2,   quotedbl,  twosuperior,    oneeighth ] };
    key   { [         3,   sterling, threesuperior,    sterling ] };
    key   { [         4,     dollar,     EuroSign,   onequarter ] };

    key   { [apostrophe,         at, dead_circumflex, dead_caron] };
    key   { [     grave,    notsign,          bar,          bar ] };

    key   { [numbersign, asciitilde,   dead_grave,   dead_breve ] };
    key   { [ backslash,        bar,          bar,    brokenbar ] };

    include "kpdl(comma)"

    include "level3(ralt_switch)"
};