Expect, ssh and two passwords

Let say that you need to run a command on many remote machine and you cannot use any of specialist tools (I mean i.e. func). In such situation expect might be helpful and I even has written about it already. Recently I had to modified my script, cause not all remote systems have the same root password.

#!/usr/bin/expect -f

set timeout -1

set machine [lindex $argv 0]
set command [lindex $argv 1]

set pass "xen4ulez\r" 
puts $machine

spawn ssh -o "NumberOfPasswordPrompts 2" -o "ServerAliveCountMax 1" -l root $machine $command
match_max 100000

expect {
    "assword:" 
      {
        exp_send $pass 
        set pass "xen!king\r"
        exp_continue
    }
}

First, the script has a password set as a variable. It allows to change later. To to this I have expanded the 'expect password' part of the script. Now the script sends the password and changes it after. If the password was fine, script executes the command. The change of the password is not important. If the password is wrong the script sends the variable $pass again, but this time it is new value. I have also added to option to ssh command for script to run a bit faster.

1. NumberOfPasswordPrompts 2 - ensure that ssh tries only twice to provide password.
2. ServerAliveCountMax 1 - ensure that ssh sends only one Server Alive message

Finally I have set StrictHostKeyChecking no and UserKnownHostsFile /dev/null in the .ssh/config file for machines I'm going to use above scripts. This two options set ssh not to bother about checking Host Key. Please remember that it might be not secure. 

Do something every minut in shell

Recently I've needed to do some statistic based on apache log file. I've wanted statistic to be update every minute and want to use only shell.

The first idea was cron.  Sounds nice but there is small problem. I want to be flexible. In theory I can add a job to crontab in the moment I want to start it and remove it (comment out) after finishing, but it's ideal.

No cron, so maybe sleep, especially sleep 60.... But there is another problem. My scripts runs few seconds, so after few minutes (approx. 60/time my scripts runs) I'll have a gap in results.

No cron, no sleep. I needed another direction. Recently I've been playing a bit with data formats. I write a bit of code and after few minutes – volia – I had a working scripts.

#!/bin/sh

min=`date  "+%M"`

while [ 1 ]
do
        ~/test.sh
        while (($min==`date  "+%M"`))
        do
                sleep 10
        done
        min=`date  "+%M"`
done 

One more thing which might be interesting for some. I used infinite loop based on  this blog entry.

How to check a website from the commandline

Sometimes it's useful to check a website connectivity using only command line. I.e. when trying a website from remote host, or when you set up a new box you are going to put behind a loadbalancer.

If you need HTTP connection, telnet will be enough:

> telnet 10.10.10.10 80

GET /index.html HTTP/1.1

host: important.site.com

In the case of HTTPS you need openssl:

> openssl s_client  -connect 10.10.10.10:443 -prexit

GET /index.html HTTP/1.1

host: important.site.com

This is entry for my extend my 'external memory', but you might find it useful.

Copy many lines in Emacs

I'm VIM not Emacs user, but use the later one to applied DNS changes. It has two nice feature helping to maintenance BIND zone files. First, built-in versioning in RCS, but even more helpful the mechanism of updating serial number.

Anyway, I found it's very hard to copy more than one line in Emacs, until I found some suggestion here and here. So to copy the current line without the newline you have to one of two (depends if you want copy or move the lines):

C-a C-Space C-e M-w

C-a C-Space C-e C-w

It means: C-a move to beginning of the line, C-Space sat the mark, C-e go to end line, M-w/C-w save/delete(kill) the region. "OK, but it only one line" you may save and you would be right. But to mark more line you just need to i.e. use arrow before marking a text (by M-w).

64 vs 32 rather then SSD vs HDD

Introduction
Some time ago I promised to present results of the performance test of an Apache serving content from HDD and SSD. The tests confirmed that SSD gives Apache serving static content significant performance boost, however the other remedy for I/O problem has been found.

It's not so easy to test a system from inside. It also applies to the internet. The "image" you see is not exactly what your customers might see. There are some tools helping measure the "real image" of your website (i.e. KeyNote), but to test a difference of performance between two server visible as one entitle from outside (i.e. behind load-balancer) such test are not ideal. (Of course it is possible to prepare a A/B test when A means serving images from one server, B from another one). The web search hadn't brought any technique and parameters useful in such test (of course it might means that the search  was not very good). Therefore, the serving speed [byte/microsecond] parameter were define as ratio of file size [bytes] and response time [microseconds], respectively %B and %D in extended Apache log.


SSD vs HDD
In the first test the SSD box (Sun X4150 with 8 Xeon E5345 @ 2.33GHz cores and 8 GB of memory) was try out against HP ProLiant D380 G5 with the same amount of memory and 4 similar processors/cores (Xeon 5148 @ 2.33GHz). The test were split into two phase. Results for both ones are collected in Table 1. The first day both boxes serve the same amount of requests. The SSD machine sent files two time faster. Next day the weight of connection to machine with SSD was increase on the load balancer, but it doesn't change the results. The SSD box was much faster again. The data presented in Table 1 ensure that both boxes worked with similar set of files (similar size and number of requests).

Table 1. Comparison of server with SSD (SUN X4150) and HDD (HP D380 G5) drives.

SSD machine		HDD machine
Day I
Average speed 24.27563 Average time 13908.68485 Average size 7683.71685 Served files 14983388		Average speed 12.23371 Average time 24674.27763 Average size 7702.66678 Served files 14987023
Day II
Average speed 23.76447 Average time 13107.95822 Average size 7839.91839 Served files 21522835		Average speed 11.10400 Average time 26001.08215 Average size 7829.47865 Served files 10758116

During next phase of research the SSD machine was tested again wider set of servers with HDD servers (machines with Apache using different MPM (prefork and worker), with 8 or 16 GB of memory and with slower and faster processors). SSD machine was faster than any of the server with a classical hard drive. Adding memory as well as changing MPM didn't change the difference in the performance. On the other hand, the machine with more and newer processor sent files much faster than the old ones, however still 50% slower than the SSD test kit.


Table 1. Comparison of Apache performance on machine with a SSD against different machine using a HDD. SUN SSD means the test boxs, OLD HP a box with 4 L5430@2.33Ghz core and NEW HP a server with 8 E5345 cores.

		OLD HP		NEW HP	SUN SSD
speed	16GB (prefork)	8GB (prefork)	8GB (worker)	16GB (worker)	8GB, (worker)
Test 1	9.73669	10.99227	10.82629	16.81683	24.18491
Test 2	9.56062	10.57697	10.51381	16.32999	24.41852
Test 3	8.69836	9.83142	9.65313	16.16164	24.39481
Test 4	9.03057	10.12731	9.98168	15.60127	24.16711

32bit vs 64bit
In the mean time another two things occurred. On the one hand, another test machine, this time with 64bit OS, was built.The first results showed significant decrease of the load and the number of I/O (read) operation. Figure 1 and 2 are good indicator how drastic it was change, even if they were prepared on latter production boxes. Moreover, a change in some of the production server settings exhibited that the machine do not properly caching content served by Apache.




Figure 1. Load on a production system before and after changing OS from 32 to 64 flavour.



Figure 2. I/O operation on a production system before and after changing OS from 32 to 64 flavour.



Further investigation of both phenomena showed that the I/O problem mentioned in the first part of article was cause by inefficient caching, what was cause by memory wasting on a server with 32bit OS. By default Apache is using the sendfile() function rather than the combination of read() and write() functions to transfer files from a storage to a network interface (so called zero copy approach). It speeds up a data transfer by avoiding switching context from kernel to user space, but it also means that Apache cache is limited because on 32-bit OS it cannot allocate more than 3GB of memory.

To confirm that a 32bit OS was causing of performance issues another test on live system was conduct. When comparing two identical servers with 16GB of memory and 8 E5345 Xeon cores, one running a 32bit and second 64bit OS, following observation was made:

• The load decreases from around 2 to 0.5 (Figure 1).
• The read from disk decreases from 2.5 to 0.6 [megabytes/second]  (Figure 2).
• The memory usages increase from 2.5-3 to 16 (full memory) GB.
• The average Apache speed increase from around 12 to 30 [bytes/microsecond] (Figure 3).

Figure 3. Comparison of the Apache speed (speed of sending static content) on 32 and 64 bit OS.    
Summary
During the tests the SSD showed its superiority to the classical HDD storages. However in the case of serving a static content from a web server using a distributed memory caching technique might be faster, more scalable  and even cheaper solution.

Another conclusion from above research, probably less and less important as there are less and less of x86 32bit servers, that 64 bit OS might be really faster, especially in the case of system with big amount of memory. 

Short (mostly shell) fomulas

If you hit this website looking for a Linux/shell/UNIX related advice you might want to visit my identi.ca account. I post "short, magic", Linux formula there now. You know something like:

How to split lines in VIM?

s/\,/,[ctrl-V][Enter]/g

where [ctrl-V][Enter] - means type Ctrl+V and next Enter (you should see s/\,/,^M/g)