64 vs 32 rather then SSD vs HDD

Introduction
Some time ago I promised to present results of the performance test of an Apache serving content from HDD and SSD. The tests confirmed that SSD gives Apache serving static content significant performance boost, however the other remedy for I/O problem has been found.

It's not so easy to test a system from inside. It also applies to the internet. The "image" you see is not exactly what your customers might see. There are some tools helping measure the "real image" of your website (i.e. KeyNote), but to test a difference of performance between two server visible as one entitle from outside (i.e. behind load-balancer) such test are not ideal. (Of course it is possible to prepare a A/B test when A means serving images from one server, B from another one). The web search hadn't brought any technique and parameters useful in such test (of course it might means that the search  was not very good). Therefore, the serving speed [byte/microsecond] parameter were define as ratio of file size [bytes] and response time [microseconds], respectively %B and %D in extended Apache log.


SSD vs HDD
In the first test the SSD box (Sun X4150 with 8 Xeon E5345 @ 2.33GHz cores and 8 GB of memory) was try out against HP ProLiant D380 G5 with the same amount of memory and 4 similar processors/cores (Xeon 5148 @ 2.33GHz). The test were split into two phase. Results for both ones are collected in Table 1. The first day both boxes serve the same amount of requests. The SSD machine sent files two time faster. Next day the weight of connection to machine with SSD was increase on the load balancer, but it doesn't change the results. The SSD box was much faster again. The data presented in Table 1 ensure that both boxes worked with similar set of files (similar size and number of requests).

Table 1. Comparison of server with SSD (SUN X4150) and HDD (HP D380 G5) drives.

SSD machine		HDD machine
Day I
Average speed 24.27563 Average time 13908.68485 Average size 7683.71685 Served files 14983388		Average speed 12.23371 Average time 24674.27763 Average size 7702.66678 Served files 14987023
Day II
Average speed 23.76447 Average time 13107.95822 Average size 7839.91839 Served files 21522835		Average speed 11.10400 Average time 26001.08215 Average size 7829.47865 Served files 10758116

During next phase of research the SSD machine was tested again wider set of servers with HDD servers (machines with Apache using different MPM (prefork and worker), with 8 or 16 GB of memory and with slower and faster processors). SSD machine was faster than any of the server with a classical hard drive. Adding memory as well as changing MPM didn't change the difference in the performance. On the other hand, the machine with more and newer processor sent files much faster than the old ones, however still 50% slower than the SSD test kit.


Table 1. Comparison of Apache performance on machine with a SSD against different machine using a HDD. SUN SSD means the test boxs, OLD HP a box with 4 L5430@2.33Ghz core and NEW HP a server with 8 E5345 cores.

		OLD HP		NEW HP	SUN SSD
speed	16GB (prefork)	8GB (prefork)	8GB (worker)	16GB (worker)	8GB, (worker)
Test 1	9.73669	10.99227	10.82629	16.81683	24.18491
Test 2	9.56062	10.57697	10.51381	16.32999	24.41852
Test 3	8.69836	9.83142	9.65313	16.16164	24.39481
Test 4	9.03057	10.12731	9.98168	15.60127	24.16711

32bit vs 64bit
In the mean time another two things occurred. On the one hand, another test machine, this time with 64bit OS, was built.The first results showed significant decrease of the load and the number of I/O (read) operation. Figure 1 and 2 are good indicator how drastic it was change, even if they were prepared on latter production boxes. Moreover, a change in some of the production server settings exhibited that the machine do not properly caching content served by Apache.




Figure 1. Load on a production system before and after changing OS from 32 to 64 flavour.



Figure 2. I/O operation on a production system before and after changing OS from 32 to 64 flavour.



Further investigation of both phenomena showed that the I/O problem mentioned in the first part of article was cause by inefficient caching, what was cause by memory wasting on a server with 32bit OS. By default Apache is using the sendfile() function rather than the combination of read() and write() functions to transfer files from a storage to a network interface (so called zero copy approach). It speeds up a data transfer by avoiding switching context from kernel to user space, but it also means that Apache cache is limited because on 32-bit OS it cannot allocate more than 3GB of memory.

To confirm that a 32bit OS was causing of performance issues another test on live system was conduct. When comparing two identical servers with 16GB of memory and 8 E5345 Xeon cores, one running a 32bit and second 64bit OS, following observation was made:

• The load decreases from around 2 to 0.5 (Figure 1).
• The read from disk decreases from 2.5 to 0.6 [megabytes/second]  (Figure 2).
• The memory usages increase from 2.5-3 to 16 (full memory) GB.
• The average Apache speed increase from around 12 to 30 [bytes/microsecond] (Figure 3).

Figure 3. Comparison of the Apache speed (speed of sending static content) on 32 and 64 bit OS.    
Summary
During the tests the SSD showed its superiority to the classical HDD storages. However in the case of serving a static content from a web server using a distributed memory caching technique might be faster, more scalable  and even cheaper solution.

Another conclusion from above research, probably less and less important as there are less and less of x86 32bit servers, that 64 bit OS might be really faster, especially in the case of system with big amount of memory. 

Short (mostly shell) fomulas

If you hit this website looking for a Linux/shell/UNIX related advice you might want to visit my identi.ca account. I post "short, magic", Linux formula there now. You know something like:

How to split lines in VIM?

s/\,/,[ctrl-V][Enter]/g

where [ctrl-V][Enter] - means type Ctrl+V and next Enter (you should see s/\,/,^M/g)

noatime

If you need I/O performance bust it might worth to turn on noatime flag on important partitions. Look down to see how it pushed down the read level on a server with Apache serving a static content.

Usually I add noatime flag during a system installation, but this time forgot about it and had to remount the file system. Thanks that mistake I got this beautiful image ;)
You can go even further and turn on nodiratime, it should decrease read even more.

GMAIL and msmtp (Mutt)

This is the example how to configure the msmtp (i.e. for Mutt) to use thegmail smtp server. Remember that you need the certificate. I have got mine from old Ubuntu installation (Saving /etc directory before reinstallation a box is good idea).

account your.user
logfile ~/.msmtp.log

tls on
tls_starttls on    
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on

host smtp.gmail.com
port 587 
from your.user@gmail.com
user your.user@gmail.com
password YOUR_password

BTW, in Ubuntu you can grab certificate by sudo apt-get install ca-certificates.

Nagios plugin

Not so long ago I heard the question if (or rather how) it is possible to write a NRPE plugin checking the resources utilization of an application. I'm using Nagios on the daily basis, but I haven't needed to write any plugin yet. When I went through existing plugins most/all of them checked resources on a server level. It makes sense, you are not so interested what exactly doing you server if your website/database is available and response fast. It especially true if you have 100, 500, 1000, ... servers. Anyway, I found the question interesting, even if was rather theoretical than practical one. After some research I found Jason Faulkner plugin which should be a good base, modified it a bit and created this script:

#!/bin/bash 
#
# Nagios plugin to monitor a process. Can easily be modified to do 
# pretty much whatever you want.
#
# Licensed under LGPL version 2
# Copyright 2006 Broadwick Corporation
# By: Jason Faulkner jasonf@broadwick.com
#
# Modified to measure CPU usage of chosen process.
#
# USAGE: cpu.sh process_name warning_level critical_level
#
# Licensed under LGPL version 2
# Copyright 2009 Wawrzyniec Niewodniczański
# Modification by: Wawrzyniec Niewodniczański wawrzek@gmail.com

process_name=$1
WARLVL=$2
CRITLVL=$3

OKMSG="STATUS OK: ${process_name} running"
CRITMSG="STATUS CRITICAL: ${process_name} using more than ${CRITLVL} % of Memory"
WARNMSG="STATUS WARNING: >1 ${process_name} using more than  ${WARLVL} % of Memory"
UNKMSG="STATUS UNKNOWN: ${process_name}, check if process is running"

PROCESS=`ps axu | grep  -v ${0}|grep -v grep | grep ${process_name}`
CPU=`echo ${PROCESS}| awk '{cpu+=$3} END {printf "%d", cpu}'`

if [[ $PROCESS != "" ]]  
then 
        if (($CPU < $WARLVL))
        then
                echo "$OKMSG"
                exit 0
        elif (( "$CPU" < $CRITLVL ))
        then 
                echo "$WARNMSG"
                exit 1
        else 
                echo "$CRITMSG"
                exit 2
        fi  
else
        echo "$UNKMSG"
        exit 3
fi

I would say that it's nothing excited. There are two important lines. The first one searching the process name in output of ps command and excluding the lines with script name and grep from the list. The another one using awk to add value of CPU usage from the list created in first line. BTW if you would prefer to check memory usage rather then processor, change {cpu+=$3} to {cpu+=$4} (or even to {mem+=$4}) in awk command. I also wrote the nagios command which I believe should work. "believe" not "know", as I haven't try it yet ;)

# 'check_cpu' command definition
define command{
        command_name    check_cpu
        command_line    /usr/lib/nagios/plugins/check_cpu $ARG1$ $ARG2$ $ARG3$
ń} 

Useful links

1. http://www.nagios.org/documentation
2. http://debianclusters.cs.uni.edu/index.php/Creating_Your_Own_Nagios_Plugin
3. http://www.ibm.com/developerworks/aix/library/au-nagios/index.html
4. http://lena.franken.de/nagios/own_service.html

Escape, Escape

I couldn't understand why following command was working on a local machine, but not through ssh (following form):

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += $5;} END {print tsum}'" 

I asked my workmate and he also had problems for some time, but finally he suggested that we needed to "escape" something. After some try we found that ssh don't like $ character so following command works.

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += \$5;} END {print tsum}'"

Stone Redskin: comparision of Apache2 performance on HDD and SSD

Introduction
Recently, I had a chance to test the performance of a static content web servers. The initial analysis showed that the most important issue were the speed of a disks, which started to have problems with handling I/O operations. The numbers of files were huge what means that hard drives were engaged in many random access operation.

The latest tests has shown that the new Solid State Disk (SSD) mass storage beat the classic Hard Drive Disk (HDD) in such circumstances (in most others too). So it was quite natural to prepare a set of test helping to measure the effect of switch from a HDD to a SSD storage on the Apache performance.

Methodology
It should be keep in mind, that I wasn't interesting in a general comparison of SSD vs HDD, but concentrated my tests on the Apache performance. The Grinder 3.2 software was used to simulate a load on the web server. The list of requested URL based on the real Apache logs taken from the one of box serving the static content. To eliminate the influence of caching, before each test the memory cache was cleaned using following command echo 3 > /proc/sys/vm/drop_caches (suggested on Linux-MM).

Hardware
The test machine was the Sun X4150 server with a 8GB memory and 2 4-core Xeon E5345 @ 2.33GHz processors working under control of the 32 bit version of CentOS 5.2 and the standard version of Apache2 (2.2.3). Finally, all data were served from ext3 partitions with the noatime flag.
Disks
Following disks were used for tests.

• RAID 1 matrix consist of 2 classical rotating HDD with the root file system and the partition storing files for Apache (on LVM2 volume).

  Vendor: Sun       Model: root              Rev: V1.0
  Type:   Direct-Access                      ANSI SCSI revision: 02
  SCSI device sda: 286494720 512-byte hdwr sectors (146685 MB)
  

• Standard Intel SSD storage with the partition holding Apache data.

  Vendor: ATA       Model: INTEL SSDSA2MH16  Rev: 045C
  Type:   Direct-Access                      ANSI SCSI revision: 05
  SCSI device sdc: 312581808 512-byte hdwr sectors (160042 MB)
  

• 2 Intela SSD Extreme disks joined into the one LVM2 volume. It was necessary to create a partition big enough to keep all data for Apache.

  Vendor: ATA       Model: SSDSA2SH064G1GC   Rev: 045C
  Type:   Direct-Access                      ANSI SCSI revision: 05
  SCSI device sdd: 125045424 512-byte hdwr sectors (64023 MB)
  

Measured parameters
In the both table following acronyms has been used to describe measured parameters. (More info about them on Grinder web site.)

• Test - Test name
• MTT (ms) - Mean Test Time
• TTSD (ms) - Test Time Standard Deviation
• TPS -Transactions Per Second
• RBPS - Response Bytes Per Second
• MTTFB (ms) - Mean Time to First Byte

3oo - first test
In the first phase of tests I compared the Apache's performance serving 300 000 request using data stored on classic HDD as well as SSD. Kernels from the 2.6 tree allow to choose a I/O scheduler. In theory the best scheduler for  SSD devices is Noop, therefore in table below I compared results for the mentioned and default (CFQ) schedulers.

Test	MTT (ms)	TTSD (ms)	TPS	RBPS	MTTFB (s)
HDD CFQ	5.53	8.17	179.51	1231607.13	5.3
HDD Noop	5.53	8.09	179.30	1230119.51	5.29
SSD CFQ	0.77	3.06	1226.55	8415044.64	0.56
SSDn Noop	0.74	2.77	1280.17	8782969.21	0.56
SSDe CFQ	0.73	2.55	1280.23	8783381.50	0.52
SSDe Noop	0.71	3.05	1326.62	9101643.04	0.53

How we expected, the SSD disks (or rather Apache with content on them) proved to be much faster. The web server performance grown about 10 times when a HDD were substituted by a SSD. Another observation worth to note is that the results obtained using both sets of the SSD disks were very similar. Extreme Edition storages were few percent faster, but the different is probably too small to be the only reason to justify the higher cost. Additionally, it was clear that the Noop scheduler didn't dramatical change the Apache performance.

One hour data
It's obvious that 300k requests may not enough to show the full and true image, therefore I repeated test with a bigger set of data based on 1 hour worthy log. During that hour the original server had responded to 1 341 489 queries, but during creation of the file with input data for Grinder I saved the list of URL twice, therefore grinder was sending 2 682 978 queries during the test.
The results are presented in the next table. To the data collected from Grinder I added one more number, TT — the total time of the test, that is how long it took Grinder to send all the requests.

Test	MTT (ms)	TTSD (ms)	TPS	RBPS	MTTFB (s)	TT (h:m)
HDD CFQ	2.65	5.29	371.71	2145301.3	2.45	02:00
SSDn CFQ	0.63	3.19	1495.3	8630105.68	0.43	00:29
SSDn Noop	0.64	2.52	1478.77	8534692.28	0.43	00:30
SSDe CFQ	0.59	2.93	1594.06	9200064.95	0.42	00:28
SSDe Noop	0.61	2.62	1530.84	8835205.22	0.42	00:29

The increase of the queries number diminished the difference between the SSD and HDD disk performance, but also in second test the former storage was firm winner. I.e. the Total Time of test was 4 time shorter for any version of the SSD compare to the traditional disks. Another interesting observation is that difference in performance of Mainstream and Extreme disks decreased. Finally, the Noop scheduler didn't improve the results of that test too.


Summary
The results shown in the current study, as well as other not presented above, confirmed the hypothesis that SSD disks might be a good remedy for observed I/O problems. In the few weeks time you might expect some kind of appendix in which I will describe if baptism of fire on the battlefield of the web come off as well as the preliminary tests.

Linux Works in Cambridge

Some time ago I created the "Linux Jobs in Cambridge" map on Google Maps, but something was wrong. Recently, I decided that the title was not very propriety. It's not the map of Linux related opportunities, but the map showing how important is Linux and general Open Sources for Cambridge. So I changed the name to "Linux Work in Cambridge" and it seems to be the right idea. There are some new very interesting entries (even one pub). Cheek it out yourself, and maybe add or correct something.

View Linux Works in Cambridge in a larger map.

Expect and operation on many computers

Recently, I had to delete a directory on around 200 computers. The directory belonged to root, so using my account with public key authentication wasn't possible. I googled a bit, found the expect and wrote the following script.

#!/usr/bin/expect -f

set machine [lindex $argv 0]
set command [lindex $argv 1]
set timeout -1
spawn ssh -l root $machine $command
match_max 100000

expect "?*assword: $"
send "password\n"

expect eof

The script sets the name of a remote machine (set machine [lindex $argv 0]) and a command (set command [lindex $argv 1]) to execute from arguments it is started with. Next tries to connect to the remote machine (spawn ssh -l root $machine $command) and when it's asked for the password (expect "?*assword: $") send it (send "password\n"). Of course you have to change the password to the root password. Finally, it waits for the EOF from ssh (expect eof). I have confess that I don't remember what exactly set timeout -1 and match_max 100000 means ;) The script can be called with loop similar to one below.

for cell in 1{0..3}{0..9} ;\
do for box in {1..4} ;\
do echo  c${bc}-box0${app} ; \
./command.script bc${bc}app-0${app} "ls /var/log/httpd" ; \
done; \
done

One more thing. The script assumes that you has connected at least one to all machines or rather that the machines has been added to your .ssh/know_hosts file. If you plan to use script to initialize the first connection you should add following line

expect "Are you sure you want to continue connecting (yes/no)?"
send "yes\n"

before the line expect "?*assword: $", but in such case all machine haven't to be present in .ssh/know_hosts file.

How to find the not commented line using Vim

The significant part of BOFH's live consist of editing config files. It's not so uncommon that you need find not commented lines (i.e. to find it something is set). With vim it's very easy:

/^[^#]

The above line command the editor to: find a line which doesn't start with # or rather: find a string which is at the beginning of a line with the first character anything else then #. This advice will work not only for vim i.e. you can use it in grep as well:

[user@server]$ grep  "^[^#]" modprobe.conf 
alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptspi

I discussed the similar case some time ago in this note: How to find line not starting with X in Vim.

Reading from rather big files in Python

Recently I needed to open the big file (apache log - 14 GB or so) and cut some information from it. Of course use of file.read() and/or file.readlines() method wasn't possible. On the other hand, using file.readline() few (rather more than 20) million times doesn't sound right. Therefore, I looked for another resolution and found that you can limit the size of readlines().

f=open('filename','r')
opensize=2**27
longlist=[]
while 1:
   shortlist=[[l.split()[n] for n in [0,4,-2,-1]] for l in f.readlines(opensize)]
   if not list:
       break
   else:
       longlist.extend(shortlist)

The script open the 'filename' file and next in the loop:

• read from that file lines of size close to 128 Mb (2**27),
• cut first, fifth, next to last and last column from each line,
• add created (temporary) list to the output list.

It's worth to note that if

shortlist

is not created the script will leave the loop (lines 6 and 7). It not obligatory, but I like to work with 2 powers, therefore opensize=2**27.

one for AWK and one for SVN

Another two useful one liner. First awk. Sometimes you need to grab last "element" of lines in a file which has different numbers of spaces (or other separator). In such case use variable $NF (or $NF-1, $NF-2...). Good example for such situation might be apache log file, where user agent description is a string with various number of spaces, so it's hard to get columns after that. But you can use something similar to:

 tail bo-access_log.2009-06-22 | \
awk '{print "size:\t"$(NF-1) "\t time:\t" $NF}'

In the example log file the time is the last and size of file next to last field. Of course you can type it in one line. But Then you have to remove '\' character from end of first line. Second advice is related to SVN. I found reverting last submitted changes quite not clear there. Revert works only with no committed changes, so I used the command similar to below one.

 svn merge -r HEAD:{2009-06-21} .

The example reverts everything what has been submitted between 21st June 2009 and 'now'. However, today I found that PREV 'variable', so the following command should do I had wanted to achieve. Interesting how could I missed it?

 svn merge -r HEAD:PREV .

And one more update. In petke comments to this entry in Aral Balkan blog I found another one liner, which looks event easier:

svn update -r 2689

Vim substitution

Let say that you want to add string 'bprdp' and the end of each line beginning with string 'bc' and ending with comma you should use following command:

:% g/^bc/s/\,$/, bprdp/

% means the whole file g/ for each line with pattern after '/' in above case pattern is ^bc line beginning with bc s/\,$/, bprdb/ substitute comma (\,) followed by end of line character ($) with ', bprdb'.

I wrote this message based on Vim regular expression and Vim Command Cheat Sheet.

Control the Vim from the edited file

One of the very nice Vim feature I've learnt recently is the possibility of controlling the Vim from a edited file. Chosen Vim commands may to be put in one of the first (specially formatted) file lines. The line format is describe in 'modeline' help keyword (:help modeline). It worth to remember that text before and after main part has to be commenting out directive. Therefore, for example the line in HTML might looks similar to:

<-- vim: set tabstop=4 noexpandtab:-->

for python:

# vim: tabstop=4 noexpandtab:

If you like to learn more please check the modeline keyword in Vim help.

update: I forgot to add that you need to set modeline in .vimrc file.

My first Perl script

It's nothing big, but it's the first one and, as Perl is write only language, I'd better add the short description. The script takes a list of files passed as arguments to the command; reads all lines (http addresses) from them and creates the list of unique domains names.

#!/usr/bin/env perl

%seen = ();
foreach (@ARGV)
{
open (LFILE,"$_");

for $line ()
{
       @sline=split(/\//,$line);
       print ("@sline[2]\n") unless $seen{@sline[2]}++;
}

close LFILE;
}

Perl tutorial from tizag.com was helpful.

DarwinPorts via proxy

Recently, I needed a perl module not present on my MacOSX computer, which was behind a proxy. The friend suggested to use the Darwin ports rather the Perl from Apple. I downloaded and installed it to found I cannot install any port. The problem was due to the combination of using a proxy and the sudo rather then the root user. I guess such combination is rather common among MacOSX-perl users. So below I present the command which allows to use the Darwin ports from a normal MacOSX account. Generally note, you have to export both the RSYNC_PROXY as well as the http_proxy in the sudo environment.

sudo sh -c "export RSYNC_PROXY=proxy.server:port; \
export http_proxy=http://proxy.server:port; \
port install perl5.10 "

How to find the line not starting with "X" in Vim

I don't know why but most of Vim's search examples say nothing about how to find the line not starting with string "X". Finally, I found how to do this. I.e. following line find anything not started from del:

^[^d][^e][^l].*

For people not advance in regex. The consecutive signs means:

• ^ - a line starting with
• [^d] - character other than d;
• [^e] - character other than e;
• [^l] - character other than l;
• .* - any string (any character repeated any times).

Total size of quite new files

Following command count the size of all files (-type f) newer than 10 days (-mtime -10). The size is printed in megabytes. "%k" argument of printf returns size in kilobytes, but "a/1024" in awk change it to megabytes.

find -type f -mtime -10 -printf "%k\n"| \
 awk 'BEGIN {a=0} {a=a+$1} END {print a/1024}'

Remote diff

When you are working as a Linux SysAdmin quite often you have to compare files from two different machines. I found (here) the script which made my life easier, but after some time decided to customize and extent it a bit. Usually I compare file in the same location therefore the first argument of my script is file path. I also gave chance user to pass an argument for the diff command (4th argument, the default is '-b').

#!/bin/bash
#
# this acts as a remote diff program, accepting two files and displaying
# a diff for them.  Zero, one, or both files can be remote.  File paths
# must be in a format `scp` understands: [[user@]host:]file

[ -n "$1" ] || [ -n "$2" ] || [ -n "$3" ] || \
{ echo "Usage: `basename $0` file1 server1 server2" && exit 1;}

if test -e $4 
then
       opt="-b"
else
       opt=$4
fi

scp "$2:$1" rdiff.1 >& /dev/null
scp "$3:$1" rdiff.2 >& /dev/null
diff $opt rdiff.1 rdiff.2
rm -f rdiff.1 rdiff.2

How to create pictures thums in the one line

The following command can be paste as a one line

for i in `find /path/to/directory/with/pictures  -iname "*.JPG"`;\
do\
convert $i -resize 800x600 `dirname $i`/thumb-`basename $i`;\
done

The find command return a whole path to a file. But we want to add thumb- before the actual name of a file. Therefore `dirname $i` ensure that convert get the proper path and `basename $i` the actual file name (preceded by thumb-). It is also worth to note the iname option of find command. It is case insensitive version of the name.

Another Awk one-liner

This one-liner printing third word from the lines beginning with "Mem:" (precisely the lines which first word is "Mem:") but adding the serial number before the word.

awk 'BEGIN {a=1} \
{if ( $1 == "Mem:" ) \
  {printf "%4d  %s\n",  a, $3; a++}}' \
free-prefork.log >mem-prefork.log

BTW. This script is to help me to plot a gnuplot graph based on "used memory" number from free command. plot "mem-prefork.log" will do the rest of job. UPDATE Mike Hommey post force me to rethink my scripts and I found easy way to eliminate if clause:

awk 'BEGIN {a=1} \
/^Mem:/  {printf "%4d  %s\n",  a, $3; a++}' \
free-prefork.log >mem-prefork.log

Gnuplot with readline on MacOSX

Recently, I tried to use gnuplot on a Mac, and, of course, it wasn't working properly. Apple prepared and shipped with MacOS its own (broken) version of readline library with didn't work with gnuplot (known bug). So I grabbed the sources of readline, applied all of patches and built mine version of readline. Next, I added the proper option to gnuplot configure file, but it wasn't pass to makefile. I looked into the Makefiles and found the the TERMLIBS option had to be change. I.e. using such command:

find . -name Makefile \
-exec sed -i.old "s/TERMLIBS\ =/TERMLIBS = -L\/usr\/local\/lib/" {} \; 

Two more things about gnuplot and MacOSX.

1. I started to think to make a gnuplot.app for MacOSX, but sure how it should work.
2. I found that X11 term is much better then Aqua, in particular, it's allow to rotate 3D (s)plots.

Header for bonnie++ csv file

Recently I started to use bonnie++ to perform some disk tests. One of the thing annoy me is that bonnie++ output has no the 'csv header' describing what is in which column. It makes overview of results in a spreadsheet rather hard. Therefore I created my own one.

,,Sequential,Output,,,,,Sequential,Input,,,Random,,Sequential,Create,,,,,Random,Create,
,,Per Chr, ,Block, ,Rewrite, ,Per Chr, ,Block, ,Seeks, ,Create, ,Read, ,Delete, ,Create, ,Read, ,Delete
Machine,Size,K/sec,%CP,K/sec,%CP,K/sec,%CP,K/sec,%CP,K/sec,%CP,/sec,%CP,files,/sec,%CP,/sec,%CP,/sec,%CP,/sec,%CP,/sec,%CP,/sec,%CP

You can save above lines in the file called i.e. bonnie-header.csv and then cat it before csv part of bonnie.out file (of course it can have different name), by:

cat bonnie-header.csv `tail -1 bonnie.out` >bonnie.csv

After that the output should looks similar to this one: BTW. I found that default number of files creating for metadata benchmarks is low, so I increased it to 128.

BigPicture

I don't know how I might not to add this earlier but I didn't. The Big Picture it's a on-line addition to Boston Globe (I guess it only on-line, but I don't reader Boston Globe, as I'm leaving in proper Cambridge, at least at the moment). I saw it first time on Bad Astronomy Blog. That story was about picture of our sky but made from above. The pictures were amazing, but what more important that Big Picture brings new great pictures month after month. It's quite interesting how many of them are space/astronomy related, i.e. this one:

Polish UK X11 keybord layout

The UK and US keyboard layout are different. Polish one, as probably most other ones based on American. It's fine in Poland, but not so fine in UK. Therefore, I decided to create Polish layout, but based on UK keyboard. I based on Mikoła Kosmulski article describing how to prepare mixture of Polish and German layouts. My kebord setting looks this way:

partial default alphanumeric_keys
xkb_symbols "basic" {

    include "latin"

    name[Group1]="Poland based on GB";

    key   { [         q,          Q ] };
    key   { [         w,          W ] };
    key   { [         e,          E,      eogonek,      Eogonek ] };
    key   { [         o,          O,       oacute,       Oacute ] };

    key   { [         a,          A,      aogonek,      Aogonek ] };
    key   { [         s,          S,       sacute,       Sacute ] };
    key   { [         f,          F ] };

    key   { [         z,          Z,    zabovedot,    Zabovedot ] };
    key   { [         x,          X,       zacute,       Zacute ] };
    key   { [         c,          C,       cacute,       Cacute ] };
    key   { [         n,          N,       nacute,       Nacute ] };

    key   { [         2,   quotedbl,  twosuperior,    oneeighth ] };
    key   { [         3,   sterling, threesuperior,    sterling ] };
    key   { [         4,     dollar,     EuroSign,   onequarter ] };

    key   { [apostrophe,         at, dead_circumflex, dead_caron] };
    key   { [     grave,    notsign,          bar,          bar ] };

    key   { [numbersign, asciitilde,   dead_grave,   dead_breve ] };
    key   { [ backslash,        bar,          bar,    brokenbar ] };

    include "kpdl(comma)"

    include "level3(ralt_switch)"
};

Trash in Ubuntu

My father had a strange problem. He couldn't delete few files from his Trash. First we figured out that he didn't have 'write' rights to them, even if he saved them from a mail. My first suggestion was to change permissions, but he couldn't (using Nautilus). Today I had a chance to sit in front of dad's computer, but I couldn't find where is the Trash. I decided to check if there were any similar bug report in the launchpad. I found this suggestion, which work for me (of course I need to make chmod -R +w ~/.local/share/Trash/files/*). And the point I would like to save here is that the trash in Ubuntu is in:

~/.local/share/Trash/files/

Cordless USB phone - not working with Ubuntu

This time no-success story. I bought the WP-01 VoIP Dect Cordless Handset on ebay (from shop4USB). In Product Description there was an info that it works with "Linux2.4 above". In reality it was working only partly, very partly. Indeed the only part working was sound (as normal sound card), it means I could hear and speak after /choose a number/pick up a call/ using Skype software. Anything ( ring, Keyboard and LCD) wasn't working. Phone was recognize by system and asign to USB HID module:
dmesg |grep hid [ 33.859714] usbcore: registered new interface driver hiddev [ 33.866820] hiddev96hidraw0: USB HID v1.00 Device [HID 06e6:c31c] on usb-0000:00:1f.4-2.2
I tried to make yealink module controlling phone.
rmmod ubhid modprobe yealink
But, how I expected, it didn't help. Finally, the phone USB info was/is
06e6:c31c Tiger Jet Network, Inc.

PS. The seller was OK and gave me my money back.

Swap - VMware effects and parallelization

VMare and swap

To present our software marketing and scientific represents use Windows laptopts with Linux in VMWare. They need Linux because our web based product (Relibase+, IsoStar and incoming WebCSD) working only on it. Using virtualization shouldn't be a problem as a machine has 2GB of memory and we can assign 1GB to guest OS. However, recently we couldn't start WebCSD, not only guest was affected but also host froze. VMWare has problem with I/O operation so we were suspicious about disk usage, but the server didn't need to much of it. Anyway I went I/O trace and decided to turn off the swap. After that server started to work as a rocket!

Parallelization of a swap partitions

I was browins through IBM developersWorks and found info that you can parallelize a swap partition.

Amazingly, all modern Linux kernels, by default (with no special kernel options or patches) allow you to parallelize swap, just like a RAID 0 stripe. By using the pri option in /etc/fstab to set multiple swap partitions to the same priority, we tell Linux to use them in parallel:

/dev/sda2 none swap sw,pri=3 0 0
/dev/sdb2 none swap sw,pri=3 0 0
/dev/sdc2 none swap sw,pri=3 0 0
/dev/sdd2 none swap sw,pri=1 0 0

Creating service starting script

One of our customer ask what to do to start IsoStar server during system start. Our customer was using Centos (RedHat derivative) so I suggested to use the command line tool called chkconfig [1, 2]. TO use this tool you need a special script starting Isostar Apache server. The sample of such script can be found here [3]. Here is the example instruction. Let me assume that isostar is going to /opt directory.

1. Copy the script from link [3] to your HDD and call it isostar_server
2. Change line:

   /path/to/command/to/start/new-service

   to:

   /opt/csd/isostar/APACHE/bin/ccdc_apache start

   and line:

   /path/to/command/to/stop/new-service

   to:

   /opt/csd/isostar/APACHE/bin/ccdc_apache stop

3. Remove following lines (from both start and stop subsection):

         #Or to run it as some other user:
   /bin/su - username -c /path/to/command/to/start/new-service
   echo "."

4. Change 'new-service' in 'echo -n' lines to 'isostar_server'.
5. Now as a root copy isostar_server file into /etc/init.d/
6. Again as a root invoke chkconfig and add isostar_server: /sbin/chkconfig --add isostar_server

There are very similar tool for Debian (Ubuntu etc.) called update-rc.d [4, 5]. You should be able to use following command with the same (very similar script): update-rc.d isostar_server start 90 2 3 4 5 . stop 10 0 1 6.

Links

[1] http://linux.die.net/man/8/chkconfig
[2] http://spiralbound.net/2006/11/15/controlling-services-with-chkconfig
[3] http://spiralbound.net/2007/07/23/example-linux-init-script
[4] http://wiki.linuxquestions.org/wiki/Update-rc.d
[5] http://www.annodex.net/cgi-bin/man/man2html?update-rc.d+8

Count a sum of sizes of selected files in a directory

I would like to count how much space taking Qt4 libraries for the MacOSX universal build of one from our application. So I used following command:

du -m * | sort -nr | grep Qt | awk '{sum+=$1} END {print sum}'

• du -m - print used size in megabytes (for directory do not forgot about -s option);
• sort -nr - sort in reverse, numerical order;
• grep Qt - left only lines with Qt (you can also first grep and next sort lines);
• {sum+=$1} - adding a value from first column of each line to variable sum;
• END {print sum} - printing variable sum on after going through all of lines.

The electronic structure of selected betaine dyes. A quantum chemical study

This thesis presents electronic absorption spectra, non linear optical properties and geometrical parameters of betaine dyes obtained by quantum chemical calculations.

Four betaines [4-(1-piridinium-phenolan), 3-(1-piridinium-phenolan), 2-(1-piridinium-phenolan) and 4-(1-piridinium-thiophenolan)] were selected for the study.

During the research various ab initio methods were applied. The Hartee-Fock method (HF) and the second order Møller-Plesset perturbation theory (MP2) were used to determine a geometrical and NLO properties. Moreover, the NLO were obtained using chosen variants of the coupled cluster metod (CC2, CCSD) and the geometry optimizations were perform using the Density Functional Theory (DFT/B3LYP) as well as complete active space methods (CASSCF and CASPT2). In the case of spectroscopic properties the CC and CASSCF/CASPT2 methods along with Time-Dependant DFT (with B3LYP, PBE0 and CAM-B3LYP functionals) and the Configuration Interaction with Singles (CIS and CIS(D)) were used.

Results obtained during the study indicate that the correct description of betaine dyes' electronic structure is an unusually demanding test for present quantum chemical methods. It is safe to say, that, for all of the investigated parameters, the electron correlation has to be take into account. It is also worth to notify that basis set selection is less important. However, diffuse and polarisation functions should be included in the case of spectroscopic and optical properties.

The presented computational result confirmed the very strong interaction between a betaine molecule and its environment. One of the outcome of this phenomena is a large difference between experimental results (usually obtained in condense phases) and theoretical data (calculated in vacuum). Another observation verified during project is the significant increase of the betaines' NLO by the conformational shifting. Finally, it is worth notify that the largest NLO response was obtained for 4-(1-piridinium-thiophenolan).

How you can quest I'm a doctor now! I defended my thesis (abstract above) 19th of June 2008 and my degree was confirmed by faculty of Chemistry board 26th of June. Thesis was written in Polish, so it isn't very useful for most of the world. However, there is the appendix with all available theoretical data of geometrical and spectroscopic parameters of betaine dyes. Additionally, some of results was published in following articles: JMM-11, JMM-13 and LETT-411 (I hope to write one, maybe more). PDF with thesis can be download here.

If I find some time I will might prepare English version of mentioned appendix, and of course, I will share my LaTeX, gnuplot, computational experience.

GRUB and why root!=root

It one of the this 'keep in mind notes'. I tried to install the Fedora 9 along the Ubuntu 8.04. The Fedora went onto the sdb3 partition or (hd1,2) in the GRUB notation. I added following lines to the Ubuntu /boot/grub/menu.lst file. (I didn't install GRUB from Fedora).

title        Fedora 9
root        (hd1,2)
kernel        /boot/vmlinuz-2.6.25-14.fc9.i686 root=/dev/sdb3 ro
initrd        /boot/initrd-2.6.25-14.fc9.i686.img

But it wanted to work. All the time, I got Error 2 : Bad file or directory type. After trying many things I figured out that problem is lack of the GRUB files at the Fedora partition, so I updated grub config file.

title        Fedora 9
root        (hd0,0)
kernel        (hd1,2)/boot/vmlinuz-2.6.25-14.fc9.i686 root=/dev/sdb3 ro
initrd        (hd1,2)/boot/initrd-2.6.25-14.fc9.i686.img

Finally, the Fedora started to boot.

Yet Another Gnuplot Script

I'm preparing presentation for the Public Defence of my PhD thesis. I'm creating few plots and of course I'm using gnuplot for them. One of plots is seems to be really simple: three 'columns' each in other colour, but it took me few hours to make it. I'm glad that I finally did it so I'm sharing my scripts with you (I needed Postscript for Greek's symbols):

set term postscript eps enhanced color
set output "nlo-rhb-cc.eps"
set ylabel '{/Symbol b} [10^{-30} esu]'
set style data histogram
set style histogram cluster gap 1
set style fill solid border -1
set boxwidth
unset xtics

plot [-0.5:.7][-15000:0] "cc.csv" using 1 ti col, '' using 2 ti col, '' using 3 ti col

If you would like to try use the data below: and data:

HF/FF MP2/FF CCSD
-14720.51945800 -6960.7083277 -10843.140 

BTW. I noted three new interesting website related to the gnuplot:

• http://www.cs.ucr.edu/~titus/pyxTutorial/
• http://www.ibm.com/developerworks/library/l-gnuplot/
• http://www.duke.edu/~hpgavin/gnuplot.html

Ian Foster blog

When I was preparing to the exam from GRID computing one of me friend sent me a link to the Ian Foster blog, and I think that it is great that you can follow thoughts one of the GRID fathers. At the moment he is at the "Open Source Grid & Cluster Conference". It is very interesting that you can grab pdf/videos/ppt from all point of programme.

And Now For Something Not-Completely Different

Sed one liners

I'm using this set of sed one liners very hardly. Today I found that I hasn't added it to my blog so I cannot find it easily. (You know I use this blog as a some kind of memory extension ;) I was trying to find sed command capitalized a variable, I couldn't find it at mentioned page. Farther searching pointed to very interesting thread at Unix .com forum. The was script doing something similar in python, perl, sed and awk. I chosen perl:

echo $myvar | perl -p -e '$_ = ucfirst'

However, python looks also nice:

echo $myvar | python -c "print raw_input().capitalize()"

Awk one is also nice, but a bit more complicated (one line):

echo $myvar |awk '{(sub("^.",substr(toupper($1),1,1),$1)); print }'

Very useful MacOSX key shortcut

The apple+tab key combination change between apps on MacOSX, but it works for one windows per application. What if you have more than one window for app? It is quite common when you are using X11 applications or try to use more than one terminal. To change between active window of one app you can use apple+', but it seems works only for windows on one desktop. It also not work for X11 app, but for them you can use apple+left/right arrows. One more thing. In some instruction related to MacOSX apple key is called comm and alt - option.

Povray export in Mercury CSD

One of the nice, new feature of Mercury CSD 2.0 is export to Povray. You can create high quality picture like one below. I used refcode MATCQI04 and Display->Voids and Calculate->Packing&Slicing option to produce the picture. If you are interesting in these as well other features of Mercury please check the article in J. Appl. Cryst. [DOI: 10.1107/S0021889807067908].

Unix tips

Today I would like to share two link related to UNIX. The first one is IBM tips how learn 10 good UNIX usage habits. The second one is The ultimate collection of UNIX cheat sheets by Scott Klarr.

Introductions not only to Quantum Chemistry

Supercomputer Institute at University of Minnesota presents set of PDF files being nice introduction to resources available there. Most of them are related to Quantum Chemistry, but there is also CSD System and Material Studio. If you are interested please check the link.

4 blogs - 4 maybe not so different subject

I would like to present 4 interesting blogs from 4 different area:

1. Bash Cures Cancer, how it is easy to guests, is a nice blog mostly about Bash tips&trick. However, you can also find infos about other FOSS.
2. I'm not sure if you can call Molecules of the Month @ 3DChem a blog. I think it's older than idea of blog (it has been started in 1996). The subtitle (molecules of the month) indicates one molecule per month, but molecules have been added in random manner - no molecule between October 2007 and January 2008 and 4 in January. Anyway, the choose of molecules are quite good and the additional information can be really useful.
3. Bad Astronomy Blog is a good blog about astronomy. I visited it first because of this entry, especially because of this picture. Just be aware that from time to time there are some politics/religion related entries.
4. Finally, Online Video Streaming Archive is looking promising place where Nature (one of the best/the best scientific journal on Earth ;) presents streaming videos that feature interviews with scientists behind the most important present research.

Dalton problems 2: Too long input record (ERI)

Problem

My CC calculations stopped with the forrtl: severe (22): input record too long, unit 9, file /tmp/niewod/RhbCCS/betaccs_rhb631+gd/CCSD_IAJB error message.

Answer

I changed four things and the job finished properly.

• First I changed a machine, but both were Itanium.
• Next I lowered the print level from 3 to 2.
• In the original jobfile I requested a direct calculation in a CC part, in the new one I put .DIRECT keyword in a main part (for all calculations).
• Finally I requested more memory (1900mb).

I got the proper results but the calculations end with error 32. It means forrtl: severe (32): invalid logical unit number, unit -10001, file unknown I reran jobs with new commands at the first machine, and it worked. Next I upgraded print level to 3 and job finished with success. So the issue could be cause by small amount of requested memory or direct/non-direct calculation in a HF,MP2 or CC calculation.

Files

• betacssi-old.dal
• betacssi-new.dal
• rhb631+gd.mol
• betaccs_rhb631+gd.out
• RhbCCS.e127210
• RhbCCS.o127210
• betaccs_h2co.out

Dalton problems 1: Direct and NonDirect HF

Short introduction

Once, I tried to create a website with notes on my Dalton's problems. I have had not time to upgraded it because Quantum Chemistry (and Dalton) has become much less important for me. Recently, I've decided that the best way to save my notes would be to add them as entries in my English blog. So they are.

Problem

I found difference between direct and non-direct Hartree-Fock results. The direct calculation didn't converge when non-direct did (look into the files below).

Answer

The reason of my problems was a very sharp convergence criteria. The screening in the direct SCF gives round-off errors, which makes it impossible to converge to 1.0D-10. (The default screening is 1.0D-14, which is usually safe, unless users ask for very sharp convergence,as I did!) Disable screening in the direct SCF, resolve my problem. I had to add: **INTEGRALS *TWOINT .IFTHRS 20 When convergence was set to 1.0D-09 the difference between direct and non-direct results disappear and both calculations converged in 28 iterations, with (nearly) the same energy:

• direct_rhb631+gd.out: -551.031670938128
• nodirect_rhb631+gd.out: -551.031670938119

I would like thanks Kenneth Ruud and Hans Jorgen Aa. Jensen for help.

Files

• nodirect.dal
• direct.dal
• rhb631+gd.mol
• direct_rhb631+gd.out
• nodirect_rhb631+gd.out

PC GAMESS benchmarks

Alex Granovsky presented the results of the PC GAMESS (version of Gamess US designed to work better on x86 architecture) benchmarks on new AMD Opteron and Intel Core2 processors. You can find them here.

The most important observations are:

• Intel CPUs are much faster,
• but AMDs scale better with number of cores.

BTW. You can find other interesting benchmark in the Performance section of the PC GAMESS website.

Tips for starting MacOSX apps

1. If you want to start a MacOSX applications from a terminal (or a shell script) you have to use the open command, i.e.: open /Application/Firefox.app (based on xahlee.org)
2. If you need to write a wrapper starting a binary file (which is inside of a MacOSX app) with some options you can use following construction: HERE=`dirname $0`; $HERE/name.x -options The same construction is useful if you need to set some environment variables.
3. Sometimes is it necessary to have a location of an app, but apps can be place in a very strange places. MacOSX binaries are located third level deep inside of an app, so the following command save location of an app in the TOPDIR variable: TOPDIR=`cd $HERE/../../../; pwd`

3D in Linux

There is a very interesting overview of present state (Nov 2007) of 3D graphic card drivers on Mitch Meyran's blog.

Two chemical links

This time two links to websites related to chemistry.

• The Supercomputing Institute from University of Minnesota prepared kit of Tutorials for Computational Chemistry and Physics Sciences.
• Vitalii Vanovschi created the website with many important chemical informations.

OpenGL and Windows

I'm starting to me more interesting in 3D graphic. Looking for some information about OpenGL on Windows I found Lucian Wischik old webpage. Maybe information collected on it isn't the latest news, but at least one thing is quite interesting. There is a register key which keep information about which library provided OpenGL functionality on your computer. HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\OpenGLDrivers. The another interesting finding is OpenGL Extension Viewer provided by RealTech-Vr. It works on Windows, MacOSX and under Wine.

Getting the Molcas CASPT2 exciation energy

This is the short bash/awk script printing an energy of excitation to a first excited stated obtain by CASPT2 method in Molcas6:

for i in *.log
do echo $i
grep "Total energy:" $i| gawk ' BEGIN {l=1} {if (l==1) {a=$3} else if (l==2) \
{b=$3; print 27.2097*(b-a)} l++} '
done

LaTeX font catalog

If you need nice, non-standard fonts for LaTeX visit Palle Jørgensen LaTeX Font Catalogue.

I Gited my PhD

From the beginning of writing my PhD thesis I've been using RCS to store revision of my tex files. I even add settings to my GVIM that Ctrl+S make a new revision. Recently, when I started to finishing Phd ;), I found lack information about files other than main tex files (tabels, figures etc). I also wanted to be sure that change prepered on one machines can be easily reach from other ones. I decided to use something more complex than RCS and chose git. It is very easy to send commits to my academic server (git push ssh://user@server/~/PhD) and getting the latest version from it (git pull ssh://user@server/~/PhD). However, I found that it isn't so easy to edit files on my server. Directory doesn't change after pushing it from remote repository. My first work around was on the server first committing changes - it means reverting files to state before pushing, and latter reverting to previous version -it means last version committed from remote machine. Does it sound complicated? Yes! So I was trying find the other way. Today I found it. On remote machine I do: git add filename(if needed) git commit -a git push ssh://user@server/~/PhD Later on server: git reset --hard HEAD And now I has the same files at server and any remote machines. Nice. And I need only ssh to do it.

Nekochan.Net

If you like IRIX and other old UNIX systems, Nekochan.Net is a place for you. There are a lot of software for IRIX, very interesting forum (not only about IRIX) and nice screenshots, like this one:

Comparing two list of files

I had to compare list of files, which was stored in two files. I wrote a small shell script.

for i in `cat ~buildman/WhiteList`
do
        grep $i ~buildman/BlackList
done

But beware it is taking each line of WhiteList as a substring. So if you have aaa in WhiteList that aaaaa from BlackList will be printed out.

Find

Unix find command is powerful tool. Recently, I learn how to use it to fix wrong permission of a whole directory tree (hundres of files). If you will be in similar situation try to use: find -nouser - looking for files (directories) not belongs to user. find -type f -not -perm 644 - looking for files with permision other that 644 find -type d -not -perm 755 - looking for directories with permision other that 755

10 nice shell command

Here you can find 10 nice, but not very other using shell command. Personally, I was using bc and lsof.

Solaris vs. Linux for Application Developers

At Sun Developer Network there is quite the article examines similarities and differences in the development environments between Linux and Solaris operating systems.

Another interesting blog about HPC

John West, the Director of the DoD Major Shared Resource Center at ERDC in Vicksburg, MS has created a blog where he discusses new developments in HPC. Check it out - very interesting info.

Geometries from Gaussian Scan

Reading geometries from Gaussian file storing information about Potential Energy Scan (PES) isn't easy. I was sure that there were some tools to do this, but I cannot find any yesterday, so I prepared my own. It is writing in python, of course ;). Right now have very basic features. It reads input file and created output xyz files (as many as stationary points was found). You are using it simply writing in command line:

scaner-0.1.py inputfile

Greping excitation energy from Gaussian output

I've already written about CIS energies, but my suggestion is more general. You can use that command also for TDDFT calculations. But if you need energies from perturbative correction of CIS method (CIS(D) method) you have to use this:

 grep "CIS(D) Exc\. E" *.log

'Symmetry' font in LaTeX

If you need nice font for symmetry group names use mathrsfs package and next (example for C2 group) $\mathscr{C}_2$ in the text. Of course you can use it for many other purpose.

Hack a Rox

I like Rox, but it has some annoying small problems. One of them is that the program which open the file is running from home directory. Now I write short script which fix it for gvim and I'm thinking how to do global...

#!/bin/sh
cd `dirname "$@"`
gvim `basename "$@"`

Vim's colorscheme

I've found nice site with sample of Vim's colorschemse. I miss only python samples.

XymTeX: example

I'm using XymTeX to creating chemical structures in PhD thesis. I'm not fully satisfe, but it create structures' counter , which can be use for numbering and refering to structures. Recently I created Reichardt Betaine subtituted by eter crown. The command which create this picture is below. Is nice, isn't it ;) \begin{XyMcompd}(800,2000)(,-1400){zw:korona}{}
\sixheterov[ace]{4==\upnobond{N}{$\oplus$}} %
{1==Ph;3==Ph;5==Ph; %
4==\cyclohexanev[ace %
{c\sixfusev[ %
{c\sixfusev[ %
{d\sixfusev{2==O;5==\null}{}{A}[ef]} %
]{1==\null;4==\null}{}{F}[de]} %
]{3==O}{}{F}[cde]} %
{d\sixfusev[ %
{d\sixfusev[ %
{c\sixfusev{3==O;6==O}{}{F}[ab]} %
]{1==\null;4==\null}{}{A}[bc]} %
]{5==O}{}{A}[bcd]} %
] %
{4D=={\hspace*{-1.5mm}$| \underline{\mathrm{O}}|^{\ominus}$};1==(yl)}}
\end{XyMcompd}
If you don't like XymTeX you can try konwerter very nice program by Piotr Wawrzyniak.

Energy converter v0.1

I've just created small Python/Tkinter program to convert values of energies. At the moment you can convert from (to) au, eV, nm and cm^-1. Not much, but it is beginning :). Program can be download from mml website.

How to list files in subdirectories

I wrote a shell script which allow to list files and only files in all of subdirectiories of current directory. I don't know if it useful but it nice piece of shell code:

#!/bin/sh
for i in `find $PWD -maxdepth 1 -type d`
do
    (cd $i; echo $i; find -maxdepth 1 -type f| wc -l)
done

The Python Challenge

I nearly become addicted. To what? To Python Challenge. It is a set of puzzle which can be resolved with Python (you can use other language, but it is design for Python). I don't remember when I have so much fun. It is great think if you what to learn Python or just test your skills. It is my 50th note and I started write this blog 1 year and 3 days ago!

Molekel 5.0 alpha

I wrote that Molekel was rebirth and version 4.6 was release. Recently, I found that the project has been moved and alpha version of Molekel 5.0 can be found hear. So, right now we have:

1. 4.3. It is very old version and the sources are close. [website]
2. 4.6. The whole project has been rewritten and it is alpha version, not all functions from 4.3 version are working [website]
3. 5.0 New, not ready yet version with open source (GPL). [website].

Physics for 'normal'

I found another interesting blog related to science. This time some physics. You can find there a lot of interesting news: i.e. the info about physics of new NBA ball or fastest waves ever photographed.

Loooong bash command

I like bash command. This one is quite long:

for ((i=1;i<36;i++)); do if [ $i -lt 10 ]; then j=0"$i" ; else j="$i" ; fi ; wget --no-check-certificate https://www.linux-magazine.pl/issue/$j/; done

It could be interesting for someone, because in one line there are loop (C-style) and if command. This command download the contents of all Polish LinuxMagazine issues. I have got most of LM issues, but searching some information thru them are quite boring. So I thinking about making some kind of 'searching machine' because one from LM website is weak. My command once more, but this in more readable form:

for ((i=1;i<36;i++)); \
do \
 if [ $i -lt 10 ]; \
    then j=0"$i"; \
    else j="$i" ; \
 fi; \ 
 wget --no-check-certificate https://www.linux-magazine.pl/issue/$j/; \
done

Mencoder (part II)

My second example of using mencoder: I had the problem with large avi file which didn't fit CD (it had 1.4 gb). I used mencoder to split this file(console rulez ;). Creating of the first part was quite easy (I found the advices here and here).

mencoder -ovc copy -oac copy -ss 0 -endpos 700mb -o movie_part1.avi movie.avi

But there were no info where I should start the second part. I watched the end of the part 1 and manually (eyelly ;) chose the best point to split the movie (for example 1:05:25). This time I could creat two parts.

mencoder -ovc copy -oac copy -endpos 01:05:25 -o movie_part1.avi movie.avi
mencoder -ovc copy -oac copy -ss 01:05:25 -o movie_part2.avi movie.avi

Mencoder (part I)

I like console. So I'm using it even to manipulate video files ;). My first example of using mencoder: I'm recoding files from my camera (Fuji FinePix 5600/5200). It use two passes technique (it should give smaller out file). I don't know if my options are optimal, I found it somewhere (I don't remember where). The input file is the first argument and output file is second argument of my script.

#!/bin/sh
mencoder $1 -ovc lavc -lavcopts \
vcodec=mpeg4:vpass=1:vbitrate=1000:vlelim=-4:vcelim=9:lumi_mask=0.05:dark_mask=0.01 \
-oac copy -o /dev/null
mencoder $1 -ovc lavc -lavcopts \
vcodec=mpeg4:mbd=2:trell:vpass=2:vbitrate=2000:vlelim=-4:vcelim=9:lumi_mask=0.05:dark_mask=0.01 \
-oac copy -o $2

Sailing through the net (I tried to find the place with mencoder option) I reached Mario Valle website. It seems very interesting. I have to read it careful and write more about it. Right now, I'm adding his blog to my list.

SwapFile

Most of people (even Klauss Knopper [1]) think that swap partition has better performance compare to swapfile (Ok. Most of people don't know what is swap file or swap partition ;). However, it isn't true, at least for 2.6 linux kernel. Check thread at Linux-Kernel Mailing list [ 2]. I think the most important is the 3rd point
Does creating the swapfile on a journaled filesystem (e.g. ext3 or reiser) incur a significant performance hit?
and answer
None at all. The kernel generates a map of swap offset -> disk blocks at swapon time and from then on uses that map to perform swap I/O directly against the underlying disk queue, bypassing all caching, metadata and filesystem code.
[1] Klaus Knopper, "Questions and Answers", Linux Magazine Polish ed., 3(25) March 2006, pp 38-40.
[2] Andre Morton et al., "Swap partition vs swap file" thread Linux-Kernel Mailing List, July 2005.

Camera.sh v2

I rewrite my script for converting raw format pictures. Now, it is converting files from raw to jpg in loop. Doing first convert of all raw files take a lot of space (ppm files are BIG). Right now I need space for one ppm file. Morover, ppm->jpg converting and resizing are made in one step. I found very nice perl program (ExifTool) to managing exif info.

#!/bin/sh

EXIFPATH=/opt/Image-ExifTool-6.36
for raf in *.raf
do 
        dcraw -w $raf
        name=`basename $raf .raf`
        jpg=`basename $name`.jpg
        ppm=`basename $name`.ppm
        echo $name
        convert -resize 800x600 $ppm $jpg
        $EXIFPATH/exiftool -TagsFromFile $raf $jpg
        rm $ppm $jpg"_original"
done

Overwriting a line in Vim

Finally, I've learn how to overwrite the whole line in Vim. First you have to yank the line you would like to copy (yy), next to mark the destination line (V) and paste the first line (P or p on it), what gives VP. I found it here.

Blog with many science video lectures

Sometime ago I found Free Science and Video Lectures Online!. It's the place where Peteris Krumins try to list video courses which are related to physics, mathematics, chemistry etc. from all over the world (MIT, Berkeley).

Fonts in Linux

There is great HowTo about fonts in Linux.

ls and find

I'm all the time learning new things about unix command. Recently, I found new feature of ls and find command.

ls -Ssh
-S lists files and directories in size order,
-s prints size,
-h does it in human readable way.

find -mtime -1
-mtime looks for files modificated later/earlier/exactly n days from now (in my example today and yesterday).

Benchmarking Filesystems - part II

There is another article benchmarking Linux's filesystems. And the winner is XFS - again.

Mcabber

I've found very nice, small but powerful console jabber client. It's name is Mcabber. I also created its port for Crux.

My dalton's problems

I created a website, where I share some info about my problems with Dalton. At the moment there is only one point (problems with HF direct convergence).

Full info about your jobs in PBS

To get full info about job in PBS you have to use -f option, but it doesn't work with -u $USER option, so you get a list of all jobs, or you have to know yours job id. To get around this problem I created this one line 'script':

qstat -f `qstat -u $USER |grep $USER |awk {'print $1'} |sed 's/[a-z.]//g'`

• qstat -u $USER - create the list of all user jobs,
• grep $USER - cut lines with usernames,
• awk {'print $1'} - prints only column with job id (first one),
• sed 's/[a-z.]//g' - delete all letters and dots from it.

After that you have only id numbers which can be use with qstat -f. You can extent this command further, i.e. to get info about host where you job is running, by adding grep host (it can be very useful at a cluster).

CCLib

The new version (0.5) of CCLib has been release. CCLib is an open source library, written in Python, for parsing and interpreting the results of computational chemistry packages. It currently parses output files from ADF, GAMESS (US), Gaussian, and PC GAMESS.

gaussfield.sh v0.1

It can be very useful to obtain value of some molecular properties in electric field. I.e. you can get dipole moment of excited states from energies in fields F and -F, hyperpolarizability from analytic polarizability etc. But it's quite annoying to create many very similar Gaussian input files, so I wrote short script to created this files:

#!/usr/bin/env bash
for i in $*
do
        name=`basename $i .com`
        sed -e '/^%/d; /^#/s/$/ Field=X-10/' $i > ${name}-x.com
        sed -e "/^%/d; /^#/s/$/ Field=X+10/" $i > ${name}+x.com
done

No CIS(D) in solvent [Gaussian]

I tried to calculate excitation energy with CIS(D) method in solvent (PCM method). I found that the CIS resalts from CIS and CIS(D) outout are different. I checked and CIS with Direct and without this option gave another results. I wrote to Gaussian Help and got replay: Yes, there are differences and the CIS=Direct results are correct.The non-direct methods, both CIS and CIS(D) use MO integrals where the solvent effects are not present and thus neither of them will give the correct results. Starting with G03 Rev. D the program correctly forces the use of CIS=Direct when combined with solvation. There is no option for doing CIS(D) with solvent and I don't expect this to change. Sorry for any confusion.

Deleting needless pbs files

When PBS queue system's finished your calculation was leaving two info file: [name].o[number] and [name].e[number]. If results were correct you don't need them. To delete all pbs file (and only them^*) use this command:

rm -f *.[eo]*

^* Of course if you have file with e or o after dot it will be delete too. In this situation you can upgrade command to

rm -f *.[eo][0-9]*

. However, if can fail too if there is number after o or e letter ;). But It's rather unusual situation. You can add this line to your .bashrc file:

 alias pbsdel="rm -f *.[eo][0-9]"

. Now you can use pbsdel to deleting pbs file.

Massive changes in input files

Imagine, you have to change basis set in 10 files. It is a lot of work. Not really, sed will be your friend. For example, I would like to change 6-31+G(d) into 6-311++G(2d,p) basis set in Gaussian, so I did:

sed -i.bak *.com -e " s/6-31+G(d)/6-311++G(2d,p)/"

. This command created backup of my file. If you would like to change the level of theory in gamess you should write:

sed scf.inp =e "s/scftyp=rhf/scftyp=rhf mplevl=2/" > mp2.inp

Of course you can change much more thinks in any of quantum chemical program.

FujiFilm S5600 digital camera and [not only] Crux

My FujiFilm S5600 (S5200 in US) works out of box with my Crux. The only inconvenience is that the mount point can depend on presences of other usb stuff. I read the udev rules manual and wrote this rule.

BUS=="scsi", KERNEL=="sd*", SYSFS{vendor}=="FUJIFILM",  NAME="%k",SYMLINK="fuji"

It makes the /dev/fuji link, which use as mount point:

/dev/fuji /media/aparat vfat  user,noauto   0    0

To edit pictures from raw format im using UFRaw. It can be GIMP plugin, what is it's big advantage. But when I have more photos I'm using this small bash script to prepare preview in jpg:

#!/bin/sh
dcraw -w *.raf
for i in *.ppm
do
    j=`basename $i .ppm`.jpg
    echo $j
    convert $i $j
    mogrify -resize 800x600 $j
done
rm *.ppm

I found that it is very important to use dcraw with -w option.

NLO in Gaussian 98/03

Sometime ago Damian Gregory published very nice article about NonLinear Optical (NLO) properties in Gaussian. Later he moved his website into wordress and forgot to transfer this article. Right now "Gaussian98/03 Nonlinear Optical (NLO) Response" is back. BTW. Damian's blog is very nice place for all interesting in so called NanoThechnology. His Nanotechnology Gallery is beautiful.

Molekel

I've just check the website of Molekel one of the most interesting program to visualization Quantum Chemistry date. An there are good news. Yes, they are working on the new version. And Yes, it is GPL program now!

How to convert color pdf (ps) to b&w file

I have a pdf file, where foreground color is blue. It is very nice when you look at it at your monitor, but no so nice when you have to print it on b&w printer. Today I found the way how to convert a color pdf (or rather ps) into a b&w file. First you have to convert a pdf into a ps

pdf2ps file.ps

Printing into a file from Acrobat Reader doesn't work. Next you have to download this small perl script and use it

 bw_convert -b file.ps file-bw.ps

Now you have file ready to print on your b&w printer.
I found link to bw_convert script on this FAQu. There are more interesting advices.

Regular expression [βzzz from Dalton]

Very often you need only one number from whole output. Going through all the file is boring, especially if you have more then one file. In these situation regular expression are a great help. For example: I needed to find out the value of β_zzz. I figured out from an output file that I needed lines with 3 ZDIPLEN words. I prepared the command which should work:

grep "Z.*Z.*Z" *.out

But I got more than I wanted:

betaccs_rhb-631+gd.out:  PUT TRIPLE: ZDIPLEN  ZDIPLEN  ZDIPLEN ON THE LIST. 
betaccs_rhb-631+gd.out: ZDIPLEN  (unrel.) 0.0000  ZDIPLEN (unrel.) 0.0000  ZDIPLEN (unrel.) 0.0000  -383.12517

So I modified my expression (brackets are only in wanted lines):

Z.*(.*Z.*Z" *.out

BTW. If you want to obtaine the correct value of β you'll have to multiply value from output file by -1.

rsp_zfs and gcc 3.4.x [Dalton]

In Dalton FAQ it is written that "g77-3.4.2 (Fedora Core 3) miscompiles DLAMC3 routine from gp_zlapack.F file leading to apparent hang in rsp_zfs test". I had the same problem (hangs of rsp_zfs) on my CRUX with gcc-3.4.4. Adding of -ffloat-store option helps. Moreover, after adding this option Dalton has been build without and failed tastes on Athlon.

Points style in Gnuplot - part II

My friend Paweł Kędzierski showed me another (better?) way how to present points style in gnuplot. It is describe in docs. (Shame on me!) The ps_symbols.gpi script can be found in gnuplot docs directory or hear. Below you see output from it: BTW I would like to add link to another nice site with gnuplot images. It was created by Petr Mikulik.

Points style in Gnuplot

It's quite annoying to find the best points styles for your postscript output from gnuplot. I tried to find the site with list of all points style, but I failed, so made my own:

set term postscript
set output "test.ps"
set xrange [0:30]
set yrange [0:30]
set key right bottom
plot   x+1   with linespoints pt 1 title "1",  
       x+2   with linespoints pt 2 title "2",
       x+3   with linespoints pt 3 title "3",
       x+4   with linespoints pt 4 title "4",
       x+5   with linespoints pt 5 title "5",
       x+6   with linespoints pt 6 title "6",
       x+7   with linespoints pt 7 title "7",
       x+8   with linespoints pt 8 title "8",
       x+9   with linespoints pt 9 title "9",
       x+10  with linespoints pt 10 title "10",
       x+11  with linespoints pt 11 title "11",
       x+12  with linespoints pt 12 title "12",
       x+13  with linespoints pt 13 title "13",
       x+14  with linespoints pt 14 title "14",
       x+15  with linespoints pt 15 title "15",
       x+16  with linespoints pt 16 title "16",
       x+17  with linespoints pt 17 title "17",
       x+18  with linespoints pt 18 title "18",
       x+19  with linespoints pt 19 title "19",
       x+20  with linespoints pt 20 title "20"

The result is below. It was obtained by: convert test.ps test.png (and next rotated):

Vim/Vi Cheat Sheet and Tutorial

Deelopers of ViEmu published very nice help for Vim. I've missed something like this.

Medical Computing

Alex Amies is writing the blog about Medical Computing. It can be interesting for some people hitting my webpage.

Dipole moment in excited state [Gaussian]

To get values of dipole moment obtain with CIS method using Gaussian program you can use this bash command:

egrep "Tot*=|Population" *.log

From all of line you need only the line after Population analysis using the CI density. You can automate it. For example to get the total value of dipole moment use this script (you can write it in one line):

grep "Tot*=" *.log|
 gawk '  BEGIN { currline = 1 }
  {if (currline%3==0){print $9}
  currline = currline + 1}'