(w)dstat

wdstat

In my .profile (on CentOS 5, just in case there were some changes in dstat) I have following alias to dstat (wdstat stands for Wawrzek's dstat):

alias wdstat="dstat -lcpymsgdn 5"

Where the options stands for:

• -l  - UNIX load (1m   5m  15m) load average in 1, 5 and 15 minutes, respectively;
• -c - cpu stats (usr sys idl wai hiq siq) percent of time spent in user and system space, idle, waiting on resource,  serving interrupts and softirqs (software interrupts);
• -p - process stats (run blk new) number of running, blocked and newly created processes;
• -y - system stats (int   csw) - number of interrupts and context switches;
• -m - memory stats (used  buff  cach  free) amount of memory used by processes, disk buffers, disk cache and free;
• -s - swap stats (used  free) - amount of used and free swap space;
• -g - page stats (in   out) number of page put in and out from swap;
• -d -disk stats (read  writ) - number of reads and writes from all disks;
• -n -network stats (recv  send) number of received and send network packages;

Further reading:

• http://dag.wiee.rs/home-made/dstat/
• http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
• http://www.linuxhowtos.org/System/procstat.htm
• http://lwn.net/Articles/520076/
• http://en.wikipedia.org/wiki/Paging#Linux

netstat, ports, hosts and awk glue

Recently, I needed to create a list of all servers connected on a given port (in following example port 80). I used a mixture of awk and other UNIX command line tools.

netstat -nt| \
 awk -F':'\
   '$5==80 {count[$8]++} \
   END{ for (i in count) { \
      cmd="host "i; \
      cmd |& getline j; \
      split(j, a, " "); \
      printf "%40s - %d\n", a[5], count[i]}}'| \
 sort -n -k 3

First netstat provided the list of all connection (netstat -nt); -n stands for numeric and -t for only TCP connections.

Next awk, with the ':' defined as a field separator (awk -F':'), used lines where local port was 80 ($5==80) to create an associated array with a key define by connected host ip and a value equal to  number of connection from it ({count[$8]++}). At the end of the script execution, awk looped over all element of the array (END{for (i in count)).  Next there was a crux of the script, the cmd was define as a run the OS host command with the awk variable i as an argument  (cmd="host" i). The |& operator created two-way pipe between awk and a execution of the previously defined cmd. The getline command was used to store cmd output into the variable j (cmd |& getline j). Next the split command split the content of the  j into separate words and saved them into the a array (split(j, a, " ")). Finally the printf formatted output (printf "%40s - %d\n", a[5], count[i])).  The actual hostname was fifth element of the a.

For continence, output lines were sorted by numeric order on third column  (sort -n -k 3). Each output line consisted of a hostname ,'-' and a number - e.g. important.com - 3456.

Python for SysAdmins

Preparing for a interview some time ago I made a list of python module interesting for SysAdmins. Today looking for something else I found that half baked note and decided to polish it enough to put it on the blog. It's mostly for myself as quick reference.

Each module is describe by one, two sentences (from Python documentation) and have a link to official online document. At the end there is a list of example function, object.

Python modules for SysAdmin

import sys

This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is always available.

https://docs.python.org/2/library/sys.html

examples:

• argv,
• exit(),
• path,
• modules,
• exec().

 

import os

This module provides a portable way of using operating system dependent functionality.

https://docs.python.org/2/library/os.html

examples:

• chdir(),
• getuid(),
• uname(),
• listdir(),
• stat(),
• rename(),
• access().

 

import os.path

This module implements some useful functions on pathnames.

https://docs.python.org/2/library/os.path.html

examples:

• isdir(), 
• isfile(), 
• exist(), 
• getmtime(), 
• abspath(), 
• join(), 
• basename(), 
• dirname().

 

import time

This module provides various time-related functions.

https://docs.python.org/2/library/time.html

examples:

• time(), 
• ctime(), 
• sleep(), 
• strftime(), 
• strptime().

 

import glob

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.

https://docs.python.org/2/library/glob.html

examples:

• glob(),
• iglob().

 

import fnmatch

This module provides support for Unix shell-style wildcards, which are not the same as regular expressions (which are documented in the re module).

https://docs.python.org/2/library/fnmatch.html

examples:

•  fnmatch().

 

import re

This module provides regular expression matching operations similar to those found in Perl. Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings.

https://docs.python.org/2/library/re.html

examples:

• compile(), 
• match(), 
• search(), 
• split(), 
• findall(), 
• sub(),
• group().

MatchObject

Match objects always have a boolean value of True. Since match() and search() return None when there is no match, you can test whether there was a match with a simple if statement:

match = re.search(pattern, string)
if match:
    process(match)

Zombie

Usually zombie process is not a big problem, but sometimes... just look at the screenshot below. It wants the whole machine as fried eggs! or maybe boiled?

Multiprocessing (in Python)

I needed to do some multi-threading in Python. As I needed effect quick I decided to use standard threading module. However, every time I had to use it I felt it was rather complicated beast. At threading module documentation page there is a link to multiprocessing module. I was tired with threading, on the one hand, but didn't have enough time to learn about greenlet or another competing project, on the other, so I decided to take a quick glance at multiprocessing module...
... And my life became much easier, sky bluer, grass, greener, oh and scripts faster ;-).
I don't do anything special with it, so just one simple code example, but this is very good tutorial you can find much more: http://pymotw.com/2/multiprocessing/communication.html.

Main part of nearly all my scripts looks the same:

import multiprocessing 

SIZE = 30
pool = multiprocessing.Pool(processes=SIZE)
pool_output = pool.map(get_values, servers)
pool.close() # no more tasks
pool.join()

Where servers is a list with servers I need get information from, and get_values is a function (sometimes with a different name).  Simple, isn't it?

shell, history and substitution

One of the most known tricks in using shell history is to use:

^old^new

to replace string old by new in last command. The only problem is that it replaces only first appearance. But there is another command replacing all string old by new:

!:gs/old/new/

Space War

Let say it is a late Christmas present for Science Fiction fans. Especially ones who like a lot of science in SF. Two articles discussion how space warfare can look. Have fun.

• www.foreignpolicy.com/articles/2012/09/28/aircraft_carriers_in_space
• http://gizmodo.com/5426453/the-physics-of-space-battles

Even more threads counting

This is small extension to one of my previous posts. This time a loop is enriched by load values (from /proc/loadavg) as well as measurement time (date).  ps command uses the same option, but there is small improvement in awk call.  Rather than count only processes per state it concatenate state with last string in command arguments — I was mostly interested in few java application and jar name was the last parameter for each one. Also awk counts only processes actually running or in uninterruptible sleep ($2 ~ /(D|R)/ at the beginning of awk command).

while [ 1 ];
do
    date;
    cat /proc/loadavg;
    ps -Leo pid,state,args |
     awk ' $2 ~ /(D|R)/ {state[$2 " - (" $1 ") " $NF]++} \
      END{ for (j in state) {printf "%s - %d\n", j, state[j]}}' |
      sort -k 2;
    echo "---";
    sleep 5;
done

Debian, Vim and alternatives

I don't know why, but default editor in Debian is not Vim, but Nano. I learned once that it can be change with a command, but has never remembered the actual name of it and since than had to check it each time in install Debian (not so often). Today decided to write it down, not to search too long next time. Maybe it's going to be useful to others:

aptitude install vim.nox
update-alternatives --set editor vim.nox

e17 and stable Debian

If you are interested in using e17 with Debian Wheezy, the best way is to use package provided by John Holland at http://www.vin-dit.org/.

Number of threads per state

awk with ps creates powerful Linux combo. For example there is one, or more, multi-threaded application on your system. You would like to know what states all those threads are. Standard ps show information about processes, but you can learn more about threads with "-L" switch. You probably hit another issue - number of threads, you don't want to analyse, let say, 300 lines. Lucky, awk is very good for such tasks. Following 'one-liner' shows number of threads and processes per state (running, sleeping, uninterrupted sleep), every 5 seconds.

while [ 1 ];
do
    ps -Leo pid,state,args| \
     awk '{state[$2]++} \
      END{ for (j in state) {printf "%s - %d\n", j, state[j]}}';
    echo "---";
    sleep 5;
done

First line ensures that command runs forever. ps uses '-e' (all processes), '-L' (include threads), '-o' (user specify output). There are 3 lines in ps output: PID, state and command arguments. (These arguments are useful for my other commands. In this case you can leave only 'state', but you has to modify awk command). Next awk takes every line and counts each state using state array with argument equal to state representation (i.e.: S for sleep, D for uninterrupted sleep, R for running). At the end of execution it prints each argument with it value. After that script displays '---'  and waits 5 seconds.
Such output can be valuable extension to top command results. It shows instant changes to number of busy processes, which affects system load.

CPU affinity, interrupts and old kernel bug

This is an old story and AFAIK were address ages ago in Linux kernel, but there are still plenty of CentOS/RHEL 5 running and you still might be hit by the problem with kernel not properly balancing interrupts from hardware, so the that below list of the links my be helpful. In bug report network card are mentioned, but it might affect other components (i.e. raid controllers from HP - cciss drivers).

If you would like to learn more there is a set of good links:

1. https://bugzilla.redhat.com/show_bug.cgi?id=432451
2. http://docs.oracle.com/cd/E24290_01/coh.371/e22838/tune_perftune.htm
3. http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
4. http://www.alexonlinux.com/msi-x-the-right-way-to-spread-interrupt-load
5. http://www.alexonlinux.com/why-interrupt-affinity-with-multiple-cores-is-not-such-a-good-thing

Regexp Groups in Python

Let say that we have a string consist from a numeric continent code  (i.e. 1 for Europe) , followed by a country code  (i.e. 44 for UK)  and ended with a region code made in similar manner (i.e. Cambridge equal 65). A continent and a country are preceded with the letter C and a location with the letter L.

So we have the string C1C44L65 and we need to know the continent, counter and region codes, but numbers might have different length (i.e. 5, but also 55 or 555) so we cannot just grab selected characters from a string, but in Python we can use following regular expression to save all that information into "groups" for further use, in example in an mathematical operations (as integers).

info = re.search(r"^C(?P[^\d]+)C(?P\d+)L(?P\d+)$", full_id)
continent = int(info.group("continent"))
country = int(info.group("country"))
region = int(info.group("region"))

Ubuntu and mix of NFS, NIS and autofs in post from past

Some time ago I had a problem with autofs, NIS and NFS during Ubuntu start. As far as I remember autofs was responsible for serving users home directories, define with NIS and  provided from remote location with  NFS. The problem that autofs started before network and NIS, so home directories weren't ready.

When I started look into the problem found that small change to autofs.conf should resolve the situation. I didn't deploy the fix on big scale (I were using CRUX on my desktop), but saved the patch to blogger and today finally found some time to share it. Maybe it will be useful for anyone.

root@test-upstart:/etc/init# diff -u autofs.conf ~wawrzekn/autofs.conf
--- autofs.conf2010-08-17 11:34:47.000000000 +0100 
+++ /home/wawrzekn/autofs.conf2011-02-06 12:19:48.678663000 +0000 
@@ -10,6 +10,15 @@ 
 respawn 
  
 pre-start script 
+while : 
+do 
+ypwhich 
+if [ $? = 0 ]  
+then  
+break 
+fi 
+sleep 4 
+done 
 if [ -f /etc/default/autofs ]; then 
 . /etc/default/autofs 
 fi

PS2 and friends

This post won't be about PlayStation 2, but about my (not so) recent problem with the UNIX shell prompt.

I had a problem with prompt on some remote Linux system. Everything looked fine until the command (with arguments) fitted one line. I was using iTerm2 on MacOSX with full screen mode, so line could be long, but anyway it started to be very annoying. When I tried to move cursor back or jump to the beginning of the line it could end up in the middle of existing text.

I spent a lot of time trying to debug problem. First I tried to find something wrong with my iTerm2 settings, but could not find a thing. Next I started to play with tput (see link 1 and 2). For a moment believed that I had broken my PS2 setting (see 3) and finally, after reading link 4, figured out that I hadn't properly escaped non printing character in colour settings. My 'remote' bash prompt looks currently as printer below and you can see a lot of \[ before not printing characters and \] after them.

\[\e[32m\]\h \[\e[0m\]: \[\e[33m\]\w \[\e[0m\]#>

Reading:

1. http://tldp.yolinux.com/HOWTO/Bash-Prompt-HOWTO.html
2. http://tldp.org/LDP/abs/html/terminalccmds.html
3. http://www.thegeekstuff.com/2008/09/bash-shell-take-control-of-ps1-ps2-ps3-ps4-and-prompt_command/
4. http://www.ibm.com/developerworks/linux/library/l-tip-prompt/

Fabric as a python module

Introduction

Fabric is a powerful tool to automate running the same command on many remote server. It is written in python, but authors concentrate on usage as command line tool rather as a python module.

There are some documentation how to do this. You can also find some examples in the internet, but  examples never too much, so below another one.

Scenario

We would like to run a command on each of our webservers, ideally in batches, but in a specific order. Our webserver are grouped in 'cells'. Each cell consist of 3 servers in different 'zones' (zone are called a, b, and c). For convince there is following convention of naming servers: "role-cell#-zone" i.e. web-04-a. To make things more interesting some cells are missing, so we cannot make simple loop. On the other hand, there is a service holding various servers information in json file accessible over http. Additionally, we have a python module called 'server' to locally access those information from our workstation. In the 'server' module we have class 'webserver' to deal with webservers. One of its method is 'get_all_server'. It return a list of all webserver.

Script

Servers

First we create the list of all servers using our external tool.

from servers import webserver
allservers = webserver.get_all_severs()


The order mentioned above is to run first all 'c' boxes next 'b' and finally 'a'. To achieve that, let creates 3 lists (in a function). Nothing magic, a bit of 're'.

def make_lists(servers):
    import re # if not imported in the main part

    wa = []
    wb = []
    wc = []
   
    for server in servers:
        if re.search('web-\d+-a', server):
            wa.append(server)
        elif re.search('web-\d+-b', server):
            wb.append(server)
        elif re.search('web-\d+-c', server):
            wc.append(server)

    separation = (wa, wb, wc)
    return separation


Fabric

So now we can start to use fabric. Let import necessary pieces:

# Fabric part
from fabric.api import *
from fabric.network import disconnect_all

set up environment

env.skip_bad_hosts = True
env.warn_only = True
env.parallel = True


and finally call the function:


@parallel(pool_size=10)
def run_remotely():
    with hide('running', 'status'):
       sudo(task)


Main part

Prepare the lists of servers (see paragraph servers):

lists = make_lists(get_servers())

Get the command to run. I don't remember why, but writing my script decided not to pass variable to 'fabric; function, but use global variable.

global task
task = raw_input("Type/paste command to run >")

Now run the provided command in selected order:

for i in [3, 2, 1]:
    execute (run_remotely, hosts=lists[i-1])


For any case disconnect all connection. (It might be not necessary).

disconnect_all

My way for binding ssh-agent with zshell

The following lines I put into my .zshrc might not be very pretty, but it is short, seems to work and I don't need to use any of graphical tools to ssh somewhere with my key (with passphrase). So it works fine with my zshell and E17.

SSHPID=`ps ax|grep -c "[s]sh-agent"`
if (( $SSHPID == 0 ))
then
    ssh-agent > ~/.ssh-env
    source ~/.ssh-env
    ssh-add
else
    source ~/.ssh-env
fi

CakePHP tutorials on TuxRadar

LinuxFormat presented a some time ago interesting tutorials to CakePHP. What even more interesting their share materials on TuxRadar webpage. The only problem I found is lack of some kind of list of contents, so I made one:

1. CakePHP Tutorial: Build Web Apps Faster
2. CakePHP Tutorial: Storage, Baking and Slugs
3. CakePHP Tutorial: Build a file sharing application
4. CakePHP Tutorial: Build a bookmark site

BTW. IF you hit this page you might be also interested in Practical PHP Programing tutorial from the same page.

List of VMs on XenServer (with UUIDs)

Xen XE command is a very powerful tool, especially if you bind it with other UNIX tools such as awk. Let say we need a map of vm uuids and theirs names. We can use an output from the vm-list command and parse if with awk to get desire result.

xe vm-list | \
awk '{if ( $0 ~ /uuid/) {uuid=$5} if ($0 ~ /name-label/) \
{$1=$2=$3="";vmname=$0; printf "%s - %s\n", vmname, uuid}}'

The script first save the fifth column from a line having uuid string in it into the variable uuid. Next it saves all columns, after the third one, from line having name-label into variable vmname. Finally it prints both variables.

The exemplary output:

ukweb2 - fbca0851-35de-2963-bf0c-7980f3c0d96f
nagios - b741def2-14cc-def4-f8ba-ff0d3ed741d9
ukmail1 - 343c8f93-e4db-d0df-bc30-7544fcd6f14e
jira - ecc3241f-ac14-0398-4e44-ba96cd1d51d2
dodb-02 - 7f223172-e43e-a200-6dc6-b108ce4f9166
RTST-Witness Server - 3c236b0a-209f-6ac9-6d46-b14f7678bfa6
hub-01 - 60ef767c-9b87-edf8-9f13-af2185e656cd
ukweb1 - 6e0e4622-ddfe-0db8-a128-f432e05565cb
dns2 - d65e40d4-ea21-1cbf-cc86-9f522f5e04ef
ixchariot - 73f78129-86db-fd9f-81b4-85768eeee487

We can modify our command to prepare a list of all host with vms bind to them. This time we use xe vm-list with params=all option. The scripts searches for lines with the name-label and saves a name (third column). Next it looks for lines with the word affinity and a uuid (we know that UUID have to start from a hexadecimal number) and prints a saved name.

xe vm-list params=all| \
awk '{if ( $0 ~ /name-label/) {$1=$2=$3=""; vmname=$0} \
if ($0 ~ /affinity.*\:\ [a-e,0-9]/) {host=$4; printf "%s \n", vmname}}'

The output might looks similar to:

Control domain on host: p1-m4 
   Control domain on host: p1-m2 
   dodb-02 
   ukweb5
   Control domain on host: p1-m3 
   Control domain on host: p1-m1 

You might wonder why the list is so short, but we have the list of machine enforce to start from a given host (affinity to a given UUID). If you have machine on share storage allowed to flow between machine you should get very short list indeed.

Crux and Mercurial view

I've spent quite some time trying to make hg view to work on my Crux. All time I got error message:

hg view /bin/sh: hgk: command not found

I could not understand what going on. I enable hgk in /etc/mercurial/hgrc or ~/.hgrc. I specified the full path to hgk.py in there as well. I even modify default python path. It didn't work.

After some time of googling, changing various variables I found somewhere (probably on Mercurial page), that some Linux distro missing hgk even if they provide hgk.py. Now I know that Crux is one of them. I copied hgk from contrib directory in source package to /usr/bin and now hg view works fine.