Search This Blog

Thursday, November 22, 2012

Fabric as a python module


Introduction

Fabric is a powerful tool to automate running the same command on many remote server. It is written in python, but authors concentrate on usage as command line tool rather as a python module.

There are some documentation how to do this. You can also find some examples in the internet, but  examples never too much, so below another one.

Scenario

We would like to run a command on each of our webservers, ideally in batches, but in a specific order. Our webserver are grouped in 'cells'. Each cell consist of 3 servers in different 'zones' (zone are called a, b, and c). For convince there is following convention of naming servers: "role-cell#-zone" i.e. web-04-a. To make things more interesting some cells are missing, so we cannot make simple loop. On the other hand, there is a service holding various servers information in json file accessible over http. Additionally, we have a python module called 'server' to locally access those information from our workstation. In the 'server' module we have class 'webserver' to deal with webservers. One of its method is 'get_all_server'. It return a list of all webserver.

Script

Servers

First we create the list of all servers using our external tool.

from servers import webserver
allservers = webserver.get_all_severs()


The order mentioned above is to run first all 'c' boxes next 'b' and finally 'a'. To achieve that, let creates 3 lists (in a function). Nothing magic, a bit of 're'.

def make_lists(servers):
    import re # if not imported in the main part


    wa = []
    wb = []
    wc = []
   
    for server in servers:
        if re.search('web-\d+-a', server):
            wa.append(server)
        elif re.search('web-\d+-b', server):
            wb.append(server)
        elif re.search('web-\d+-c', server):
            wc.append(server)

    separation = (wa, wb, wc)
    return separation


Fabric

So now we can start to use fabric. Let import necessary pieces:

# Fabric part
from fabric.api import *
from fabric.network import disconnect_all


set up environment

env.skip_bad_hosts = True
env.warn_only = True
env.parallel = True


and finally call the function:


@parallel(pool_size=10)
def run_remotely():
    with hide('running', 'status'):
       sudo(task)


Main part

Prepare the lists of servers (see paragraph servers):

lists = make_lists(get_servers())

Get the command to run. I don't remember why, but writing my script decided not to pass variable to 'fabric; function, but use global variable.

global task
task = raw_input("Type/paste command to run >")


Now run the provided command in selected order:

for i in [3, 2, 1]:
    execute (run_remotely, hosts=lists[i-1])


For any case disconnect all connection. (It might be not necessary).

disconnect_all

Wednesday, November 21, 2012

My way for binding ssh-agent with zshell

The following lines I put into my .zshrc might not be very pretty, but it is short, seems to work and I don't need to use any of graphical tools to ssh somewhere with my key (with passphrase). So it works fine with my zshell and E17.

SSHPID=`ps ax|grep -c "[s]sh-agent"`
if (( $SSHPID == 0 ))
then
    ssh-agent > ~/.ssh-env
    source ~/.ssh-env
    ssh-add
else
    source ~/.ssh-env
fi



Sunday, February 05, 2012

CakePHP tutorials on TuxRadar

LinuxFormat presented a some time ago interesting tutorials to CakePHP. What even more interesting their share materials on TuxRadar webpage. The only problem I found is lack of some kind of list of contents, so I made one:

  1. CakePHP Tutorial: Build Web Apps Faster
  2. CakePHP Tutorial: Storage, Baking and Slugs
  3. CakePHP Tutorial: Build a file sharing application
  4. CakePHP Tutorial: Build a bookmark site

BTW. IF you hit this page you might be also interested in Practical PHP Programing tutorial from the same page.

List of VMs on XenServer (with UUIDs)

Xen XE command is a very powerful tool, especially if you bind it with other UNIX tools such as awk. Let say we need a map of vm uuids and theirs names. We can use an output from the vm-list command and parse if with awk to get desire result.
xe vm-list | \
awk '{if ( $0 ~ /uuid/) {uuid=$5} if ($0 ~ /name-label/) \
{$1=$2=$3="";vmname=$0; printf "%s - %s\n", vmname, uuid}}'

The script first save the fifth column from a line having uuid string in it into the variable uuid. Next it saves all columns, after the third one, from line having name-label into variable vmname. Finally it prints both variables.

The exemplary output:

ukweb2 - fbca0851-35de-2963-bf0c-7980f3c0d96f
nagios - b741def2-14cc-def4-f8ba-ff0d3ed741d9
ukmail1 - 343c8f93-e4db-d0df-bc30-7544fcd6f14e
jira - ecc3241f-ac14-0398-4e44-ba96cd1d51d2
dodb-02 - 7f223172-e43e-a200-6dc6-b108ce4f9166
RTST-Witness Server - 3c236b0a-209f-6ac9-6d46-b14f7678bfa6
hub-01 - 60ef767c-9b87-edf8-9f13-af2185e656cd
ukweb1 - 6e0e4622-ddfe-0db8-a128-f432e05565cb
dns2 - d65e40d4-ea21-1cbf-cc86-9f522f5e04ef
ixchariot - 73f78129-86db-fd9f-81b4-85768eeee487

We can modify our command to prepare a list of all host with vms bind to them. This time we use xe vm-list with params=all option. The scripts searches for lines with the name-label and saves a name (third column). Next it looks for lines with the word affinity and a uuid (we know that UUID have to start from a hexadecimal number) and prints a saved name.

xe vm-list params=all| \
awk '{if ( $0 ~ /name-label/) {$1=$2=$3=""; vmname=$0} \
if ($0 ~ /affinity.*\:\ [a-e,0-9]/) {host=$4; printf "%s \n", vmname}}'

The output might looks similar to:

Control domain on host: p1-m4 
   Control domain on host: p1-m2 
   dodb-02 
   ukweb5
   Control domain on host: p1-m3 
   Control domain on host: p1-m1 

You might wonder why the list is so short, but we have the list of machine enforce to start from a given host (affinity to a given UUID). If you have machine on share storage allowed to flow between machine you should get very short list indeed.

Tuesday, January 24, 2012

Crux and Mercurial view

I've spent quite some time trying to make hg view to work on my Crux. All time I got error message:

hg view /bin/sh: hgk: command not found

I could not understand what going on. I enable hgk in /etc/mercurial/hgrc or ~/.hgrc. I specified the full path to hgk.py in there as well. I even modify default python path. It didn't work.

After some time of googling, changing various variables I found somewhere (probably on Mercurial page), that some Linux distro missing hgk even if they provide hgk.py. Now I know that Crux is one of them. I copied hgk from contrib directory in source package to /usr/bin and now hg view works fine.

Saturday, January 14, 2012

Aptitude advance usage

Some time ago on the Debian mailing list there was a discussion (there are many discussion over there ;). That one was extremely interesting, not because of the problem (I don't remember it), but because of the Jörg-Volker. Peetz mail with information on advance aptitude usage:

"I would try the following:


- to find out what is installed
    aptitude search '~iapache'


- why it is installed
    aptitude why apache2-mpm-worker
  maybe this one is only recommended by another package


- and what depends on this package
    aptitude search '~i~Dapache2-mpm-worker'


- finally, see what would happen, if it is removed:
    aptitude -s purge apache2-mpm-worker"


In the same thread Bernd Semler suggested following command:

apt-cache rdepends $packagename


The original thread can be found here: http://lists.debian.org/debian-user/2011/10/msg01472.html


Monday, January 09, 2012

XenDebian.py to install Debian on XenServer/XCP

Recently, I've needed to install many Debian VMs onto XenServers and, of course, wanted to automate it. One of my colleagues pointed that rather use existing tools like Cobbler I should take opportunity of working with world leading XenAPI developers and learnt it by writing some code. I took that advice and started to write a python script.

During that I decided that I could improve XenAPI documentation. I spent some extra time on my program and tried to write the code clear and with as many comments as possible, so other can learn from it and reuse it. I hoped to write even more documentation (some tutorial) based on my experiences, unfortunately I haven't had enough time.

Please find short XenDebian.py documentation on Xen wiki:

http://wiki.xen.org/wiki/XenDebian.py

and code on GitHub:

https://github.com/wawrzek/XenDebian


The script is called XenDebian, but with minor modification (new preseed file and change distro name in few places) you should be able to use it with Ubuntu. With a few more modification it should works for any distribution.

Finally, thanks to Project Kronos you would be soon able to install use XenDebian to install many Debian on Debian!

Wednesday, January 04, 2012

xe-patch

Recently I wrote a very small script to apply XenServer patches from command line.

#!/bin/bash

unzip $1
filename=`basename $1 .zip`.xsupdate
echo "Applying $filename"
xe patch-pool-apply uuid=`xe patch-upload file-name=$filename`

To use if first you need to download a patch (you might try to find any new patch here), and next use the script:

./xe-patch hotfix.zip