Get Public IP address of Azure VM from shell on VM

Sometimes you need to get an IP address of the VM from inside of it. You can do this relatively simple from Azure VM with curl and jq thanks to Metadata endpoint as described in this document https://azure.microsoft.com/en-us/blog/announcing-general-availability-of-azure-instance-metadata-service/

And if you need the public IP address of interface eth0 this is the command:

curl -H Metadata:true http://169.254.169.254/metadata/instance?api-version=2017-04-02| jq '.network.interface[0].ipv4.ipAddress[0].publicIpAddress'

One liner to list interactive Unix users

If you need simple command to list all interactive users this awk one-liner might be helpful. It prints all usernames without few keywords in shell. May trigger false positives, so be careful.

awk -F: '$NF !~ /(false|nologin|halt|shutdown|sync)/ {print $1}'  /etc/passwd

Terraform debugging (Azure Storage long names)

I had strange problem with Terraform on Azure. Everything looked good, but ever time I got. I tried to changed few things. Didn't help. All the time:

* module.storage.azurerm_storage_container.vhds: 1 error(s) occurred:

* module.storage.azurerm_storage_container.vhds: Resource \
'azurerm_storage_account.vhds' not found for variable \
'azurerm_storage_account.vhds.name'

I couldn't find any explanation to me problems, so I started to look to increase amount of information from Terraform. I found that TF_LOG controls log level. I ran

TF_LOG=TRACE terraform plan| grep ERROR

And found following line:

2017/10/27 20:31:37 [ERROR] root.storage: eval: *terraform.EvalValidateResource, \
err: Warnings: []. Errors: [name can only consist of lowercase letters and numbers, \
and must be between 3 and 24 characters long]

Yes, the create name was a bit long, but aaaa... The original error message is not the most obvious one.

OpenSSL and Azure VPN

I had to set up an Azure Point-to-Site VPN, but didn't want to do it from Windows machine (I'm Linux/MacOSX kind of the guy) but luckily I found this Aris Plakias' article which describe in a plain language with good example how to prepare all necessary certificates using OpenSSL.

Actually I've created this small script so I can easily repeat client creation step.

#/bin/sh
name=$1
openssl genrsa -out ${name}1Cert.key 2048
openssl req -new -out ${name}1Cert.req -key ${name}1Cert.key -subj /CN="MyAzureVPN"
openssl x509 -req -sha256 -in ${name}1Cert.req -out ${name}1Cert.cer -CAkey MyAzureVPN.key -CA MyAzureVPN.cer -days 180 -CAcreateserial -CAserial serial
openssl pkcs12 -export -out ${name}1Cert.pfx -inkey ${name}1Cert.key -in ${name}1Cert.cer -certfile MyAzureVPN.cer

Firefox update in Crux

Updating Firefox, without rebuilding all other ports in Crux, is not the easiest task. Quite often you need to update one of packages Firefox depends one, but not all of them.

In my case, limited space on root partition is an additional problem. In the same time I have another, much bigger partition attached.

To address both issues I prepared this small script. It updates require dependencies (autoconf, sqlite, libpng, nspr, nss) and then updates Firefox, but with working directory in no default location (PKGMK_WORK_DIR).

prt-get update autoconf
prt-get update sqlite3
prt-get update libpng
prt-get update nspr
prt-get update nss
PKGMK_WORK_DIR=/media/pictures prt-get update firefox

Bringing my Crux Enligthement+ repo back to live [Part I]

I wasn't doing anything with my home Linux machines for some time and recently decided to change it a bit. There are quite a few projects I hope to push a  forward. For example put all my pictures from CDROMs/DVDs into Dropbox, enable internal streaming from my RPi, install Linux on my Dell laptop.To achieve the last one I tried Kubuntu, but it's not so easy properly configure KDE nowadays (KDE5). I tried Enlightenment from Ubuntu PPA, but it connman seemed to be broken. So it seemed that I had to use Crux. Another option could be Arch, but I had already spent quite a lot of time configuring Enlightenment for Crux.

The package repository is located at wawrzek.name/crux/wawrzek/, and I keep my work in github wawrzek/crux-ports repository.

The first problem I encountered was issues with access to some .httpup related info:

Connecting to http://wawrzek.name/crux/wawrzek/
Updating collection wawrzek
 Edit: wawrzek/.httpup-repo.current
Failed to download http://wawrzek.name/crux/wawrzek/.httpup-repo.current: The requested URL returned error: 403 Forbidden
 Edit: wawrzek/.httpup-urlinfo
Failed to download http://wawrzek.name/crux/wawrzek/.httpup-urlinfo: The requested URL returned error: 403 Forbidden
Finished successfully

I host my repo on Linux virtual machine and I could easily confirm that all files has right access privileges, so I eliminated OS level problem. All other files in the same directory were properly accessible, so I started to suspect Apache configuration and that was a case. Apache Debian/Ubuntu configuration has block preventing web clients accessing .htaccess and .htpasswd, but definition is rather generous (^\.ht or everything what starts with string ".ht"). I decided that the simplist resolution is going to replace it with more specific rule (^\.ht(access|passwd)) what matches only ".htaccess" or ".htpasswd". Updated block of configuration is below:

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<FilesMatch "^\.ht(access|passwd)">
        Require all denied
</FilesMatch>

Control terminal name and comment block in VIM

Somewhere (I think it was Stackoverflow) I found simple command to control the name of terminal from commandline which works very nice with iterm2 tabs on MacOSX and decided to add to my zsh environment this function:

termname() {
 echo -en "\e]1; $1 \a"
}

And if we are saying about Stackoverflow one of the most useful Vim suggestion I've ever found is this instruction to comment/uncomment a multiple lines in VIM.

Many AWS accounts and Zsh

That might not be a common problem, but I have to deal with many AWS accounts in the same time. For example I might to have to run an Ansible playbook for one account and a CLI commend for other one in parallel. To make my life easier i wrote this ZSH function. (It's probably going to work in other advance shell).

There are two ways to use it.

• awsenv list - returns the list of all available accounts/environments.
• awsenv - sets, based on values provided in ~/.aws/credentaials and ~/.aws/config, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.

The list is create with simple AWK script (assuming that any lines with "[]" is OK.

The actual command to set environment uses two AWK scripts. First one looks for requested account name and set variable "a" to 1. When "a" equals 1 it prints shell export command for access_key_id and for secret_access_key to  standard output, which is redirected to $TMP_FILE. Then it sources, prints and deletes that file.

Please note that in current form script requires access_key_id being define before secret_access_key.  Printing the value of all variables, especially secret_aceess_key could be consider as security weakness, so you might want to modify/remove "cat $TMP_FILE line.

# vim: set filetype=sh
awsenv () {
    if [[  $1 == list ]]
        then
        print $1
        awk  '/\[.*\]/ {print $1}'  ~/.aws/credentials
    else
        TMP_FILE=/tmp/current_aws
        awk \
           'BEGIN{a=0};\
           /\['$1'\]/ {a=1};\
           /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
           /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
            ~/.aws/credentials > $TMP_FILE
        awk \
            'BEGIN{a=0};\
            /\[profile '$1'\]/ {a=1};\
            /region/ {if (a==1){printf "export AWS_DEFAULT_%s=%s\n", toupper($1), $3; a=0}}'\
            ~/.aws/config >> $TMP_FILE
        source $TMP_FILE
        cat $TMP_FILE
        rm $TMP_FILE
    fi
}


Finally, initial version of this script and some discusion on profiles in older boto versions is in this post http://larryn.blogspot.co.uk/2015/03/how-to-deal-with-aws-profiles.html

Real "Get All Launch Configuration" In Boto

During my recent Ansible tests I created 'some' number of launch configuration, enough to reach account limit and I wanted/had to clean it. Boto sounded like a good candidate to do this. This few lines should address my problem.

import boto
import boto.ec2
import boto.ec2.autoscale

asg = boto.ec2.autoscale.connect_to_region('us-east-1')
results = asg.get_all_launch_configurations()

But it didn't. I could even found my launch configurations in the results sets. I figured out quickly that my results set is rather big and by default get_all_lunch_configurations method paging results (AFAIR default value is 20). Using example from this post on SDB I create following function doing what above method name promises - get all lunch configuration.


def get_all_launch_configuration(connection)
    """get_all_launch_configuration(connection) -
       returns results set of all launch configuration,
       regardless it size. Function require established
       boto.ec2.autoscaling connection."""

    results = connection.get_all_launch_configurations()
    token = results.next_token
    while True:
        if token:
            r =  asg.get_all_launch_configurations(next_token=token)
            token = r.next_token
            results.extend(r)
        else:
            break
    return results
       

A small example of advance aptitude usage

Few years ago I 'preserved' few example of advance aptitude usage I found on a Debian mailing list. Recently I had a chance to use them again. I was looking for a version of packages with distinguish string in a name (let say it was 'apache').

aptitude versions \$(aptitude search ~iapache| awk '{print \$2}')

• So first I ran aptitude search for 'apache' term, but only among installed packages - aptitude search ~iapache
• From the results I cut package name (second column) - awk '{print \$2}'. Please note I escaped dollar character, because I run this command using Ansible (see this blog entry for more details).
• The outcome of those two command allows me to query package versions - aptitude versions

Links

• http://larryn.blogspot.co.uk/2012/01/some-time-ago-on-debian-mailing-list.html
• http://larryn.blogspot.co.uk/2015/04/the-cassandra-ansible-pipe-and.html

Crux as a second system in GRUB2

I tried to install CRUX as a secondary system on a DELL XPS 13 (Ubuntu Edition). I run setup, configured fstab and rc.conf, compiled a kernel and then tried to add new system to the GRUB2 menu in Ubuntu. The update-grub script recognised the new Linux entry, added it, but CRUX didn't start — kernel panicked. It could not find root partition. I double checked my kernel and had all important options mentioned in CRUX installation compiled in.
Then I looked at menu entry created by os-prober for CRUX and noticed that it had following line:

        linux /boot/vmlinuz root=/#ROOT_DEVICE# ro quiet

I analysed what the probe script do and found that it checked boot loader configuration files from a new/additional Linux. Then I remembered that I had not bothered to configure LILO, because I had plan to use Ubuntu GRUB2 rather than it. I updated /etc/lilo.conf (without runing lilo command itself).
After that update-grub properly created CRUX entry in boot menu and I could start my CRUX installation.

zsh and ssh-agent

 INTRODUCTION

One of the post which gets some attention on this blog is My Way For Binding SSH Agent With Zshell. The method presented there is far from ideal and it stopped to work for me some time ago. After that I wrote a new version. I think it is much better and should work with bash or other shells. I tested it on Ubuntu and Crux.

ZSSH

I have the .zssh file in my home directory. It is sources by my .zshrc file. The .zssh consists of 3 functions.

SSHAGENT

The first function is responsible for starting ssh-agent.

sshagent () {
    SSHAGENT=$(ps ax|grep "[s]sh-agent"| grep -cv Z)
    if (( $SSHAGENT == 0 ))
    then
        sshupdate
    else
        SSHPID="$(ps -eo pid,command | awk '/ ssh-[a]gent/ {print $1}');"
        SSHPID_ENV=$(awk  '/Agent/ {print $NF}' ~/.ssh-env)
        if [[ $SSHPID == $SSHPID_ENV ]]
        then
            source ~/.ssh-env
        else
            killall ssh-agent
            sshupdate
        fi
    fi
}


It checks if a ssh-agent runs already and it isn't a zombie. (On one of my systems, after starting a desktop environment, I always had a zombie ssh-agent running.) If there is no ssh-agent running the function calls sshupdate, another function described below. If the agent is present and live in a system the function then compares ssh-agent pid with the information saved in the ~/.ssh-env file. (See sshupdate paragraph for more information.) If informations are consistence it sources .ssh-env.  If not it kills all ssh-agent and the calls sshupdate.

SSHUPDATE

This is a very simply function calling ssh-agent and saving its output to a file.

sshupdate () {
    ssh-agent > ~/.ssh-env
    source ~/.ssh-env
}

The output then can be sourced by other functions or processes. Oh, and if you don't remember/know the output of ssh-agent looks like that:

SSH_AUTH_SOCK=/tmp/ssh-BnXafqRnOSHx/agent.1884;
export SSH_AUTH_SOCK;
SSH_AGENT_PID=1885; 
export SSH_AGENT_PID;
echo Agent pid 1885;

SSHADD

Finally the function responsible for adding your ssh key.

sshadd () {
    if (( $(ssh-add -l | grep -c $USER) == 0 ))
    then
        ssh-add
    else
        ssh-add -l
    fi
}

It checks the number of added keys. If a key from you home directory, or having your username in the path, is not present it adds it. Otherwise it lists all added keys.

USAGE

sshagent is called from your .zshrc, so it should be present during every session. sshadd need to be called by you, when you need it first time.

FURTHER UPDATES

What if you have more than one key and you would like ti add all of them in the same time. Then you could try to use the 'file' program to find ssh keys in the.ssh, or other, directory and source all of them.

The Cassandra, the Ansible, a pipe and a complicate command

I've been working on Cassandra ring downsizing (not a fanny task). To make things a bit easier I've been using Ansible for configuration management. In theory you don't need to change the value of 'initial_token' in Cassandra configuration, and everything else stays the same during ring resize. Therefore, Ansible is not really necessary, but I believe it's good to have consistence in your configuration.

Cassandra role

In first place I added a variable with the Cassandra ring size as a max_nodes  to its role.

  - {role: cassandra, max_nodes: 13}

This value is used to create a Ansible local fact. To learn more on local facts please read this Curtis Collicutt article, which is the best for my script. Please note that the following code is a Jinja template not ready script.

#!/bin/sh
NODE=$(hostname |grep -o -P "\d+")
TOKEN=$(echo '2^127/{{ max_nodes }}' | bc)
cat <
{
    "node_number": $NODE,
    "token": $TOKEN
}
EOF

The above script print onto standard output a json with two variables.

• node_number - is created based on hostname. The number a hostname correspond to a ring position (i.e. cassandra-4 is forth node).
• token - is 'a main part' of token value calculation. The rest is done in Cassandra configuration template:

    initial_token: {{ '%d' % ( ansible_local.cassandra.token *  (ansible_local.cassandra.node_number - 1)) }}

That's sounds complicated and it is. As often there is a historical explanation to that situation. Initially, the whole calculation was done in the template. It worked for 16 node ring, but not for 15. It didn't work, because division two integral  number (2**127/15) results in float one in jinja what lead to rounding error!

Ansible commands

As a bonus two example of using Ansible to run a bit more complicated command on many hosts.

1. Reads token position from configuration file and initialize node move in a screen session. Please note escape character in front of '$'.
   ansible \
    -i inventory/ec.py \
    -m shell \
    -a "screen -d -m \
      nodetool -h localhost move \
      \$(awk '/initial_token/ {print \$2}' \
        /etc/cassandra/default.conf/cassandra.yaml)"\
   tag_pool_cassandra
   
   
2. Much simpler. Checks that screen runs. AFAIK you have to use shell module if you want to use UNIX pipe.
   ansible \
    -i inventory/ec2.py \
    -m shell \
    -a "ps -ef| grep SCREEN"\
   tag_pool_cassandra
   

How to deal with AWS profiles

I don't know how common it is to be a part of an organisation having many AWS (Amazon Web Services) accounts, but it's make things tricky. Amazon make it relatively easy to use many 'named profiles' (account) with AWS CLI. (If you haven't try see this documentation). Boto (python AWS interface) developers also added easy way to use the same profiles in version 2.29. (How to use the same profiles in Boto and other SDKs check this article.) But release 2.29 is not so old and what if you got stacked with older version (for example the one from the latest Ubuntu LTS)? I was in such situation and wrote this small function to use it with profiles from the ~/.boto (not ~/.aws/) file.

def set_account(environment):
    """set_credentials(environment) -
    sets credentials for given environment/account.
    """
    for i in boto.config.items(environment):
        boto.config.set('Credentials', i[0], i[1])

So if your profile is called 'prod':

import boto

set_account('prod') 
conn = boto.connect_ec2()

Another program having issues with many AWS accounts is Ansible. (However, authors claims it's a feature not a bug). My first approach was to add the above function in the ec2.py inventory script and  further extended it by adding following lines:

• to the __init__ method of the Ec2Inventory class:

   
  set_credentials(self.args.environment)
   

• and to the parse_cli function:
  

   
  parser.add_argument('-e', '--environment', type=str, required=True,
                             help="select an environment/profile to run.")
   
  

Such prepared script is not ready to use with Ansible yet - a wrapper around it is needed as well. For example the bash script called ec2-prod.sh to use Ansible in the 'prod' environment.

#!/bin/sh
cd $(dirname $0)
./ec2.py -e prod --refresh-cache

If you need to know why the wrapper is needed check the Ansible inventory code.

Such approach is not ideal if you have many account to work with, you will need a wrapper for each one. What worse, it doesn't work with unified AWS config approach and require to keep a unique version of the inventory script. Therefore, I tried to find a better resolution. I could not find anything interesting and decided to write a small shell script to read ~/.aws/credentials and exports AWS keys for selected profile. The script is a simple wrapper around a bit complicated awk command. To use it you have to source, not execute, it, because the script should execute in a current shell.

#!/bin/bash

TMP_FILE=/tmp/current_aws
awk  \
 'BEGIN{a=0};\
 /\['$1'\]/ {a=1};\
 /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
 /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
 ~/.aws/credentials > $TMP_FILE

source $TMP_FILE
rm $TMP_FILE

The script ensure that:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

are exported. This works not only with Ansible, but also can simplified AWS CLI usage, because the --profile option can be dropped. It might also help with other tools.

Links:

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
• http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
• http://boto.readthedocs.org/en/latest/
• http://www.ansible.com/home
• http://docs.ansible.com/intro_dynamic_inventory.html

Datadog and many dataseries stacked together

Recently, I've started to use Datadog. It has nice features, but I have also found some annoying lacks. One of them is no easy way to prepare a graph with a stack of different series in one graph, for example nice representation of CPU time spent in different states.

Luckily, as you can see above it can be done. You just need to change some things in JSON and have something similar to what I got below. The main point is to have all dataseries in the argument of one "q".

{
  "viz": "timeseries",
  "requests": [
    {

      "q": "avg:system.cpu.system{host:host-01}, avg:system.cpu.user{,host:host-01}, avg:system.cpu.iowait{host:host-01}, avg:system.cpu.stolen{host:host-01}, avg:system.cpu.idle{host:host-01}",
    },
      "type": "area"
  ],
  "events": []
}

Count processes per state per application

In previous posts (here and here) I discussed how to count thread in a given state for a give process. Recently, I had another problem - I needed to count number of processes per application per state. My previous commands wouldn't work, so I wrote an alternative version.

while [ 1 ];
do
    date;
    cat /proc/loadavg;
    ps -Leo state,args |
     awk ' $1 ~ /(D|R)/ {state[$0]++} \
      END{ for (j in state) {printf "%s - %d\n", j, state[j]}}' |
      sort -k 2;
    echo "---";
    sleep 5;
done 

There is not PID and args are included in the output list as a whole. worried for number of processes.

One more thought. Dropping "$1 ~ /(D|R)/" can be useful in case of problem with total number of processes. But then the whole command should be a bit modified, so the results are sorted by number of processes. Simplified version would look like this one:

while [ 1 ];
do
    ps -Leo state,args |
     awk ' $1 ~ /(D|R)/ {state[$0]++} \
      END{ for (j in state) {printf "%d - %s\n", state[j], j}}' |
      sort -n;
    echo "---";
    sleep 5;
done 

What is the (UNIX) load?

The "load" is use widely to describe stress/work applied onto a UNIX system. The simple rule is "lower than better". In the older days of uniprocessor machine load 1 was  kind of a borderline. In the new brave world of multi-core/processor machines load 1 means nothing.  Many people suggests that load equal or lower to number of processors/cores is good. That sound sensible, but not always is accurate.
Why? To answer that we have to comeback to question asked in the subject.

What is the "load"?

The load as the exponentially damped/weighted moving average of the number of processes, including threads, using or waiting for CPU and, at least at Linux, in uninterruptible sleep state in last 1, 5 and 15 minutes (see Wikipedia). The last part means that all processes/threads waiting for a disk (or other I/O device) will increase the load, without increasing a CPU usage. It leads to situation when the load lower than number of core/processes is danger. Let imagine few processes trying to dump important information on disks. Especially if all interrupts have affinity to one processor only (see this post) or just data are store in many small files. On the other hand, machine with very high load might be very responsive. Plenty of processes waiting to write information onto a disk not using a lot of memory and CPU in the same time. Just look at this picture:

If you want to know even more details of how the load is actually calculated
read this impressive white paper.


Links:
http://en.wikipedia.org/wiki/Load_%28computing%29
http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
http://larryn.blogspot.co.uk/2013/05/cpu-affinity-interrupts-and-old-kernel.html

Install CyanogenMod at Nook HD+

Recently I decided to try the new Cyanomogen (CM11) on my Nook HD+. Initial reading indicated that I had to reinstall using Recovery rather than internal updater. I tried to login to Recovery so much, that I recovered official B&N OS which replaced CM.

I needed to start from beginning. I did some research and found that post. It looked good so I gave it a try. First download ClockworkMod attached to the post, but later I downloaded latest CM snapshot from  there and  added  Google Apps for CM11 from there. I put everything as described on SD Card and kicked off installation. It flew like an Albatross. (To be honest I don't know why I did write Albatross - maybe because of this?) 



Anyway CM11 works good at Nook HD+.

Links:

• http://wiki.cyanogenmod.org/w/Ovation_Info
• http://download.cyanogenmod.org/?type=snapshot&device=ovation
• http://wiki.cyanogenmod.org/w/Google_Apps
• http://forum.xda-developers.com/showpost.php?p=42406126&postcount=7
• http://forum.xda-developers.com/attachment.php?attachmentid=2849350&d=1405272804

More fabric as a library

Recently I had to prepare a tool doing some remote commands, so of course I decided to use fabric, but I have big problem to control hosts. I remembered that I had written a short article on Fabric in here some time ago. But it didn't help. I asked on the Fabric mailing lists, but there was no help.

Manual host name control

In this tool I didn't need to run many parallel SSH connection, so I decided to control remote host name from inside the loop in the function my setting env.host_string each time (this is very useful functionality). Like in following example:

#!/usr/bin/env python

"""Example code to use Fabric as a library. 

It shows how to set up host manually.

Author: Wawrzek Niewodniczanski < main at wawrzek dot name >

"""

# import sys to deal with scripts arguments and of course fabric 

import sys

import fabric
from fabric.api import run, hide, env

env.hosts = ['host1', 'host2'] 

# Main function to run remote task 

def run_task(task='uname'):
    """run_task([task]) -
    runs a command on a remote server. If task is not specify it will run 'uname'."""

    # hide some information (this is not necessary).

    with hide('running', 'status'):
        run(task) 

# Main loop

# take all arguments and run them on all hosts specify in env.hosts variable

# if not arguments run 'uname' 

if len(sys.argv) > 1:

    tasks = sys.argv[1:]

    for task in tasks:

        for host in env.hosts:
            env.host_string = host
            run_task(task)

else:

    for host in env.hosts:

        run_task() 

Fabric in full control

The problem bugged me since then. Yesterday I found some of my old code. Analysed it and quickly found small, but profound difference with mu recent fabric usage. Rhe code above called the run_task function wrongly. Rather than dealt it in the normal way I supposed to use execute.

#!/usr/bin/env python

"""Example code to use Fabric as a library. 

It shows how to set up host manually.

Author: Wawrzek Niewodniczanski < main at wawrzek dot name >

"""

# import sys to deal with scripts arguments and of course fabric 

import sys

import fabric
from fabric.api import run, hide, env, execute

env.hosts = ['host1', 'host2'] 

# Main function to run remote task 

def run_task(task='uname'):
    """run_task([task]) -
    runs a command on a remote server. If task is not specify it will run 'uname'."""

    # hide some information (this is not necessary).

    with hide('running', 'status'):
        run(task) 

# Main loop

# take all arguments and run them on all hosts specify in env.hosts variable

# if not arguments run 'uname' 

if len(sys.argv) > 1:

    tasks = sys.argv[1:]

    for task in tasks:

        execute(run_task, task)

else:

    execute(run_task)

Links:

http://www.fabfile.org/
http://larryn.blogspot.co.uk/2012/11/fabric-as-python-module.html
http://lists.nongnu.org/archive/html/fab-user/2014-10/msg00002.html

Resource for vim

Just some link with useful Vim's advices.

General Vim advices

• https://code.google.com/p/vimcolorschemetest/ - many beautiful colourschemes.
• http://tnerual.eriogerg.free.fr/vimqrc.html - many shortcuts (to keep under the pillow).
• http://vimregex.com/ - all you might want to know about regex in Vim.
• http://dailyvim.blogspot.co.uk - plenty of short yet useful Vim advices.
• http://vimcasts.org/ - great Vim screencast with good support.

Vim and Python

• https://dev.launchpad.net/UltimateVimPythonSetup/
• http://justinlilly.com/vim/vim_and_python.html
• http://haridas.in/vim-as-your-ide.html
• http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/
• http://www.sontek.net/blog/2011/05/07/turning_vim_into_a_modern_python_ide.html
• http://www.vim.org/scripts/script.php?script_id=1494
• https://github.com/scrooloose/syntastic
• http://vim.wikia.com/wiki/Omni_completion
• https://code.google.com/p/vimpdb/

(w)dstat

wdstat

In my .profile (on CentOS 5, just in case there were some changes in dstat) I have following alias to dstat (wdstat stands for Wawrzek's dstat):

alias wdstat="dstat -lcpymsgdn 5"

Where the options stands for:

• -l  - UNIX load (1m   5m  15m) load average in 1, 5 and 15 minutes, respectively;
• -c - cpu stats (usr sys idl wai hiq siq) percent of time spent in user and system space, idle, waiting on resource,  serving interrupts and softirqs (software interrupts);
• -p - process stats (run blk new) number of running, blocked and newly created processes;
• -y - system stats (int   csw) - number of interrupts and context switches;
• -m - memory stats (used  buff  cach  free) amount of memory used by processes, disk buffers, disk cache and free;
• -s - swap stats (used  free) - amount of used and free swap space;
• -g - page stats (in   out) number of page put in and out from swap;
• -d -disk stats (read  writ) - number of reads and writes from all disks;
• -n -network stats (recv  send) number of received and send network packages;

Further reading:

• http://dag.wiee.rs/home-made/dstat/
• http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
• http://www.linuxhowtos.org/System/procstat.htm
• http://lwn.net/Articles/520076/
• http://en.wikipedia.org/wiki/Paging#Linux

netstat, ports, hosts and awk glue

Recently, I needed to create a list of all servers connected on a given port (in following example port 80). I used a mixture of awk and other UNIX command line tools.

netstat -nt| \
 awk -F':'\
   '$5==80 {count[$8]++} \
   END{ for (i in count) { \
      cmd="host "i; \
      cmd |& getline j; \
      split(j, a, " "); \
      printf "%40s - %d\n", a[5], count[i]}}'| \
 sort -n -k 3

First netstat provided the list of all connection (netstat -nt); -n stands for numeric and -t for only TCP connections.

Next awk, with the ':' defined as a field separator (awk -F':'), used lines where local port was 80 ($5==80) to create an associated array with a key define by connected host ip and a value equal to  number of connection from it ({count[$8]++}). At the end of the script execution, awk looped over all element of the array (END{for (i in count)).  Next there was a crux of the script, the cmd was define as a run the OS host command with the awk variable i as an argument  (cmd="host" i). The |& operator created two-way pipe between awk and a execution of the previously defined cmd. The getline command was used to store cmd output into the variable j (cmd |& getline j). Next the split command split the content of the  j into separate words and saved them into the a array (split(j, a, " ")). Finally the printf formatted output (printf "%40s - %d\n", a[5], count[i])).  The actual hostname was fifth element of the a.

For continence, output lines were sorted by numeric order on third column  (sort -n -k 3). Each output line consisted of a hostname ,'-' and a number - e.g. important.com - 3456.

Python for SysAdmins

Preparing for a interview some time ago I made a list of python module interesting for SysAdmins. Today looking for something else I found that half baked note and decided to polish it enough to put it on the blog. It's mostly for myself as quick reference.

Each module is describe by one, two sentences (from Python documentation) and have a link to official online document. At the end there is a list of example function, object.

Python modules for SysAdmin

import sys

This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is always available.

https://docs.python.org/2/library/sys.html

examples:

• argv,
• exit(),
• path,
• modules,
• exec().

 

import os

This module provides a portable way of using operating system dependent functionality.

https://docs.python.org/2/library/os.html

examples:

• chdir(),
• getuid(),
• uname(),
• listdir(),
• stat(),
• rename(),
• access().

 

import os.path

This module implements some useful functions on pathnames.

https://docs.python.org/2/library/os.path.html

examples:

• isdir(), 
• isfile(), 
• exist(), 
• getmtime(), 
• abspath(), 
• join(), 
• basename(), 
• dirname().

 

import time

This module provides various time-related functions.

https://docs.python.org/2/library/time.html

examples:

• time(), 
• ctime(), 
• sleep(), 
• strftime(), 
• strptime().

 

import glob

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.

https://docs.python.org/2/library/glob.html

examples:

• glob(),
• iglob().

 

import fnmatch

This module provides support for Unix shell-style wildcards, which are not the same as regular expressions (which are documented in the re module).

https://docs.python.org/2/library/fnmatch.html

examples:

•  fnmatch().

 

import re

This module provides regular expression matching operations similar to those found in Perl. Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings.

https://docs.python.org/2/library/re.html

examples:

• compile(), 
• match(), 
• search(), 
• split(), 
• findall(), 
• sub(),
• group().

MatchObject

Match objects always have a boolean value of True. Since match() and search() return None when there is no match, you can test whether there was a match with a simple if statement:

match = re.search(pattern, string)
if match:
    process(match)

Zombie

Usually zombie process is not a big problem, but sometimes... just look at the screenshot below. It wants the whole machine as fried eggs! or maybe boiled?

Multiprocessing (in Python)

I needed to do some multi-threading in Python. As I needed effect quick I decided to use standard threading module. However, every time I had to use it I felt it was rather complicated beast. At threading module documentation page there is a link to multiprocessing module. I was tired with threading, on the one hand, but didn't have enough time to learn about greenlet or another competing project, on the other, so I decided to take a quick glance at multiprocessing module...
... And my life became much easier, sky bluer, grass, greener, oh and scripts faster ;-).
I don't do anything special with it, so just one simple code example, but this is very good tutorial you can find much more: http://pymotw.com/2/multiprocessing/communication.html.

Main part of nearly all my scripts looks the same:

import multiprocessing 

SIZE = 30
pool = multiprocessing.Pool(processes=SIZE)
pool_output = pool.map(get_values, servers)
pool.close() # no more tasks
pool.join()

Where servers is a list with servers I need get information from, and get_values is a function (sometimes with a different name).  Simple, isn't it?

shell, history and substitution

One of the most known tricks in using shell history is to use:

^old^new

to replace string old by new in last command. The only problem is that it replaces only first appearance. But there is another command replacing all string old by new:

!:gs/old/new/

Space War

Let say it is a late Christmas present for Science Fiction fans. Especially ones who like a lot of science in SF. Two articles discussion how space warfare can look. Have fun.

• www.foreignpolicy.com/articles/2012/09/28/aircraft_carriers_in_space
• http://gizmodo.com/5426453/the-physics-of-space-battles

Even more threads counting

This is small extension to one of my previous posts. This time a loop is enriched by load values (from /proc/loadavg) as well as measurement time (date).  ps command uses the same option, but there is small improvement in awk call.  Rather than count only processes per state it concatenate state with last string in command arguments — I was mostly interested in few java application and jar name was the last parameter for each one. Also awk counts only processes actually running or in uninterruptible sleep ($2 ~ /(D|R)/ at the beginning of awk command).

while [ 1 ];
do
    date;
    cat /proc/loadavg;
    ps -Leo pid,state,args |
     awk ' $2 ~ /(D|R)/ {state[$2 " - (" $1 ") " $NF]++} \
      END{ for (j in state) {printf "%s - %d\n", j, state[j]}}' |
      sort -k 2;
    echo "---";
    sleep 5;
done

Debian, Vim and alternatives

I don't know why, but default editor in Debian is not Vim, but Nano. I learned once that it can be change with a command, but has never remembered the actual name of it and since than had to check it each time in install Debian (not so often). Today decided to write it down, not to search too long next time. Maybe it's going to be useful to others:

aptitude install vim.nox
update-alternatives --set editor vim.nox

e17 and stable Debian

If you are interested in using e17 with Debian Wheezy, the best way is to use package provided by John Holland at http://www.vin-dit.org/.

Number of threads per state

awk with ps creates powerful Linux combo. For example there is one, or more, multi-threaded application on your system. You would like to know what states all those threads are. Standard ps show information about processes, but you can learn more about threads with "-L" switch. You probably hit another issue - number of threads, you don't want to analyse, let say, 300 lines. Lucky, awk is very good for such tasks. Following 'one-liner' shows number of threads and processes per state (running, sleeping, uninterrupted sleep), every 5 seconds.

while [ 1 ];
do
    ps -Leo pid,state,args| \
     awk '{state[$2]++} \
      END{ for (j in state) {printf "%s - %d\n", j, state[j]}}';
    echo "---";
    sleep 5;
done

First line ensures that command runs forever. ps uses '-e' (all processes), '-L' (include threads), '-o' (user specify output). There are 3 lines in ps output: PID, state and command arguments. (These arguments are useful for my other commands. In this case you can leave only 'state', but you has to modify awk command). Next awk takes every line and counts each state using state array with argument equal to state representation (i.e.: S for sleep, D for uninterrupted sleep, R for running). At the end of execution it prints each argument with it value. After that script displays '---'  and waits 5 seconds.
Such output can be valuable extension to top command results. It shows instant changes to number of busy processes, which affects system load.

CPU affinity, interrupts and old kernel bug

This is an old story and AFAIK were address ages ago in Linux kernel, but there are still plenty of CentOS/RHEL 5 running and you still might be hit by the problem with kernel not properly balancing interrupts from hardware, so the that below list of the links my be helpful. In bug report network card are mentioned, but it might affect other components (i.e. raid controllers from HP - cciss drivers).

If you would like to learn more there is a set of good links:

1. https://bugzilla.redhat.com/show_bug.cgi?id=432451
2. http://docs.oracle.com/cd/E24290_01/coh.371/e22838/tune_perftune.htm
3. http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
4. http://www.alexonlinux.com/msi-x-the-right-way-to-spread-interrupt-load
5. http://www.alexonlinux.com/why-interrupt-affinity-with-multiple-cores-is-not-such-a-good-thing

Regexp Groups in Python

Let say that we have a string consist from a numeric continent code  (i.e. 1 for Europe) , followed by a country code  (i.e. 44 for UK)  and ended with a region code made in similar manner (i.e. Cambridge equal 65). A continent and a country are preceded with the letter C and a location with the letter L.

So we have the string C1C44L65 and we need to know the continent, counter and region codes, but numbers might have different length (i.e. 5, but also 55 or 555) so we cannot just grab selected characters from a string, but in Python we can use following regular expression to save all that information into "groups" for further use, in example in an mathematical operations (as integers).

info = re.search(r"^C(?P[^\d]+)C(?P\d+)L(?P\d+)$", full_id)
continent = int(info.group("continent"))
country = int(info.group("country"))
region = int(info.group("region"))

Ubuntu and mix of NFS, NIS and autofs in post from past

Some time ago I had a problem with autofs, NIS and NFS during Ubuntu start. As far as I remember autofs was responsible for serving users home directories, define with NIS and  provided from remote location with  NFS. The problem that autofs started before network and NIS, so home directories weren't ready.

When I started look into the problem found that small change to autofs.conf should resolve the situation. I didn't deploy the fix on big scale (I were using CRUX on my desktop), but saved the patch to blogger and today finally found some time to share it. Maybe it will be useful for anyone.

root@test-upstart:/etc/init# diff -u autofs.conf ~wawrzekn/autofs.conf
--- autofs.conf2010-08-17 11:34:47.000000000 +0100 
+++ /home/wawrzekn/autofs.conf2011-02-06 12:19:48.678663000 +0000 
@@ -10,6 +10,15 @@ 
 respawn 
  
 pre-start script 
+while : 
+do 
+ypwhich 
+if [ $? = 0 ]  
+then  
+break 
+fi 
+sleep 4 
+done 
 if [ -f /etc/default/autofs ]; then 
 . /etc/default/autofs 
 fi

PS2 and friends

This post won't be about PlayStation 2, but about my (not so) recent problem with the UNIX shell prompt.

I had a problem with prompt on some remote Linux system. Everything looked fine until the command (with arguments) fitted one line. I was using iTerm2 on MacOSX with full screen mode, so line could be long, but anyway it started to be very annoying. When I tried to move cursor back or jump to the beginning of the line it could end up in the middle of existing text.

I spent a lot of time trying to debug problem. First I tried to find something wrong with my iTerm2 settings, but could not find a thing. Next I started to play with tput (see link 1 and 2). For a moment believed that I had broken my PS2 setting (see 3) and finally, after reading link 4, figured out that I hadn't properly escaped non printing character in colour settings. My 'remote' bash prompt looks currently as printer below and you can see a lot of \[ before not printing characters and \] after them.

\[\e[32m\]\h \[\e[0m\]: \[\e[33m\]\w \[\e[0m\]#>

Reading:

1. http://tldp.yolinux.com/HOWTO/Bash-Prompt-HOWTO.html
2. http://tldp.org/LDP/abs/html/terminalccmds.html
3. http://www.thegeekstuff.com/2008/09/bash-shell-take-control-of-ps1-ps2-ps3-ps4-and-prompt_command/
4. http://www.ibm.com/developerworks/linux/library/l-tip-prompt/

Fabric as a python module

Introduction

Fabric is a powerful tool to automate running the same command on many remote server. It is written in python, but authors concentrate on usage as command line tool rather as a python module.

There are some documentation how to do this. You can also find some examples in the internet, but  examples never too much, so below another one.

Scenario

We would like to run a command on each of our webservers, ideally in batches, but in a specific order. Our webserver are grouped in 'cells'. Each cell consist of 3 servers in different 'zones' (zone are called a, b, and c). For convince there is following convention of naming servers: "role-cell#-zone" i.e. web-04-a. To make things more interesting some cells are missing, so we cannot make simple loop. On the other hand, there is a service holding various servers information in json file accessible over http. Additionally, we have a python module called 'server' to locally access those information from our workstation. In the 'server' module we have class 'webserver' to deal with webservers. One of its method is 'get_all_server'. It return a list of all webserver.

Script

Servers

First we create the list of all servers using our external tool.

from servers import webserver
allservers = webserver.get_all_severs()


The order mentioned above is to run first all 'c' boxes next 'b' and finally 'a'. To achieve that, let creates 3 lists (in a function). Nothing magic, a bit of 're'.

def make_lists(servers):
    import re # if not imported in the main part

    wa = []
    wb = []
    wc = []
   
    for server in servers:
        if re.search('web-\d+-a', server):
            wa.append(server)
        elif re.search('web-\d+-b', server):
            wb.append(server)
        elif re.search('web-\d+-c', server):
            wc.append(server)

    separation = (wa, wb, wc)
    return separation


Fabric

So now we can start to use fabric. Let import necessary pieces:

# Fabric part
from fabric.api import *
from fabric.network import disconnect_all

set up environment

env.skip_bad_hosts = True
env.warn_only = True
env.parallel = True


and finally call the function:


@parallel(pool_size=10)
def run_remotely():
    with hide('running', 'status'):
       sudo(task)


Main part

Prepare the lists of servers (see paragraph servers):

lists = make_lists(get_servers())

Get the command to run. I don't remember why, but writing my script decided not to pass variable to 'fabric; function, but use global variable.

global task
task = raw_input("Type/paste command to run >")

Now run the provided command in selected order:

for i in [3, 2, 1]:
    execute (run_remotely, hosts=lists[i-1])


For any case disconnect all connection. (It might be not necessary).

disconnect_all

My way for binding ssh-agent with zshell

The following lines I put into my .zshrc might not be very pretty, but it is short, seems to work, and I don't need to use any of the graphical tools to ssh somewhere with my key (with passphrase). So it works fine with my zshell and E17.

SSHPID=`ps ax|grep -c "[s]sh-agent"`
if (( $SSHPID == 0 ))
then
    ssh-agent > ~/.ssh-env
    source ~/.ssh-env
    ssh-add
else
    source ~/.ssh-env
fi

CakePHP tutorials on TuxRadar

LinuxFormat presented a some time ago interesting tutorials to CakePHP. What even more interesting their share materials on TuxRadar webpage. The only problem I found is lack of some kind of list of contents, so I made one:

1. CakePHP Tutorial: Build Web Apps Faster
2. CakePHP Tutorial: Storage, Baking and Slugs
3. CakePHP Tutorial: Build a file sharing application
4. CakePHP Tutorial: Build a bookmark site

BTW. IF you hit this page you might be also interested in Practical PHP Programing tutorial from the same page.

List of VMs on XenServer (with UUIDs)

Xen XE command is a very powerful tool, especially if you bind it with other UNIX tools such as awk. Let say we need a map of vm uuids and theirs names. We can use an output from the vm-list command and parse if with awk to get desire result.

xe vm-list | \
awk '{if ( $0 ~ /uuid/) {uuid=$5} if ($0 ~ /name-label/) \
{$1=$2=$3="";vmname=$0; printf "%s - %s\n", vmname, uuid}}'

The script first save the fifth column from a line having uuid string in it into the variable uuid. Next it saves all columns, after the third one, from line having name-label into variable vmname. Finally it prints both variables.

The exemplary output:

ukweb2 - fbca0851-35de-2963-bf0c-7980f3c0d96f
nagios - b741def2-14cc-def4-f8ba-ff0d3ed741d9
ukmail1 - 343c8f93-e4db-d0df-bc30-7544fcd6f14e
jira - ecc3241f-ac14-0398-4e44-ba96cd1d51d2
dodb-02 - 7f223172-e43e-a200-6dc6-b108ce4f9166
RTST-Witness Server - 3c236b0a-209f-6ac9-6d46-b14f7678bfa6
hub-01 - 60ef767c-9b87-edf8-9f13-af2185e656cd
ukweb1 - 6e0e4622-ddfe-0db8-a128-f432e05565cb
dns2 - d65e40d4-ea21-1cbf-cc86-9f522f5e04ef
ixchariot - 73f78129-86db-fd9f-81b4-85768eeee487

We can modify our command to prepare a list of all host with vms bind to them. This time we use xe vm-list with params=all option. The scripts searches for lines with the name-label and saves a name (third column). Next it looks for lines with the word affinity and a uuid (we know that UUID have to start from a hexadecimal number) and prints a saved name.

xe vm-list params=all| \
awk '{if ( $0 ~ /name-label/) {$1=$2=$3=""; vmname=$0} \
if ($0 ~ /affinity.*\:\ [a-e,0-9]/) {host=$4; printf "%s \n", vmname}}'

The output might looks similar to:

Control domain on host: p1-m4 
   Control domain on host: p1-m2 
   dodb-02 
   ukweb5
   Control domain on host: p1-m3 
   Control domain on host: p1-m1 

You might wonder why the list is so short, but we have the list of machine enforce to start from a given host (affinity to a given UUID). If you have machine on share storage allowed to flow between machine you should get very short list indeed.

Crux and Mercurial view

I've spent quite some time trying to make hg view to work on my Crux. All time I got error message:

hg view /bin/sh: hgk: command not found

I could not understand what going on. I enable hgk in /etc/mercurial/hgrc or ~/.hgrc. I specified the full path to hgk.py in there as well. I even modify default python path. It didn't work.

After some time of googling, changing various variables I found somewhere (probably on Mercurial page), that some Linux distro missing hgk even if they provide hgk.py. Now I know that Crux is one of them. I copied hgk from contrib directory in source package to /usr/bin and now hg view works fine.

Aptitude advance usage

Some time ago on the Debian mailing list there was a discussion (there are many discussion over there ;). That one was extremely interesting, not because of the problem (I don't remember it), but because of the Jörg-Volker. Peetz mail with information on advance aptitude usage:

"I would try the following:


- to find out what is installed
    aptitude search '~iapache'


- why it is installed
    aptitude why apache2-mpm-worker
  maybe this one is only recommended by another package


- and what depends on this package
    aptitude search '~i~Dapache2-mpm-worker'


- finally, see what would happen, if it is removed:
    aptitude -s purge apache2-mpm-worker"


In the same thread Bernd Semler suggested following command:

apt-cache rdepends $packagename


The original thread can be found here: http://lists.debian.org/debian-user/2011/10/msg01472.html

XenDebian.py to install Debian on XenServer/XCP

Recently, I've needed to install many Debian VMs onto XenServers and, of course, wanted to automate it. One of my colleagues pointed that rather use existing tools like Cobbler I should take opportunity of working with world leading XenAPI developers and learnt it by writing some code. I took that advice and started to write a python script.

During that I decided that I could improve XenAPI documentation. I spent some extra time on my program and tried to write the code clear and with as many comments as possible, so other can learn from it and reuse it. I hoped to write even more documentation (some tutorial) based on my experiences, unfortunately I haven't had enough time.

Please find short XenDebian.py documentation on Xen wiki:

http://wiki.xen.org/wiki/XenDebian.py

and code on GitHub:

https://github.com/wawrzek/XenDebian


The script is called XenDebian, but with minor modification (new preseed file and change distro name in few places) you should be able to use it with Ubuntu. With a few more modification it should works for any distribution.

Finally, thanks to Project Kronos you would be soon able to install use XenDebian to install many Debian on Debian!

xe-patch

Recently I wrote a very small script to apply XenServer patches from command line.

#!/bin/bash

unzip $1
filename=`basename $1 .zip`.xsupdate
echo "Applying $filename"
xe patch-pool-apply uuid=`xe patch-upload file-name=$filename`

To use if first you need to download a patch (you might try to find any new patch here), and next use the script:

./xe-patch hotfix.zip

The power of find - exec and friends (grep, sed)

This is quite old post. I started to write it in October 2010, but could not properly polish it for a long time.

Today I would like to present an example of usage of find (and grep) rooted in Crux. Let say that I want to find all packages in version 20100511 (this was true scenario when I wanted to update e17 related ports). Translating into less Crux specific language it means that I had to find all Pkgfile files (simple find), which had string 20100511 (simple grep). I needed only file names not a matching string so I used -l option for grep.

find . -name Pkgfile -exec grep -l 20100511 {} \;

I not only needed to find all old files but  to updated them as well (to version 1.0.0.beta that time). I used the same find but exchanged grep to sed (with option -i for  "in place").

find . -name Pkgfile -exec sed -i 's/20100511/1.0.0.beta/ {} \;

Let push our example one step further. I wanted to find dependence for packages, therefore I ran following command.

grep -i depen `find . -name Pkgfile -exec grep -l beta {} \;

What used  previous command to create a list of files to hunted through for word beta. (I wasn't sure if word "dependence" begun lower or upper case so used option -i for --ignore-case). 

In UNIX world there are always more than one way of doing things and in our scenario the find -exec can be replace with a separate command xargs. Xargs might be very useful in many cases because can be use to create unix command from standard input. Using xargs rather then find -exec my first example would be:

find . -name Pkgfile  | xargs grep -l beta $1 

Let use xargs for another task related to above example. In my scenario I had not only to update the version, but also to change the sources of the packages. To do that I used find, xargs and sed in a for loop.

for file in `find . -name Pkgfile  | xargs grep -l beta $1 2&> /dev/null` ; \
do \
 sed -i 's/pitillo.mine.nu\/crux\/distfiles/download.enlightenment.org\/releases/' $file; \
done

The above command might be one liner, but can be paste line by line. It used the command from the previous example to create the $file array consist of names of files with word "beta". Elements from $file were use as input for sed command.

Mount LVM from domU in dom0

Mounting logical volumes (LVM) from domX (guest) within dom0 (XenServer) is very not recommender and suggested. But sometimes it might be the only options, and it's cool hacker (hacker in good old meaning, not cracker) trick. So to mount your volume you should take following steps:

1. xe vm-disk-list vm=test4 - list disk of the VM.
2. xe vm-list - find info (UUID) about dom0.
3. xe vbd-create device=xvda unpluggable=true vdi-uuid=79a7a556-a6ba-48cf-8c82-30fa5bb9597c vm-uuid=36895434-e6d7-4fea-8271-d5477ca23c6d - create unpluggable VBD (xvda) with disk (uuid from point 1) on dom0 (uuid from point 2).
4. xe vbd-plug uuid=0a4151b6-2b59-2fdb-c0a9-492520a8d52c - plug created vbd.
5. mount /dev/xvda1 /mnt - mount disk (any 'physical partition').
6. vi /etc/lvm/lvm.conf - if you need anything on LVM you have to edit lvm.conf 

# Ignore /dev/xvd* devices to prevent deadlocking when live-snapshotting
# dom0-attached LVHD VDIs
#filter = "r|/dev/xvd.|","r|/dev/VG_Xen.*/LV.*|"

7. vgchange -a y test4 - activate new vg.
8. mount /dev/test4/root /test/mnt - mount partition.

XenServer, xe and more greping

In my last post I described how you can use grep to extend listing of XenServer parameters. Continuing this thread today, I show how to get values of specific parametes, let say memory.

You can get memory using following command, but I bet it's not you expect.

# xe vm-list params=memory
memory (MRO)    : 


memory (MRO)    : 

More interesting values you can get using params=all. The problem is that you will get hundreds of other values too. This is not helpful, but of course  grep can help us:

# xe vm-list params=all| grep memory
                 memory-actual ( RO): 536870912
                 memory-target ( RO): 
               memory-overhead ( RO): 6291456
             memory-static-max ( RW): 536870912
            memory-dynamic-max ( RW): 536870912
            memory-dynamic-min ( RW): 536870912
             memory-static-min ( RW): 536870912
               recommendations ( RO): 
                        memory (MRO): 
                 memory-actual ( RO): 500170752
                 memory-target ( RO): 
               memory-overhead ( RO): 15728640
             memory-static-max ( RW): 789839872
            memory-dynamic-max ( RW): 500170752
            memory-dynamic-min ( RW): 500170752
             memory-static-min ( RW): 395313152
                        memory (MRO): 
                 memory-actual ( RO): 1073741824
                 memory-target ( RO): 
               memory-overhead ( RO): 10485760
             memory-static-max ( RW): 1073741824
            memory-dynamic-max ( RW): 1073741824
            memory-dynamic-min ( RW): 1073741824
             memory-static-min ( RW): 1073741824
               recommendations ( RO): 
                        memory (MRO): 

It's not so useful yet, because we have only memory related value, but we don't know relation between them and VMs. Luckily, we can very easily extend our grep.

# xe vm-list params=all| grep "label\|memory"
                    name-label ( RW): at10
                 memory-actual ( RO): 536870912
                 memory-target ( RO): 
               memory-overhead ( RO): 6291456
             memory-static-max ( RW): 536870912
            memory-dynamic-max ( RW): 536870912
            memory-dynamic-min ( RW): 536870912
             memory-static-min ( RW): 536870912
               recommendations ( RO): 
                        memory (MRO): 
                    name-label ( RW): Control domain on host: dt33
                 memory-actual ( RO): 500170752
                 memory-target ( RO): 
               memory-overhead ( RO): 15728640
             memory-static-max ( RW): 789839872
            memory-dynamic-max ( RW): 500170752
            memory-dynamic-min ( RW): 500170752
             memory-static-min ( RW): 395313152
                        memory (MRO): 
                    name-label ( RW): at11
                 memory-actual ( RO): 1073741824
                 memory-target ( RO): 
               memory-overhead ( RO): 10485760
             memory-static-max ( RW): 1073741824
            memory-dynamic-max ( RW): 1073741824
            memory-dynamic-min ( RW): 1073741824
             memory-static-min ( RW): 1073741824
               recommendations ( RO): 
                        memory (MRO): 

Now we know that the at10 has 0.5GB, the at11 1GB and the dom0 might have between 386 and 771 MB.

Summary

Of course similar construction in grep query might be use for other variables.E.g. to have the list of live VMs try:

xe vm-list params=all| grep -i -E '(label|\s+live)'

XenServer CLI and grep by name

One of the annoying thing in XenServer command line tool (xe) is lack of regexp/wildcards in filtering vm (and other objects) by name. However, there is easy way to work around thanks to grep. Standard output have 3 lines per vm, with one line before (-B 1 in grep) and one after (-A 1) line with name. So you can use following line to find information about VM with given name:

xe vm-list | grep -B 1 -A 1 your-vm-name

You can also extend this command to get uuid of requested vm:

xe vm-list | grep -B 1  your-vm-name| awk '/uuid/ {print $5}'

What can be use i.e. to take the snapshot:

vm-uuid=`xe vm-list | grep -B 1  dowa| awk '/uuid/ {print $5}'`
xe vm-clone name-name-label='dowa-01-clone' uuid=$vm-uuid

Debian, DHCP and IPv6

It was big surprise for that in Debian /etc/network/interface you cannot us iface ethX inet6 dhcp option to use DHCPv6 client to set IP address. If you need IPv6 address from DHCP server, you should can either define interface manually:

iface eth1 inet6 manual
 up /sbin/ip link set eth1 up
 post-up /etc/init.d/wide-dhcpv6-client start
 pre-down /etc/init.d/wide-dhcpv6-client stop
 down /sbin/ip link set eth1 down

or plug it into IPv4 definition (i.e. inet dhcp).

inet dhcp
iface eth1 inet dhcp
 post-up /etc/init.d/wide-dhcpv6-client start
 pre-down /etc/init.d/wide-dhcpv6-client stop

Another awk example

This time how to print all columns others than first one, but only for lines begins with i.e. exim4-config.

hub-02:~# awk '/^exim4-config/ {$1=""; print }' test2

And I got:

 exim4/dc_smarthost string smtp.uk
 exim4/dc_relay_domains string
 exim4/dc_relay_nets string
 exim4/mailname string hub-01.uk
 exim4/dc_localdelivery select mbox format in /var/mail/
 exim4/dc_local_interfaces string 127.0.0.1
 exim4/dc_minimaldns boolean false
 exim4/dc_other_hostnames string
 exim4/dc_eximconfig_configtype select mail sent by smarthost; received via SMTP or fetchmail
 exim4/no_config boolean true
 exim4/hide_mailname boolean false
 exim4/dc_postmaster string toor
 exim4/dc_readhost string
 exim4/use_split_config boolean true

BTW. I need this list to filter responses used to configure Exim in Debian. And I need the answers to reuse them in pressed.
Oh, I obtained the list using:

debconf-get-selections  --install > test2

Xenserver & ipmitool

I found that to enable access BMC information locally from Xenserver you need to load following modules:

•  modprobe ipmi_msghandler
•  modprobe ipmi_devintf
•  modprobe ipmi_si

 and latter as normal:
  ipmitool -I open channel info


Tested on Dell R310 without Drac module.



Another thing worth to note  about Dell and it's BMC — it is not enable by default and you need to press Ctrl+E in the right moment and turn it on. 

UPDATE

Dell support was wrong I can turn on access to BMC over LAN without rebooting box:

• ipmitool -I open lan set 1 access on
• ipmitool -I open lan set 1 user
• ipmitool -I open lan set 1 ipsrc dhcp

I'm not sure if I needed all three command. I'll check another day.

IPMITOOL man:  http://ipmitool.sourceforge.net/manpage.html

not so talkative expect

I'm not sure how I could miss it, but just recently I learned how to set expect not to send everything to the standard output. In the expect file you have to set log_user 0. you can change it but by log_user 1.

This is very helpful to write bash scripts i.e. using my remote_command.expect script - awking output is much easier.

Expect, telnet and Dell switch

Recently I've tried to find a way to obtain MAC addresses of computers attached to a Dell Switch. I've found that i.e. show bridge address-table ethernet 1/g3 returns  MAC addresses behind the port 3. I found small problem - it's not so easy to query all ports (or a chosen set of ports). Therefore, I decided to use expect. You can find the script below.
It connects to a switch run the command and finally print a mac address in the format with ':' between number doublets.
However, there are two issues. I don't know how to avoid sending everything to the standard output. Moreover, if there are more than one MAC address connected to the port (i.e. virtual machines), only first address will be print on the bottom of the output. To be precise the address  appears in the line 6 of output (see line started with set temp_mac).

#!/usr/bin/expect -f

set timeout -1

set machine [lindex $argv 0]
set port [lindex $argv 1]
set command "show bridge address-table ethernet 1/g$port\n"

#Connect to the server
spawn telnet $machine 

expect "User:"
exp_send "admin\r"
expect "Password:"
exp_send "myXEN\r"
expect "?*>"
exp_send "enable\r"
expect "?*#"
exp_send $command
expect "?*#"
set temp_mac [ lindex [ lindex [ split $expect_out(0,string) "\n"] 6] 1]
exp_send "exit\r"
exp_send "quit\r"

puts "\n"

# Creating mac address in DHCP format (with ':')
set mac [ string range $temp_mac 0 1 ]
append mac "."
append mac [ string range $temp_mac 2 6 ]
append mac "."
append mac [ string range $temp_mac 7 11 ]
append mac "."
append mac [ string range $temp_mac 12 13 ]

puts  "$machine/$port:  [ string map  {. :} $mac]"

exit

Oh one more thing. To make a list of all following it's good to run the script in loop and the following one should be good base to start with.

for ((i=1;i<48;i++)) do mac-dell.expect esw44-1 $i| grep "esw44-1/$i"; done

Upgrade Dell BIOS from XenServer 5.6

I've just found that to upgrade the Dell BIOS for R310 (and I guess other servers) on XenServer 5.6 you have to provide the lockfile file which comes from the procmail RPM.

Expect, ssh and two passwords

Let say that you need to run a command on many remote machine and you cannot use any of specialist tools (I mean i.e. func). In such situation expect might be helpful and I even has written about it already. Recently I had to modified my script, cause not all remote systems have the same root password.

#!/usr/bin/expect -f

set timeout -1

set machine [lindex $argv 0]
set command [lindex $argv 1]

set pass "xen4ulez\r" 
puts $machine

spawn ssh -o "NumberOfPasswordPrompts 2" -o "ServerAliveCountMax 1" -l root $machine $command
match_max 100000

expect {
    "assword:" 
      {
        exp_send $pass 
        set pass "xen!king\r"
        exp_continue
    }
}

First, the script has a password set as a variable. It allows to change later. To to this I have expanded the 'expect password' part of the script. Now the script sends the password and changes it after. If the password was fine, script executes the command. The change of the password is not important. If the password is wrong the script sends the variable $pass again, but this time it is new value. I have also added to option to ssh command for script to run a bit faster.

1. NumberOfPasswordPrompts 2 - ensure that ssh tries only twice to provide password.
2. ServerAliveCountMax 1 - ensure that ssh sends only one Server Alive message

Finally I have set StrictHostKeyChecking no and UserKnownHostsFile /dev/null in the .ssh/config file for machines I'm going to use above scripts. This two options set ssh not to bother about checking Host Key. Please remember that it might be not secure. 

Do something every minut in shell

Recently I've needed to do some statistic based on apache log file. I've wanted statistic to be update every minute and want to use only shell.

The first idea was cron.  Sounds nice but there is small problem. I want to be flexible. In theory I can add a job to crontab in the moment I want to start it and remove it (comment out) after finishing, but it's ideal.

No cron, so maybe sleep, especially sleep 60.... But there is another problem. My scripts runs few seconds, so after few minutes (approx. 60/time my scripts runs) I'll have a gap in results.

No cron, no sleep. I needed another direction. Recently I've been playing a bit with data formats. I write a bit of code and after few minutes – volia – I had a working scripts.

#!/bin/sh

min=`date  "+%M"`

while [ 1 ]
do
        ~/test.sh
        while (($min==`date  "+%M"`))
        do
                sleep 10
        done
        min=`date  "+%M"`
done 

One more thing which might be interesting for some. I used infinite loop based on  this blog entry.

How to check a website from the commandline

Sometimes it's useful to check a website connectivity using only command line. I.e. when trying a website from remote host, or when you set up a new box you are going to put behind a loadbalancer.

If you need HTTP connection, telnet will be enough:

> telnet 10.10.10.10 80

GET /index.html HTTP/1.1

host: important.site.com

In the case of HTTPS you need openssl:

> openssl s_client  -connect 10.10.10.10:443 -prexit

GET /index.html HTTP/1.1

host: important.site.com

This is entry for my extend my 'external memory', but you might find it useful.