Control terminal name and comment block in VIM

Somewhere (I think it was Stackoverflow) I found simple command to control the name of terminal from commandline which works very nice with iterm2 tabs on MacOSX and decided to add to my zsh environment this function:

termname() {
 echo -en "\e]1; $1 \a"
}

And if we are saying about Stackoverflow one of the most useful Vim suggestion I've ever found is this instruction to comment/uncomment a multiple lines in VIM.

Many AWS accounts and Zsh

That might not be a common problem, but I have to deal with many AWS accounts in the same time. For example I might to have to run an Ansible playbook for one account and a CLI commend for other one in parallel. To make my life easier i wrote this ZSH function. (It's probably going to work in other advance shell).

There are two ways to use it.

• awsenv list - returns the list of all available accounts/environments.
• awsenv - sets, based on values provided in ~/.aws/credentaials and ~/.aws/config, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.

The list is create with simple AWK script (assuming that any lines with "[]" is OK.

The actual command to set environment uses two AWK scripts. First one looks for requested account name and set variable "a" to 1. When "a" equals 1 it prints shell export command for access_key_id and for secret_access_key to  standard output, which is redirected to $TMP_FILE. Then it sources, prints and deletes that file.

Please note that in current form script requires access_key_id being define before secret_access_key.  Printing the value of all variables, especially secret_aceess_key could be consider as security weakness, so you might want to modify/remove "cat $TMP_FILE line.

# vim: set filetype=sh
awsenv () {
    if [[  $1 == list ]]
        then
        print $1
        awk  '/\[.*\]/ {print $1}'  ~/.aws/credentials
    else
        TMP_FILE=/tmp/current_aws
        awk \
           'BEGIN{a=0};\
           /\['$1'\]/ {a=1};\
           /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
           /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
            ~/.aws/credentials > $TMP_FILE
        awk \
            'BEGIN{a=0};\
            /\[profile '$1'\]/ {a=1};\
            /region/ {if (a==1){printf "export AWS_DEFAULT_%s=%s\n", toupper($1), $3; a=0}}'\
            ~/.aws/config >> $TMP_FILE
        source $TMP_FILE
        cat $TMP_FILE
        rm $TMP_FILE
    fi
}


Finally, initial version of this script and some discusion on profiles in older boto versions is in this post http://larryn.blogspot.co.uk/2015/03/how-to-deal-with-aws-profiles.html

Real "Get All Launch Configuration" In Boto

During my recent Ansible tests I created 'some' number of launch configuration, enough to reach account limit and I wanted/had to clean it. Boto sounded like a good candidate to do this. This few lines should address my problem.

import boto
import boto.ec2
import boto.ec2.autoscale

asg = boto.ec2.autoscale.connect_to_region('us-east-1')
results = asg.get_all_launch_configurations()

But it didn't. I could even found my launch configurations in the results sets. I figured out quickly that my results set is rather big and by default get_all_lunch_configurations method paging results (AFAIR default value is 20). Using example from this post on SDB I create following function doing what above method name promises - get all lunch configuration.


def get_all_launch_configuration(connection)
    """get_all_launch_configuration(connection) -
       returns results set of all launch configuration,
       regardless it size. Function require established
       boto.ec2.autoscaling connection."""

    results = connection.get_all_launch_configurations()
    token = results.next_token
    while True:
        if token:
            r =  asg.get_all_launch_configurations(next_token=token)
            token = r.next_token
            results.extend(r)
        else:
            break
    return results
       

A small example of advance aptitude usage

Few years ago I 'preserved' few example of advance aptitude usage I found on a Debian mailing list. Recently I had a chance to use them again. I was looking for a version of packages with distinguish string in a name (let say it was 'apache').

aptitude versions \$(aptitude search ~iapache| awk '{print \$2}')

• So first I ran aptitude search for 'apache' term, but only among installed packages - aptitude search ~iapache
• From the results I cut package name (second column) - awk '{print \$2}'. Please note I escaped dollar character, because I run this command using Ansible (see this blog entry for more details).
• The outcome of those two command allows me to query package versions - aptitude versions

Links

• http://larryn.blogspot.co.uk/2012/01/some-time-ago-on-debian-mailing-list.html
• http://larryn.blogspot.co.uk/2015/04/the-cassandra-ansible-pipe-and.html

Crux as a second system in GRUB2

I tried to install CRUX as a secondary system on a DELL XPS 13 (Ubuntu Edition). I run setup, configured fstab and rc.conf, compiled a kernel and then tried to add new system to the GRUB2 menu in Ubuntu. The update-grub script recognised the new Linux entry, added it, but CRUX didn't start — kernel panicked. It could not find root partition. I double checked my kernel and had all important options mentioned in CRUX installation compiled in.
Then I looked at menu entry created by os-prober for CRUX and noticed that it had following line:

        linux /boot/vmlinuz root=/#ROOT_DEVICE# ro quiet

I analysed what the probe script do and found that it checked boot loader configuration files from a new/additional Linux. Then I remembered that I had not bothered to configure LILO, because I had plan to use Ubuntu GRUB2 rather than it. I updated /etc/lilo.conf (without runing lilo command itself).
After that update-grub properly created CRUX entry in boot menu and I could start my CRUX installation.

zsh and ssh-agent

 INTRODUCTION

One of the post which gets some attention on this blog is My Way For Binding SSH Agent With Zshell. The method presented there is far from ideal and it stopped to work for me some time ago. After that I wrote a new version. I think it is much better and should work with bash or other shells. I tested it on Ubuntu and Crux.

ZSSH

I have the .zssh file in my home directory. It is sources by my .zshrc file. The .zssh consists of 3 functions.

SSHAGENT

The first function is responsible for starting ssh-agent.

sshagent () {
    SSHAGENT=$(ps ax|grep "[s]sh-agent"| grep -cv Z)
    if (( $SSHAGENT == 0 ))
    then
        sshupdate
    else
        SSHPID="$(ps -eo pid,command | awk '/ ssh-[a]gent/ {print $1}');"
        SSHPID_ENV=$(awk  '/Agent/ {print $NF}' ~/.ssh-env)
        if [[ $SSHPID == $SSHPID_ENV ]]
        then
            source ~/.ssh-env
        else
            killall ssh-agent
            sshupdate
        fi
    fi
}


It checks if a ssh-agent runs already and it isn't a zombie. (On one of my systems, after starting a desktop environment, I always had a zombie ssh-agent running.) If there is no ssh-agent running the function calls sshupdate, another function described below. If the agent is present and live in a system the function then compares ssh-agent pid with the information saved in the ~/.ssh-env file. (See sshupdate paragraph for more information.) If informations are consistence it sources .ssh-env.  If not it kills all ssh-agent and the calls sshupdate.

SSHUPDATE

This is a very simply function calling ssh-agent and saving its output to a file.

sshupdate () {
    ssh-agent > ~/.ssh-env
    source ~/.ssh-env
}

The output then can be sourced by other functions or processes. Oh, and if you don't remember/know the output of ssh-agent looks like that:

SSH_AUTH_SOCK=/tmp/ssh-BnXafqRnOSHx/agent.1884;
export SSH_AUTH_SOCK;
SSH_AGENT_PID=1885; 
export SSH_AGENT_PID;
echo Agent pid 1885;

SSHADD

Finally the function responsible for adding your ssh key.

sshadd () {
    if (( $(ssh-add -l | grep -c $USER) == 0 ))
    then
        ssh-add
    else
        ssh-add -l
    fi
}

It checks the number of added keys. If a key from you home directory, or having your username in the path, is not present it adds it. Otherwise it lists all added keys.

USAGE

sshagent is called from your .zshrc, so it should be present during every session. sshadd need to be called by you, when you need it first time.

FURTHER UPDATES

What if you have more than one key and you would like ti add all of them in the same time. Then you could try to use the 'file' program to find ssh keys in the.ssh, or other, directory and source all of them.

The Cassandra, the Ansible, a pipe and a complicate command

I've been working on Cassandra ring downsizing (not a fanny task). To make things a bit easier I've been using Ansible for configuration management. In theory you don't need to change the value of 'initial_token' in Cassandra configuration, and everything else stays the same during ring resize. Therefore, Ansible is not really necessary, but I believe it's good to have consistence in your configuration.

Cassandra role

In first place I added a variable with the Cassandra ring size as a max_nodes  to its role.

  - {role: cassandra, max_nodes: 13}

This value is used to create a Ansible local fact. To learn more on local facts please read this Curtis Collicutt article, which is the best for my script. Please note that the following code is a Jinja template not ready script.

#!/bin/sh
NODE=$(hostname |grep -o -P "\d+")
TOKEN=$(echo '2^127/{{ max_nodes }}' | bc)
cat <
{
    "node_number": $NODE,
    "token": $TOKEN
}
EOF

The above script print onto standard output a json with two variables.

• node_number - is created based on hostname. The number a hostname correspond to a ring position (i.e. cassandra-4 is forth node).
• token - is 'a main part' of token value calculation. The rest is done in Cassandra configuration template:

    initial_token: {{ '%d' % ( ansible_local.cassandra.token *  (ansible_local.cassandra.node_number - 1)) }}

That's sounds complicated and it is. As often there is a historical explanation to that situation. Initially, the whole calculation was done in the template. It worked for 16 node ring, but not for 15. It didn't work, because division two integral  number (2**127/15) results in float one in jinja what lead to rounding error!

Ansible commands

As a bonus two example of using Ansible to run a bit more complicated command on many hosts.

1. Reads token position from configuration file and initialize node move in a screen session. Please note escape character in front of '$'.
   ansible \
    -i inventory/ec.py \
    -m shell \
    -a "screen -d -m \
      nodetool -h localhost move \
      \$(awk '/initial_token/ {print \$2}' \
        /etc/cassandra/default.conf/cassandra.yaml)"\
   tag_pool_cassandra
   
   
2. Much simpler. Checks that screen runs. AFAIK you have to use shell module if you want to use UNIX pipe.
   ansible \
    -i inventory/ec2.py \
    -m shell \
    -a "ps -ef| grep SCREEN"\
   tag_pool_cassandra
   

How to deal with AWS profiles

I don't know how common it is to be a part of an organisation having many AWS (Amazon Web Services) accounts, but it's make things tricky. Amazon make it relatively easy to use many 'named profiles' (account) with AWS CLI. (If you haven't try see this documentation). Boto (python AWS interface) developers also added easy way to use the same profiles in version 2.29. (How to use the same profiles in Boto and other SDKs check this article.) But release 2.29 is not so old and what if you got stacked with older version (for example the one from the latest Ubuntu LTS)? I was in such situation and wrote this small function to use it with profiles from the ~/.boto (not ~/.aws/) file.

def set_account(environment):
    """set_credentials(environment) -
    sets credentials for given environment/account.
    """
    for i in boto.config.items(environment):
        boto.config.set('Credentials', i[0], i[1])

So if your profile is called 'prod':

import boto

set_account('prod') 
conn = boto.connect_ec2()

Another program having issues with many AWS accounts is Ansible. (However, authors claims it's a feature not a bug). My first approach was to add the above function in the ec2.py inventory script and  further extended it by adding following lines:

• to the __init__ method of the Ec2Inventory class:

   
  set_credentials(self.args.environment)
   

• and to the parse_cli function:
  

   
  parser.add_argument('-e', '--environment', type=str, required=True,
                             help="select an environment/profile to run.")
   
  

Such prepared script is not ready to use with Ansible yet - a wrapper around it is needed as well. For example the bash script called ec2-prod.sh to use Ansible in the 'prod' environment.

#!/bin/sh
cd $(dirname $0)
./ec2.py -e prod --refresh-cache

If you need to know why the wrapper is needed check the Ansible inventory code.

Such approach is not ideal if you have many account to work with, you will need a wrapper for each one. What worse, it doesn't work with unified AWS config approach and require to keep a unique version of the inventory script. Therefore, I tried to find a better resolution. I could not find anything interesting and decided to write a small shell script to read ~/.aws/credentials and exports AWS keys for selected profile. The script is a simple wrapper around a bit complicated awk command. To use it you have to source, not execute, it, because the script should execute in a current shell.

#!/bin/bash

TMP_FILE=/tmp/current_aws
awk  \
 'BEGIN{a=0};\
 /\['$1'\]/ {a=1};\
 /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
 /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
 ~/.aws/credentials > $TMP_FILE

source $TMP_FILE
rm $TMP_FILE

The script ensure that:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

are exported. This works not only with Ansible, but also can simplified AWS CLI usage, because the --profile option can be dropped. It might also help with other tools.

Links:

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
• http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
• http://boto.readthedocs.org/en/latest/
• http://www.ansible.com/home
• http://docs.ansible.com/intro_dynamic_inventory.html

Datadog and many dataseries stacked together

Recently, I've started to use Datadog. It has nice features, but I have also found some annoying lacks. One of them is no easy way to prepare a graph with a stack of different series in one graph, for example nice representation of CPU time spent in different states.

Luckily, as you can see above it can be done. You just need to change some things in JSON and have something similar to what I got below. The main point is to have all dataseries in the argument of one "q".

{
  "viz": "timeseries",
  "requests": [
    {

      "q": "avg:system.cpu.system{host:host-01}, avg:system.cpu.user{,host:host-01}, avg:system.cpu.iowait{host:host-01}, avg:system.cpu.stolen{host:host-01}, avg:system.cpu.idle{host:host-01}",
    },
      "type": "area"
  ],
  "events": []
}

Count processes per state per application

In previous posts (here and here) I discussed how to count thread in a given state for a give process. Recently, I had another problem - I needed to count number of processes per application per state. My previous commands wouldn't work, so I wrote an alternative version.

while [ 1 ];
do
    date;
    cat /proc/loadavg;
    ps -Leo state,args |
     awk ' $1 ~ /(D|R)/ {state[$0]++} \
      END{ for (j in state) {printf "%s - %d\n", j, state[j]}}' |
      sort -k 2;
    echo "---";
    sleep 5;
done 

There is not PID and args are included in the output list as a whole. worried for number of processes.

One more thought. Dropping "$1 ~ /(D|R)/" can be useful in case of problem with total number of processes. But then the whole command should be a bit modified, so the results are sorted by number of processes. Simplified version would look like this one:

while [ 1 ];
do
    ps -Leo state,args |
     awk ' $1 ~ /(D|R)/ {state[$0]++} \
      END{ for (j in state) {printf "%d - %s\n", state[j], j}}' |
      sort -n;
    echo "---";
    sleep 5;
done 

What is the (UNIX) load?

The "load" is use widely to describe stress/work applied onto a UNIX system. The simple rule is "lower than better". In the older days of uniprocessor machine load 1 was  kind of a borderline. In the new brave world of multi-core/processor machines load 1 means nothing.  Many people suggests that load equal or lower to number of processors/cores is good. That sound sensible, but not always is accurate.
Why? To answer that we have to comeback to question asked in the subject.

What is the "load"?

The load as the exponentially damped/weighted moving average of the number of processes, including threads, using or waiting for CPU and, at least at Linux, in uninterruptible sleep state in last 1, 5 and 15 minutes (see Wikipedia). The last part means that all processes/threads waiting for a disk (or other I/O device) will increase the load, without increasing a CPU usage. It leads to situation when the load lower than number of core/processes is danger. Let imagine few processes trying to dump important information on disks. Especially if all interrupts have affinity to one processor only (see this post) or just data are store in many small files. On the other hand, machine with very high load might be very responsive. Plenty of processes waiting to write information onto a disk not using a lot of memory and CPU in the same time. Just look at this picture:

If you want to know even more details of how the load is actually calculated
read this impressive white paper.


Links:
http://en.wikipedia.org/wiki/Load_%28computing%29
http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
http://larryn.blogspot.co.uk/2013/05/cpu-affinity-interrupts-and-old-kernel.html

Install CyanogenMod at Nook HD+

Recently I decided to try the new Cyanomogen (CM11) on my Nook HD+. Initial reading indicated that I had to reinstall using Recovery rather than internal updater. I tried to login to Recovery so much, that I recovered official B&N OS which replaced CM.

I needed to start from beginning. I did some research and found that post. It looked good so I gave it a try. First download ClockworkMod attached to the post, but later I downloaded latest CM snapshot from  there and  added  Google Apps for CM11 from there. I put everything as described on SD Card and kicked off installation. It flew like an Albatross. (To be honest I don't know why I did write Albatross - maybe because of this?) 



Anyway CM11 works good at Nook HD+.

Links:

• http://wiki.cyanogenmod.org/w/Ovation_Info
• http://download.cyanogenmod.org/?type=snapshot&device=ovation
• http://wiki.cyanogenmod.org/w/Google_Apps
• http://forum.xda-developers.com/showpost.php?p=42406126&postcount=7
• http://forum.xda-developers.com/attachment.php?attachmentid=2849350&d=1405272804

More fabric as a library

Recently I had to prepare a tool doing some remote commands, so of course I decided to use fabric, but I have big problem to control hosts. I remembered that I had written a short article on Fabric in here some time ago. But it didn't help. I asked on the Fabric mailing lists, but there was no help.

Manual host name control

In this tool I didn't need to run many parallel SSH connection, so I decided to control remote host name from inside the loop in the function my setting env.host_string each time (this is very useful functionality). Like in following example:

#!/usr/bin/env python

"""Example code to use Fabric as a library. 

It shows how to set up host manually.

Author: Wawrzek Niewodniczanski < main at wawrzek dot name >

"""

# import sys to deal with scripts arguments and of course fabric 

import sys

import fabric
from fabric.api import run, hide, env

env.hosts = ['host1', 'host2'] 

# Main function to run remote task 

def run_task(task='uname'):
    """run_task([task]) -
    runs a command on a remote server. If task is not specify it will run 'uname'."""

    # hide some information (this is not necessary).

    with hide('running', 'status'):
        run(task) 

# Main loop

# take all arguments and run them on all hosts specify in env.hosts variable

# if not arguments run 'uname' 

if len(sys.argv) > 1:

    tasks = sys.argv[1:]

    for task in tasks:

        for host in env.hosts:
            env.host_string = host
            run_task(task)

else:

    for host in env.hosts:

        run_task() 

Fabric in full control

The problem bugged me since then. Yesterday I found some of my old code. Analysed it and quickly found small, but profound difference with mu recent fabric usage. Rhe code above called the run_task function wrongly. Rather than dealt it in the normal way I supposed to use execute.

#!/usr/bin/env python

"""Example code to use Fabric as a library. 

It shows how to set up host manually.

Author: Wawrzek Niewodniczanski < main at wawrzek dot name >

"""

# import sys to deal with scripts arguments and of course fabric 

import sys

import fabric
from fabric.api import run, hide, env, execute

env.hosts = ['host1', 'host2'] 

# Main function to run remote task 

def run_task(task='uname'):
    """run_task([task]) -
    runs a command on a remote server. If task is not specify it will run 'uname'."""

    # hide some information (this is not necessary).

    with hide('running', 'status'):
        run(task) 

# Main loop

# take all arguments and run them on all hosts specify in env.hosts variable

# if not arguments run 'uname' 

if len(sys.argv) > 1:

    tasks = sys.argv[1:]

    for task in tasks:

        execute(run_task, task)

else:

    execute(run_task)

Links:

http://www.fabfile.org/
http://larryn.blogspot.co.uk/2012/11/fabric-as-python-module.html
http://lists.nongnu.org/archive/html/fab-user/2014-10/msg00002.html

Resource for vim

Just some link with useful Vim's advices.

General Vim advices

• https://code.google.com/p/vimcolorschemetest/ - many beautiful colourschemes.
• http://tnerual.eriogerg.free.fr/vimqrc.html - many shortcuts (to keep under the pillow).
• http://vimregex.com/ - all you might want to know about regex in Vim.
• http://dailyvim.blogspot.co.uk - plenty of short yet useful Vim advices.
• http://vimcasts.org/ - great Vim screencast with good support.

Vim and Python

• https://dev.launchpad.net/UltimateVimPythonSetup/
• http://justinlilly.com/vim/vim_and_python.html
• http://haridas.in/vim-as-your-ide.html
• http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/
• http://www.sontek.net/blog/2011/05/07/turning_vim_into_a_modern_python_ide.html
• http://www.vim.org/scripts/script.php?script_id=1494
• https://github.com/scrooloose/syntastic
• http://vim.wikia.com/wiki/Omni_completion
• https://code.google.com/p/vimpdb/

(w)dstat

wdstat

In my .profile (on CentOS 5, just in case there were some changes in dstat) I have following alias to dstat (wdstat stands for Wawrzek's dstat):

alias wdstat="dstat -lcpymsgdn 5"

Where the options stands for:

• -l  - UNIX load (1m   5m  15m) load average in 1, 5 and 15 minutes, respectively;
• -c - cpu stats (usr sys idl wai hiq siq) percent of time spent in user and system space, idle, waiting on resource,  serving interrupts and softirqs (software interrupts);
• -p - process stats (run blk new) number of running, blocked and newly created processes;
• -y - system stats (int   csw) - number of interrupts and context switches;
• -m - memory stats (used  buff  cach  free) amount of memory used by processes, disk buffers, disk cache and free;
• -s - swap stats (used  free) - amount of used and free swap space;
• -g - page stats (in   out) number of page put in and out from swap;
• -d -disk stats (read  writ) - number of reads and writes from all disks;
• -n -network stats (recv  send) number of received and send network packages;

Further reading:

• http://dag.wiee.rs/home-made/dstat/
• http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
• http://www.linuxhowtos.org/System/procstat.htm
• http://lwn.net/Articles/520076/
• http://en.wikipedia.org/wiki/Paging#Linux

netstat, ports, hosts and awk glue

Recently, I needed to create a list of all servers connected on a given port (in following example port 80). I used a mixture of awk and other UNIX command line tools.

netstat -nt| \
 awk -F':'\
   '$5==80 {count[$8]++} \
   END{ for (i in count) { \
      cmd="host "i; \
      cmd |& getline j; \
      split(j, a, " "); \
      printf "%40s - %d\n", a[5], count[i]}}'| \
 sort -n -k 3

First netstat provided the list of all connection (netstat -nt); -n stands for numeric and -t for only TCP connections.

Next awk, with the ':' defined as a field separator (awk -F':'), used lines where local port was 80 ($5==80) to create an associated array with a key define by connected host ip and a value equal to  number of connection from it ({count[$8]++}). At the end of the script execution, awk looped over all element of the array (END{for (i in count)).  Next there was a crux of the script, the cmd was define as a run the OS host command with the awk variable i as an argument  (cmd="host" i). The |& operator created two-way pipe between awk and a execution of the previously defined cmd. The getline command was used to store cmd output into the variable j (cmd |& getline j). Next the split command split the content of the  j into separate words and saved them into the a array (split(j, a, " ")). Finally the printf formatted output (printf "%40s - %d\n", a[5], count[i])).  The actual hostname was fifth element of the a.

For continence, output lines were sorted by numeric order on third column  (sort -n -k 3). Each output line consisted of a hostname ,'-' and a number - e.g. important.com - 3456.

Python for SysAdmins

Preparing for a interview some time ago I made a list of python module interesting for SysAdmins. Today looking for something else I found that half baked note and decided to polish it enough to put it on the blog. It's mostly for myself as quick reference.

Each module is describe by one, two sentences (from Python documentation) and have a link to official online document. At the end there is a list of example function, object.

Python modules for SysAdmin

import sys

This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is always available.

https://docs.python.org/2/library/sys.html

examples:

• argv,
• exit(),
• path,
• modules,
• exec().

 

import os

This module provides a portable way of using operating system dependent functionality.

https://docs.python.org/2/library/os.html

examples:

• chdir(),
• getuid(),
• uname(),
• listdir(),
• stat(),
• rename(),
• access().

 

import os.path

This module implements some useful functions on pathnames.

https://docs.python.org/2/library/os.path.html

examples:

• isdir(), 
• isfile(), 
• exist(), 
• getmtime(), 
• abspath(), 
• join(), 
• basename(), 
• dirname().

 

import time

This module provides various time-related functions.

https://docs.python.org/2/library/time.html

examples:

• time(), 
• ctime(), 
• sleep(), 
• strftime(), 
• strptime().

 

import glob

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.

https://docs.python.org/2/library/glob.html

examples:

• glob(),
• iglob().

 

import fnmatch

This module provides support for Unix shell-style wildcards, which are not the same as regular expressions (which are documented in the re module).

https://docs.python.org/2/library/fnmatch.html

examples:

•  fnmatch().

 

import re

This module provides regular expression matching operations similar to those found in Perl. Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings.

https://docs.python.org/2/library/re.html

examples:

• compile(), 
• match(), 
• search(), 
• split(), 
• findall(), 
• sub(),
• group().

MatchObject

Match objects always have a boolean value of True. Since match() and search() return None when there is no match, you can test whether there was a match with a simple if statement:

match = re.search(pattern, string)
if match:
    process(match)

Zombie

Usually zombie process is not a big problem, but sometimes... just look at the screenshot below. It wants the whole machine as fried eggs! or maybe boiled?

Multiprocessing (in Python)

I needed to do some multi-threading in Python. As I needed effect quick I decided to use standard threading module. However, every time I had to use it I felt it was rather complicated beast. At threading module documentation page there is a link to multiprocessing module. I was tired with threading, on the one hand, but didn't have enough time to learn about greenlet or another competing project, on the other, so I decided to take a quick glance at multiprocessing module...
... And my life became much easier, sky bluer, grass, greener, oh and scripts faster ;-).
I don't do anything special with it, so just one simple code example, but this is very good tutorial you can find much more: http://pymotw.com/2/multiprocessing/communication.html.

Main part of nearly all my scripts looks the same:

import multiprocessing 

SIZE = 30
pool = multiprocessing.Pool(processes=SIZE)
pool_output = pool.map(get_values, servers)
pool.close() # no more tasks
pool.join()

Where servers is a list with servers I need get information from, and get_values is a function (sometimes with a different name).  Simple, isn't it?

shell, history and substitution

One of the most known tricks in using shell history is to use:

^old^new

to replace string old by new in last command. The only problem is that it replaces only first appearance. But there is another command replacing all string old by new:

!:gs/old/new/