Search This Blog

Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Tuesday, June 30, 2026

Cruxpy as an example of the python package release process

As mentioned beforeCruxPy helped me to automate the process of keeping the repository of my Crux ports rolling. It also gave me a chance to prepare a pipeline for a Python package with GitHub Action (GHA) workflows.

On every commit the GHA runs 2 jobs. First, pytest. When the tests pass, it checks the version stored in the pyproject.toml file, and compares it to the latest tag. If the version from the file is higher than the current tag, the new one is pushed. These jobs are stored in the commit.yaml file.

When there is a new tag pushed to the code, another workflow, from the release.yaml file, kicks off. It has only one job - to create a GitHub release.

The completion of the "New Release" job triggers the third workflow saved in the deploy.yaml file. It is responsible for publishing the python package to the pypi repository.

If you are just starting your adventure with GHA, you might want to check triggers to understand them better.  This pipeline uses three distinct types:

  • commit is defined as push to every branch (excluding tags)
  • release uses a push of a tag, that matches 3-number semantic version
  • deployment relays on the internal GitHub workflow dependencies.

This structure allows me to push changes directly to the main branch safely. Nothing becomes visible outside of repository until the version is bumped!

Wednesday, September 17, 2025

Crux Port Automation

Crux ports 

To simplify keeping my Crux port repository up to date, I set GitHub Actions. There are 2 at the moment. The first one is the simple check of basic rules (trailing white spaces, lack of big files etc.) implemented with pre-commit.

The second one provides the crucial automation. The script takes the repository code, downloads the cruxpy library (see below for details). Then it gets Crux httpup-repgen script from the private s3 bucket. (Stored there to avoid 3rd party dependencies). It uses a very simple python script (portspage.py) to create a page for ports and then port-sync.sh to create the REPO file and upload the whole repo to the server. The SSH keys required to log in to the remote machine are stored in GitHub secret.

CruxPy

Initially, my automation based on standard Crux scripts. I was copying the repository from my desktop, even if the source code was stored in GitHub. Then I introduced GitHub Actions. The automation kept working from the remote machine fine. But there was a small issue. The last update date of ports in web UI was populated with the timestamp of the last file update on the GHA runner. They all were the same, because the code was downloaded every time. To fix it, I decide to prepare the CruxPy, a simple Python module.

Initially, I thought of preparing an equivalent of portspage.sh script, but with ports update date using git file metadata. Drafting the solution, I decided that the object-oriented approach suites the problem well; a port is an object, the repository website is as well. And in the future, the port class can be also used to prepare a repo one, and write CLI tools.

Having basic classes, I thought that it would be a great opportunity to prepare my first Crux package and add some automation. Which deserve a separate article.

Links

Wednesday, September 02, 2015

Real "Get All Launch Configuration" In Boto

During my recent Ansible tests I created 'some' number of launch configuration, enough to reach account limit and I wanted/had to clean it. Boto sounded like a good candidate to do this. This few lines should address my problem.

import boto
import boto.ec2
import boto.ec2.autoscale

asg = boto.ec2.autoscale.connect_to_region('us-east-1')
results = asg.get_all_launch_configurations()

But it didn't. I could even found my launch configurations in the results sets. I figured out quickly that my results set is rather big and by default get_all_lunch_configurations method paging results (AFAIR default value is 20). Using example from this post on SDB I create following function doing what above method name promises - get all lunch configuration.


def get_all_launch_configuration(connection)
    """get_all_launch_configuration(connection) -
       returns results set of all launch configuration,
       regardless it size. Function require established
       boto.ec2.autoscaling connection."""

    results = connection.get_all_launch_configurations()
    token = results.next_token
    while True:
        if token:
            r =  asg.get_all_launch_configurations(next_token=token)
            token = r.next_token
            results.extend(r)
        else:
            break

    return results
       

Friday, March 06, 2015

How to deal with AWS profiles

I don't know how common it is to be a part of an organisation having many AWS (Amazon Web Services) accounts, but it's make things tricky. Amazon make it relatively easy to use many 'named profiles' (account) with AWS CLI. (If you haven't try see this documentation). Boto (python AWS interface) developers also added easy way to use the same profiles in version 2.29. (How to use the same profiles in Boto and other SDKs check this article.) But release 2.29 is not so old and what if you got stacked with older version (for example the one from the latest Ubuntu LTS)? I was in such situation and wrote this small function to use it with profiles from the ~/.boto (not ~/.aws/) file.

def set_account(environment):
    """set_credentials(environment) -
    sets credentials for given environment/account.
    """
    for i in boto.config.items(environment):
        boto.config.set('Credentials', i[0], i[1])

So if your profile is called 'prod':

import boto

set_account('prod') 
conn = boto.connect_ec2()

Another program having issues with many AWS accounts is Ansible. (However, authors claims it's a feature not a bug). My first approach was to add the above function in the ec2.py inventory script and  further extended it by adding following lines:
  • to the __init__ method of the Ec2Inventory class:
     
    set_credentials(self.args.environment)
     
  • and to the parse_cli function:
     
    parser.add_argument('-e', '--environment', type=str, required=True,
                               help="select an environment/profile to run.")
     
    
Such prepared script is not ready to use with Ansible yet - a wrapper around it is needed as well. For example the bash script called ec2-prod.sh to use Ansible in the 'prod' environment.

#!/bin/sh
cd $(dirname $0)
./ec2.py -e prod --refresh-cache

If you need to know why the wrapper is needed check the Ansible inventory code.

Such approach is not ideal if you have many account to work with, you will need a wrapper for each one. What worse, it doesn't work with unified AWS config approach and require to keep a unique version of the inventory script. Therefore, I tried to find a better resolution. I could not find anything interesting and decided to write a small shell script to read ~/.aws/credentials and exports AWS keys for selected profile. The script is a simple wrapper around a bit complicated awk command. To use it you have to source, not execute, it, because the script should execute in a current shell.

#!/bin/bash

TMP_FILE=/tmp/current_aws
awk  \
 'BEGIN{a=0};\
 /\['$1'\]/ {a=1};\
 /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
 /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
 ~/.aws/credentials > $TMP_FILE

source $TMP_FILE
rm $TMP_FILE

The script ensure that:
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
are exported. This works not only with Ansible, but also can simplified AWS CLI usage, because the --profile option can be dropped. It might also help with other tools.

Links:

Sunday, November 23, 2014

More fabric as a library

Recently I had to prepare a tool doing some remote commands, so of course I decided to use fabric, but I have big problem to control hosts. I remembered that I had written a short article on Fabric in here some time ago. But it didn't help. I asked on the Fabric mailing lists, but there was no help.


Manual host name control

In this tool I didn't need to run many parallel SSH connection, so I decided to control remote host name from inside the loop in the function my setting env.host_string each time (this is very useful functionality). Like in following example:


#!/usr/bin/env python
"""Example code to use Fabric as a library. 
It shows how to set up host manually.
 
Author: Wawrzek Niewodniczanski < main at wawrzek dot name >
"""
 
# import sys to deal with scripts arguments and of course fabric 
import sys
import fabric
from fabric.api import run, hide, env

env.hosts = ['host1', 'host2'] 
 
# Main function to run remote task 
def run_task(task='uname'):
    """run_task([task]) -
    runs a command on a remote server. If task is not specify it will run 'uname'."""
    # hide some information (this is not necessary).
    with hide('running', 'status'):
        run(task) 
 
# Main loop
# take all arguments and run them on all hosts specify in env.hosts variable
# if not arguments run 'uname' 
if len(sys.argv) > 1:
    tasks = sys.argv[1:]
    for task in tasks:
        for host in env.hosts:
            env.host_string = host
            run_task(task)
else:
    for host in env.hosts:
        run_task() 



Fabric in full control

The problem bugged me since then. Yesterday I found some of my old code. Analysed it and quickly found small, but profound difference with mu recent fabric usage. Rhe code above called the run_task function wrongly. Rather than dealt it in the normal way I supposed to use execute.

#!/usr/bin/env python
"""Example code to use Fabric as a library. 
It shows how to set up host manually.
 
Author: Wawrzek Niewodniczanski < main at wawrzek dot name >
"""
 
# import sys to deal with scripts arguments and of course fabric 
import sys
import fabric
from fabric.api import run, hide, env, execute

env.hosts = ['host1', 'host2'] 
 
# Main function to run remote task 
def run_task(task='uname'):
    """run_task([task]) -
    runs a command on a remote server. If task is not specify it will run 'uname'."""
    # hide some information (this is not necessary).
    with hide('running', 'status'):
        run(task) 
 
# Main loop
# take all arguments and run them on all hosts specify in env.hosts variable
# if not arguments run 'uname' 
if len(sys.argv) > 1:
    tasks = sys.argv[1:]
    for task in tasks:
        execute(run_task, task)
else:
    execute(run_task)


Links:

http://www.fabfile.org/
http://larryn.blogspot.co.uk/2012/11/fabric-as-python-module.html
http://lists.nongnu.org/archive/html/fab-user/2014-10/msg00002.html

Wednesday, June 25, 2014

Python for SysAdmins

Preparing for a interview some time ago I made a list of python module interesting for SysAdmins. Today looking for something else I found that half baked note and decided to polish it enough to put it on the blog. It's mostly for myself as quick reference.

Each module is describe by one, two sentences (from Python documentation) and have a link to official online document. At the end there is a list of example function, object.

Python modules for SysAdmin

import sys

This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is always available.

https://docs.python.org/2/library/sys.html

examples:

  • argv,
  • exit(),
  • path,
  • modules,
  • exec().

 

import os

This module provides a portable way of using operating system dependent functionality.

https://docs.python.org/2/library/os.html

examples:

  • chdir(),
  • getuid(),
  • uname(),
  • listdir(),
  • stat(),
  • rename(),
  • access().

 

import os.path

This module implements some useful functions on pathnames.

https://docs.python.org/2/library/os.path.html

examples:

  • isdir(), 
  • isfile(), 
  • exist(), 
  • getmtime(), 
  • abspath(), 
  • join(), 
  • basename(), 
  • dirname().

 

import time

This module provides various time-related functions.

https://docs.python.org/2/library/time.html

examples:

  • time(), 
  • ctime(), 
  • sleep(), 
  • strftime(), 
  • strptime().

 

import glob

The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.

https://docs.python.org/2/library/glob.html

examples:

  • glob(),
  • iglob().

 

import fnmatch

This module provides support for Unix shell-style wildcards, which are not the same as regular expressions (which are documented in the re module).

https://docs.python.org/2/library/fnmatch.html

examples:

  •  fnmatch().

 

import re

This module provides regular expression matching operations similar to those found in Perl. Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings.

https://docs.python.org/2/library/re.html

examples:

  • compile(), 
  • match(), 
  • search(), 
  • split(), 
  • findall(), 
  • sub(),
  • group().

MatchObject

Match objects always have a boolean value of True. Since match() and search() return None when there is no match, you can test whether there was a match with a simple if statement:

match = re.search(pattern, string)
if match:
    process(match)


Tuesday, March 25, 2014

Multiprocessing (in Python)


I needed to do some multi-threading in Python. As I needed effect quick I decided to use standard threading module. However, every time I had to use it I felt it was rather complicated beast. At threading module documentation page there is a link to multiprocessing module. I was tired with threading, on the one hand, but didn't have enough time to learn about greenlet or another competing project, on the other, so I decided to take a quick glance at multiprocessing module...
... And my life became much easier, sky bluer, grass, greener, oh and scripts faster ;-).
I don't do anything special with it, so just one simple code example, but this is very good tutorial you can find much more: http://pymotw.com/2/multiprocessing/communication.html.

Main part of nearly all my scripts looks the same:

import multiprocessing 


SIZE = 30
pool = multiprocessing.Pool(processes=SIZE)
pool_output = pool.map(get_values, servers)
pool.close() # no more tasks
pool.join()

Where servers is a list with servers I need get information from, and get_values is a function (sometimes with a different name).  Simple, isn't it?

Saturday, April 20, 2013

Regexp Groups in Python

Let say that we have a string consist from a numeric continent code  (i.e. 1 for Europe) , followed by a country code  (i.e. 44 for UK)  and ended with a region code made in similar manner (i.e. Cambridge equal 65). A continent and a country are preceded with the letter C and a location with the letter L.

So we have the string C1C44L65 and we need to know the continent, counter and region codes, but numbers might have different length (i.e. 5, but also 55 or 555) so we cannot just grab selected characters from a string, but in Python we can use following regular expression to save all that information into "groups" for further use, in example in an mathematical operations (as integers).

info = re.search(r"^C(?P[^\d]+)C(?P\d+)L(?P\d+)$", full_id)
continent = int(info.group("continent"))
country = int(info.group("country"))
region = int(info.group("region"))

Thursday, November 22, 2012

Fabric as a python module


Introduction

Fabric is a powerful tool to automate running the same command on many remote server. It is written in python, but authors concentrate on usage as command line tool rather as a python module.

There are some documentation how to do this. You can also find some examples in the internet, but  examples never too much, so below another one.

Scenario

We would like to run a command on each of our webservers, ideally in batches, but in a specific order. Our webserver are grouped in 'cells'. Each cell consist of 3 servers in different 'zones' (zone are called a, b, and c). For convince there is following convention of naming servers: "role-cell#-zone" i.e. web-04-a. To make things more interesting some cells are missing, so we cannot make simple loop. On the other hand, there is a service holding various servers information in json file accessible over http. Additionally, we have a python module called 'server' to locally access those information from our workstation. In the 'server' module we have class 'webserver' to deal with webservers. One of its method is 'get_all_server'. It return a list of all webserver.

Script

Servers

First we create the list of all servers using our external tool.

from servers import webserver
allservers = webserver.get_all_severs()


The order mentioned above is to run first all 'c' boxes next 'b' and finally 'a'. To achieve that, let creates 3 lists (in a function). Nothing magic, a bit of 're'.

def make_lists(servers):
    import re # if not imported in the main part


    wa = []
    wb = []
    wc = []
   
    for server in servers:
        if re.search('web-\d+-a', server):
            wa.append(server)
        elif re.search('web-\d+-b', server):
            wb.append(server)
        elif re.search('web-\d+-c', server):
            wc.append(server)

    separation = (wa, wb, wc)
    return separation


Fabric

So now we can start to use fabric. Let import necessary pieces:

# Fabric part
from fabric.api import *
from fabric.network import disconnect_all


set up environment

env.skip_bad_hosts = True
env.warn_only = True
env.parallel = True


and finally call the function:


@parallel(pool_size=10)
def run_remotely():
    with hide('running', 'status'):
       sudo(task)


Main part

Prepare the lists of servers (see paragraph servers):

lists = make_lists(get_servers())

Get the command to run. I don't remember why, but writing my script decided not to pass variable to 'fabric; function, but use global variable.

global task
task = raw_input("Type/paste command to run >")


Now run the provided command in selected order:

for i in [3, 2, 1]:
    execute (run_remotely, hosts=lists[i-1])


For any case disconnect all connection. (It might be not necessary).

disconnect_all

Monday, January 09, 2012

XenDebian.py to install Debian on XenServer/XCP

Recently, I've needed to install many Debian VMs onto XenServers and, of course, wanted to automate it. One of my colleagues pointed that rather use existing tools like Cobbler I should take opportunity of working with world leading XenAPI developers and learnt it by writing some code. I took that advice and started to write a python script.

During that I decided that I could improve XenAPI documentation. I spent some extra time on my program and tried to write the code clear and with as many comments as possible, so other can learn from it and reuse it. I hoped to write even more documentation (some tutorial) based on my experiences, unfortunately I haven't had enough time.

Please find short XenDebian.py documentation on Xen wiki:

http://wiki.xen.org/wiki/XenDebian.py

and code on GitHub:

https://github.com/wawrzek/XenDebian


The script is called XenDebian, but with minor modification (new preseed file and change distro name in few places) you should be able to use it with Ubuntu. With a few more modification it should works for any distribution.

Finally, thanks to Project Kronos you would be soon able to install use XenDebian to install many Debian on Debian!

Tuesday, August 04, 2009

Reading from rather big files in Python

Recently I needed to open the big file (apache log - 14 GB or so) and cut some information from it. Of course use of file.read() and/or file.readlines() method wasn't possible. On the other hand, using file.readline() few (rather more than 20) million times doesn't sound right. Therefore, I looked for another resolution and found that you can limit the size of readlines().
f=open('filename','r')
opensize=2**27
longlist=[]
while 1:
   shortlist=[[l.split()[n] for n in [0,4,-2,-1]] for l in f.readlines(opensize)]
   if not list:
       break
   else:
       longlist.extend(shortlist)

The script open the 'filename' file and next in the loop:
  • read from that file lines of size close to 128 Mb (2**27),
  • cut first, fifth, next to last and last column from each line,
  • add created (temporary) list to the output list.
It's worth to note that if
shortlist
is not created the script will leave the loop (lines 6 and 7). It not obligatory, but I like to work with 2 powers, therefore opensize=2**27.

Wednesday, March 11, 2009

Control the Vim from the edited file

One of the very nice Vim feature I've learnt recently is the possibility of controlling the Vim from a edited file. Chosen Vim commands may to be put in one of the first (specially formatted) file lines. The line format is describe in 'modeline' help keyword (:help modeline). It worth to remember that text before and after main part has to be commenting out directive. Therefore, for example the line in HTML might looks similar to:
<-- vim: set tabstop=4 noexpandtab:-->
for python:
# vim: tabstop=4 noexpandtab:
If you like to learn more please check the modeline keyword in Vim help.

update: I forgot to add that you need to set modeline in .vimrc file.

Tuesday, April 29, 2008

Sed one liners

I'm using this set of sed one liners very hardly. Today I found that I hasn't added it to my blog so I cannot find it easily. (You know I use this blog as a some kind of memory extension ;) I was trying to find sed command capitalized a variable, I couldn't find it at mentioned page. Farther searching pointed to very interesting thread at Unix .com forum. The was script doing something similar in python, perl, sed and awk. I chosen perl:
echo $myvar | perl -p -e '$_ = ucfirst'
However, python looks also nice:
echo $myvar | python -c "print raw_input().capitalize()"
Awk one is also nice, but a bit more complicated (one line):
echo $myvar |awk '{(sub("^.",substr(toupper($1),1,1),$1)); print }'

Friday, December 29, 2006

Geometries from Gaussian Scan

Reading geometries from Gaussian file storing information about Potential Energy Scan (PES) isn't easy. I was sure that there were some tools to do this, but I cannot find any yesterday, so I prepared my own. It is writing in python, of course ;). Right now have very basic features. It reads input file and created output xyz files (as many as stationary points was found). You are using it simply writing in command line:
scaner-0.1.py inputfile

Monday, November 20, 2006

Energy converter v0.1

I've just created small Python/Tkinter program to convert values of energies. At the moment you can convert from (to) au, eV, nm and cm-1. Not much, but it is beginning :). Program can be download from mml website.

Saturday, November 18, 2006

The Python Challenge

I nearly become addicted. To what? To Python Challenge. It is a set of puzzle which can be resolved with Python (you can use other language, but it is design for Python). I don't remember when I have so much fun. It is great think if you what to learn Python or just test your skills. It is my 50th note and I started write this blog 1 year and 3 days ago!

Thursday, July 20, 2006

CCLib

The new version (0.5) of CCLib has been release. CCLib is an open source library, written in Python, for parsing and interpreting the results of computational chemistry packages. It currently parses output files from ADF, GAMESS (US), Gaussian, and PC GAMESS.