The "load" is use widely to describe stress/work applied onto a UNIX system. The simple rule is "lower than better". In the older days of uniprocessor machine load 1 was kind of a borderline. In the new brave world of multi-core/processor machines load 1 means nothing. Many people suggests that load equal or lower to number of processors/cores is good. That sound sensible, but not always is accurate.
Why? To answer that we have to comeback to question asked in the subject.
What is the "load"?
The load as the exponentially damped/weighted moving average of the number of processes, including threads, using or waiting for CPU and, at least at Linux, in uninterruptible sleep state in last 1, 5 and 15 minutes (see Wikipedia). The last part means that all processes/threads waiting for a disk (or other I/O device) will increase the load, without increasing a CPU usage. It leads to situation when the load lower than number of core/processes is danger. Let imagine few processes trying to dump important information on disks. Especially if all interrupts have affinity to one processor only (see this post) or just data are store in many small files. On the other hand, machine with very high load might be very responsive. Plenty of processes waiting to write information onto a disk not using a lot of memory and CPU in the same time. Just look at this picture:
If you want to know even more details of how the load is actually calculated
read this impressive white paper.
Links:
http://en.wikipedia.org/wiki/Load_%28computing%29
http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
http://larryn.blogspot.co.uk/2013/05/cpu-affinity-interrupts-and-old-kernel.html
Kind of my extended memory with thoughts mostly on Linux and related technologies. You might also find some other stuff, a bit of SF, astronomy as well as old (quantum) chemistry posts.
Search This Blog
Tuesday, December 30, 2014
Saturday, December 06, 2014
Install CyanogenMod at Nook HD+
Recently I decided to try the new Cyanomogen (CM11) on my Nook HD+. Initial reading indicated that I had to reinstall using Recovery rather than internal updater. I tried to login to Recovery so much, that I recovered official B&N OS which replaced CM.
I needed to start from beginning. I did some research and found that post. It looked good so I gave it a try. First download ClockworkMod attached to the post, but later I downloaded latest CM snapshot from there and added Google Apps for CM11 from there. I put everything as described on SD Card and kicked off installation. It flew like an Albatross. (To be honest I don't know why I did write Albatross - maybe because of this?)
Anyway CM11 works good at Nook HD+.
I needed to start from beginning. I did some research and found that post. It looked good so I gave it a try. First download ClockworkMod attached to the post, but later I downloaded latest CM snapshot from there and added Google Apps for CM11 from there. I put everything as described on SD Card and kicked off installation. It flew like an Albatross. (To be honest I don't know why I did write Albatross - maybe because of this?)
Anyway CM11 works good at Nook HD+.
Links:
- http://wiki.cyanogenmod.org/w/Ovation_Info
- http://download.cyanogenmod.org/?type=snapshot&device=ovation
- http://wiki.cyanogenmod.org/w/Google_Apps
- http://forum.xda-developers.com/showpost.php?p=42406126&postcount=7
- http://forum.xda-developers.com/attachment.php?attachmentid=2849350&d=1405272804
Sunday, November 23, 2014
More fabric as a library
Recently I had to prepare a tool doing some remote commands, so of course I decided to use fabric, but I have big problem to control hosts. I remembered that I had written a short article on Fabric in here some time ago. But it didn't help. I asked on the Fabric mailing lists, but there was no help.
http://larryn.blogspot.co.uk/2012/11/fabric-as-python-module.html
http://lists.nongnu.org/archive/html/fab-user/2014-10/msg00002.html
Manual host name control
In this tool I didn't need to run many parallel SSH connection, so I decided to control remote host name from inside the loop in the function my setting env.host_string each time (this is very useful functionality). Like in following example:#!/usr/bin/env python
"""Example code to use Fabric as a library.
It shows how to set up host manually.
Author: Wawrzek Niewodniczanski < main at wawrzek dot name >
"""
# import sys to deal with scripts arguments and of course fabric
import sys
import fabric from fabric.api import run, hide, env env.hosts = ['host1', 'host2']
# Main function to run remote task
def run_task(task='uname'): """run_task([task]) - runs a command on a remote server. If task is not specify it will run 'uname'."""
# hide some information (this is not necessary).
with hide('running', 'status'): run(task)
# Main loop
# take all arguments and run them on all hosts specify in env.hosts variable
# if not arguments run 'uname'
if len(sys.argv) > 1:
tasks = sys.argv[1:]
for task in tasks:
for host in env.hosts: env.host_string = host run_task(task)
else:
for host in env.hosts:
run_task()
Fabric in full control
The problem bugged me since then. Yesterday I found some of my old code. Analysed it and quickly found small, but profound difference with mu recent fabric usage. Rhe code above called the run_task function wrongly. Rather than dealt it in the normal way I supposed to use execute.#!/usr/bin/env python
"""Example code to use Fabric as a library.
It shows how to set up host manually.
Author: Wawrzek Niewodniczanski < main at wawrzek dot name >
"""
# import sys to deal with scripts arguments and of course fabric
import sys
import fabric from fabric.api import run, hide, env, execute env.hosts = ['host1', 'host2']
# Main function to run remote task
def run_task(task='uname'): """run_task([task]) - runs a command on a remote server. If task is not specify it will run 'uname'."""
# hide some information (this is not necessary).
with hide('running', 'status'): run(task)
# Main loop
# take all arguments and run them on all hosts specify in env.hosts variable
# if not arguments run 'uname'
if len(sys.argv) > 1:
tasks = sys.argv[1:]
for task in tasks:
execute(run_task, task)
else:
execute(run_task)
Links:
http://www.fabfile.org/http://larryn.blogspot.co.uk/2012/11/fabric-as-python-module.html
http://lists.nongnu.org/archive/html/fab-user/2014-10/msg00002.html
Labels:
linux,
programming,
python,
UNIX
Friday, November 21, 2014
Resource for vim
Just some link with useful Vim's advices.
General Vim advices
- https://code.google.com/p/vimcolorschemetest/ - many beautiful colourschemes.
- http://tnerual.eriogerg.free.fr/vimqrc.html - many shortcuts (to keep under the pillow).
- http://vimregex.com/ - all you might want to know about regex in Vim.
- http://dailyvim.blogspot.co.uk - plenty of short yet useful Vim advices.
- http://vimcasts.org/ - great Vim screencast with good support.
Vim and Python
- https://dev.launchpad.net/UltimateVimPythonSetup/
- http://justinlilly.com/vim/vim_and_python.html
- http://haridas.in/vim-as-your-ide.html
- http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/
- http://www.sontek.net/blog/2011/05/07/turning_vim_into_a_modern_python_ide.html
- http://www.vim.org/scripts/script.php?script_id=1494
- https://github.com/scrooloose/syntastic
- http://vim.wikia.com/wiki/Omni_completion
- https://code.google.com/p/vimpdb/
Thursday, August 21, 2014
(w)dstat
wdstat
In my .profile (on CentOS 5, just in case there were some changes in dstat) I have following alias to dstat (wdstat stands for Wawrzek's dstat):alias wdstat="dstat -lcpymsgdn 5"
Where the options stands for:
- -l - UNIX load (1m 5m 15m) load average in 1, 5 and 15 minutes, respectively;
- -c - cpu stats (usr sys idl wai hiq siq) percent of time spent in user and system space, idle, waiting on resource, serving interrupts and softirqs (software interrupts);
- -p - process stats (run blk new) number of running, blocked and newly created processes;
- -y - system stats (int csw) - number of interrupts and context switches;
- -m - memory stats (used buff cach free) amount of memory used by processes, disk buffers, disk cache and free;
- -s - swap stats (used free) - amount of used and free swap space;
- -g - page stats (in out) number of page put in and out from swap;
- -d -disk stats (read writ) - number of reads and writes from all disks;
- -n -network stats (recv send) number of received and send network packages;
Further reading:
- http://dag.wiee.rs/home-made/dstat/
- http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
- http://www.linuxhowtos.org/System/procstat.htm
- http://lwn.net/Articles/520076/
- http://en.wikipedia.org/wiki/Paging#Linux
Thursday, August 07, 2014
netstat, ports, hosts and awk glue
Recently, I needed to create a list of all servers connected on a given port (in following example port 80). I used a mixture of awk and other UNIX command line tools.
First netstat provided the list of all connection (netstat -nt); -n stands for numeric and -t for only TCP connections.
Next awk, with the ':' defined as a field separator (awk -F':'), used lines where local port was 80 ($5==80) to create an associated array with a key define by connected host ip and a value equal to number of connection from it ({count[$8]++}). At the end of the script execution, awk looped over all element of the array (END{for (i in count)). Next there was a crux of the script, the cmd was define as a run the OS host command with the awk variable i as an argument (cmd="host" i). The |& operator created two-way pipe between awk and a execution of the previously defined cmd. The getline command was used to store cmd output into the variable j (cmd |& getline j). Next the split command split the content of the j into separate words and saved them into the a array (split(j, a, " ")). Finally the printf formatted output (printf "%40s - %d\n", a[5], count[i])). The actual hostname was fifth element of the a.
For continence, output lines were sorted by numeric order on third column (sort -n -k 3). Each output line consisted of a hostname ,'-' and a number - e.g. important.com - 3456.
netstat -nt| \ awk -F':'\ '$5==80 {count[$8]++} \ END{ for (i in count) { \ cmd="host "i; \ cmd |& getline j; \ split(j, a, " "); \ printf "%40s - %d\n", a[5], count[i]}}'| \ sort -n -k 3
First netstat provided the list of all connection (netstat -nt); -n stands for numeric and -t for only TCP connections.
Next awk, with the ':' defined as a field separator (awk -F':'), used lines where local port was 80 ($5==80) to create an associated array with a key define by connected host ip and a value equal to number of connection from it ({count[$8]++}). At the end of the script execution, awk looped over all element of the array (END{for (i in count)). Next there was a crux of the script, the cmd was define as a run the OS host command with the awk variable i as an argument (cmd="host" i). The |& operator created two-way pipe between awk and a execution of the previously defined cmd. The getline command was used to store cmd output into the variable j (cmd |& getline j). Next the split command split the content of the j into separate words and saved them into the a array (split(j, a, " ")). Finally the printf formatted output (printf "%40s - %d\n", a[5], count[i])). The actual hostname was fifth element of the a.
For continence, output lines were sorted by numeric order on third column (sort -n -k 3). Each output line consisted of a hostname ,'-' and a number - e.g. important.com - 3456.
Wednesday, June 25, 2014
Python for SysAdmins
Preparing for a interview some time ago I made a list of python module interesting for SysAdmins. Today looking for something else I found that half baked note and decided to polish it enough to put it on the blog. It's mostly for myself as quick reference.
Each module is describe by one, two sentences (from Python documentation) and have a link to official online document. At the end there is a list of example function, object.
https://docs.python.org/2/library/sys.html
https://docs.python.org/2/library/os.html
https://docs.python.org/2/library/os.path.html
https://docs.python.org/2/library/time.html
https://docs.python.org/2/library/glob.html
https://docs.python.org/2/library/fnmatch.html
https://docs.python.org/2/library/re.html
match = re.search(pattern, string)
if match:
process(match)
Each module is describe by one, two sentences (from Python documentation) and have a link to official online document. At the end there is a list of example function, object.
Python modules for SysAdmin
import sys
This module provides access to some variables used or maintained by the interpreter and to functions that interact strongly with the interpreter. It is always available.https://docs.python.org/2/library/sys.html
examples:
- argv,
- exit(),
- path,
- modules,
- exec().
import os
This module provides a portable way of using operating system dependent functionality.https://docs.python.org/2/library/os.html
examples:
- chdir(),
- getuid(),
- uname(),
- listdir(),
- stat(),
- rename(),
- access().
import os.path
This module implements some useful functions on pathnames.https://docs.python.org/2/library/os.path.html
examples:
- isdir(),
- isfile(),
- exist(),
- getmtime(),
- abspath(),
- join(),
- basename(),
- dirname().
import time
This module provides various time-related functions.https://docs.python.org/2/library/time.html
examples:
- time(),
- ctime(),
- sleep(),
- strftime(),
- strptime().
import glob
The glob module finds all the pathnames matching a specified pattern according to the rules used by the Unix shell. No tilde expansion is done, but *, ?, and character ranges expressed with [] will be correctly matched.https://docs.python.org/2/library/glob.html
examples:
- glob(),
- iglob().
import fnmatch
This module provides support for Unix shell-style wildcards, which are not the same as regular expressions (which are documented in the re module).https://docs.python.org/2/library/fnmatch.html
examples:
- fnmatch().
import re
This module provides regular expression matching operations similar to those found in Perl. Both patterns and strings to be searched can be Unicode strings as well as 8-bit strings.https://docs.python.org/2/library/re.html
examples:
- compile(),
- match(),
- search(),
- split(),
- findall(),
- sub(),
- group().
MatchObject
Match objects always have a boolean value of True. Since match() and search() return None when there is no match, you can test whether there was a match with a simple if statement:match = re.search(pattern, string)
if match:
process(match)
Friday, March 28, 2014
Zombie
Usually zombie process is not a big problem, but sometimes... just look at the screenshot below. It wants the whole machine as fried eggs! or maybe boiled?
Tuesday, March 25, 2014
Multiprocessing (in Python)
I needed to do some multi-threading in Python. As I needed effect quick I decided to use standard threading module. However, every time I had to use it I felt it was rather complicated beast. At threading module documentation page there is a link to multiprocessing module. I was tired with threading, on the one hand, but didn't have enough time to learn about greenlet or another competing project, on the other, so I decided to take a quick glance at multiprocessing module...
... And my life became much easier, sky bluer, grass, greener, oh and scripts faster ;-).
I don't do anything special with it, so just one simple code example, but this is very good tutorial you can find much more: http://pymotw.com/2/multiprocessing/communication.html.
Main part of nearly all my scripts looks the same:
import multiprocessing
SIZE = 30 pool = multiprocessing.Pool(processes=SIZE) pool_output = pool.map(get_values, servers) pool.close() # no more tasks pool.join()
Where servers is a list with servers I need get information from, and get_values is a function (sometimes with a different name). Simple, isn't it?
Tuesday, February 04, 2014
shell, history and substitution
One of the most known tricks in using shell history is to use:
^old^new
to replace string old by new in last command. The only problem is that it replaces only first appearance. But there is another command replacing all string old by new:
!:gs/old/new/
Subscribe to:
Posts (Atom)