Search This Blog

Wednesday, December 29, 2010

Expect, ssh and two passwords

Let say that you need to run a command on many remote machine and you cannot use any of specialist tools (I mean i.e. func). In such situation expect might be helpful and I even has written about it already. Recently I had to modified my script, cause not all remote systems have the same root password.
#!/usr/bin/expect -f

set timeout -1

set machine [lindex $argv 0]
set command [lindex $argv 1]

set pass "xen4ulez\r" 
puts $machine

spawn ssh -o "NumberOfPasswordPrompts 2" -o "ServerAliveCountMax 1" -l root $machine $command
match_max 100000

expect {
    "assword:" 
      {
        exp_send $pass 
        set pass "xen!king\r"
        exp_continue
    }
}
First, the script has a password set as a variable. It allows to change later. To to this I have expanded the 'expect password' part of the script. Now the script sends the password and changes it after. If the password was fine, script executes the command. The change of the password is not important. If the password is wrong the script sends the variable $pass again, but this time it is new value. I have also added to option to ssh command for script to run a bit faster.
  1. NumberOfPasswordPrompts 2 - ensure that ssh tries only twice to provide password.
  2. ServerAliveCountMax 1 - ensure that ssh sends only one Server Alive message
Finally I have set StrictHostKeyChecking no and UserKnownHostsFile /dev/null in the .ssh/config file for machines I'm going to use above scripts. This two options set ssh not to bother about checking Host Key. Please remember that it might be not secure. 

Thursday, July 29, 2010

Do something every minut in shell

Recently I've needed to do some statistic based on apache log file. I've wanted statistic to be update every minute and want to use only shell.

The first idea was cron.  Sounds nice but there is small problem. I want to be flexible. In theory I can add a job to crontab in the moment I want to start it and remove it (comment out) after finishing, but it's ideal.

No cron, so maybe sleep, especially sleep 60.... But there is another problem. My scripts runs few seconds, so after few minutes (approx. 60/time my scripts runs) I'll have a gap in results.

No cron, no sleep. I needed another direction. Recently I've been playing a bit with data formats. I write a bit of code and after few minutes – volia – I had a working scripts.

#!/bin/sh

min=`date  "+%M"`

while [ 1 ]
do
        ~/test.sh
        while (($min==`date  "+%M"`))
        do
                sleep 10
        done
        min=`date  "+%M"`
done 
  
One more thing which might be interesting for some. I used infinite loop based on  this blog entry.

Wednesday, June 23, 2010

How to check a website from the commandline

Sometimes it's useful to check a website connectivity using only command line. I.e. when trying a website from remote host, or when you set up a new box you are going to put behind a loadbalancer.

If you need HTTP connection, telnet will be enough:

> telnet 10.10.10.10 80

GET /index.html HTTP/1.1
host: important.site.com

In the case of HTTPS you need openssl:

> openssl s_client  -connect 10.10.10.10:443 -prexit

GET /index.html HTTP/1.1
host: important.site.com

This is entry for my extend my 'external memory', but you might find it useful.

Tuesday, February 09, 2010

Copy many lines in Emacs

I'm VIM not Emacs user, but use the later one to applied DNS changes. It has two nice feature helping to maintenance BIND zone files. First, built-in versioning in RCS, but even more helpful the mechanism of updating serial number.

Anyway, I found it's very hard to copy more than one line in Emacs, until I found some suggestion here and here. So to copy the current line without the newline you have to one of two (depends if you want copy or move the lines):
C-a C-Space C-e M-w
C-a C-Space C-e C-w
It means: C-a move to beginning of the line, C-Space sat the mark, C-e go to end line, M-w/C-w save/delete(kill) the region. "OK, but it only one line" you may save and you would be right. But to mark more line you just need to i.e. use arrow before marking a text (by M-w).

Thursday, January 28, 2010

64 vs 32 rather then SSD vs HDD


Introduction
Some time ago I promised to present results of the performance test of an Apache serving content from HDD and SSD. The tests confirmed that SSD gives Apache serving static content significant performance boost, however the other remedy for I/O problem has been found.

It's not so easy to test a system from inside. It also applies to the internet. The "image" you see is not exactly what your customers might see. There are some tools helping measure the "real image" of your website (i.e. KeyNote), but to test a difference of performance between two server visible as one entitle from outside (i.e. behind load-balancer) such test are not ideal. (Of course it is possible to prepare a A/B test when A means serving images from one server, B from another one). The web search hadn't brought any technique and parameters useful in such test (of course it might means that the search  was not very good). Therefore, the serving speed [byte/microsecond] parameter were define as ratio of file size [bytes] and response time [microseconds], respectively %B and %D in extended Apache log.


SSD vs HDD
In the first test the SSD box (Sun X4150 with 8 Xeon E5345 @ 2.33GHz cores and 8 GB of memory) was try out against HP ProLiant D380 G5 with the same amount of memory and 4 similar processors/cores (Xeon 5148 @ 2.33GHz). The test were split into two phase. Results for both ones are collected in Table 1. The first day both boxes serve the same amount of requests. The SSD machine sent files two time faster. Next day the weight of connection to machine with SSD was increase on the load balancer, but it doesn't change the results. The SSD box was much faster again. The data presented in Table 1 ensure that both boxes worked with similar set of files (similar size and number of requests).

Table 1. Comparison of server with SSD (SUN X4150) and HDD (HP D380 G5) drives.
SSD machine HDD machine
Day I
Average speed 24.27563
Average time 13908.68485
Average size 7683.71685
Served files 14983388
Average speed 12.23371
Average time 24674.27763
Average size 7702.66678
Served files 14987023
Day II
Average speed 23.76447
Average time 13107.95822
Average size 7839.91839
Served files 21522835
Average speed 11.10400
Average time 26001.08215
Average size 7829.47865
Served files 10758116

During next phase of research the SSD machine was tested again wider set of servers with HDD servers (machines with Apache using different MPM (prefork and worker), with 8 or 16 GB of memory and with slower and faster processors). SSD machine was faster than any of the server with a classical hard drive. Adding memory as well as changing MPM didn't change the difference in the performance. On the other hand, the machine with more and newer processor sent files much faster than the old ones, however still 50% slower than the SSD test kit.


Table 1. Comparison of Apache performance on machine with a SSD against different machine using a HDD. SUN SSD means the test boxs, OLD HP a box with 4 L5430@2.33Ghz core and NEW HP a server with 8 E5345 cores.
OLD HPNEW HPSUN SSD
speed 16GB (prefork) 8GB (prefork) 8GB (worker) 16GB (worker) 8GB, (worker)
Test 1 9.73669 10.99227 10.82629 16.81683 24.18491
Test 2 9.56062 10.57697 10.51381 16.32999 24.41852
Test 3 8.69836 9.83142 9.65313 16.16164 24.39481
Test 4 9.03057 10.12731 9.98168 15.60127 24.16711



32bit vs 64bit
In the mean time another two things occurred. On the one hand, another test machine, this time with 64bit OS, was built.The first results showed significant decrease of the load and the number of I/O (read) operation. Figure 1 and 2 are good indicator how drastic it was change, even if they were prepared on latter production boxes. Moreover, a change in some of the production server settings exhibited that the machine do not properly caching content served by Apache.




Figure 1. Load on a production system before and after changing OS from 32 to 64 flavour.



Figure 2. I/O operation on a production system before and after changing OS from 32 to 64 flavour.



Further investigation of both phenomena showed that the I/O problem mentioned in the first part of article was cause by inefficient caching, what was cause by memory wasting on a server with 32bit OS. By default Apache is using the sendfile() function rather than the combination of read() and write() functions to transfer files from a storage to a network interface (so called zero copy approach). It speeds up a data transfer by avoiding switching context from kernel to user space, but it also means that Apache cache is limited because on 32-bit OS it cannot allocate more than 3GB of memory.

To confirm that a 32bit OS was causing of performance issues another test on live system was conduct. When comparing two identical servers with 16GB of memory and 8 E5345 Xeon cores, one running a 32bit and second 64bit OS, following observation was made:
  • The load decreases from around 2 to 0.5 (Figure 1).
  • The read from disk decreases from 2.5 to 0.6 [megabytes/second]  (Figure 2).
  • The memory usages increase from 2.5-3 to 16 (full memory) GB.
  • The average Apache speed increase from around 12 to 30 [bytes/microsecond] (Figure 3).



Figure 3. Comparison of the Apache speed (speed of sending static content) on 32 and 64 bit OS. 
 
  

Summary
During the tests the SSD showed its superiority to the classical HDD storages. However in the case of serving a static content from a web server using a distributed memory caching technique might be faster, more scalable  and even cheaper solution.

Another conclusion from above research, probably less and less important as there are less and less of x86 32bit servers, that 64 bit OS might be really faster, especially in the case of system with big amount of memory. 

Thursday, January 07, 2010

Short (mostly shell) fomulas

If you hit this website looking for a Linux/shell/UNIX related advice you might want to visit my identi.ca account. I post "short, magic", Linux formula there now. You know something like:

How to split lines in ?
s/\,/,[ctrl-V][Enter]/g
where [ctrl-V][Enter] - means type Ctrl+V and next Enter (you should see s/\,/,^M/g)

Friday, November 13, 2009

noatime

If you need I/O performance bust it might worth to turn on noatime flag on important partitions. Look down to see how it pushed down the read level on a server with Apache serving a static content.


Usually I add noatime flag during a system installation, but this time forgot about it and had to remount the file system. Thanks that mistake I got this beautiful image ;)
You can go even further and turn on nodiratime, it should decrease read even more.

GMAIL and msmtp (Mutt)

This is the example how to configure the msmtp (i.e. for Mutt) to use thegmail smtp server. Remember that you need the certificate. I have got mine from old Ubuntu installation (Saving /etc directory before reinstallation a box is good idea).
account your.user
logfile ~/.msmtp.log

tls on
tls_starttls on    
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on

host smtp.gmail.com
port 587 
from your.user@gmail.com
user your.user@gmail.com
password YOUR_password
BTW, in Ubuntu you can grab certificate by sudo apt-get install ca-certificates.

Sunday, October 25, 2009

Nagios plugin

Not so long ago I heard the question if (or rather how) it is possible to write a NRPE plugin checking the resources utilization of an application. I'm using Nagios on the daily basis, but I haven't needed to write any plugin yet. When I went through existing plugins most/all of them checked resources on a server level. It makes sense, you are not so interested what exactly doing you server if your website/database is available and response fast. It especially true if you have 100, 500, 1000, ... servers. Anyway, I found the question interesting, even if was rather theoretical than practical one. After some research I found Jason Faulkner plugin which should be a good base, modified it a bit and created this script:

#!/bin/bash 
#
# Nagios plugin to monitor a process. Can easily be modified to do 
# pretty much whatever you want.
#
# Licensed under LGPL version 2
# Copyright 2006 Broadwick Corporation
# By: Jason Faulkner jasonf@broadwick.com
#
# Modified to measure CPU usage of chosen process.
#
# USAGE: cpu.sh process_name warning_level critical_level
#
# Licensed under LGPL version 2
# Copyright 2009 Wawrzyniec Niewodniczański
# Modification by: Wawrzyniec Niewodniczański wawrzek@gmail.com

process_name=$1
WARLVL=$2
CRITLVL=$3

OKMSG="STATUS OK: ${process_name} running"
CRITMSG="STATUS CRITICAL: ${process_name} using more than ${CRITLVL} % of Memory"
WARNMSG="STATUS WARNING: >1 ${process_name} using more than  ${WARLVL} % of Memory"
UNKMSG="STATUS UNKNOWN: ${process_name}, check if process is running"

PROCESS=`ps axu | grep  -v ${0}|grep -v grep | grep ${process_name}`
CPU=`echo ${PROCESS}| awk '{cpu+=$3} END {printf "%d", cpu}'`

if [[ $PROCESS != "" ]]  
then 
        if (($CPU < $WARLVL))
        then
                echo "$OKMSG"
                exit 0
        elif (( "$CPU" < $CRITLVL ))
        then 
                echo "$WARNMSG"
                exit 1
        else 
                echo "$CRITMSG"
                exit 2
        fi  
else
        echo "$UNKMSG"
        exit 3
fi
I would say that it's nothing excited. There are two important lines. The first one searching the process name in output of ps command and excluding the lines with script name and grep from the list. The another one using awk to add value of CPU usage from the list created in first line. BTW if you would prefer to check memory usage rather then processor, change {cpu+=$3} to {cpu+=$4} (or even to {mem+=$4}) in awk command. I also wrote the nagios command which I believe should work. "believe" not "know", as I haven't try it yet ;)

# 'check_cpu' command definition
define command{
        command_name    check_cpu
        command_line    /usr/lib/nagios/plugins/check_cpu $ARG1$ $ARG2$ $ARG3$
ń} 
 
Useful links
  1. http://www.nagios.org/documentation
  2. http://debianclusters.cs.uni.edu/index.php/Creating_Your_Own_Nagios_Plugin
  3. http://www.ibm.com/developerworks/aix/library/au-nagios/index.html
  4. http://lena.franken.de/nagios/own_service.html

Monday, September 21, 2009

Escape, Escape

I couldn't understand why following command was working on a local machine, but not through ssh (following form):

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += $5;} END {print tsum}'" 
 
I asked my workmate and he also had problems for some time, but finally he suggested that we needed to "escape" something. After some try we found that ssh don't like $ character so following command works.
 
ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += \$5;} END {print tsum}'"

Thursday, August 27, 2009

Stone Redskin: comparision of Apache2 performance on HDD and SSD

Introduction
Recently, I had a chance to test the performance of a static content web servers. The initial analysis showed that the most important issue were the speed of a disks, which started to have problems with handling I/O operations. The numbers of files were huge what means that hard drives were engaged in many random access operation.

The latest tests has shown that the new Solid State Disk (SSD) mass storage beat the classic Hard Drive Disk (HDD) in such circumstances (in most others too). So it was quite natural to prepare a set of test helping to measure the effect of switch from a HDD to a SSD storage on the Apache performance.

Methodology
It should be keep in mind, that I wasn't interesting in a general comparison of SSD vs HDD, but concentrated my tests on the Apache performance. The Grinder 3.2 software was used to simulate a load on the web server. The list of requested URL based on the real Apache logs taken from the one of box serving the static content. To eliminate the influence of caching, before each test the memory cache was cleaned using following command echo 3 > /proc/sys/vm/drop_caches (suggested on Linux-MM).

Hardware
The test machine was the Sun X4150 server with a 8GB memory and 2 4-core Xeon E5345 @ 2.33GHz processors working under control of the 32 bit version of CentOS 5.2 and the standard version of Apache2 (2.2.3). Finally, all data were served from ext3 partitions with the noatime flag.
Disks
Following disks were used for tests.
  • RAID 1 matrix consist of 2 classical rotating HDD with the root file system and the partition storing files for Apache (on LVM2 volume).
    Vendor: Sun       Model: root              Rev: V1.0
    Type:   Direct-Access                      ANSI SCSI revision: 02
    SCSI device sda: 286494720 512-byte hdwr sectors (146685 MB)
    
  • Standard Intel SSD storage with the partition holding Apache data.
    Vendor: ATA       Model: INTEL SSDSA2MH16  Rev: 045C
    Type:   Direct-Access                      ANSI SCSI revision: 05
    SCSI device sdc: 312581808 512-byte hdwr sectors (160042 MB)
    
  • 2 Intela SSD Extreme disks joined into the one LVM2 volume. It was necessary to create a partition big enough to keep all data for Apache.
    Vendor: ATA       Model: SSDSA2SH064G1GC   Rev: 045C
    Type:   Direct-Access                      ANSI SCSI revision: 05
    SCSI device sdd: 125045424 512-byte hdwr sectors (64023 MB)
    
Measured parameters
In the both table following acronyms has been used to describe measured parameters. (More info about them on Grinder web site.)
  • Test - Test name
  • MTT (ms) - Mean Test Time
  • TTSD (ms) - Test Time Standard Deviation
  • TPS -Transactions Per Second
  • RBPS - Response Bytes Per Second
  • MTTFB (ms) - Mean Time to First Byte
3oo - first test
In the first phase of tests I compared the Apache's performance serving 300 000 request using data stored on classic HDD as well as SSD. Kernels from the 2.6 tree allow to choose a I/O scheduler. In theory the best scheduler for  SSD devices is Noop, therefore in table below I compared results for the mentioned and default (CFQ) schedulers.
Test MTT (ms) TTSD (ms) TPS RBPS MTTFB (s)
HDD CFQ 5.53 8.17 179.51 1231607.13 5.3
HDD Noop 5.53 8.09 179.30 1230119.51 5.29
SSD CFQ 0.77 3.06 1226.55 8415044.64 0.56
SSDn Noop 0.74 2.77 1280.17 8782969.21 0.56
SSDe CFQ 0.73 2.55 1280.23 8783381.50 0.52
SSDe Noop 0.71 3.05 1326.62 9101643.04 0.53
How we expected, the SSD disks (or rather Apache with content on them) proved to be much faster. The web server performance grown about 10 times when a HDD were substituted by a SSD. Another observation worth to note is that the results obtained using both sets of the SSD disks were very similar. Extreme Edition storages were few percent faster, but the different is probably too small to be the only reason to justify the higher cost. Additionally, it was clear that the Noop scheduler didn't dramatical change the Apache performance.
One hour data
It's obvious that 300k requests may not enough to show the full and true image, therefore I repeated test with a bigger set of data based on 1 hour worthy log. During that hour the original server had responded to 1 341 489 queries, but during creation of the file with input data for Grinder I saved the list of URL twice, therefore grinder was sending 2 682 978 queries during the test.
The results are presented in the next table. To the data collected from Grinder I added one more number, TT — the total time of the test, that is how long it took Grinder to send all the requests.
Test MTT (ms) TTSD (ms) TPS RBPS MTTFB (s) TT (h:m)
HDD CFQ 2.65 5.29 371.71 2145301.3 2.45 02:00
SSDn CFQ 0.63 3.19 1495.3 8630105.68 0.43 00:29
SSDn Noop 0.64 2.52 1478.77 8534692.28 0.43 00:30
SSDe CFQ 0.59 2.93 1594.06 9200064.95 0.42 00:28
SSDe Noop 0.61 2.62 1530.84 8835205.22 0.42 00:29
The increase of the queries number diminished the difference between the SSD and HDD disk performance, but also in second test the former storage was firm winner. I.e. the Total Time of test was 4 time shorter for any version of the SSD compare to the traditional disks. Another interesting observation is that difference in performance of Mainstream and Extreme disks decreased. Finally, the Noop scheduler didn't improve the results of that test too.


Summary
The results shown in the current study, as well as other not presented above, confirmed the hypothesis that SSD disks might be a good remedy for observed I/O problems. In the few weeks time you might expect some kind of appendix in which I will describe if baptism of fire on the battlefield of the web come off as well as the preliminary tests.

Tuesday, August 25, 2009

Linux Works in Cambridge

Some time ago I created the "Linux Jobs in Cambridge" map on Google Maps, but something was wrong. Recently, I decided that the title was not very propriety. It's not the map of Linux related opportunities, but the map showing how important is Linux and general Open Sources for Cambridge. So I changed the name to "Linux Work in Cambridge" and it seems to be the right idea. There are some new very interesting entries (even one pub). Cheek it out yourself, and maybe add or correct something.

View Linux Works in Cambridge in a larger map.

Friday, August 21, 2009

Expect and operation on many computers

Recently, I had to delete a directory on around 200 computers. The directory belonged to root, so using my account with public key authentication wasn't possible. I googled a bit, found the expect and wrote the following script.
#!/usr/bin/expect -f

set machine [lindex $argv 0]
set command [lindex $argv 1]
set timeout -1
spawn ssh -l root $machine $command
match_max 100000

expect "?*assword: $"
send "password\n"

expect eof
The script sets the name of a remote machine (set machine [lindex $argv 0]) and a command (set command [lindex $argv 1]) to execute from arguments it is started with. Next tries to connect to the remote machine (spawn ssh -l root $machine $command) and when it's asked for the password (expect "?*assword: $") send it (send "password\n"). Of course you have to change the password to the root password. Finally, it waits for the EOF from ssh (expect eof). I have confess that I don't remember what exactly set timeout -1 and match_max 100000 means ;) The script can be called with loop similar to one below.
for cell in 1{0..3}{0..9} ;\
do for box in {1..4} ;\
do echo  c${bc}-box0${app} ; \
./command.script bc${bc}app-0${app} "ls /var/log/httpd" ; \
done; \
done
One more thing. The script assumes that you has connected at least one to all machines or rather that the machines has been added to your .ssh/know_hosts file. If you plan to use script to initialize the first connection you should add following line
expect "Are you sure you want to continue connecting (yes/no)?"
send "yes\n"
before the line expect "?*assword: $", but in such case all machine haven't to be present in .ssh/know_hosts file.

Tuesday, August 18, 2009

How to find the not commented line using Vim

The significant part of BOFH's live consist of editing config files. It's not so uncommon that you need find not commented lines (i.e. to find it something is set). With vim it's very easy:
/^[^#]
The above line command the editor to: find a line which doesn't start with # or rather: find a string which is at the beginning of a line with the first character anything else then #. This advice will work not only for vim i.e. you can use it in grep as well:
[user@server]$ grep  "^[^#]" modprobe.conf 
alias eth0 tg3
alias eth1 tg3
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptspi
I discussed the similar case some time ago in this note: How to find line not starting with X in Vim.

Tuesday, August 04, 2009

Reading from rather big files in Python

Recently I needed to open the big file (apache log - 14 GB or so) and cut some information from it. Of course use of file.read() and/or file.readlines() method wasn't possible. On the other hand, using file.readline() few (rather more than 20) million times doesn't sound right. Therefore, I looked for another resolution and found that you can limit the size of readlines().
f=open('filename','r')
opensize=2**27
longlist=[]
while 1:
   shortlist=[[l.split()[n] for n in [0,4,-2,-1]] for l in f.readlines(opensize)]
   if not list:
       break
   else:
       longlist.extend(shortlist)

The script open the 'filename' file and next in the loop:
  • read from that file lines of size close to 128 Mb (2**27),
  • cut first, fifth, next to last and last column from each line,
  • add created (temporary) list to the output list.
It's worth to note that if
shortlist
is not created the script will leave the loop (lines 6 and 7). It not obligatory, but I like to work with 2 powers, therefore opensize=2**27.

Monday, June 22, 2009

one for AWK and one for SVN

Another two useful one liner. First awk. Sometimes you need to grab last "element" of lines in a file which has different numbers of spaces (or other separator). In such case use variable $NF (or $NF-1, $NF-2...). Good example for such situation might be apache log file, where user agent description is a string with various number of spaces, so it's hard to get columns after that. But you can use something similar to:
 tail bo-access_log.2009-06-22 | \
awk '{print "size:\t"$(NF-1) "\t time:\t" $NF}'
In the example log file the time is the last and size of file next to last field. Of course you can type it in one line. But Then you have to remove '\' character from end of first line. Second advice is related to SVN. I found reverting last submitted changes quite not clear there. Revert works only with no committed changes, so I used the command similar to below one.
 svn merge -r HEAD:{2009-06-21} .
The example reverts everything what has been submitted between 21st June 2009 and 'now'. However, today I found that PREV 'variable', so the following command should do I had wanted to achieve. Interesting how could I missed it?
 svn merge -r HEAD:PREV .
And one more update. In petke comments to this entry in Aral Balkan blog I found another one liner, which looks event easier:
svn update -r 2689

Wednesday, May 06, 2009

Vim substitution

Let say that you want to add string 'bprdp' and the end of each line beginning with string 'bc' and ending with comma you should use following command:
:% g/^bc/s/\,$/, bprdp/
% means the whole file g/ for each line with pattern after '/' in above case pattern is ^bc line beginning with bc s/\,$/, bprdb/ substitute comma (\,) followed by end of line character ($) with ', bprdb'.

I wrote this message based on Vim regular expression and Vim Command Cheat Sheet.

Wednesday, March 11, 2009

Control the Vim from the edited file

One of the very nice Vim feature I've learnt recently is the possibility of controlling the Vim from a edited file. Chosen Vim commands may to be put in one of the first (specially formatted) file lines. The line format is describe in 'modeline' help keyword (:help modeline). It worth to remember that text before and after main part has to be commenting out directive. Therefore, for example the line in HTML might looks similar to:
<-- vim: set tabstop=4 noexpandtab:-->
for python:
# vim: tabstop=4 noexpandtab:
If you like to learn more please check the modeline keyword in Vim help.

update: I forgot to add that you need to set modeline in .vimrc file.

Tuesday, March 10, 2009

My first Perl script

It's nothing big, but it's the first one and, as Perl is write only language, I'd better add the short description. The script takes a list of files passed as arguments to the command; reads all lines (http addresses) from them and creates the list of unique domains names.
#!/usr/bin/env perl

%seen = ();
foreach (@ARGV)
{
open (LFILE,"$_");

for $line ()
{
       @sline=split(/\//,$line);
       print ("@sline[2]\n") unless $seen{@sline[2]}++;
}

close LFILE;
}
Perl tutorial from tizag.com was helpful.

Monday, March 09, 2009

DarwinPorts via proxy

Recently, I needed a perl module not present on my MacOSX computer, which was behind a proxy. The friend suggested to use the Darwin ports rather the Perl from Apple. I downloaded and installed it to found I cannot install any port. The problem was due to the combination of using a proxy and the sudo rather then the root user. I guess such combination is rather common among MacOSX-perl users. So below I present the command which allows to use the Darwin ports from a normal MacOSX account. Generally note, you have to export both the RSYNC_PROXY as well as the http_proxy in the sudo environment.
sudo sh -c "export RSYNC_PROXY=proxy.server:port; \
export http_proxy=http://proxy.server:port; \
port install perl5.10 "