The power of find - exec and friends (grep, sed)

This is quite old post. I started to write it in October 2010, but could not properly polish it for a long time.

Today I would like to present an example of usage of find (and grep) rooted in Crux. Let say that I want to find all packages in version 20100511 (this was true scenario when I wanted to update e17 related ports). Translating into less Crux specific language it means that I had to find all Pkgfile files (simple find), which had string 20100511 (simple grep). I needed only file names not a matching string so I used -l option for grep.

find . -name Pkgfile -exec grep -l 20100511 {} \;

I not only needed to find all old files but  to updated them as well (to version 1.0.0.beta that time). I used the same find but exchanged grep to sed (with option -i for  "in place").

find . -name Pkgfile -exec sed -i 's/20100511/1.0.0.beta/ {} \;

Let push our example one step further. I wanted to find dependence for packages, therefore I ran following command.

grep -i depen `find . -name Pkgfile -exec grep -l beta {} \;

What used  previous command to create a list of files to hunted through for word beta. (I wasn't sure if word "dependence" begun lower or upper case so used option -i for --ignore-case). 

In UNIX world there are always more than one way of doing things and in our scenario the find -exec can be replace with a separate command xargs. Xargs might be very useful in many cases because can be use to create unix command from standard input. Using xargs rather then find -exec my first example would be:

find . -name Pkgfile  | xargs grep -l beta $1 

Let use xargs for another task related to above example. In my scenario I had not only to update the version, but also to change the sources of the packages. To do that I used find, xargs and sed in a for loop.

for file in `find . -name Pkgfile  | xargs grep -l beta $1 2&> /dev/null` ; \
do \
 sed -i 's/pitillo.mine.nu\/crux\/distfiles/download.enlightenment.org\/releases/' $file; \
done

The above command might be one liner, but can be paste line by line. It used the command from the previous example to create the $file array consist of names of files with word "beta". Elements from $file were use as input for sed command.

Mount LVM from domU in dom0

Mounting logical volumes (LVM) from domX (guest) within dom0 (XenServer) is very not recommender and suggested. But sometimes it might be the only options, and it's cool hacker (hacker in good old meaning, not cracker) trick. So to mount your volume you should take following steps:

1. xe vm-disk-list vm=test4 - list disk of the VM.
2. xe vm-list - find info (UUID) about dom0.
3. xe vbd-create device=xvda unpluggable=true vdi-uuid=79a7a556-a6ba-48cf-8c82-30fa5bb9597c vm-uuid=36895434-e6d7-4fea-8271-d5477ca23c6d - create unpluggable VBD (xvda) with disk (uuid from point 1) on dom0 (uuid from point 2).
4. xe vbd-plug uuid=0a4151b6-2b59-2fdb-c0a9-492520a8d52c - plug created vbd.
5. mount /dev/xvda1 /mnt - mount disk (any 'physical partition').
6. vi /etc/lvm/lvm.conf - if you need anything on LVM you have to edit lvm.conf 

# Ignore /dev/xvd* devices to prevent deadlocking when live-snapshotting
# dom0-attached LVHD VDIs
#filter = "r|/dev/xvd.|","r|/dev/VG_Xen.*/LV.*|"

7. vgchange -a y test4 - activate new vg.
8. mount /dev/test4/root /test/mnt - mount partition.

XenServer, xe and more greping

In my last post I described how you can use grep to extend listing of XenServer parameters. Continuing this thread today, I show how to get values of specific parametes, let say memory.

You can get memory using following command, but I bet it's not you expect.

# xe vm-list params=memory
memory (MRO)    : 


memory (MRO)    : 

More interesting values you can get using params=all. The problem is that you will get hundreds of other values too. This is not helpful, but of course  grep can help us:

# xe vm-list params=all| grep memory
                 memory-actual ( RO): 536870912
                 memory-target ( RO): 
               memory-overhead ( RO): 6291456
             memory-static-max ( RW): 536870912
            memory-dynamic-max ( RW): 536870912
            memory-dynamic-min ( RW): 536870912
             memory-static-min ( RW): 536870912
               recommendations ( RO): 
                        memory (MRO): 
                 memory-actual ( RO): 500170752
                 memory-target ( RO): 
               memory-overhead ( RO): 15728640
             memory-static-max ( RW): 789839872
            memory-dynamic-max ( RW): 500170752
            memory-dynamic-min ( RW): 500170752
             memory-static-min ( RW): 395313152
                        memory (MRO): 
                 memory-actual ( RO): 1073741824
                 memory-target ( RO): 
               memory-overhead ( RO): 10485760
             memory-static-max ( RW): 1073741824
            memory-dynamic-max ( RW): 1073741824
            memory-dynamic-min ( RW): 1073741824
             memory-static-min ( RW): 1073741824
               recommendations ( RO): 
                        memory (MRO): 

It's not so useful yet, because we have only memory related value, but we don't know relation between them and VMs. Luckily, we can very easily extend our grep.

# xe vm-list params=all| grep "label\|memory"
                    name-label ( RW): at10
                 memory-actual ( RO): 536870912
                 memory-target ( RO): 
               memory-overhead ( RO): 6291456
             memory-static-max ( RW): 536870912
            memory-dynamic-max ( RW): 536870912
            memory-dynamic-min ( RW): 536870912
             memory-static-min ( RW): 536870912
               recommendations ( RO): 
                        memory (MRO): 
                    name-label ( RW): Control domain on host: dt33
                 memory-actual ( RO): 500170752
                 memory-target ( RO): 
               memory-overhead ( RO): 15728640
             memory-static-max ( RW): 789839872
            memory-dynamic-max ( RW): 500170752
            memory-dynamic-min ( RW): 500170752
             memory-static-min ( RW): 395313152
                        memory (MRO): 
                    name-label ( RW): at11
                 memory-actual ( RO): 1073741824
                 memory-target ( RO): 
               memory-overhead ( RO): 10485760
             memory-static-max ( RW): 1073741824
            memory-dynamic-max ( RW): 1073741824
            memory-dynamic-min ( RW): 1073741824
             memory-static-min ( RW): 1073741824
               recommendations ( RO): 
                        memory (MRO): 

Now we know that the at10 has 0.5GB, the at11 1GB and the dom0 might have between 386 and 771 MB.

Summary

Of course similar construction in grep query might be use for other variables.E.g. to have the list of live VMs try:

xe vm-list params=all| grep -i -E '(label|\s+live)'

XenServer CLI and grep by name

One of the annoying thing in XenServer command line tool (xe) is lack of regexp/wildcards in filtering vm (and other objects) by name. However, there is easy way to work around thanks to grep. Standard output have 3 lines per vm, with one line before (-B 1 in grep) and one after (-A 1) line with name. So you can use following line to find information about VM with given name:

xe vm-list | grep -B 1 -A 1 your-vm-name

You can also extend this command to get uuid of requested vm:

xe vm-list | grep -B 1  your-vm-name| awk '/uuid/ {print $5}'

What can be use i.e. to take the snapshot:

vm-uuid=`xe vm-list | grep -B 1  dowa| awk '/uuid/ {print $5}'`
xe vm-clone name-name-label='dowa-01-clone' uuid=$vm-uuid

Debian, DHCP and IPv6

It was big surprise for that in Debian /etc/network/interface you cannot us iface ethX inet6 dhcp option to use DHCPv6 client to set IP address. If you need IPv6 address from DHCP server, you should can either define interface manually:

iface eth1 inet6 manual
 up /sbin/ip link set eth1 up
 post-up /etc/init.d/wide-dhcpv6-client start
 pre-down /etc/init.d/wide-dhcpv6-client stop
 down /sbin/ip link set eth1 down

or plug it into IPv4 definition (i.e. inet dhcp).

inet dhcp
iface eth1 inet dhcp
 post-up /etc/init.d/wide-dhcpv6-client start
 pre-down /etc/init.d/wide-dhcpv6-client stop

Another awk example

This time how to print all columns others than first one, but only for lines begins with i.e. exim4-config.

hub-02:~# awk '/^exim4-config/ {$1=""; print }' test2

And I got:

 exim4/dc_smarthost string smtp.uk
 exim4/dc_relay_domains string
 exim4/dc_relay_nets string
 exim4/mailname string hub-01.uk
 exim4/dc_localdelivery select mbox format in /var/mail/
 exim4/dc_local_interfaces string 127.0.0.1
 exim4/dc_minimaldns boolean false
 exim4/dc_other_hostnames string
 exim4/dc_eximconfig_configtype select mail sent by smarthost; received via SMTP or fetchmail
 exim4/no_config boolean true
 exim4/hide_mailname boolean false
 exim4/dc_postmaster string toor
 exim4/dc_readhost string
 exim4/use_split_config boolean true

BTW. I need this list to filter responses used to configure Exim in Debian. And I need the answers to reuse them in pressed.
Oh, I obtained the list using:

debconf-get-selections  --install > test2

Xenserver & ipmitool

I found that to enable access BMC information locally from Xenserver you need to load following modules:

•  modprobe ipmi_msghandler
•  modprobe ipmi_devintf
•  modprobe ipmi_si

 and latter as normal:
  ipmitool -I open channel info


Tested on Dell R310 without Drac module.



Another thing worth to note  about Dell and it's BMC — it is not enable by default and you need to press Ctrl+E in the right moment and turn it on. 

UPDATE

Dell support was wrong I can turn on access to BMC over LAN without rebooting box:

• ipmitool -I open lan set 1 access on
• ipmitool -I open lan set 1 user
• ipmitool -I open lan set 1 ipsrc dhcp

I'm not sure if I needed all three command. I'll check another day.

IPMITOOL man:  http://ipmitool.sourceforge.net/manpage.html

not so talkative expect

I'm not sure how I could miss it, but just recently I learned how to set expect not to send everything to the standard output. In the expect file you have to set log_user 0. you can change it but by log_user 1.

This is very helpful to write bash scripts i.e. using my remote_command.expect script - awking output is much easier.

Expect, telnet and Dell switch

Recently I've tried to find a way to obtain MAC addresses of computers attached to a Dell Switch. I've found that i.e. show bridge address-table ethernet 1/g3 returns  MAC addresses behind the port 3. I found small problem - it's not so easy to query all ports (or a chosen set of ports). Therefore, I decided to use expect. You can find the script below.
It connects to a switch run the command and finally print a mac address in the format with ':' between number doublets.
However, there are two issues. I don't know how to avoid sending everything to the standard output. Moreover, if there are more than one MAC address connected to the port (i.e. virtual machines), only first address will be print on the bottom of the output. To be precise the address  appears in the line 6 of output (see line started with set temp_mac).

#!/usr/bin/expect -f

set timeout -1

set machine [lindex $argv 0]
set port [lindex $argv 1]
set command "show bridge address-table ethernet 1/g$port\n"

#Connect to the server
spawn telnet $machine 

expect "User:"
exp_send "admin\r"
expect "Password:"
exp_send "myXEN\r"
expect "?*>"
exp_send "enable\r"
expect "?*#"
exp_send $command
expect "?*#"
set temp_mac [ lindex [ lindex [ split $expect_out(0,string) "\n"] 6] 1]
exp_send "exit\r"
exp_send "quit\r"

puts "\n"

# Creating mac address in DHCP format (with ':')
set mac [ string range $temp_mac 0 1 ]
append mac "."
append mac [ string range $temp_mac 2 6 ]
append mac "."
append mac [ string range $temp_mac 7 11 ]
append mac "."
append mac [ string range $temp_mac 12 13 ]

puts  "$machine/$port:  [ string map  {. :} $mac]"

exit

Oh one more thing. To make a list of all following it's good to run the script in loop and the following one should be good base to start with.

for ((i=1;i<48;i++)) do mac-dell.expect esw44-1 $i| grep "esw44-1/$i"; done

Upgrade Dell BIOS from XenServer 5.6

I've just found that to upgrade the Dell BIOS for R310 (and I guess other servers) on XenServer 5.6 you have to provide the lockfile file which comes from the procmail RPM.

Expect, ssh and two passwords

Let say that you need to run a command on many remote machine and you cannot use any of specialist tools (I mean i.e. func). In such situation expect might be helpful and I even has written about it already. Recently I had to modified my script, cause not all remote systems have the same root password.

#!/usr/bin/expect -f

set timeout -1

set machine [lindex $argv 0]
set command [lindex $argv 1]

set pass "xen4ulez\r" 
puts $machine

spawn ssh -o "NumberOfPasswordPrompts 2" -o "ServerAliveCountMax 1" -l root $machine $command
match_max 100000

expect {
    "assword:" 
      {
        exp_send $pass 
        set pass "xen!king\r"
        exp_continue
    }
}

First, the script has a password set as a variable. It allows to change later. To to this I have expanded the 'expect password' part of the script. Now the script sends the password and changes it after. If the password was fine, script executes the command. The change of the password is not important. If the password is wrong the script sends the variable $pass again, but this time it is new value. I have also added to option to ssh command for script to run a bit faster.

1. NumberOfPasswordPrompts 2 - ensure that ssh tries only twice to provide password.
2. ServerAliveCountMax 1 - ensure that ssh sends only one Server Alive message

Finally I have set StrictHostKeyChecking no and UserKnownHostsFile /dev/null in the .ssh/config file for machines I'm going to use above scripts. This two options set ssh not to bother about checking Host Key. Please remember that it might be not secure. 

Do something every minut in shell

Recently I've needed to do some statistic based on apache log file. I've wanted statistic to be update every minute and want to use only shell.

The first idea was cron.  Sounds nice but there is small problem. I want to be flexible. In theory I can add a job to crontab in the moment I want to start it and remove it (comment out) after finishing, but it's ideal.

No cron, so maybe sleep, especially sleep 60.... But there is another problem. My scripts runs few seconds, so after few minutes (approx. 60/time my scripts runs) I'll have a gap in results.

No cron, no sleep. I needed another direction. Recently I've been playing a bit with data formats. I write a bit of code and after few minutes – volia – I had a working scripts.

#!/bin/sh

min=`date  "+%M"`

while [ 1 ]
do
        ~/test.sh
        while (($min==`date  "+%M"`))
        do
                sleep 10
        done
        min=`date  "+%M"`
done 

One more thing which might be interesting for some. I used infinite loop based on  this blog entry.

How to check a website from the commandline

Sometimes it's useful to check a website connectivity using only command line. I.e. when trying a website from remote host, or when you set up a new box you are going to put behind a loadbalancer.

If you need HTTP connection, telnet will be enough:

> telnet 10.10.10.10 80

GET /index.html HTTP/1.1

host: important.site.com

In the case of HTTPS you need openssl:

> openssl s_client  -connect 10.10.10.10:443 -prexit

GET /index.html HTTP/1.1

host: important.site.com

This is entry for my extend my 'external memory', but you might find it useful.

Copy many lines in Emacs

I'm VIM not Emacs user, but use the later one to applied DNS changes. It has two nice feature helping to maintenance BIND zone files. First, built-in versioning in RCS, but even more helpful the mechanism of updating serial number.

Anyway, I found it's very hard to copy more than one line in Emacs, until I found some suggestion here and here. So to copy the current line without the newline you have to one of two (depends if you want copy or move the lines):

C-a C-Space C-e M-w

C-a C-Space C-e C-w

It means: C-a move to beginning of the line, C-Space sat the mark, C-e go to end line, M-w/C-w save/delete(kill) the region. "OK, but it only one line" you may save and you would be right. But to mark more line you just need to i.e. use arrow before marking a text (by M-w).

64 vs 32 rather then SSD vs HDD

Introduction
Some time ago I promised to present results of the performance test of an Apache serving content from HDD and SSD. The tests confirmed that SSD gives Apache serving static content significant performance boost, however the other remedy for I/O problem has been found.

It's not so easy to test a system from inside. It also applies to the internet. The "image" you see is not exactly what your customers might see. There are some tools helping measure the "real image" of your website (i.e. KeyNote), but to test a difference of performance between two server visible as one entitle from outside (i.e. behind load-balancer) such test are not ideal. (Of course it is possible to prepare a A/B test when A means serving images from one server, B from another one). The web search hadn't brought any technique and parameters useful in such test (of course it might means that the search  was not very good). Therefore, the serving speed [byte/microsecond] parameter were define as ratio of file size [bytes] and response time [microseconds], respectively %B and %D in extended Apache log.


SSD vs HDD
In the first test the SSD box (Sun X4150 with 8 Xeon E5345 @ 2.33GHz cores and 8 GB of memory) was try out against HP ProLiant D380 G5 with the same amount of memory and 4 similar processors/cores (Xeon 5148 @ 2.33GHz). The test were split into two phase. Results for both ones are collected in Table 1. The first day both boxes serve the same amount of requests. The SSD machine sent files two time faster. Next day the weight of connection to machine with SSD was increase on the load balancer, but it doesn't change the results. The SSD box was much faster again. The data presented in Table 1 ensure that both boxes worked with similar set of files (similar size and number of requests).

Table 1. Comparison of server with SSD (SUN X4150) and HDD (HP D380 G5) drives.

SSD machine		HDD machine
Day I
Average speed 24.27563 Average time 13908.68485 Average size 7683.71685 Served files 14983388		Average speed 12.23371 Average time 24674.27763 Average size 7702.66678 Served files 14987023
Day II
Average speed 23.76447 Average time 13107.95822 Average size 7839.91839 Served files 21522835		Average speed 11.10400 Average time 26001.08215 Average size 7829.47865 Served files 10758116

During next phase of research the SSD machine was tested again wider set of servers with HDD servers (machines with Apache using different MPM (prefork and worker), with 8 or 16 GB of memory and with slower and faster processors). SSD machine was faster than any of the server with a classical hard drive. Adding memory as well as changing MPM didn't change the difference in the performance. On the other hand, the machine with more and newer processor sent files much faster than the old ones, however still 50% slower than the SSD test kit.


Table 1. Comparison of Apache performance on machine with a SSD against different machine using a HDD. SUN SSD means the test boxs, OLD HP a box with 4 L5430@2.33Ghz core and NEW HP a server with 8 E5345 cores.

		OLD HP		NEW HP	SUN SSD
speed	16GB (prefork)	8GB (prefork)	8GB (worker)	16GB (worker)	8GB, (worker)
Test 1	9.73669	10.99227	10.82629	16.81683	24.18491
Test 2	9.56062	10.57697	10.51381	16.32999	24.41852
Test 3	8.69836	9.83142	9.65313	16.16164	24.39481
Test 4	9.03057	10.12731	9.98168	15.60127	24.16711

32bit vs 64bit
In the mean time another two things occurred. On the one hand, another test machine, this time with 64bit OS, was built.The first results showed significant decrease of the load and the number of I/O (read) operation. Figure 1 and 2 are good indicator how drastic it was change, even if they were prepared on latter production boxes. Moreover, a change in some of the production server settings exhibited that the machine do not properly caching content served by Apache.




Figure 1. Load on a production system before and after changing OS from 32 to 64 flavour.



Figure 2. I/O operation on a production system before and after changing OS from 32 to 64 flavour.



Further investigation of both phenomena showed that the I/O problem mentioned in the first part of article was cause by inefficient caching, what was cause by memory wasting on a server with 32bit OS. By default Apache is using the sendfile() function rather than the combination of read() and write() functions to transfer files from a storage to a network interface (so called zero copy approach). It speeds up a data transfer by avoiding switching context from kernel to user space, but it also means that Apache cache is limited because on 32-bit OS it cannot allocate more than 3GB of memory.

To confirm that a 32bit OS was causing of performance issues another test on live system was conduct. When comparing two identical servers with 16GB of memory and 8 E5345 Xeon cores, one running a 32bit and second 64bit OS, following observation was made:

• The load decreases from around 2 to 0.5 (Figure 1).
• The read from disk decreases from 2.5 to 0.6 [megabytes/second]  (Figure 2).
• The memory usages increase from 2.5-3 to 16 (full memory) GB.
• The average Apache speed increase from around 12 to 30 [bytes/microsecond] (Figure 3).

Figure 3. Comparison of the Apache speed (speed of sending static content) on 32 and 64 bit OS.    
Summary
During the tests the SSD showed its superiority to the classical HDD storages. However in the case of serving a static content from a web server using a distributed memory caching technique might be faster, more scalable  and even cheaper solution.

Another conclusion from above research, probably less and less important as there are less and less of x86 32bit servers, that 64 bit OS might be really faster, especially in the case of system with big amount of memory. 

Short (mostly shell) fomulas

If you hit this website looking for a Linux/shell/UNIX related advice you might want to visit my identi.ca account. I post "short, magic", Linux formula there now. You know something like:

How to split lines in VIM?

s/\,/,[ctrl-V][Enter]/g

where [ctrl-V][Enter] - means type Ctrl+V and next Enter (you should see s/\,/,^M/g)

noatime

If you need I/O performance bust it might worth to turn on noatime flag on important partitions. Look down to see how it pushed down the read level on a server with Apache serving a static content.

Usually I add noatime flag during a system installation, but this time forgot about it and had to remount the file system. Thanks that mistake I got this beautiful image ;)
You can go even further and turn on nodiratime, it should decrease read even more.

GMAIL and msmtp (Mutt)

This is the example how to configure the msmtp (i.e. for Mutt) to use thegmail smtp server. Remember that you need the certificate. I have got mine from old Ubuntu installation (Saving /etc directory before reinstallation a box is good idea).

account your.user
logfile ~/.msmtp.log

tls on
tls_starttls on    
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on

host smtp.gmail.com
port 587 
from your.user@gmail.com
user your.user@gmail.com
password YOUR_password

BTW, in Ubuntu you can grab certificate by sudo apt-get install ca-certificates.

Nagios plugin

Not so long ago I heard the question if (or rather how) it is possible to write a NRPE plugin checking the resources utilization of an application. I'm using Nagios on the daily basis, but I haven't needed to write any plugin yet. When I went through existing plugins most/all of them checked resources on a server level. It makes sense, you are not so interested what exactly doing you server if your website/database is available and response fast. It especially true if you have 100, 500, 1000, ... servers. Anyway, I found the question interesting, even if was rather theoretical than practical one. After some research I found Jason Faulkner plugin which should be a good base, modified it a bit and created this script:

#!/bin/bash 
#
# Nagios plugin to monitor a process. Can easily be modified to do 
# pretty much whatever you want.
#
# Licensed under LGPL version 2
# Copyright 2006 Broadwick Corporation
# By: Jason Faulkner jasonf@broadwick.com
#
# Modified to measure CPU usage of chosen process.
#
# USAGE: cpu.sh process_name warning_level critical_level
#
# Licensed under LGPL version 2
# Copyright 2009 Wawrzyniec Niewodniczański
# Modification by: Wawrzyniec Niewodniczański wawrzek@gmail.com

process_name=$1
WARLVL=$2
CRITLVL=$3

OKMSG="STATUS OK: ${process_name} running"
CRITMSG="STATUS CRITICAL: ${process_name} using more than ${CRITLVL} % of Memory"
WARNMSG="STATUS WARNING: >1 ${process_name} using more than  ${WARLVL} % of Memory"
UNKMSG="STATUS UNKNOWN: ${process_name}, check if process is running"

PROCESS=`ps axu | grep  -v ${0}|grep -v grep | grep ${process_name}`
CPU=`echo ${PROCESS}| awk '{cpu+=$3} END {printf "%d", cpu}'`

if [[ $PROCESS != "" ]]  
then 
        if (($CPU < $WARLVL))
        then
                echo "$OKMSG"
                exit 0
        elif (( "$CPU" < $CRITLVL ))
        then 
                echo "$WARNMSG"
                exit 1
        else 
                echo "$CRITMSG"
                exit 2
        fi  
else
        echo "$UNKMSG"
        exit 3
fi

I would say that it's nothing excited. There are two important lines. The first one searching the process name in output of ps command and excluding the lines with script name and grep from the list. The another one using awk to add value of CPU usage from the list created in first line. BTW if you would prefer to check memory usage rather then processor, change {cpu+=$3} to {cpu+=$4} (or even to {mem+=$4}) in awk command. I also wrote the nagios command which I believe should work. "believe" not "know", as I haven't try it yet ;)

# 'check_cpu' command definition
define command{
        command_name    check_cpu
        command_line    /usr/lib/nagios/plugins/check_cpu $ARG1$ $ARG2$ $ARG3$
ń} 

Useful links

1. http://www.nagios.org/documentation
2. http://debianclusters.cs.uni.edu/index.php/Creating_Your_Own_Nagios_Plugin
3. http://www.ibm.com/developerworks/aix/library/au-nagios/index.html
4. http://lena.franken.de/nagios/own_service.html

Escape, Escape

I couldn't understand why following command was working on a local machine, but not through ssh (following form):

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += $5;} END {print tsum}'" 

I asked my workmate and he also had problems for some time, but finally he suggested that we needed to "escape" something. After some try we found that ssh don't like $ character so following command works.

ssh server \
"ls -l /var/log/httpd/*-20* \
| awk 'BEGIN {tsum=0} /sizetime/ {tsum += \$5;} END {print tsum}'"