2010-12-30

CentOS, sar and reporting activity for block devices

Just recently I wanted to watch activity for some block devices on one of our CentOS systems. From manual:

man sar
...
       -d     Report activity for each block device (kernels 2.4 and newer only).  
... 
Let's try:

server:/home/przemol>sar -d
Requested activities not available in file

Ops ! There is something wrong !
Since the sar activity is called (by default) every 10 minutes let's start with cron:

server:/home/przemol>su - 
server # cat /etc/cron.d/sysstat
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 -A

server # cat /usr/lib/sa/sa1

#!/bin/sh
# /usr/lib/sa/sa1.sh
# (C) 1999-2006 Sebastien Godard (sysstat  wanadoo.fr)
#
umask 0022
ENDIR=/usr/lib/sa
cd ${ENDIR}
if [ $# = 0 ]
then
# Note: Stats are written at the end of previous file *and* at the
# beginning of the new one (when there is a file rotation) only if
# outfile has been specified as '-' on the command line...
        exec ${ENDIR}/sadc -F -L 1 1 -
else
        exec ${ENDIR}/sadc -F -L $* -
fi 

As you can see sadc is responsible for real work se we have to look at sadc closer:

server # man sadc
NAME
       sadc - System activity data collector.

SYNOPSIS
       /usr/lib/sa/sadc [ -d ] [ -F ] [ -I ] [ -L ] [ -V ] [ interval [ count ] ] [ outfile ]

DESCRIPTION
       The  sadc  command  samples system data a specified number of times ( count ) at a specified interval measured in seconds ( interval ). It writes in binary format to the specified outfile or to the standard output. If outfile is
       set to -, then sadc uses the standard system activity daily data file, the /var/log/sa/sadd file, where the dd parameter indicates the current day.  By default sadc collects all the data available from  the  kernel.   Exceptions
       are interrupts and disks data, for which the relevant options must be explicitly passed to sadc (see options below).

       When  the count parameter is not specified, sadc writes its data endlessly.  When both interval and count are not specified, a dummy record, which is used at system startup to mark the time when the counter restarts from 0, will
       be written.  For example, one of the system startup script may write the restart mark to the daily data file by the command entry:

       /usr/lib/sa/sadc -

       The sadc command is intended to be used as a backend to the sar command.
       Note: The sadc command only reports on local activities.

OPTIONS
       -d     Tell sadc to report statistics for disks. By default sadc does not report disks activity to prevent data files from growing too large....
We have to -d option to sadc so change /usr/lib/sa/sa1 from:

         exec ${ENDIR}/sadc -F -L 1 1 -

else

        exec ${ENDIR}/sadc -F -L $* -


to

        exec ${ENDIR}/sadc -d -F -L 1 1 -

else

        exec ${ENDIR}/sadc -d -F -L $* -

Then (it is important step !) since current file from /var/log/sa doesn't contains any disk data we have to remove/archive it.
Now:

server # sar -d|head
Linux 2.6.18-194.26.1.el5 (server)      12/30/2010

12:00:01 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
12:10:01 AM    dev8-0      4.83      0.00    116.55     24.12      0.05     10.47      5.07      2.45
12:10:01 AM    dev8-1      4.83      0.00    116.55     24.12      0.05     10.47      5.07      2.45
12:10:01 AM    dev8-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:10:01 AM   dev8-16      2.01      2.99     12.21      7.55      0.01      5.16      2.12      0.43
12:10:01 AM   dev8-17      2.01      2.99     12.21      7.55      0.01      5.16      2.12      0.43
12:10:01 AM   dev8-32      1.24      2.93      9.74     10.24      0.01      5.73      2.75      0.34 


2010-07-26

Brian Cantrill is leaving SUN/Oracle

Brian Cantrill, one of the three Dtrace inventors, is leaving SUN/Oracle.
I wonder how it is possible to loose such people like Brian.
It is really sad ... :-(

2010-06-07

Performance: The "Not a Problem" Problem

I have just found amazing discuss between Jim Mauro, Brendan Gregg, and Roch Bourbonnais. I love it because my feeling are exactly the same and for a few last years I have been struggling with "users" who reports e.g. high CPU usage as a problem. Now I hope after watching this video some of them may see the light ... ;-)