Linux Development – Page 14

Oracle CHM(IPD/OS) graphical report with gnuplot

another post on sar graphical report with gnuplot

ipd/os is a very useful tool to monitor/collect system resource status, and from release 11.2.0.3, it’s now merged as a standard component of oracle clusterware.

the new name: oracle CHM(Cluster Healthy Monitor) shows it’s now a tool for clusterware/rac, it’s very useful to record resource usage and status upto 3days.

but ipd/os report is just plain txt with boring data, it’s a better way to show as a graphical report, gnuplot helps.

FOLLOWING DATA IS JUST USED FOR DEMO, DOESNT IMPLICATE ANY PERFORMANCE STATUS OF RUNNING PRODUCTION ENV.

here’s a sample data from classic ipd/os output(the newest chm output may contains tiny different on formatting), it contains sample data from 07:00:00-10:00:00 for node01.

ipdos_data_node01 <this sample file is very large 100M+…. not upload on this server…>
Whoever would like to get the sample data and want to try, please click here:

#1. get the time column.

grep Clock ipdos_data_node01.txt  | awk '{print $5}'| sed s-'--g &gt; time_node01.txt

1	grep Clock ipdos_data_node01.txt \| awk '{print $5}'\| sed s-'--g > time_node01.txt

Sample: time_node01

Following, to retrieve necessary data on cpu/mem usage and iowait #.

#2. format the text, just collect usable data.

grep "#cpus" ipdos_data_node01.txt &gt;  resource_data_node01
sed -i s-;'.*'--g resource_data_node01

1 2	grep "#cpus" ipdos_data_node01.txt > resource_data_node01 sed -i s-;'.*'--g resource_data_node01

Sample:resource_data_node01

#3. get the cpu usage column.

awk '{print $4}' resource_data_node01 &gt;  cpu_node01.txt

1	awk '{print $4}' resource_data_node01 > cpu_node01.txt

Sample:cpu_node01

#4. get the iowait number column.

awk '{print $20}' resource_data_node01 &gt; iowait_data_node01.txt

1	awk '{print $20}' resource_data_node01 > iowait_data_node01.txt

Sample:iowait_data_node01

#5. get memory usage column.
ipd/os just provide column memory free/total/cache, like:

physmemfree: 33068 physmemtotal: 5975632 mcache: 1869232

1	physmemfree: 33068 physmemtotal: 5975632 mcache: 1869232

to calculate the usage%, need to use (physmemtotal – physmemfree – mcache) / physmemtotal *100%

so we need to get the 3 columns.

awk '{print "echo "scale=2;("$10"-"$8"-"$12")/"$10"*100"|bc -l"}'  resource_data_node01 &gt; mem_usage_node01.txt

1	awk '{print "echo "scale=2;("$10"-"$8"-"$12")/"$10"*100"\|bc -l"}' resource_data_node01 > mem_usage_node01.txt

Sample:mem_usage_node01

run the file to get result..

.  mem_usage_node01.txt &gt; mem_node01.txt

1	. mem_usage_node01.txt > mem_node01.txt

Sample: mem_node01

#6 get the report for cpu/mem usage.

paste time_node01.txt cpu_node01.txt  mem_node01.txt  &gt; ipdos_cpu_mem_gnuplot_node01.txt

1	paste time_node01.txt cpu_node01.txt mem_node01.txt > ipdos_cpu_mem_gnuplot_node01.txt

Sample: ipdos_cpu_mem_gnuplot_node01
run gnuplot with following cmd:

set autoscale
set xdata time
set timefmt "%H.%M.%S"
set ylabel "Performance"
set xlabel "Time"
set title "IPD/OS DATA:CPU/MEM"
set xrange ["07.00":"10:00"]
set yrange ["00:00" : "99:99" ]
set grid
set style data lines
plot "ipdos_cpu_mem_gnuplot_node01.txt" using 1:2 title "CPU", '' using 1:3 title "MEM"

set autoscale

set xdata time

set timefmt "%H.%M.%S"

set ylabel "Performance"

set xlabel "Time"

set title "IPD/OS DATA:CPU/MEM"

set xrange ["07.00":"10:00"]

set yrange ["00:00" : "99:99" ]

set grid

set style data lines

plot "ipdos_cpu_mem_gnuplot_node01.txt" using 1:2 title "CPU", '' using 1:3 title "MEM"

#7 get report for iowait usage.

paste  time_node01.txt  iowait_data_node01.txt  &gt;ipdos_iow_gnuplot_node01.txt

1	paste time_node01.txt iowait_data_node01.txt >ipdos_iow_gnuplot_node01.txt

Sample: ipdos_iow_gnuplot_node01

run gnuplot with following cmd:

set autoscale
set xdata time
set timefmt "%H.%M.%S"
set ylabel "iowait"
set xlabel "Time"
set title "IPD/OS DATA:IOWAIT"
set xrange ["07.00":"10:00"]
set yrange ["0" : "9999" ]
set grid
set style data lines
plot "ipdos_iow_gnuplot_node01.txt" using 1:2 title "iowait"

set autoscale

set xdata time

set timefmt "%H.%M.%S"

set ylabel "iowait"

set xlabel "Time"

set title "IPD/OS DATA:IOWAIT"

set xrange ["07.00":"10:00"]

set yrange ["0" : "9999" ]

set grid

set style data lines

plot "ipdos_iow_gnuplot_node01.txt" using 1:2 title "iowait"

#8. comparison on different nodes.
we can show data on 1node for a duration, and also it helps to compare different node status.
following is a sample to show a four-node cluster running cpu usage:
4 samples, each records the cpu usage,
Sample:
ipdos_cpu_mem_gnuplot_node01
ipdos_cpu_mem_gnuplot_node02
ipdos_cpu_mem_gnuplot_node03
ipdos_cpu_mem_gnuplot_node04

*** why not combine the 4 samples, merge into 1 text file?
well, if you watch into the 4files, they’re of different size(line#), different issues matter…
when some node evicted/rebooted, os data that time may lose on that node,
or when the sys stress is very very high, ipd/os will dump data using a longer interval (>1s).
So it’s fine to use different files, of course they all dumps the same duration 07:00:00-10:00:00.
gnuplot accepts different file inputs.
run gnuplot with following cmd:

set autoscale
set xdata time
set timefmt "%H.%M.%S"
set ylabel "Performance"
set xlabel "Time"
set title "IPD/OS DATA CLUSTER "
set xrange ["07.00":"10:00"]
set yrange ["00:00" : "99:99" ]
set grid
set style data lines
plot "ipdos_cpu_mem_gnuplot_node01.txt" using 1:2 title "node01",  "ipdos_cpu_mem_gnuplot_node02.txt" using 1:2 title "node02",   "ipdos_cpu_mem_gnuplot_node03.txt"  using 1:2 title "node03", "ipdos_cpu_mem_gnuplot_node04.txt"  using 1:2 title "node04"

set autoscale

set xdata time

set timefmt "%H.%M.%S"

set ylabel "Performance"

set xlabel "Time"

set title "IPD/OS DATA CLUSTER "

set xrange ["07.00":"10:00"]

set yrange ["00:00" : "99:99" ]

set grid

set style data lines

plot "ipdos_cpu_mem_gnuplot_node01.txt" using 1:2 title "node01", "ipdos_cpu_mem_gnuplot_node02.txt" using 1:2 title "node02", "ipdos_cpu_mem_gnuplot_node03.txt" using 1:2 title "node03", "ipdos_cpu_mem_gnuplot_node04.txt" using 1:2 title "node04"

well seems it’s not very clear to put all data in this graphic, we can also just compare a short time.

set xrange ["07.30":"08:17"]

1	set xrange ["07.30":"08:17"]

and then plot the line…

sar graphical report with gnuplot

sar is very useful to monitor sys.res, following is a demo to paint gra. report using gnuplot.

gnuplot can be obtained from sf.(while it’s not GPL compatible license) http://gnuplot.sourceforge.net/

for Fedora users, just using yum install gnuplot to make the tool running.

#1, run sar command, and collect a sample of output. as: sar.txt

sar &gt; sar.txt

1	sar > sar.txt

sample data: sar.txt
#2, handle the file, format for the tool, to :sar2.txt

cat sar.txt | sed -e '1,3d'| grep -v Average | cut -d' ' -f1,10- &gt; sar2.txt

1	cat sar.txt \| sed -e '1,3d'\| grep -v Average \| cut -d' ' -f1,10- > sar2.txt

sample data: sar2.txt
#3. run gnuplot with following cmd.

set autoscale
set xdata time
set timefmt "%H:%M:%S"
set style data lines
plot "sar2.txt" using 1:2 title "%user" , '' using 1:3 title "%nice", ''  using 1:4 title "%system", '' using 1:5 title "%iowait", '' using 1:6 title "%steal", '' using 1:7 title "%idle"

set autoscale

set xdata time

set timefmt "%H:%M:%S"

set style data lines

plot "sar2.txt" using 1:2 title "%user" , '' using 1:3 title "%nice", '' using 1:4 title "%system", '' using 1:5 title "%iowait", '' using 1:6 title "%steal", '' using 1:7 title "%idle"

#4. paste the result of demo.

pureDarwin Xmas running on Fedora15+qemu-kvm

PureDarwin was a nice project… but no good updates in last year

I tried qemu running on following guides：

http://www.puredarwin.org/developers/qemu

1. Download the img of puredarwinXmas:

puredarwinxmas.tar.bz2 (md5: fd0ade4da224475e5dc33c2c11d9d0bc)

2. Open qemu…
”

Command used:
qemu -hda ‘puredarwinxmas.vmdk’ -cdrom ‘puredarwinxmas.vmdk’ -boot d -m 512
>> press F8 when asked and echo following line on the boot: prompt.
mach_kernel.voodoo

>> Then have to launch X11 manually:
/usr/X11/bin/startx

”

paste the screenshot here…

Fedora 15: chmsee failed to open .chm file(on XPCOM component)

I’m not using chmsee on Fedora mainline, just fetch codes from googlecodes.

It’s now hitting bug.. which already tracked as bug 833640 on RH side.

Error Message:

Open book failed!
You may need to rebuild the chmsee XPCOM component.

1 2	Open book failed! You may need to rebuild the chmsee XPCOM component.

Problem:

Firefox 12 not upgraded to 13, but thunderbird already to 13 on Fedora15.
no updates on yum….

Need to manually upgrade xulrunner/firefox.
http://koji.fedoraproject.org/koji/buildinfo?buildID=325852
https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=833640

I’m using Fedora 15.x86, as the yum mainline may not update in a few while(it may be outofdate… )
Following packages help..
http://kojipkgs.fedoraproject.org//packages/xulrunner/13.0.1/1.fc15/i686/xulrunner-13.0.1-1.fc15.i686.rpm
http://kojipkgs.fedoraproject.org//packages/xulrunner/13.0.1/1.fc15/i686/xulrunner-devel-13.0.1-1.fc15.i686.rpm
http://kojipkgs.fedoraproject.org//packages/firefox/13.0.1/1.fc15/i686/firefox-13.0.1-1.fc15.i686.rpm

LXR-0.11.1 + glimpse-4.18.6 on Fedora 15… reading linux kernel 0.12

Recently I’m reading linux kernel 0.12. I setup lxr+glimpse on my Fedora 15 node.
Following is the steps:

1. Prepare…

uname=2.6.43.8-1.fc15.i686

1	uname=2.6.43.8-1.fc15.i686

with following packages installed:

httpd-2.2.22-1.fc15.i686
mod_perl-2.0.4-14.fc15.i686
mysql-server-5.5.23-1.fc15.i686
ctags-5.8-3.fc15.i686
perl-5.12.4-166.fc15.i686
perl-CPAN-1.9402-166.fc15.noarch
perl-DBD-MySQL-4.018-7.fc15.i686

httpd-2.2.22-1.fc15.i686

mod_perl-2.0.4-14.fc15.i686

mysql-server-5.5.23-1.fc15.i686

ctags-5.8-3.fc15.i686

perl-5.12.4-166.fc15.i686

perl-CPAN-1.9402-166.fc15.noarch

perl-DBD-MySQL-4.018-7.fc15.i686

2. Install glimpse…
http://webglimpse.net/trial/glimpse-4.18.6.tar.gz

download/congfigure/make/make install
If any package lost, use yum/cpan to install.
You can inst as default(/usr/local)…
I use prefix=/usr/suse/lxr/glimpse

3. Install lxr…
http://sourceforge.net/projects/lxr/
I download lxr-0.11.1.tgz, no necessary to compile
extra/untar the package and

mv lxr-0.11.1/ /usr/suse/lxr/lxr

1	mv lxr-0.11.1/ /usr/suse/lxr/lxr

4. Configuration…
I configure a single tree here.

cd /usr/suse/lxr/lxr   (my ${LXR/ROOT})
mkdir lxrconf.d
./scripts/initial-config.sh
./scripts/initdb-config.sh

cd /usr/suse/lxr/lxr (my ${LXR/ROOT})

mkdir lxrconf.d

./scripts/initial-config.sh

./scripts/initdb-config.sh

.htaccess file will be generated into lxrroot, no need to modify…
config file will be generated into lxrconf.d/

5. handle the configuration files…
modify apache-lxrserver.conf, erase following caret: “alias ^/lxr” –> “alias /lxr”

cd lxrconf.d/
cp apache-lxrserver.conf /etc/httpd/conf.d/

1 2	cd lxrconf.d/ cp apache-lxrserver.conf /etc/httpd/conf.d/

mysql import lxrconf.d/initdb-mysql-custom.sql with correct user/passwd/dbname.
6. copy lxrconf.d/lxr.conf to ${LXRROOT} and configure…
I paste a copy of my lxr.conf here
lxr.conf

7. prepare some templates html file, ensure httpserver can access the files…

cd ${LXRROOT}
cp -r templates/html/ to ./html
chmod 777  ./html -R

cd ${LXRROOT}

cp -r templates/html/ to ./html

chmod 777 ./html -R

8. all ready..

mkdir ${LXRROOT}/src
cd ${LXRROOT}/src
echo "0.12" &gt; version

mkdir ${LXRROOT}/src

cd ${LXRROOT}/src

echo "0.12" > version

ln or cp folder of kernel0.12 here, name as “0.12”

9. go to folder ${LXRROOT}, generate Index…

 ./genxref --url=http://localhost/lxr --version=0.12

1	./genxref --url=http://localhost/lxr --version=0.12

we can see records into db…

10. re/start httpd server.
access http://localhost/lxr/source

all should be ok now…

[debug]
1. Cannot show ident correctly, httpd server errorlog shows
[Thu Jul 5 18:24:33 2012] warning: LXR::Template, line 105: Template file ‘htmlident’ => ‘/usr/suse/lxr/lxr/http/html-ident.html’ does not exist
[Thu Jul 5 18:24:33 2012] fatal: ModPerl::ROOT::ModPerl::RegistryPrefork::usr_suse_lxr_lxr_ident, line 188: ‘htmlident’ template not configured at /usr/suse/lxr/lxr/ident line 188, <FILETYPE> line 4.

just ensure lxr.conf matches step#7. and give folder html enough privilege.

2. No need to copy ${LXRROOT}/lib files to other perl @INC path.
/etc/httpd/conf.d/apache-lxrserver.conf will call following file to include…
/usr/suse/lxr/lxr/lxrconf.d/apache2-require.pl

—————————-

Well , glimpse here can be replaced with swish-e, I have another post to provide swish-e RPMs for fedora15/16.
Any problems on similar configuration, plz feel free to drop me a message.

-s

[update] Aug.17th

About the “src” folder:
In the configure file, writes:

        , 'variables' =&gt;
                { 'v' =&gt;
                        { 'name' =&gt; 'Version'
                        , 'range' =&gt; [ readfile('src/version') ]
                        , 'default' =&gt; '0.12'
                        }
                , 'a' =&gt;
                        { 'name'  =&gt; 'Architecture'
                        , 'range' =&gt; [qw(i386 alpha arm m68k mips ppc sparc sparc64)]
                        }
        }

, 'variables' =>

{ 'v' =>

{ 'name' => 'Version'

, 'range' => [ readfile('src/version') ]

, 'default' => '0.12'

}

, 'a' =>

{ 'name' => 'Architecture'

, 'range' => [qw(i386 alpha arm m68k mips ppc sparc sparc64)]

}

I didnt comment out “range ” values for new versions of linux kernel contains such code.

About the Step8. first time I use “>” and when new version trees come, I use “>>” instead.. like following

echo "0.12" &gt; version
echo 2.6.32 &gt;&gt; version

1 2	echo "0.12" > version echo 2.6.32 >> version

So, if other users who dont handle version intro with this file, please just configure above code, and hard-code modify “range” array.

above I mentioned “ln or cp” trees here, just as Andre suggested it’s not a good idea to put source code inside the app folder.
well I’m now not installing lxr in a standard prefix(like: /usr/local/ or /usr/), so I would like the data integrated

I place source code trees in other place and use soft link to keep update.

[suse@suse /usr/suse/lxr/lxr/src]$ls -al
total 12
drwxrwxr-x  2 apache apache 4096 Aug 16 14:21 .
drwxr-xr-x 14 apache apache 4096 Jul 18 14:56 ..
lrwxrwxrwx  1 apache apache   38 Jul  4 15:40 0.12 -&gt; /home/suse/workspace/kernel/linux-0.12
lrwxrwxrwx  1 root   root     25 Aug 16 14:21 2.6.11 -&gt; /codes/linux/linux-2.6.11
lrwxrwxrwx  1 root   root     26 Jul 18 09:11 2.6.32 -&gt; /codes/linux/linux-stable/
lrwxrwxrwx  1 root   root     42 Jul  6 13:04 3.0 -&gt; /home/suse/workspace/kernel/linux-3.0-rc1/
-rw-rw-r--  1 apache apache   23 Aug 16 14:23 versions

[suse@suse /usr/suse/lxr/lxr/src]$ls -al

total 12

drwxrwxr-x 2 apache apache 4096 Aug 16 14:21 .

drwxr-xr-x 14 apache apache 4096 Jul 18 14:56 ..

lrwxrwxrwx 1 apache apache 38 Jul 4 15:40 0.12 -> /home/suse/workspace/kernel/linux-0.12

lrwxrwxrwx 1 root root 25 Aug 16 14:21 2.6.11 -> /codes/linux/linux-2.6.11

lrwxrwxrwx 1 root root 26 Jul 18 09:11 2.6.32 -> /codes/linux/linux-stable/

lrwxrwxrwx 1 root root 42 Jul 6 13:04 3.0 -> /home/suse/workspace/kernel/linux-3.0-rc1/

-rw-rw-r-- 1 apache apache 23 Aug 16 14:23 versions

I think the soft link is alternatively a good choice to avoid data loosing when app update/deinstall.

And of course when new install/upgrade needed, I think we can move away the src folder and softlink back here.

About the err mesg#1 and step7. on folder “/lxr/lxrconf.d/html/”

here I didnt successfully configured the server as I didnt grant enough privilege to the “/lxr/lxrconf.d/html/” dir. it’s why failed.
I mv the folder to “/lxr/http” and grant privilege, it’s ok to access now.

I think it ok not move the folder, just chown/chmod to give enough privilege to let httpserver to access the folder.
My change is not a must, I just think it ok to delete dir “”/lxr/lxrconf.d/” after server successfully running. and the server not rely on this folder now.

and before step#7, invoke “cd ” to ${LXRROOT}, already updated.

Thanks Andre pointing out these issues,

-s