OSW的一些改良

1. 增加进程运行状态检测，异常时自动重启

    由于osw不是守护进程，因此在停机维护以后，很有可能忘记启动，后续在进行问题分析的时候，缺少了有效的数据，因此可以将osw配置成crontab，这样就避免了主机重启以后采集数据丢失的情况。

[oracle@rac11g1 oswbb]$ cat oswcheck.sh 

#!/bin/sh

######################################################################

# Copyright (c)  2016 by Ducw

# oswcheck.sh

# This program check OSWatcher run status, if not run, start OSWatcher

# oswcheck crontab config

# 1 * * * * /oracle/oswbb/oswcheck.sh > /oracle/oswbb/oswcheck.log 2>/oracle/oswbb/oswcheck.err

######################################################################

OSWRUNFLAG=`ps -ef | grep OSWatcher.sh | grep -v grep | wc -l`

CHECKDATA=`date "+%Y-%m-%d %H:%M:%S"`

if [[ ${OSWRUNFLAG} -eq 1 ]]; then

  echo "================================================================================="

  echo "OSWatcher is running at "${CHECKDATA}

  echo "================================================================================="

  echo ""

else

  echo "================================================================================="

  echo "OSWatcher is not running at "${CHECKDATA}

  echo "================================================================================="

  echo ""

  echo "Begin to start startOSWbb.sh"

  cd /oracle/oswbb

  nohup ./startOSWbb.sh 10 72 &

fi

配置crontab

[oracle@rac11g1 oswbb]$ crontab -l

1* * * * /oracle/oswbb/oswcheck.sh > /oracle/oswbb/oswcheck.log 2>/oracle/oswbb/oswcheck.err

2. 调整 ps 输出进程信息

    默认情况下，会输出所有的进程信息，针对数据库主机，信息会比较多，每次开sr，巨大的文件上传也是一个问题。因此只输出CPU使用率较高的前100的进程信息或内存使用前100的进程，这样速度就会快很多。当然这也存在某些情况下，丢失部分有用信息。

[oracle@rac11g1 oswbb]$ cat psmemsub.sh

      HP-UX|HI-UX)

UNIX95=1 ps -e -o user,pid,ppid,pri,pcpu,cpu,vsz,sz,wchan,state,etime,args | head -1 >> $1

UNIX95=1 ps -e -o user,pid,pcpu,ppid,pri,cpu,vsz,sz,wchan,state,etime,args | sort -nr -k 3 | head -100 >> $1

3. 新增心跳网卡监控

    这段信息来至MOS，心跳网卡的监控在RAC环境中尤为重要。

    设置私网间通讯检查： 

a) 拷贝Exampleprivate.net 为 private.net 到同一个目录下。 

b).在private.net中找到您对应的系统平台，替换下面的private_nodename1 ， private_nodename2 为具体的私网IP或者主机名 

traceroute -r -F private_nodename1 

traceroute -r -F private_nodename2 

c). 将private.net中其它的平台部分删除。 

d). 千万不要删除下面的内容： 

rm locks/lock.file 

4. 快速分析某段时间的CPU/内存的使用率

    linux中的paste命令，可以将多个文件的记录进行拼接，同时显示出来。oswtop的数据形式如下，包含时间信息，以及cpu使用率信息。因此我们可以截取这两部的内容，拼接到一个文件，然后对特定字段进行排序操作。这样就可以帮助我们快速定位负载高的时间点。当然也可以使用java图形工具的形式。只是感觉脚本定制的方式更便捷，也不受环境的约束。

    当然这种方法也可以适用于内存等信息的分析。

zzz ***Wed Jan 27 08:53:08 CST 2016

top - 08:53:10 up 9 min,  3 users,  load average: 1.22, 2.01, 1.27

Tasks: 259 total,   2 running, 257 sleeping,   0 stopped,   0 zombie

Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   4050948k total,  3643476k used,   407472k free,   217272k buffers

Swap:  4095992k total,        0k used,  4095992k free,  1929572k cached

[root@rac11g1 oswtop]# cat rac11g1_top_16.01.27.0800.dat | grep ^zzz > top_time.txt

[root@rac11g1 oswtop]# cat rac11g1_top_16.01.27.0800.dat | grep ^Cpu > top_cpu.txt

[root@rac11g1 oswtop]#  paste -d ' ' top_time.txt top_cpu.txt > result.txt

[root@rac11g1 oswtop]# cat result.txt 

zzz ***Wed Jan 27 08:49:08 CST 2016 Cpu(s):  3.0%us,  1.0%sy,  0.0%ni, 94.1%id,  2.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:49:38 CST 2016 Cpu(s):  1.0%us,  1.0%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:50:08 CST 2016 Cpu(s):  2.0%us,  0.0%sy,  0.0%ni, 96.1%id,  0.0%wa,  1.0%hi,  1.0%si,  0.0%st

zzz ***Wed Jan 27 08:50:38 CST 2016 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:51:08 CST 2016 Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:51:38 CST 2016 Cpu(s): 14.0%us,  1.0%sy,  0.0%ni, 85.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:52:08 CST 2016 Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:52:38 CST 2016 Cpu(s):  1.0%us,  1.9%sy,  0.0%ni, 78.6%id, 17.5%wa,  1.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:53:08 CST 2016 Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:53:38 CST 2016 Cpu(s):  2.0%us,  2.0%sy,  0.0%ni, 95.1%id,  0.0%wa,  1.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:54:09 CST 2016 Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  1.0%si,  0.0%st

zzz ***Wed Jan 27 08:54:39 CST 2016 Cpu(s):  1.0%us,  1.0%sy,  0.0%ni, 98.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:55:09 CST 2016 Cpu(s):  2.0%us,  2.0%sy,  0.0%ni, 96.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:55:39 CST 2016 Cpu(s):  1.0%us,  2.0%sy,  0.0%ni, 96.1%id,  0.0%wa,  1.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:56:09 CST 2016 Cpu(s):  5.9%us,  2.0%sy,  0.0%ni, 87.3%id,  4.9%wa,  0.0%hi,  0.0%si,  0.0%st

[root@rac11g1 oswtop]# sort -n -k 9 result.txt  经过排序后的内容

......

zzz ***Wed Jan 27 08:55:39 CST 2016 Cpu(s):  1.0%us,  2.0%sy,  0.0%ni, 96.1%id,  0.0%wa,  1.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:50:08 CST 2016 Cpu(s):  2.0%us,  0.0%sy,  0.0%ni, 96.1%id,  0.0%wa,  1.0%hi,  1.0%si,  0.0%st

zzz ***Wed Jan 27 08:53:38 CST 2016 Cpu(s):  2.0%us,  2.0%sy,  0.0%ni, 95.1%id,  0.0%wa,  1.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:55:09 CST 2016 Cpu(s):  2.0%us,  2.0%sy,  0.0%ni, 96.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:49:08 CST 2016 Cpu(s):  3.0%us,  1.0%sy,  0.0%ni, 94.1%id,  2.0%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:56:09 CST 2016 Cpu(s):  5.9%us,  2.0%sy,  0.0%ni, 87.3%id,  4.9%wa,  0.0%hi,  0.0%si,  0.0%st

zzz ***Wed Jan 27 08:51:38 CST 2016 Cpu(s): 14.0%us,  1.0%sy,  0.0%ni, 85.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

    由于文件是时间顺序生成的，因此时间和对应的资源使用率可以一致对应。剩下的工作就是自由发挥了。当然你也可以写出更优雅的shell脚本。