ASM Configuration

Exadata shows error on kfod

Following is an example which configured on Exadata issue.

kfod shows segment fault.

Dump file /oraTB/base/diag/kfod/user_oracle/host_1620649457_93/incident/incdir_81/ora_95934_140163329665856_i81.trc
[TOC00000]
Jump to table of contents
Dump continued from file: /oraTB/base/diag/kfod/user_oracle/host_1620649457_93/trace/ora_95934_140163329665856.trc
[TOC00001]
oci-24550 [11] [[si_signo=11] [si_errno=0] [si_code=1] [si_int=1364841332] [si_ptr=0x7f7a5159d374] [si_addr=0x18]] [] [] [] [] [] [] [] [] [] []
[TOC00001-END]
[TOC00002]
========= Dump for incident 81 (oci 24550) ========
Starting a Diag Context default dump (level=3)

—– Incident Context Dump —–
Address: 0x7fff8c1392f0
Incident ID: 81
Problem Key: oci 24550
Error: oci-24550 [11] [[si_signo=11] [si_errno=0] [si_code=1] [si_int=1364841332] [si_ptr=0x7f7a5159d374] [si_addr=0x18]] [] [] [] [] [] [] [] [] [] []
[00]: dbgePostErrorDirectVaList_int [diag_dde]
[01]: dbgePostErrorDirect [diag_dde]
[02]: kpeDbgSignalHandler []<– Signaling
[03]: skgesig_sigactionHandler []
[04]: __sighandler []
[05]: kgfdTerm [KGF]
[06]: kfodddTerm []
[07]: kfodddInit []
[08]: kfodExecute []
[09]: kfod_main []
[10]: lpmcall []
[11]: lpmpmai []
[12]: main []
[13]: __libc_start_main []

check kfod log:
Trace file /oraTB/base/diag/kfod/user_oracle/host_1620649457_93/trace/ora_95934_140163329665856.trc
OSSIPC:SKGXP:[1378fb0.0]{0}: ifconfig_int(3047a8c0) failed with 0.
OSSIPC:SKGXP:[1378fb0.1]{0}: skgxpcini: Invalid IP address [192.168.71.48] provided
OS system dependent operation:if_not_found failed with status: 0
OS failure message: Error 0
failure occurred at: skgxpvaddr9
additional information: requested interface 192.1xx.71.48 not found. Check output from ifconfig command
DDE: Flood control is not active
2015-06-22T22:39:25.137052+18:00
Incident 81 created, dump file: /oraTB/base/diag/kfod/user_oracle/host_1620649457_93/incident/incdir_81/ora_95934_140163329665856_i81.trc
oci-24550 [11] [[si_signo=11] [si_errno=0] [si_code=1] [si_int=1364841332] [si_ptr=0x7f7a5159d374] [si_addr=0x18]] [] [] [] [] [] [] [] [] [] []

Fix:
check that this ip 71.48 is on 1st node, while current ip should be 71.49.

change following cellinit ip to 49

#cat /etc/oracle/cell/network-config/cellinit.ora
ipaddress1=192.168.71.48/21

ASMLib support on RedHat EL6

recently oracle adds support on ASMlib+RHEL6 kernel.

will seems some policies already being changed.

if someone interested, refer to

http://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel6-1940776.html

To enable ASMlib supports, please download following file from oracle web-site

and get supports ‘kmod-oracleasm’ from RedHat RHN or related support.

good luck.

Use kfed to read ASM disk metadata

又有几时没有更新了，又挤着写东西
kfed可以read/merge，这样可以把一些Disk Metadata以一种可以读的方式展示出来。

养成好习惯还是要的。

[bee@hostname0db01 ~]$ kfed read 'o/192.168.3.23/N5_CD_06_hostname0cel07'
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483696 ; 0x008: TYPE=0x8 NUMB=0x30
kfbh.check:                  2399392961 ; 0x00c: 0x8f03d4c1
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum:                       48 ; 0x024: 0x0030
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: N5_CD_06_hostname0cel07 ; 0x028: length=23
kfdhdb.grpname:                  NORMAL ; 0x048: length=6
kfdhdb.fgname:           hostname0cel07 ; 0x068: length=14
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32951617 ; 0x0a8: HOUR=0x1 DAYS=0xa MNTH=0x3 YEAR=0x7db
kfdhdb.crestmp.lo:            773907456 ; 0x0ac: USEC=0x0 MSEC=0x39 SECS=0x22 MINS=0xb
kfdhdb.mntstmp.hi:             32951617 ; 0x0b0: HOUR=0x1 DAYS=0xa MNTH=0x3 YEAR=0x7db
kfdhdb.mntstmp.lo:            792303616 ; 0x0b4: USEC=0x0 MSEC=0x266 SECS=0x33 MINS=0xb
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  4194304 ; 0x0bc: 0x00400000
kfdhdb.mfact:                    454272 ; 0x0c0: 0x0006ee80
kfdhdb.dsksize:                    5120 ; 0x0c4: 0x00001400
kfdhdb.pmcnt:                         2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      0 ; 0x0d4: 0x00000000
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000
kfdhdb.dbcompat:              186646528 ; 0x0e0: 0x0b200000
kfdhdb.grpstmp.hi:             32951617 ; 0x0e4: HOUR=0x1 DAYS=0xa MNTH=0x3 YEAR=0x7db
kfdhdb.grpstmp.lo:            766188544 ; 0x0e8: USEC=0x0 MSEC=0x2c7 SECS=0x1a MINS=0xb
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000
kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000

[bee@hostname0db01 ~]$ kfed read 'o/192.168.3.23/N5_CD_06_hostname0cel07'

kfbh.endian: 1 ; 0x000: 0x01

kfbh.hard: 130 ; 0x001: 0x82

kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD

kfbh.datfmt: 1 ; 0x003: 0x01

kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0

kfbh.block.obj: 2147483696 ; 0x008: TYPE=0x8 NUMB=0x30

kfbh.check: 2399392961 ; 0x00c: 0x8f03d4c1

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8

kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000

kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000

kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000

kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000

kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000

kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000

kfdhdb.compat: 186646528 ; 0x020: 0x0b200000

kfdhdb.dsknum: 48 ; 0x024: 0x0030

kfdhdb.grptyp: 2 ; 0x026: KFDGTP_NORMAL

kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname: N5_CD_06_hostname0cel07 ; 0x028: length=23

kfdhdb.grpname: NORMAL ; 0x048: length=6

kfdhdb.fgname: hostname0cel07 ; 0x068: length=14

kfdhdb.capname: ; 0x088: length=0

kfdhdb.crestmp.hi: 32951617 ; 0x0a8: HOUR=0x1 DAYS=0xa MNTH=0x3 YEAR=0x7db

kfdhdb.crestmp.lo: 773907456 ; 0x0ac: USEC=0x0 MSEC=0x39 SECS=0x22 MINS=0xb

kfdhdb.mntstmp.hi: 32951617 ; 0x0b0: HOUR=0x1 DAYS=0xa MNTH=0x3 YEAR=0x7db

kfdhdb.mntstmp.lo: 792303616 ; 0x0b4: USEC=0x0 MSEC=0x266 SECS=0x33 MINS=0xb

kfdhdb.secsize: 512 ; 0x0b8: 0x0200

kfdhdb.blksize: 4096 ; 0x0ba: 0x1000

kfdhdb.ausize: 4194304 ; 0x0bc: 0x00400000

kfdhdb.mfact: 454272 ; 0x0c0: 0x0006ee80

kfdhdb.dsksize: 5120 ; 0x0c4: 0x00001400

kfdhdb.pmcnt: 2 ; 0x0c8: 0x00000002

kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001

kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000

kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000

kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000

kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000

kfdhdb.dbcompat: 186646528 ; 0x0e0: 0x0b200000

kfdhdb.grpstmp.hi: 32951617 ; 0x0e4: HOUR=0x1 DAYS=0xa MNTH=0x3 YEAR=0x7db

kfdhdb.grpstmp.lo: 766188544 ; 0x0e8: USEC=0x0 MSEC=0x2c7 SECS=0x1a MINS=0xb

kfdhdb.vfstart: 0 ; 0x0ec: 0x00000000

kfdhdb.vfend: 0 ; 0x0f0: 0x00000000

kfdhdb.spfile: 0 ; 0x0f4: 0x00000000

kfdhdb.spfflg: 0 ; 0x0f8: 0x00000000

ERROR: no PST quorum in group: required 2, found 0

今天在Xdata上犯了个小错误。。上午忙活了半天
几个DG一直没办法MOUNT，显示问题很难看。只觉得是disk missing 又没说具体哪个disk，
然后 mount force也不顶用。

SQL&gt;   ALTER DISKGROUP high  MOUNT force
NOTE: cache registered group HIGH number=1 incarn=0x9679682f
NOTE: cache began mount (first) of group HIGH number=1
       incarn=0x9679682f
Sun Dec 26 22:01:38 2010
ERROR: no PST quorum in group: required 2, found 0
NOTE: cache dismounting (not clean) group 1/0x9679682F (HIGH)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x9679682F (HIGH)
NOTE: cache ending mount (fail) of group HIGH number=1
       incarn=0x9679682f
NOTE: cache deleting context for group HIGH 1/0x9679682f
GMON dismounting group 1 at 31 for pid 29, osid 19112
ERROR: diskgroup HIGH was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "HIGH" cannot be mounted
ORA-15063: ASM discovered an insufficient number
           of disks for diskgroup "HIGH"
ERROR:   ALTER DISKGROUP high  MOUNT force

SQL> ALTER DISKGROUP high MOUNT force

NOTE: cache registered group HIGH number=1 incarn=0x9679682f

NOTE: cache began mount (first) of group HIGH number=1

incarn=0x9679682f

Sun Dec 26 22:01:38 2010

ERROR: no PST quorum in group: required 2, found 0

NOTE: cache dismounting (not clean) group 1/0x9679682F (HIGH)

NOTE: dbwr not being msg'd to dismount

NOTE: lgwr not being msg'd to dismount

NOTE: cache dismounted group 1/0x9679682F (HIGH)

NOTE: cache ending mount (fail) of group HIGH number=1

incarn=0x9679682f

NOTE: cache deleting context for group HIGH 1/0x9679682f

GMON dismounting group 1 at 31 for pid 29, osid 19112

ERROR: diskgroup HIGH was not mounted

ORA-15032: not all alterations performed

ORA-15017: diskgroup "HIGH" cannot be mounted

ORA-15063: ASM discovered an insufficient number

of disks for diskgroup "HIGH"

ERROR: ALTER DISKGROUP high MOUNT force

找了半天发现是重启instance时候不小心，discovery_string改掉了，
是我以前本用来建OCRVD时候，DG太多用来过滤的，于是只好改了回来。。

 asm_diskstring           = "/o/*/PCW_CD_01*"
 ...
 ALTER SYSTEM SET asm_diskstring='/o/*/*' SCOPE=BOTH SID='*';

asm_diskstring = "/o/*/PCW_CD_01*"

...

ALTER SYSTEM SET asm_diskstring='/o/*/*' SCOPE=BOTH SID='*';

然后mount就一切顺利了。。。看来有时候小细节不能忽视啊。。