개요
Tibero 인스턴스에서 Hang 상태 또는 장애 발생 시, gdb (GNU Debugger) 툴을 활용해 Dump 파일을 수동으로 생성하는 방안에 대해 설명합니다.
방법
1.
LOCK 상태 만들기
Session 1 | Session 2 |
$ tbsql tibero1/tmax
SQL> create table t1 (a number);
SQL> insert into t1 values(100);
SQL> commit;
SQL> update t1 set a=101 where a=100; | |
SQL> update t1 set a=102 where a=100;
→ Hang 상태 |
2.
Process ID 확인
Tibero 인스턴스에 포함된 각 프로세스의 PID(Process ID)를 확인합니다.
$ tbdown pid
1779391: MONP
1779393: MGWP
1779394: FGWP0000
1779395: FGWP0001
1779396: FGWP0002
1779397: PEWP0000
1779398: PEWP0001
1779399: PEWP0002
1779400: PEWP0003
1779401: PEWP0004
1779402: PEWP0005
1779403: AGNT
1779404: DBWR
1779405: RCWP
Plain Text
복사
3.
문제 프로세스 확인
문제가 되는 세션의 상태를 확인합니다.
SQL> col PROG_NAME for a15
SQL> select SID,serial#, serial#,prog_name,client_pid,WLOCK_WAIT from v$session;
SID SERIAL# SERIAL# PROG_NAME CLIENT_PID WLOCK_WAIT
---------- ---------- ---------- --------------- ---------- ------------------
92 278 278 tbsql 1780262
93 286 286 tbsql 1780294 WLOCK_TX
102 4946 4946 tbsql 1791952
3 rows selected.
SQL
복사
4.
gdb 디버거 실행 및 연결
문제가 되는 프로세스에 서버 프로세스에 gdb로 attach 합니다.
$ gdb tbsvr 1779394
GNU gdb (GDB) Rocky Linux 8.2-20.el8.0.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from tbsvr...done.
Attaching to program: /home/viera/tibero7/bin/tbsvr, process 1779394
[New LWP 1779467]
[New LWP 1779469]
[New LWP 1779471]
[New LWP 1779473]
[New LWP 1779475]
[New LWP 1779479]
[New LWP 1779482]
[New LWP 1779493]
[New LWP 1779506]
[New LWP 1779518]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f410dad3307 in epoll_wait () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-251.el8_10.11.x86_64 libaio-0.3.112-1.el8.x86_64
libgcc-8.5.0-22.el8_10.x86_64 libstdc++-8.5.0-22.el8_10.x86_64
SQL
복사
SIGUSR2 무시 설정
(gdb) handle SIGUSR2 nostop
Signal Stop Print Pass to program Description
SIGUSR2 No Yes Yes User defined signal 2
SQL
복사
5.
스레드 상태 확인
스레드 목록을 확인합니다.
(gdb) thread apply all bt
Thread 11 (Thread 0x7f408980b700 (LWP 1779518)):
#0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6
#1 0x0000000000f751aa in tb_sem_wait_internal (wait_mode=SEM_WAIT_MODE_BITQ, usec=0, my_tid=<optimized out>)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:663
#2 tb_sem_wait (my_tid=<optimized out>, timeout=timeout@entry=0, wait_mode=wait_mode@entry=SEM_WAIT_MODE_BITQ)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:953
. . 중략 . .
Thread 3 (Thread 0x7f408d17b700 (LWP 1779469)):
#0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6
#1 0x0000000000f751aa in tb_sem_wait_internal (wait_mode=SEM_WAIT_MODE_BITQ, usec=0, my_tid=<optimized out>)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:663
#2 tb_sem_wait (my_tid=<optimized out>, timeout=timeout@entry=0, wait_mode=wait_mode@entry=SEM_WAIT_MODE_BITQ)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:953
#3 0x0000000000f6de5d in bitq_read_dbginfo (bitmask=..., timeout=timeout@entry=0,
event_type=event_type@entry=WE_WLOCK_TX, id1=458778, id2=id2@entry=67,
file=file@entry=0x6f4b684 "wlock.c", line=1267, func=0x6f34e80 <func.74998> "wlock_acq_internal_local")
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/bitqueue.c:470
#4 0x0000000000fc0031 in wlock_acq_internal_local (old_mode=0x4, we=WE_WLOCK_TX, timeout=-10,
smart=<optimized out>, scope=SO_STMT, mode=WLOCK_PIN, id2=67, id1=458778, type=WLOCK_TX)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:1267
#5 wlock_acq_internal (type=type@entry=WLOCK_TX, id1=458778, id2=67, mode=mode@entry=WLOCK_PIN,
--Type <RET> for more, q to quit, c to continue without paging--
scope=scope@entry=SO_STMT, smart=smart@entry=0 '\000', timeout=timeout@entry=-10, we=we@entry=WE_NOEVENT,
old_mode=old_mode@entry=0x0) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:3824
#6 0x0000000002170107 in wlock_acquire_with_timeout (we=WE_NOEVENT, timeout=-10, scope=SO_STMT,
mode=WLOCK_PIN, id2=<optimized out>, id1=<optimized out>, type=WLOCK_TX)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/include/wlock.h:326
#7 tx_wait_internal (xid=..., max_secs=max_secs@entry=-1, we=we@entry=WE_NOEVENT)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/tx/tx.c:990
#8 0x0000000001e8fc2f in td_dsgmt_tx_wait (tsid=3, rowid=..., sgmt_id=<optimized out>, max_secs=-1, xid=...)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/td/td_dsgmt.c:163
#9 td_dsgmt_alloc_itl (snapshot=0x7f410a648c30, databuf=0x0, databuf@entry=0x7f410a72c888, rowid=...,
rowid@entry=..., max_secs=max_secs@entry=-1, dblk=dblk@entry=0x7f40be807000, dl=dl@entry=0x7f408d176cc0,
rp=rp@entry=0x7f408d176cd0, cleanouted=cleanouted@entry=0 '\000')
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/td/td_dsgmt.c:1097
#10 0x000000000266fa40 in xi_tscan_pick_current (exp_ctx=exp_ctx@entry=0x7f408d176f00, ts_id=<optimized out>,
lrowid=..., bufpool=<optimized out>, use=BU_DBX_UPD, wait_secs=wait_secs@entry=-1,
has_undo=has_undo@entry=0 '\000', stat=0x7f4088926090, is_tcc=is_tcc@entry=0x7f408d176dc0 "")
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/ex/xi_tscan.c:436
#11 0x000000000253cc3b in op_dml_update_ (exp_ctx=exp_ctx@entry=0x7f408d176f00,
exn_dml=exn_dml@entry=0x7f4088925950, idx_ctx=idx_ctx@entry=0x7f408d176ea0)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/ex/op_dml_update.c:295
. . 중략 . .
SQL
복사
6.
문제가 되는 스레드로 전환
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f408d17b700 (LWP 1779469))]
#0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6
SQL
복사
7.
스레드의 호출 스택 확인
(gdb) bt
#0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6
#1 0x0000000000f751aa in tb_sem_wait_internal (wait_mode=SEM_WAIT_MODE_BITQ, usec=0, my_tid=<optimized out>)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:663
#2 tb_sem_wait (my_tid=<optimized out>, timeout=timeout@entry=0, wait_mode=wait_mode@entry=SEM_WAIT_MODE_BITQ)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:953
#3 0x0000000000f6de5d in bitq_read_dbginfo (bitmask=..., timeout=timeout@entry=0,
event_type=event_type@entry=WE_WLOCK_TX, id1=458778, id2=id2@entry=67,
file=file@entry=0x6f4b684 "wlock.c", line=1267, func=0x6f34e80 <func.74998> "wlock_acq_internal_local")
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/bitqueue.c:470
#4 0x0000000000fc0031 in wlock_acq_internal_local (old_mode=0x4, we=WE_WLOCK_TX, timeout=-10,
smart=<optimized out>, scope=SO_STMT, mode=WLOCK_PIN, id2=67, id1=458778, type=WLOCK_TX)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:1267
#5 wlock_acq_internal (type=type@entry=WLOCK_TX, id1=458778, id2=67, mode=mode@entry=WLOCK_PIN,
scope=scope@entry=SO_STMT, smart=smart@entry=0 '\000', timeout=timeout@entry=-10, we=we@entry=WE_NOEVENT,
old_mode=old_mode@entry=0x0) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:3824
#6 0x0000000002170107 in wlock_acquire_with_timeout (we=WE_NOEVENT, timeout=-10, scope=SO_STMT,
mode=WLOCK_PIN, id2=<optimized out>, id1=<optimized out>, type=WLOCK_TX)
at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/include/wlock.h:326
#7 tx_wait_internal (xid=..., max_secs=max_secs@entry=-1, we=we@entry=WE_NOEVENT)
. . 중략 . .
SQL
복사
8.
Dump 파일 생성
(gdb) call wlock_dump_ddl(&dump_dstream, 0)
Thread 4 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 10 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 11 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 9 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 6 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 8 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 2 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 5 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 3 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 7 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 10 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 9 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 2 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 6 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 5 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 4 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 3 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 8 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 7 "tbsvr" received signal SIGUSR2, User defined signal 2.
Thread 11 "tbsvr" received signal SIGUSR2, User defined signal 2.
SQL
복사
9.
Dump 파일 확인
PATH : $TB_HOME/instance/$TB_SID/dump/tracedump
$ ls -al | grep dump_test
-rw-r--r-- 1 viera viera 18627 Feb 10 14:25 dump_test.trc
SQL
복사
잠금 경합(WLOCK_TX)**으로 인해 세션 93이 세션 92를 기다리는 상황
$ cat dump_test.trc
. . 중략 . .
WLOCK = 00007f40918119b0 TYPE = WLOCK_TX ID1 = 458778 ID2 = 67
owner_cnt
X = 1
owners :
ELEM = 00007f40917e8bc8 TID = 92 MODE = X SO_SCOPE = SO_TX
PTRS. SO=00007f40f377d998/00007f40f3d10228, ELEMS=00007f4091811a08/00007f4091811a08
converters :
waiters :
ELEM = 00007f410a6919d0 TID = 93 MODE = PIN SO_SCOPE = SO_STMT
PTRS. SO=00007f410a646a70/00007f410a691a50, ELEMS=00007f4091811a48/00007f4091811a48
bitmask :
. . 중략 . .
SQL
복사