현상
간헐적으로 Internal Error with condition: 'jmpbuf != NULL' (1 args) 에러와 메모리 부족인 3018 에러가 발생합니다.
out 파일
Thread #1596 (Session #1596): tb_thread_exit: (signo:0, sigtype:15)
Process #141 get restart thread (#1596) msg
[1596] Internal Error with condition: 'jmpbuf != NULL' (1 args) (tbsvr_err.h:177:tb_throw) (pid=31591, sessid=1596, tid=1596) ec = -3018 = 4294964278 = 0xfffff436
[1596] client : ip [] process [] logon time : [2025/02/24 15:20:57]
[1596] prev sql : "NULL"
[1596] current sql : "NULL"
[1596] *** 2025/02/24 15:24:01.936 ***
[1596] callstack dump from tbsvr_FGWP139 for [WTHR, 1596, 48970/31591]
[1596] SEE DUMP tbsvr.callstack.31591 (lwpid 48970) BY PSTACK
SQL
복사
callstack
Thread 2 (Thread 0x7f653f229700 (LWP 48970)):
#0 0x00007f6842b4b9fd in nanosleep () from /lib64/libc.so.6
#1 0x00007f6842b4b894 in sleep () from /lib64/libc.so.6
#2 0x0000000000d73046 in tb_system (cmdstring=0xdc584c0 <cmd> "/usr/bin/pstack 31591 >> /data/tibero/tibero6/instance/EGENLMS1/tbsvr.callstack.31591 2>&1", ds=0x3fc4220 <debug_dstream>, kill_child=<optimized out>, timeout_in_sec=297) at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/frame/callstack.c:297
#3 callstack_dump_internal (ds=ds@entry=0x3fc4220 <debug_dstream>, tid=tid@entry=1596, os_thr_id=os_thr_id@entry=48970, pid=pid@entry=31591, dump_dest_dir=dump_dest_dir@entry=0x0) at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/frame/callstack.c:440
#4 0x0000000000d7358e in callstack_dump_ex (ds=0x3fc4220 <debug_dstream>, tid=1596, tid@entry=-1, dump_dest_dir=dump_dest_dir@entry=0x0) at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/frame/callstack.c:472
#5 0x0000000000d719f4 in tb_tracedump (tracedump_cause=tracedump_cause@entry=TRACEDUMP_ASSERT) at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/frame/tbsvr_tracedump.c:91
#6 0x000000000101093e in tb_assert_msg_detail (cond=cond@entry=0x2589baf "jmpbuf != NULL", argc=argc@entry=1, filename=filename@entry=0x2587d16 "tbsvr_err.h", line=line@entry=177, func=func@entry=0x258a81e <__func__.69735> "tb_throw") at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/minimal/tb_assert.c:115
#7 0x000000000067cdab in tb_throw (ec=ec@entry=-3018, file=file@entry=0x282b7c4 "shp_alloc.c", line=line@entry=915, func=func@entry=0x2829ac0 <__func__.75447> :q/slab_alloc.c:225
#11 0x00000000018686f3 in sess_init_post () at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/sess/sess.c:1307
#12 0x0000000000d89e45 in svr_init_tid (my_tid=my_tid@entry=1596, name=name@entry=0x7f6542797cdc "WTHR") at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/frame/tbsvr_main.c:1341
#13 0x0000000000d43926 in wthr_init (args=0x7f6542797cd8) at /home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/frame/tbsvr_cthr.c:3256
#14 0x00007f6845430ea5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f6842b84b0d in clone () from /lib64/libc.so.6
SQL
복사
sys.log
[02-25T10:25:01.729132] [FRM-1581] [I] WTHR(1345) recovery
[02-25T10:25:01.729151] [FRM-1581] [I] thr cleanup WTHR(1345) sess(1345)
...
[02-25T10:25:01.731140] [ALC-1345] [I] Can't extend the allocator space. name: (SHP parent allocator) SLAB, ext_size: 16777216, total size: 721420288, max size: 732332120
[02-25T10:25:01.731164] [ALC-1345] [I] OUT_OF_SHP: shp_malloc fail. tid=1345, sessid=1345, req_size=496 file=/home/chef/workspace/Build_centos_7/tibero6/src/tbsvr/alloc/slab_alloc.c, line=225
[02-25T10:25:01.731181] [FRM-1345] [I] THROW. ec=ERROR_OUT_OF_SHP_FOR_CHUNKTYPE(-3018) [ Out of memory (unable to allocate 496 bytes of shared pool memory for SLAB_ALLOCATOR).] (csr_id:4294967295) [shp_alloc.c:915:shp_malloc_dbginfo]
[02-25T10:25:01.731198] [COM-1345] [I] Internal Error with condition 'jmpbuf != NULL' (1 args) (tbsvr_err.h:177:tb_throw) (pid=31566, sessid=1345, tid=1345)
SQL
복사
v$sga
NAME TOTAL USED
---------------------------------------- ---------- ----------
SHARED MEMORY 1.2885E+10 1.2885E+10
FIXED MEMORY 1.142E+10 1.142E+10
SHARED POOL MEMORY 1464573952 1191182336
SHARED POOL ALLOCATORS (LC) 201326592 193214960
SHARED POOL ALLOCATORS (DD) 134217728 6665448
SHARED POOL ALLOCATORS (SLAB) 721420288 720454048
SHARED POOL ALLOCATORS (MISC) 134217728 10522200
SHARED POOL ALLOCATORS (Total) 1191182336 930856656
Database Buffers 8589934592 8589934592
Redo Buffers 104857600 104857600
SQL
복사
원인
v$sga 및 slog 확인 시, SGA 가용 공간은 일부 남았지만 slab 영역이 더 이상 할당할 수 없는 상태로 확인됩니다.
Tibero에서는 특정 스레드가 종료되는 현상이 발생되면 다른 스레드에서 대신 cleanup을 진행하게 되고 cleanup에 성공할 경우, 종료되었던 스레드를 다시 재 생성합니다.
스레드를 재 생성하는 과정에도 내부에서 사용할 자원을 초기화하는데, 이 과정에서 slab 영역을 사용하게 되며 스레드 재 생성 시 slab 영역 부족으로 인해 'jmpbuf != NULL' 에러가 발생한 것으로 확인됩니다.
해결
메모리를 증설해 해결합니다.