Informix 11.70.FC4 (Innovator-C edition) crash on Linux

[c***@libero.it]

Informix 11.70.FC4IE
O.S. HP-UX Version B.11.31

The crash happened after 6 months without instance stops.
Memory limitation forced to 2 GB (Innovator-C max accepted):

RESIDENT 1
SHMBASE 0x0
SHMVIRTSIZE 1041176
SHMADD 32768
EXTSHMADD 32768
SHMTOTAL 2097152
SHMVIRT_ALLOCSEG 0.000000
SHMNOACCESS cpu,num=3,noage

Below some details from the af generated by the crash:

03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428KB + 1074492KB > 2097152KB
03:04:01 total allowed by configuration parameter SHMTOTAL
03:04:01 out of virtual shared memory

03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Grid Command Failed
GRID
SQL Statement Error SQLERR:-208 ISAM:0
select * from grid_cdr_node_tab where gn_need_sync = 'y'
03:04:01 ER Monitor received an error from the Internal Replicate Sync subcomponent.
Disabling Monitoring of this subcomponent.
03:04:01 stack trace for pid 17874 written to /opt/IBM/informix/tmp/af.e0638911
03:04:02 Assert Failed: No Exception Handler
03:04:02 IBM Informix Dynamic Server Version 11.70.FC4IE
03:04:02 Who: Session(1696417, ***@rp34, -1, c00000009d1df468)
Thread(1891451, sqlexec, c0000000976ed918, 9)
File: mtex.c Line: 498
03:04:02 Results: Exception Caught. Type: MT_EX_OS, Context: mem
03:04:02 Action: Please notify IBM Informix Technical Support.
03:04:02 See Also: /opt/IBM/informix/tmp/af.e0638911, shmem.e0638911.0
03:04:14 Error writing '/opt/IBM/informix/tmp/shmem.e0638911.0' errno = 28
03:04:14 mtex.c, line 498, thread 1891451, proc id 17874, No Exception Handler.
03:04:15 Fatal error in ADM VP at mt.c:14022
03:04:15 Unexpected virtual processor termination, pid = 17874, exit = 0x100
03:04:15 PANIC: Attempting to bring system down
08:17:36 IBM Informix Dynamic Server Started.
08:17:37 Segment locked: addr=c000000040000000, size=1080758272
Fri Dec 14 08:17:38 2012
08:17:38 Event alarms enabled. ALARMPROG = '/opt/IBM/informix/etc/alarmprogram.sh'
08:17:38 Booting Language <c> from module <>
08:17:38 Loading Module <CNULL>
08:17:38 Booting Language <builtin> from module <>
08:17:38 Loading Module <BUILTINNULL>
08:17:44 DR: DRAUTO is 0 (Off)
08:17:44 DR: ENCRYPT_HDR is 0 (HDR encryption Disabled)
08:17:44 Fast poll /dev/poll enabled.
08:17:44 IBM Informix Dynamic Server Version 11.70.FC4IE Software Serial Number AAA#B000000
08:17:44 HPUX Version B.11.31 -> Using flag/select style KAIO
08:17:44 HP KAIO concurrent requests changed from 1000 to 4000
08:17:45 (11) connection rejected - no calls allowed for sqlexec
08:17:46 (12) connection rejected - no calls allowed for sqlexec
08:17:47 (13) connection rejected - no calls allowed for sqlexec
08:17:47 (14) connection rejected - no calls allowed for sqlexec
08:17:47 (15) connection rejected - no calls allowed for sqlexec
08:17:47 (16) connection rejected - no calls allowed for sqlexec
08:17:47 (17) connection rejected - no calls allowed for sqlexec
08:17:47 (18) connection rejected - no calls allowed for sqlexec
08:17:47 (19) connection rejected - no calls allowed for sqlexec
08:17:47 (20) connection rejected - no calls allowed for sqlexec
08:17:47 (21) connection rejected - no calls allowed for sqlexec
08:17:47 (22) connection rejected - no calls allowed for sqlexec
08:17:47 (23) connection rejected - no calls allowed for sqlexec
08:17:48 (24) connection rejected - no calls allowed for sqlexec
08:17:48 (25) connection rejected - no calls allowed for sqlexec
08:17:48 (26) connection rejected - no calls allowed for sqlexec
08:17:48 (27) connection rejected - no calls allowed for sqlexec
08:17:48 (28) connection rejected - no calls allowed for sqlexec
08:17:48 (29) connection rejected - no calls allowed for sqlexec
08:17:48 (30) connection rejected - no calls allowed for sqlexec
08:17:48 (31) connection rejected - no calls allowed for sqlexec
08:17:48 (32) connection rejected - no calls allowed for sqlexec
08:17:48 (33) connection rejected - no calls allowed for sqlexec
08:17:48 (34) connection rejected - no calls allowed for sqlexec
08:17:48 (35) connection rejected - no calls allowed for sqlexec
08:17:48 (36) connection rejected - no calls allowed for sqlexec
08:17:48 (37) connection rejected - no calls allowed for sqlexec
08:17:48 (38) connection rejected - no calls allowed for sqlexec
08:17:48 IBM Informix Dynamic Server Initialized -- Shared Memory Initialized.
08:17:48 Physical Recovery Started at Page (24:545022).
08:17:49 Physical Recovery Complete: 2719 Pages Examined, 2719 Pages Restored.
08:17:49 Logical Recovery Started.
08:17:49 10 recovery worker threads will be started.
08:17:52 Logical Recovery has reached the transaction cleanup phase.
08:17:52 Logical Recovery Complete.
13212 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks

------------------------------------------------------------------------------

03:04:02 Stack for thread: 1891451 sqlexec

base: 0xc0000000aa3fb000
len: 102400
pc: 0x0000000000000000
tos: 0xc0000000aa40d990
state: running
vp: 9

0x40000000008a9f00 (ormix11.70IE/bin/oninit)afstack
0x400000000082d390 (ormix11.70IE/bin/oninit)mt_ex_throw_sig
0x4000000002bb35c0 (ormix11.70IE/bin/oninit)afsig_handler
0x400000000134fad0 (ormix11.70IE/bin/oninit)mkridnode
0x400000000212aa00 (ormix11.70IE/bin/oninit)rowidlabel2expr
0x4000000000db79a0 (ormix11.70IE/bin/oninit)col2expr
0x4000000000db56d0 (ormix11.70IE/bin/oninit)dotted2expr
0x4000000000c25d70 (ormix11.70IE/bin/oninit)expr2expr
0x4000000002294d00 (ormix11.70IE/bin/oninit)op_bind_rtn
0x400000000228c7c0 (ormix11.70IE/bin/oninit)buildexpr
0x4000000000c26640 (ormix11.70IE/bin/oninit)expr2expr
0x4000000001988a30 (ormix11.70IE/bin/oninit)cond2expr
0x4000000002157e60 (ormix11.70IE/bin/oninit)upd2cb
0x4000000000da4500 (ormix11.70IE/bin/oninit)stmt2cb
0x40000000010e6b20 (ormix11.70IE/bin/oninit)p_sp2cb
0x40000000022ae530 (ormix11.70IE/bin/oninit)parsecmd
0x40000000022ada40 (ormix11.70IE/bin/oninit)sqcmd
0x4000000000bee1a0 (ormix11.70IE/bin/oninit)sq_cmnd
0x40000000022b31f0 (ormix11.70IE/bin/oninit)sq_prepare
0x40000000010ae0c0 (ormix11.70IE/bin/oninit)sqmain
0x4000000002723f70 (ormix11.70IE/bin/oninit)listen_verify
0x400000000271ef90 (ormix11.70IE/bin/oninit)spawn_thread
0x4000000000870750 (ormix11.70IE/bin/oninit)startup
0x4000000003111100 (ormix11.70IE/bin/oninit)coroutine_start

Any idea about this problem?

Thanks
Francesco

[c***@libero.it]

Sorry about the title...the operating system is HP-UX, not Linux.
However, below the onstat -g ses of the session that generated the crash:

/opt/IBM/informix/bin/onstat -g ses 1696417:

IBM Informix Dynamic Server Version 11.70.FC4IE -- On-Line -- Up 122 days 11:57:26 -- 2097152 Kbytes

session effective #RSAM total used dynamic
id user user tty pid hostname threads memory memory explain
1696417 replica - - -1 rp34 1 241664 232864 off

Program :
-

tid name rstcb flags curstk status
1891451 sqlexec c0000000976ed918 ---P--- 1774272832 running-

Memory pools count 2
name class addr totalsize freesize #allocfrag #freefrag
1696417 V c0000000a6786040 237568 8016 292 29
1696417*O0 V c0000000b4f6f040 4096 784 1 1

name free used name free used
overhead 0 6624 scb 0 176
opentable 0 10160 filetable 0 1840
ru 0 608 log 0 16544
temprec 0 21696 keys 0 640
ralloc 0 124688 gentcb 0 1728
ostcb 0 4048 sqscb 0 27280
sql 0 176 hashfiletab 0 560
osenv 0 2816 sqtcb 0 10928
fragman 0 1552 sapi 0 80
udr 0 192 rsam_seqscan 0 448

sqscb info
scb sqscb optofc pdqpriority optcompind directives
c0000000c105cc70 c0000000983f2030 0 0 2 1

Sess SQL Current Iso Lock SQL ISAM F.E.
Id Stmt type Database Lvl Mode ERR ERR Vers Explain
1696417 SELECT imxr CR Wait 3000 -208 0 9.28 Off

Current SQL statement (34593) :
update MMFG:MN_ORDINI_ARCHIVIO set id_mn_ord_archivio = ? , trasmissione =
? , anno = ? , stagione = ? , collezione = ? , tipo_ordine = ? ,
nazione_mn = ? , canale = ? , modello_mn = ? , variante_mn = ? , nome = ?
, listino_acquisto = ? , prezzo_acquisto_eu = ? , listino_vendita = ? ,
prezzo_vendita_eu = ? , data_saldo = ? , stato_chiusura = ? , dta_chiusura
= ? , stato = ? , qta_or1 = ? , qta_or2 = ? , qta_or3 = ? , qta_or4 = ? ,
qta_or5 = ? , qta_or6 = ? , qta_or7 = ? , qta_or8 = ? , dta_ins = ? ,
ute_ins = ? , dta_upd = ? , ute_upd = ? , raggr_saldo = ? , qta_op1 = ? ,
qta_op2 = ? , qta_op3 = ? , qta_op4 = ? , qta_op5 = ? , qta_op6 = ? ,
qta_op7 = ? , qta_op8 = ? , flg_annullo = ? , modulo = ? , destinazione =
? where rowid = ? ;

Host variables :
address type flags value
-----------------------------------------
0xc0000000cf6b0040 CHAR 0x002
0xc0000000cf6b00d0 CHAR 0x002
0xc0000000cf6b0160 CHAR 0x002
0xc0000000cf6b01f0 CHAR 0x002
0xc0000000cf6b0280 CHAR 0x002
0xc0000000cf6b0310 CHAR 0x002
0xc0000000cf6b03a0 CHAR 0x002
0xc0000000cf6b0430 CHAR 0x002
0xc0000000cf6b04c0 CHAR 0x002
0xc0000000cf6b0550 CHAR 0x002
0xc0000000cf6b05e0 CHAR 0x002
0xc0000000cf6b0670 CHAR 0x002
0xc0000000cf6b0700 CHAR 0x002
0xc0000000cf6b0790 CHAR 0x002
0xc0000000cf6b0820 CHAR 0x002
0xc0000000cf6b08b0 CHAR 0x002
0xc0000000cf6b0940 CHAR 0x002
0xc0000000cf6b09d0 CHAR 0x002
0xc0000000cf6b0a60 CHAR 0x002
0xc0000000cf6b0af0 CHAR 0x002
0xc0000000cf6b0b80 CHAR 0x002
0xc0000000cf6b0c10 CHAR 0x002
0xc0000000cf6b0ca0 CHAR 0x002
0xc0000000cf6b0d30 CHAR 0x002
0xc0000000cf6b0dc0 CHAR 0x002
0xc0000000cf6b0e50 CHAR 0x002
0xc0000000cf6b0ee0 CHAR 0x002
0xc0000000cf6b0f70 CHAR 0x002
0xc0000000cf6b1000 CHAR 0x002
0xc0000000cf6b1090 CHAR 0x002
0xc0000000cf6b1120 CHAR 0x002
0xc0000000cf6b11b0 CHAR 0x002
0xc0000000cf6b1240 CHAR 0x002
0xc0000000cf6b12d0 CHAR 0x002
0xc0000000cf6b1360 CHAR 0x002
0xc0000000cf6b13f0 CHAR 0x002
0xc0000000cf6b1480 CHAR 0x002
0xc0000000cf6b1510 CHAR 0x002
0xc0000000cf6b15a0 CHAR 0x002
0xc0000000cf6b1630 CHAR 0x002
0xc0000000cf6b16c0 CHAR 0x002
0xc0000000cf6b1750 CHAR 0x002
0xc0000000cf6b17e0 CHAR 0x002
0xc0000000cf6b1870 CHAR 0x002

Last parsed SQL statement :
update MMFG:MN_ORDINI_ARCHIVIO set id_mn_ord_archivio = ? , trasmissione =
? , anno = ? , stagione = ? , collezione = ? , tipo_ordine = ? ,
nazione_mn = ? , canale = ? , modello_mn = ? , variante_mn = ? , nome = ?
, listino_acquisto = ? , prezzo_acquisto_eu = ? , listino_vendita = ? ,
prezzo_vendita_eu = ? , data_saldo = ? , stato_chiusura = ? , dta_chiusura
= ? , stato = ? , qta_or1 = ? , qta_or2 = ? , qta_or3 = ? , qta_or4 = ? ,
qta_or5 = ? , qta_or6 = ? , qta_or7 = ? , qta_or8 = ? , dta_ins = ? ,
ute_ins = ? , dta_upd = ? , ute_upd = ? , raggr_saldo = ? , qta_op1 = ? ,
qta_op2 = ? , qta_op3 = ? , qta_op4 = ? , qta_op5 = ? , qta_op6 = ? ,
qta_op7 = ? , qta_op8 = ? , flg_annullo = ? , modulo = ? , destinazione =
? where rowid = ? ;

-----------------------------------------------------------------------------

The architecture is composed by 2 nodes only: one (primary) replicate some tables on a secondary-server // the replication is mono-directional. Can the message below be related to the panic?

03:04:01 Grid Command Failed
GRID
SQL Statement Error SQLERR:-208 ISAM:0
select * from grid_cdr_node_tab where gn_need_sync = 'y'
03:04:01 ER Monitor received an error from the Internal Replicate Sync subcomponent.
Disabling Monitoring of this subcomponent.

Still thanks

[jrenaut]

On Tuesday, December 18, 2012 3:43:34 AM UTC-6, ***@libero.it wrote:
<stuff cut>

Post by c***@libero.it
The architecture is composed by 2 nodes only: one (primary) replicate some tables on a secondary-server // the replication is mono-directional. Can the message below be related to the panic?
03:04:01 Grid Command Failed
GRID
SQL Statement Error SQLERR:-208 ISAM:0
select * from grid_cdr_node_tab where gn_need_sync = 'y'
03:04:01 ER Monitor received an error from the Internal Replicate Sync subcomponent.
Disabling Monitoring of this subcomponent.
Still thanks

Actually, I would more likely guess the problem had to do with these messages:

03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428KB + 1074492KB > 2097152KB
03:04:01 total allowed by configuration parameter SHMTOTAL
03:04:01 out of virtual shared memory

03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.

So it looks like you are out of or nearly out of memory available to the server and are using most/all of it. So the asserting thread (which also showed a -208 error in the onstat -g ses output which is a memory allocation failed error) likely tried to allocate memory for a structure, got a 208 error, but then the code path didn't correctly check to see if the structure was allocated or not, and then drove on and later tried to reference it and crashed. So it's probably a defect, but to run into it you need to have little to no free memory available to the running threads on the instance.

Jacques Renaut
IBM Informix Advanced Support
APD Team

[Art Kagel]

So, what Jacques is saying is to resolve this issue finally, you have to
upgrade to an edition that will permit you to support more complex queries
and/or more users. That will mean paying for licenses for Growth Edition
or Ultimate Edition.

Art

Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.

Post by jrenaut
<stuff cut>

Post by c***@libero.it
The architecture is composed by 2 nodes only: one (primary) replicate

some tables on a secondary-server // the replication is mono-directional.
Can the message below be related to the panic?

Post by c***@libero.it
03:04:01 Grid Command Failed
GRID
SQL Statement Error SQLERR:-208 ISAM:0
select * from grid_cdr_node_tab where gn_need_sync = 'y'
03:04:01 ER Monitor received an error from the Internal Replicate Sync

subcomponent.

Post by c***@libero.it
Disabling Monitoring of this subcomponent.
Still thanks

03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428KB + 1074492KB > 2097152KB
03:04:01 total allowed by configuration parameter SHMTOTAL
03:04:01 out of virtual shared memory
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
So it looks like you are out of or nearly out of memory available to the
server and are using most/all of it. So the asserting thread (which also
showed a -208 error in the onstat -g ses output which is a memory
allocation failed error) likely tried to allocate memory for a structure,
got a 208 error, but then the code path didn't correctly check to see if
the structure was allocated or not, and then drove on and later tried to
reference it and crashed. So it's probably a defect, but to run into it
you need to have little to no free memory available to the running threads
on the instance.
Jacques Renaut
IBM Informix Advanced Support
APD Team
_______________________________________________
Informix-list mailing list
http://www.iiug.org/mailman/listinfo/informix-list

[c***@libero.it]

So, what Jacques is saying is to resolve this issue finally, you have to upgrade to an edition that will permit you to support more complex queries and/or more users.  That will mean paying for licenses for Growth Edition or Ultimate Edition.
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/

Sure this is a way; is there another one, trying to arrange the $ONCONFIG parameters (see above) quite differently?

[Art Kagel]

Take a look at all parameters that allocate memory in the virtual and
resident memory segments. That includes all of the SHM* parameters the
BUFFERPOOL parameters, LOCKS (128 bytes per lock in a 64bit server),
DS_NONPDQ_QUERY_MEM, DS_TOTAL_MEMORY, others. Also the environment
variable DBUPSPACE which affects user session memory allocation. Try to
minimize the memory taken by various features. Try to eliminate sorting.
SImplify complex queries where possible.

Art

Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/

Disclaimer: Please keep in mind that my own opinions are my own opinions
and do not reflect on my employer, Advanced DataTools, the IIUG, nor any
other organization with which I am associated either explicitly,
implicitly, or by inference. Neither do those opinions reflect those of
other individuals affiliated with any entity with which I am affiliated nor
those of the entities themselves.

Post by Art Kagel

Post by Art Kagel
So, what Jacques is saying is to resolve this issue finally, you have to

upgrade to an edition that will permit you to support more complex queries
and/or more users. That will mean paying for licenses for Growth Edition
or Ultimate Edition.

Post by Art Kagel
Art
Art S. Kagel
Advanced DataTools (www.advancedatatools.com)
Blog: http://informix-myview.blogspot.com/

Sure this is a way; is there another one, trying to arrange the $ONCONFIG
parameters (see above) quite differently?
_______________________________________________
Informix-list mailing list
http://www.iiug.org/mailman/listinfo/informix-list

[Ian Goddard]

Post by jrenaut
<stuff cut>

Post by c***@libero.it
The architecture is composed by 2 nodes only: one (primary) replicate some tables on a secondary-server // the replication is mono-directional. Can the message below be related to the panic?
03:04:01 Grid Command Failed
GRID
SQL Statement Error SQLERR:-208 ISAM:0
select * from grid_cdr_node_tab where gn_need_sync = 'y'
03:04:01 ER Monitor received an error from the Internal Replicate Sync subcomponent.
Disabling Monitoring of this subcomponent.
Still thanks

03:04:01 Warning: The IBM IDS Innovator-C Edition license restriction limits
03:04:01 the total shared memory size for this server to 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428 KB + 1074492 KB > 2097152 KB.
03:04:01 Size of resident + virtual segments 1055428KB + 1074492KB > 2097152KB
03:04:01 total allowed by configuration parameter SHMTOTAL
03:04:01 out of virtual shared memory

It might be as well to check the application code.

A long time back - Informix 5ish - I had a weekly invoice run crash two
weeks in a row (Friday lunchtime, just to add to the aggravation).
After the second crash I found it was the back-end exceeding its OS
imposed allocation. Although it was a 3rd party package we had been
provided with the source so I spent the afternoon digging through the
it. I eventually found a statement being prepared & executed but with
no release of the resources afterwards and buried deep in nested loops.

It didn't hit until the level of business being invoiced became
sufficient to exhaust the memory limit. As the OP's system has been
running for some time it's possible that some such growth of job size is
happening here.

[c***@libero.it]

Thanks all for the answers!

Happy new year!
Francesco