Discussion:
How to speed up hdr
(too old to reply)
Francesco Alfano
2013-08-13 12:51:01 UTC
Permalink
Hi,
I have a job that needs 26 minutes on a Informix server in Standard
configuration.

When I put a server in Hdr configuration with another on a dedicated 1Gb
ethernet line the same job requires 118 minutes.

Using iftop utility I have saw that the ethernet speed is almost 150KB/s
( iftop -i eth1 -B ) on a 1Gb cable.

The server is a 2 dual processor with 24 cores and informix takes almost
2% of cpu during job elaboration.

How can I speed up informix ?
Art Kagel
2013-08-13 15:00:36 UTC
Permalink
Make the HDE asynchronous with zero delay. There is domething wrong with a
gb network that only gets 150kB per second! Sure that's not 150 mB per sec?

Art
Post by Francesco Alfano
Hi,
I have a job that needs 26 minutes on a Informix server in Standard
configuration.
When I put a server in Hdr configuration with another on a dedicated 1Gb
ethernet line the same job requires 118 minutes.
Using iftop utility I have saw that the ethernet speed is almost 150KB/s
( iftop -i eth1 -B ) on a 1Gb cable.
The server is a 2 dual processor with 24 cores and informix takes almost
2% of cpu during job elaboration.
How can I speed up informix ?
_______________________________________________
Informix-list mailing list
http://www.iiug.org/mailman/listinfo/informix-list
Fernando Nunes
2013-08-13 23:28:54 UTC
Permalink
That's an interesting scenario, but you didn't provide all the details:
1- Do you have isolated environments with nothing more running? Did you
repeat the tests a few times?
2- How are your databases configured? Buffered or unbuffered logging?
3- Does your process execute many transactions (COMMITs)?
4- What are your instance parameters (DR*, LOGBUFFER...)?
5- Apart from HDR configurations, are the standard instance and primary
instance configured te same way and running with the same resources?
6- During the process do you see a lot of flags "G" in onstat -u?

Almost everyone tends to give too much importance to bandwidth and almost
forget around round trip times. You clearly have a very good bandwidth and
I expect the round trip time to be very good (check please). I'm used to
see this every day (not yet saying that's what's happening to you).
The typical situation are ETL lookup stages. Lot's of incredible fast
queries that in the end seem to take forever. Why? Because it's like
playing ping-pong... the ball rarely is touching somewhere. But it takes
time constantly traveling.

How can this apply to you? Well, in theory you may be facing a lot of
message interchange between the two servers. Things that would contribute
to this are:
1- unbuffered logging
2-many commits
3- small LOGBUFFER
4- HDR configured in synchronous mode
5- slow secondary

Some things you can try:

1- SET BUFFERED LOG (run in the session that executes the batch)
2- Make sure you're not using synchronous mode
3- Increase your LOGBUFFER

Regards
Post by Francesco Alfano
Hi,
I have a job that needs 26 minutes on a Informix server in Standard
configuration.
When I put a server in Hdr configuration with another on a dedicated 1Gb
ethernet line the same job requires 118 minutes.
Using iftop utility I have saw that the ethernet speed is almost 150KB/s
( iftop -i eth1 -B ) on a 1Gb cable.
The server is a 2 dual processor with 24 cores and informix takes almost
2% of cpu during job elaboration.
How can I speed up informix ?
_______________________________________________
Informix-list mailing list
http://www.iiug.org/mailman/listinfo/informix-list
--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...
Francesco Alfano
2013-08-14 09:39:10 UTC
Permalink
Post by Fernando Nunes
1- Do you have isolated environments with nothing more running? Did you
repeat the tests a few times?
Yes, Informix is the only program which runs in both machine.
Post by Fernando Nunes
2- How are your databases configured? Buffered or unbuffered logging?
Database Name Owner In Dbspace Status
sysmaster informix dati U
sysutils informix dati U
eda root dati U*

U* -- Mode ANSI (unbuffered logging)
Post by Fernando Nunes
3- Does your process execute many transactions (COMMITs)?
Yes, it does many commit.
Post by Fernando Nunes
4- What are your instance parameters (DR*, LOGBUFFER...)?
# Data Replication Variables
DRINTERVAL -1
DRTIMEOUT 30
DRLOSTFOUND /opt/informix/etc/dr.lostfound

LOCKS 100000
BUFFERS 50000
PHYSBUFF 10000
LOGBUFF 5000

MULTIPROCESSOR 1
SINGLE_CPU_VP 0
VPCLASS CPU,num=10
VPCLASS AIO,num=2

NETTYPE soctcp,3,250,NET
Post by Fernando Nunes
5- Apart from HDR configurations, are the standard instance and primary
instance configured te same way and running with the same resources?
Both machines are the same, and they are directly connected with
ethernet cable.
Post by Fernando Nunes
6- During the process do you see a lot of flags "G" in onstat -u?
No,

Userthreads
address flags sessid user tty wait tout locks nreads
nwrites
4d8ca018 ---P--D 1 root - 0 0 0 17 21
4d8ca618 ---P--F 0 root - 0 0 0 0 5
4d8cac18 ---P--F 0 root - 0 0 0 0 0
4d8cb218 ---P--F 0 root - 0 0 0 0 0
4d8cb818 ---P--F 0 root - 0 0 0 0 0
4d8cbe18 ---P--F 0 root - 0 0 0 0 0
4d8cc418 ---P--F 0 root - 0 0 0 0 0
4d8cca18 ---P--F 0 root - 0 0 0 0 0
4d8cd018 ---P--F 0 root - 0 0 0 0 0
4d8cd618 ---P--- 8 root - 0 0 0 0 0
4d8cdc18 ---P--B 9 root - 0 0 0 0 0
4d8ce218 --BPX-- 29 informix 0 0 0 1 92
1593
4d8ce818 ---P--D 22 root - 0 0 0 0 0
4d8cee18 ---P--D 13 root - 0 0 0 16 5
4d8cf418 Y--P--D 20 root - 4e2aa7a0 0 0 2107 3
4d8cfa18 ---P--D 21 root - 0 0 0 0 0
16 active, 128 total, 25 maximum concurrent
Post by Fernando Nunes
Almost everyone tends to give too much importance to bandwidth and
almost forget around round trip times. You clearly have a very good
bandwidth and I expect the round trip time to be very good (check
please).
The file transfer speed between the two machines is around 100MB/s

I'm used to see this every day (not yet saying that's what's
Post by Fernando Nunes
happening to you).
The typical situation are ETL lookup stages.
What is ETL ?

Lot's of incredible fast
Post by Fernando Nunes
queries that in the end seem to take forever. Why? Because it's like
playing ping-pong... the ball rarely is touching somewhere. But it takes
time constantly traveling.
How can this apply to you? Well, in theory you may be facing a lot of
message interchange between the two servers.
So I see a low utilization of the network due to a high exchange of
messages, rather than data?

One strange thing I noticed is that the amount of data concerned by the
test program are approximately 190 Mbytes, while network traffic
involves 1,80 Gbytes, why ?


Things that would
Post by Fernando Nunes
1- unbuffered logging
Yes
Post by Fernando Nunes
2-many commits
Yes, (but in standard mode it takes only 26 minutes)
Post by Fernando Nunes
3- small LOGBUFFER
4- HDR configured in synchronous mode
Yes, but I prefer it for security
Post by Fernando Nunes
5- slow secondary
The machines are identical.
Post by Fernando Nunes
1- SET BUFFERED LOG (run in the session that executes the batch)
how can I do it ?
Post by Fernando Nunes
2- Make sure you're not using synchronous mode
I use it.
Post by Fernando Nunes
3- Increase your LOGBUFFER
Regards
Hi,
I have a job that needs 26 minutes on a Informix server in Standard
configuration.
When I put a server in Hdr configuration with another on a dedicated 1Gb
ethernet line the same job requires 118 minutes.
Using iftop utility I have saw that the ethernet speed is almost 150KB/s
( iftop -i eth1 -B ) on a 1Gb cable.
The server is a 2 dual processor with 24 cores and informix takes almost
2% of cpu during job elaboration.
How can I speed up informix ?
_______________________________________________
Informix-list mailing list
http://www.iiug.org/mailman/listinfo/informix-list
--
Fernando Nunes
Portugal
http://informix-technology.blogspot.com
My email works... but I don't check it frequently...
Thanks for your help.
Fernando Nunes
2013-08-14 14:21:42 UTC
Permalink
Post by Fernando Nunes
2- How are your databases configured? Buffered or unbuffered logging?
Database Name Owner In Dbspace Status
sysmaster informix dati U
sysutils informix dati U
eda root dati U*
U* -- Mode ANSI (unbuffered logging)
Hmmm... ANSI... That changes a couple of things...
Post by Fernando Nunes
3- Does your process execute many transactions (COMMITs)?
Yes, it does many commit.
On each commit the logical log buffer is flushed (writen to disk) and at
the same time copied to "HDR buffers". There are 12 of these. Not
configurable.
Post by Fernando Nunes
4- What are your instance parameters (DR*, LOGBUFFER...)?
# Data Replication Variables
DRINTERVAL -1
DRTIMEOUT 30
DRLOSTFOUND /opt/informix/etc/dr.lostfound
LOCKS 100000
BUFFERS 50000
PHYSBUFF 10000
LOGBUFF 5000
MULTIPROCESSOR 1
SINGLE_CPU_VP 0
VPCLASS CPU,num=10
VPCLASS AIO,num=2
NETTYPE soctcp,3,250,NET
5- Apart from HDR configurations, are the standard instance and primary
Post by Fernando Nunes
instance configured te same way and running with the same resources?
Both machines are the same, and they are directly connected with ethernet
cable.
6- During the process do you see a lot of flags "G" in onstat -u?
No,
Userthreads
address flags sessid user tty wait tout locks nreads
nwrites
4d8ca018 ---P--D 1 root - 0 0 0 17 21
4d8ca618 ---P--F 0 root - 0 0 0 0 5
4d8cac18 ---P--F 0 root - 0 0 0 0 0
4d8cb218 ---P--F 0 root - 0 0 0 0 0
4d8cb818 ---P--F 0 root - 0 0 0 0 0
4d8cbe18 ---P--F 0 root - 0 0 0 0 0
4d8cc418 ---P--F 0 root - 0 0 0 0 0
4d8cca18 ---P--F 0 root - 0 0 0 0 0
4d8cd018 ---P--F 0 root - 0 0 0 0 0
4d8cd618 ---P--- 8 root - 0 0 0 0 0
4d8cdc18 ---P--B 9 root - 0 0 0 0 0
4d8ce218 --BPX-- 29 informix 0 0 0 1 92 1593
4d8ce818 ---P--D 22 root - 0 0 0 0 0
4d8cee18 ---P--D 13 root - 0 0 0 16 5
4d8cf418 Y--P--D 20 root - 4e2aa7a0 0 0 2107 3
4d8cfa18 ---P--D 21 root - 0 0 0 0 0
16 active, 128 total, 25 maximum concurrent
I trust you, but did you take several snapshots?
IN the case above I'd do "onstat -g ses 29 -r 1 | grep sqlexec"
And I suppose you got thi in the HDR enabled environment?
"G" flags means the session is waiting for a free logical log buffer (there
are 3 of those, not configurable).
This will happen when during the logical log buffer flush, there are no
free HDR buffers. Normally this can be caused by too frequent flushes or by
a slow secondary (that doesn't keep up with the HDR buffers "cleaning").
The "problem" with unbuffered or ANSI databases is that each commit will
trigger a flush, even if you're not on synchronous mode.
In your case you're in synchronous...
Post by Fernando Nunes
Post by Fernando Nunes
Almost everyone tends to give too much importance to bandwidth and
almost forget around round trip times. You clearly have a very good
bandwidth and I expect the round trip time to be very good (check
please).
The file transfer speed between the two machines is around 100MB/s
That's to be expected with 1GB ethernet, but again... that's not the
problem in most cases.
Post by Fernando Nunes
I'm used to see this every day (not yet saying that's what's
Post by Fernando Nunes
happening to you).
The typical situation are ETL lookup stages.
What is ETL ?
Extraction, Transformantion and Loading. An acronym used to referr to
batches that extract data from OLTP systems and load it in Data warehouses
after transforming it.
It was just an example... Many of this jobs process a large set of rows,
and for each sometime have to do a lookup in a reference table.
Post by Fernando Nunes
Lot's of incredible fast
Post by Fernando Nunes
queries that in the end seem to take forever. Why? Because it's like
playing ping-pong... the ball rarely is touching somewhere. But it takes
time constantly traveling.
How can this apply to you? Well, in theory you may be facing a lot of
message interchange between the two servers.
So I see a low utilization of the network due to a high exchange of
messages, rather than data?
It's a possibility...
Post by Fernando Nunes
One strange thing I noticed is that the amount of data concerned by the
test program are approximately 190 Mbytes, while network traffic involves
1,80 Gbytes, why ?
What are you comparing? 190MB is what? The amount of data in the tables
affected? The ammount of data that is INSERTEd/UPDATEd/DELETEd?
Something interesting to compare would be the logical log space consumed
with the network traffic.
Post by Fernando Nunes
Things that would
Post by Fernando Nunes
1- unbuffered logging
Yes
Post by Fernando Nunes
2-many commits
Yes, (but in standard mode it takes only 26 minutes)
In standard mode you don't send part of a logical log buffer after each
commit...
Would it be possible to reduce the number of commits, just for testing
purposes?
Post by Fernando Nunes
3- small LOGBUFFER
Your logbuffer is large. But it's not being used (almost ure about it).
Check "onstat -l" and see how many pages per flush your doing....
Post by Fernando Nunes
4- HDR configured in synchronous mode
Yes, but I prefer it for security
I can understand that, but you'll pay some price. Question i, if the
"price" you're seeing is justifiable or if there is something we can do to
improve it.
Post by Fernando Nunes
5- slow secondary
The machines are identical.
Ok...
Post by Fernando Nunes
Post by Fernando Nunes
1- SET BUFFERED LOG (run in the session that executes the batch)
how can I do it ?
You could alter the code or do it in sysdbopen() procedure. But it will not
change anything because you're using ANSI mode
Post by Fernando Nunes
2- Make sure you're not using synchronous mode
I use it.
3- Increase your LOGBUFFER
Probably will not make any difference becaue of the constant flushing.
I'm not sure if I'm able to help you any further... did you open a PMR? Or
wait for more knowledgeable help here...The group is regularly monitored by
the MACH 11 architect... He knows more about it, than I'll ever learn...
Regards.
Post by Fernando Nunes
Post by Fernando Nunes
Regards
Hi,
I have a job that needs 26 minutes on a Informix server in Standard
configuration.
When I put a server in Hdr configuration with another on a dedicated
1Gb
ethernet line the same job requires 118 minutes.
Using iftop utility I have saw that the ethernet speed is almost
150KB/s
( iftop -i eth1 -B ) on a 1Gb cable.
The server is a 2 dual processor with 24 cores and informix takes
almost
2% of cpu during job elaboration.
How can I speed up informix ?
______________________________**_________________
Informix-list mailing list
http://www.iiug.org/mailman/**listinfo/informix-list<http://www.iiug.org/mailman/listinfo/informix-list>
--
Fernando Nunes
Portugal
http://informix-technology.**blogspot.com<http://informix-technology.blogspot.com>
My email works... but I don't check it frequently...
Thanks for your help.
--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...
Francesco Alfano
2013-08-19 08:06:32 UTC
Permalink
Thanks for your response and your suggestions.

I have done several tests in order to try to understand the functioning
of hdr :

The first test is a sequence of select and delete:
without hdr it takes 28 seconds, with hdr enabled it takes 2 minutes and
41 seconds.

The second test have involved the data move from one table to another,
that is it performs the following steps:
1) select data form orginal table ;
2) insert data in destination table ;
3) delete data from original table ;
4) commit.

Without hdr it takes 11 minutes and 20 seconds, with hdr enabled it
takes 11 minutes and 49 seconds, a very small difference !!!

Why ?

Thanks for your time.
Post by Fernando Nunes
2- How are your databases configured? Buffered or unbuffered
logging?
Database Name Owner In Dbspace Status
sysmaster informix dati U
sysutils informix dati U
eda root dati U*
U* -- Mode ANSI (unbuffered logging)
Hmmm... ANSI... That changes a couple of things...
3- Does your process execute many transactions (COMMITs)?
Yes, it does many commit.
On each commit the logical log buffer is flushed (writen to disk) and at
the same time copied to "HDR buffers". There are 12 of these. Not
configurable.
4- What are your instance parameters (DR*, LOGBUFFER...)?
# Data Replication Variables
DRINTERVAL -1
DRTIMEOUT 30
DRLOSTFOUND /opt/informix/etc/dr.lostfound
LOCKS 100000
BUFFERS 50000
PHYSBUFF 10000
LOGBUFF 5000
MULTIPROCESSOR 1
SINGLE_CPU_VP 0
VPCLASS CPU,num=10
VPCLASS AIO,num=2
NETTYPE soctcp,3,250,NET
5- Apart from HDR configurations, are the standard instance and
primary
instance configured te same way and running with the same resources?
Both machines are the same, and they are directly connected with
ethernet cable.
6- During the process do you see a lot of flags "G" in onstat -u?
No,
Userthreads
address flags sessid user tty wait tout locks
nreads nwrites
4d8ca018 ---P--D 1 root - 0 0 0 17
21
4d8ca618 ---P--F 0 root - 0 0 0 0
5
4d8cac18 ---P--F 0 root - 0 0 0 0
0
4d8cb218 ---P--F 0 root - 0 0 0 0
0
4d8cb818 ---P--F 0 root - 0 0 0 0
0
4d8cbe18 ---P--F 0 root - 0 0 0 0
0
4d8cc418 ---P--F 0 root - 0 0 0 0
0
4d8cca18 ---P--F 0 root - 0 0 0 0
0
4d8cd018 ---P--F 0 root - 0 0 0 0
0
4d8cd618 ---P--- 8 root - 0 0 0 0
0
4d8cdc18 ---P--B 9 root - 0 0 0 0
0
4d8ce218 --BPX-- 29 informix 0 0 0 1 92 1593
4d8ce818 ---P--D 22 root - 0 0 0 0
0
4d8cee18 ---P--D 13 root - 0 0 0 16
5
4d8cf418 Y--P--D 20 root - 4e2aa7a0 0 0 2107
3
4d8cfa18 ---P--D 21 root - 0 0 0 0
0
16 active, 128 total, 25 maximum concurrent
I trust you, but did you take several snapshots?
IN the case above I'd do "onstat -g ses 29 -r 1 | grep sqlexec"
And I suppose you got thi in the HDR enabled environment?
"G" flags means the session is waiting for a free logical log buffer
(there are 3 of those, not configurable).
This will happen when during the logical log buffer flush, there are no
free HDR buffers. Normally this can be caused by too frequent flushes or
by a slow secondary (that doesn't keep up with the HDR buffers "cleaning").
The "problem" with unbuffered or ANSI databases is that each commit will
trigger a flush, even if you're not on synchronous mode.
In your case you're in synchronous...
Almost everyone tends to give too much importance to bandwidth and
almost forget around round trip times. You clearly have a very good
bandwidth and I expect the round trip time to be very good (check
please).
The file transfer speed between the two machines is around 100MB/s
That's to be expected with 1GB ethernet, but again... that's not the
problem in most cases.
I'm used to see this every day (not yet saying that's what's
happening to you).
The typical situation are ETL lookup stages.
What is ETL ?
Extraction, Transformantion and Loading. An acronym used to referr to
batches that extract data from OLTP systems and load it in Data
warehouses after transforming it.
It was just an example... Many of this jobs process a large set of rows,
and for each sometime have to do a lookup in a reference table.
Lot's of incredible fast
queries that in the end seem to take forever. Why? Because it's like
playing ping-pong... the ball rarely is touching somewhere. But
it takes
time constantly traveling.
How can this apply to you? Well, in theory you may be facing a
lot of
message interchange between the two servers.
So I see a low utilization of the network due to a high exchange of
messages, rather than data?
It's a possibility...
One strange thing I noticed is that the amount of data concerned by
the test program are approximately 190 Mbytes, while network traffic
involves 1,80 Gbytes, why ?
What are you comparing? 190MB is what? The amount of data in the tables
affected? The ammount of data that is INSERTEd/UPDATEd/DELETEd?
Something interesting to compare would be the logical log space consumed
with the network traffic.
Things that would
1- unbuffered logging
Yes
2-many commits
Yes, (but in standard mode it takes only 26 minutes)
In standard mode you don't send part of a logical log buffer after each
commit...
Would it be possible to reduce the number of commits, just for testing
purposes?
3- small LOGBUFFER
Your logbuffer is large. But it's not being used (almost ure about it).
Check "onstat -l" and see how many pages per flush your doing....
4- HDR configured in synchronous mode
Yes, but I prefer it for security
I can understand that, but you'll pay some price. Question i, if the
"price" you're seeing is justifiable or if there is something we can do
to improve it.
5- slow secondary
The machines are identical.
Ok...
1- SET BUFFERED LOG (run in the session that executes the batch)
how can I do it ?
You could alter the code or do it in sysdbopen() procedure. But it will
not change anything because you're using ANSI mode
2- Make sure you're not using synchronous mode
I use it.
3- Increase your LOGBUFFER
Probably will not make any difference becaue of the constant flushing.
I'm not sure if I'm able to help you any further... did you open a PMR?
Or wait for more knowledgeable help here...The group is regularly
monitored by the MACH 11 architect... He knows more about it, than I'll
ever learn...
Regards.
Regards
On Tue, Aug 13, 2013 at 1:51 PM, Francesco Alfano
Hi,
I have a job that needs 26 minutes on a Informix server in
Standard
configuration.
When I put a server in Hdr configuration with another on a
dedicated 1Gb
ethernet line the same job requires 118 minutes.
Using iftop utility I have saw that the ethernet speed is
almost 150KB/s
( iftop -i eth1 -B ) on a 1Gb cable.
The server is a 2 dual processor with 24 cores and informix
takes almost
2% of cpu during job elaboration.
How can I speed up informix ?
_________________________________________________
Informix-list mailing list
http://www.iiug.org/mailman/__listinfo/informix-list
<http://www.iiug.org/mailman/listinfo/informix-list>
--
Fernando Nunes
Portugal
http://informix-technology.__blogspot.com
<http://informix-technology.blogspot.com>
My email works... but I don't check it frequently...
Thanks for your help.
--
Fernando Nunes
Portugal
http://informix-technology.blogspot.com
My email works... but I don't check it frequently...
--
--
Best Regards
Francesco Alfano
System Engineer
Phone: +39 (091) 409092
Fax: +39 (091) 6766932
e-mail: ***@reroberto.it
e-mail: ***@yahoo.it
Loading...