AIX slow buffer reads

I doubt disk/io is the problem.

*) Are the plans *exactly* the same?

*) Are you running explain analyze? There are some platform specific
interactions caused by timing.

*) Are you transferring the data across the network? rule out
(horribly difficult to diagnose/fix) network effects.

merlin

Me either.
Like I said, AIX do not touch the storage when runing the query.
It became CPU-bound after data got into cache.


The plan I sent refers to the AIX box:
http://explain.depesz.com/s/5oz
At Debian, the plan looks pretty much the same.

Yes. I´m not concerned about timing because the difference (8s against 0.3s) is huge.

Not likely... Both boxes are in the same Bladecenter, using the same storage.


[]´s, Andre Volpato

Have you confirmed that the hardware is ok on both servers?

Have both OS's been tuned by people that know how to tune the respective
OS's? AIX is very different than Linux, and needs to be tuned accordingly.

On AIX can you trace why it is CPU bound? What else is taking the CPU
time, anything?

Also, can you provide the output of pg_config from your AIX build?

----- Mensagem original -----
(...)
The hardware was recently instaled and checked by the vendor team.
AIX box is on JS22:
PostgreSQL 8.4.4, AIX 5.3-9 64bits, SAN IBM DS3400, 8x450GB SAS 15K Raid-5
8GB RAM (DDR2 667)

# lsconf
System Model: IBM,7998-61X
Processor Type: PowerPC_POWER6
Processor Implementation Mode: POWER 6
Processor Version: PV_6
Number Of Processors: 4
Processor Clock Speed: 4005 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
Memory Size: 7680 MB

Debian box is on HS21:
PostgreSQL 8.4.4, Debian 4.3.2 64bits, SAN IBM DS3400, 5x300GB SAS 15K Raid-0
7GB RAM (DDR2 667)
We are forced to use RedHat on this machine, so we are virtualizing the Debian box.

# cpuinfo
processor : [0-7]
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5420 @ 2.50GHz
stepping : 6
cpu MHz : 2500.148
cache size : 6144 KB


We´ve been tuning AIX for the last 3 weeks, and lots of tuneables got changed.
On Debian, we have far more experience, and it´s been a chalenge to understand how AIX works.

Most important tunes:
page_steal_method=1
lru_file_repage=0
kernel_heap_psize=64k
maxperm%=90
maxclient%=90
minperm%=20

Disk:
chdev -l hdisk8 -a queue_depth=24
chdev -l hdisk8 -a reserve_policy=no_reserve
chdev -l hdisk8 -a algorithm=round_robin
chdev -l hdisk8 -a max_transfer=0x400000

HBA:
chdev -l fcs0 -P -a max_xfer_size=0x400000 -a num_cmd_elems=1024

Postgres:
shared_buffers = 2304MB
effective_io_concurrency = 5
wal_sync_method = fdatasync
wal_buffers = 2MB
checkpoint_segments = 32
checkpoint_timeout = 10min
random_page_cost = 2.5
effective_cache_size = 7144MB

Like I said, there´s more but this is the most important.


We´re using iostat, svmon and vmstat to trace CPU, swap and IO activity.
On 'topas' we saw no disk activity at all, but we get a Wait% about 70%, and about 700 pages/s read in PageIn, no PageOut, no PgspIn and no PgspOut.
It´s a dedicated server, no process runing besides postgres.


# pg_config
BINDIR = /usr/local/pgsql/bin
DOCDIR = /usr/local/pgsql/share/doc
HTMLDIR = /usr/local/pgsql/share/doc
INCLUDEDIR = /usr/local/pgsql/include
PKGINCLUDEDIR = /usr/local/pgsql/include
INCLUDEDIR-SERVER = /usr/local/pgsql/include/server
LIBDIR = /usr/local/pgsql/lib
PKGLIBDIR = /usr/local/pgsql/lib
LOCALEDIR = /usr/local/pgsql/share/locale
MANDIR = /usr/local/pgsql/share/man
SHAREDIR = /usr/local/pgsql/share
SYSCONFDIR = /usr/local/pgsql/etc
PGXS = /usr/local/pgsql/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--enable-integer-datetimes' '--with-readline' '--with-threads' '--with-zlib' '--with-html' 'CC=gcc -maix64' 'LDFLAGS=-Wl,-bbigtoc'
CC = gcc -maix64
CPPFLAGS =
CFLAGS = -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv
CFLAGS_SL =
LDFLAGS = -Wl,-bbigtoc -Wl,-blibpath:/usr/local/pgsql/lib:/usr/lib:/lib
LDFLAGS_SL = -Wl,-bnoentry -Wl,-H512 -Wl,-bM:SRE
LIBS = -lpgport -lz -lreadline -lld -lm
VERSION = PostgreSQL 8.4.4

[]´s, Andre Volpato

I wonder if effective_io_concurrency has anything to do with it. It was
implemented and mainly tested on Linux, and I am unsure if it will do
anything on AIX. The plan you posted for the query does a bitmap index
scans which is what effective_io_concurrency will speed up.

Can you post the output of explain analyze for that query on both AIX
and Linux? That will show where the time is being spent.

If it is being spent in the bitmap index scan, try setting
effective_io_concurrency to 0 for Linux, and see what effect that has.

----- Mensagem original -----

I changed the querys in order to make a more valuable comparison.

Debian first run (23s):
http://explain.depesz.com/s/1fT

AIX first run (40s):
http://explain.depesz.com/s/CRG

Debian cached consecutive runs (8s)
http://explain.depesz.com/s/QAi

AIX cached consecutive runs (12s)
http://explain.depesz.com/s/xJU

Both boxes are runing with DDR2 667, so RAM speed seems to be the bootleneck now.
We´re about to try RedHat EL6 in the next few days.
I disabled effective_io_concurrency at AIX but it made no changes on bitmap index times.



[]´s, Andre Volpato

Brad's point is that it probably doesn't do anything at all on AIX, and
is already disabled accordingly. But on Linux, it is doing something,
and that might be contributing to why it's executing so much better on
that platform. If you disable that parameter on your Debian box, that
should give you an idea whether that particular speed-up is a major
component to the difference you're seeing or not.

Also, if the system check was done by the "vendor team" team, don't
trust them at all. It doesn't sound like a disk problem is involved in
your case yet, but be sure to do your own basic disk benchmarking too
rather than believing what you're sold. There's a quick intro to that
at http://www.westnet.com/~gsmith/content/postgresql/pg-disktesting.htm
and a much longer treatment of the subject in my book if you want a lot
more details. I don't have any AIX-specific tuning advice in there though.

AFAICT from googling, AIX does have posix_fadvise, though maybe it
doesn't do anything useful ...

regards, tom lane

If there is an easy way to check if it does do anything useful? If so,
I can check it out.

If you don't see any performance change in bitmap scans between
effective_io_concurrency = 0 and effective_io_concurrency = maybe 4 or
so, then you could probably conclude it's a no-op.

regards, tom lane

----- Mensagem original -----

Cant do it right now, but will do it ASAP and post here.


I´m gonna read your sugestion, thanks.
We tested the disks also, and we did a lot of tuning to get acceptable transfer rates at AIX.

Yesterday I tried your "stream-scaling" and get around 7000MB/s (single thread) and 10000MB/s (eight threads) at AIX, and a little less than that at Debian, since its a virtual box.
I found that even my notebook is close to that transfer rates, and both boxes are limited by DDR2 speeds.

[]´s, André Volpato

----- Mensagem original -----

Here´s new explains based on Debian box:

(1) effective_io_concurrency = 5
# /etc/init.d/postgresql stop
# echo 3 > /proc/sys/vm/drop_caches
# /etc/init.d/postgresql start

http://explain.depesz.com/s/br

(2) effective_io_concurrency = 0
# /etc/init.d/postgresql stop
# echo 3 > /proc/sys/vm/drop_caches
# /etc/init.d/postgresql start

http://explain.depesz.com/s/3A0

BitmapAnd really gets improved a little bit in (1), but Bitmap index scans got a lot worse.

[]´s, Andre Volpato