On Mon, Aug 5, 2013 at 11:49 AM, Merlin Moncure wrote:
*) What I think is happening:
I think we are again getting burned by getting de-scheduled while
holding the free list lock. I've been chasing this problem for a long
time now (for example, see:
http://postgresql.1045698.n5.nabble.com/High-SYS-CPU-need-advise-td5732045.html)
but not I've got a reproducible case. What is happening this:

1. in RelationGetBufferForTuple (hio.c): fire LockRelationForExtension
2. call ReadBufferBI. this goes down the chain until StrategyGetBuffer()
3. Lock free list, go into clock sweep loop
4. while holding clock sweep, hit 'hot' buffer, spin on it
5. get de-scheduled
6. now enter the 'hot buffer spin lock lottery'
7. more/more backends pile on, linux scheduler goes bezerk, reducing
chances of winning #6
8. finally win the lottery. lock released. everything back to normal.
This is an interesting theory, but where's the evidence? I've seen
spinlock contention come from enough different places to be wary of
arguments that start with "it must be happening because...".

IMHO, the thing to do here is run perf record -g during one of the
trouble periods. The performance impact is quite low. You could
probably even set up a script that runs perf for five minute intervals
at a time and saves all of the perf.data files. When one of these
spikes happens, grab the one that's relevant.

If you see that s_lock is where all the time is going, then you've
proved it's a PostgreSQL spinlock rather than something in the kernel
or a shared library. If you can further see what's calling s_lock
(which should hopefully be possible with perf -g), then you've got it
nailed dead to rights.

...Robert

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 14 of 21 | next ›
Discussion Overview
grouppgsql-hackers @
categoriespostgresql
postedAug 5, '13 at 3:49p
activeAug 22, '13 at 1:07p
posts21
users9
websitepostgresql.org...
irc#postgresql

People

Translate

site design / logo © 2017 Grokbase