FAQ
We have experienced a sudden and dramatic decrease in performance
sometime over the weekend (after Sat but before Monday 4 am). In
following Gaja's tuning philosophy, I've found that the top 2 waits are
usually (always 2 of the top 3) SQL*Net Message to client/SQL*Net more
data to client. Everybody swears there have been no changes. SA's say
no harware or kernel changes. AppDev say no code changes. DBA (me) says
no database changes.

WAN folks say no WAN issues and ping is responding at expected speed.
SA's say LAN card has had no errors during this time frame and is
processing a good number of bytes but nowhere near it's capacity.
The application has some very good timing points where there is no
human element in response time, but there is a big "unknown" category
that is a larger chunk of time than previously. We suspect that is
machine wait time of some kind.

We just bounced the instance because someone wanted to try it and after
being back up for 20 minutes, early indicators are that performance is
back to normal. We'll see how long that lasts.

We have seen a few client sessions getting errors that indicate
connectivity problems (listener not responding, etc) so we wrote a .com
file that is repeatedly connecting to the database and will run
overnight and stop if there are any errors.

Metalink search for SQL*Net waits gives both "tuning advice" and "you
can't tune much" notes. I strongly suspect some kind of hardware
failure, but don't know where since everyone involved says everything is
working fine.

Environment Notes:
Server
8.1.7.3
Tru64 5.1A (upgrade to A was done a few weeks ago)
Compaq GS160 with 16 CPU's and 32 GB RAM (RAM is from memory, so that
may be off)

Client
Open VMS version 7.2
Client is 8.0.5

Any ideas on a next step for finding out the cause (solution) to this
drop in performance???

Help
Stephen

--
Please see the official ORACLE-L FAQ: http://www.orafaq.com
--
Author: Stephen Andert
INET: StephenAndert_at_firsthealth.com

Fat City Network Services -- 858-538-5051 http://www.fatcity.com
San Diego, California -- Mailing list and web hosting services
---------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from). You may
also send the HELP command for other information (like subscribing).

Search Discussions

  • Anjo Kolk at Oct 30, 2002 at 1:23 am
    This is called "Blame Storming". Every component works "fine" but
    response time sucks and the problem is some other area. So how do we
    turn "Blame Storming" into "Brain Storming"?

    Check out the network components. One of the problems is that the
    network people look at utilization, instead of response time. They will
    find that utilization of certain components may be low (due to some
    problem), and assume that the problem is some where else. Can they tell
    how long a packet is on your network connection?

    Anjo.

    -----Original Message-----
    Andert
    Sent: Wednesday, October 30, 2002 1:28 AM
    To: Multiple recipients of list ORACLE-L

    We have experienced a sudden and dramatic decrease in performance
    sometime over the weekend (after Sat but before Monday 4 am). In
    following Gaja's tuning philosophy, I've found that the top 2 waits are
    usually (always 2 of the top 3) SQL*Net Message to client/SQL*Net more
    data to client. Everybody swears there have been no changes. SA's say
    no harware or kernel changes. AppDev say no code changes. DBA (me) says
    no database changes.

    WAN folks say no WAN issues and ping is responding at expected speed.
    SA's say LAN card has had no errors during this time frame and is
    processing a good number of bytes but nowhere near it's capacity.
    The application has some very good timing points where there is no
    human element in response time, but there is a big "unknown" category
    that is a larger chunk of time than previously. We suspect that is
    machine wait time of some kind.

    We just bounced the instance because someone wanted to try it and after
    being back up for 20 minutes, early indicators are that performance is
    back to normal. We'll see how long that lasts.

    We have seen a few client sessions getting errors that indicate
    connectivity problems (listener not responding, etc) so we wrote a .com
    file that is repeatedly connecting to the database and will run
    overnight and stop if there are any errors.

    Metalink search for SQL*Net waits gives both "tuning advice" and "you
    can't tune much" notes. I strongly suspect some kind of hardware
    failure, but don't know where since everyone involved says everything is
    working fine.

    Environment Notes:
    Server
    8.1.7.3
    Tru64 5.1A (upgrade to A was done a few weeks ago)
    Compaq GS160 with 16 CPU's and 32 GB RAM (RAM is from memory, so that
    may be off)

    Client
    Open VMS version 7.2
    Client is 8.0.5

    Any ideas on a next step for finding out the cause (solution) to this
    drop in performance???

    Help
    Stephen

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: StephenAndert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Anjo Kolk
    INET: anjo_at_oraperf.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Mark Richard at Oct 30, 2002 at 2:33 am
    Stephen,

    Sounds like you've already done plenty of hunting. Do you know what these
    waits were like previously?

    Perhaps someone can correct me if I'm mistaken but does a "listener not
    responding" error indicate the problem is getting to the server in general,
    not the instance itself. Perhaps something happened to your network
    routing over the weekend? Maybe a cross-network backup was executed over
    the weekend or something like that.

    Good luck finding the gremlin - some of those issues can be very good at
    hiding themselves.

    "Stephen Andert"

    ealth.com> cc:
    Sent by: Subject: SQL*Net Message to client/SQL*Net more data to client
    root_at_fatcity.com

    30/10/2002 11:28
    Please respond to
    ORACLE-L

    We have experienced a sudden and dramatic decrease in performance
    sometime over the weekend (after Sat but before Monday 4 am). In
    following Gaja's tuning philosophy, I've found that the top 2 waits are
    usually (always 2 of the top 3) SQL*Net Message to client/SQL*Net more
    data to client. Everybody swears there have been no changes. SA's say
    no harware or kernel changes. AppDev say no code changes. DBA (me) says
    no database changes.

    WAN folks say no WAN issues and ping is responding at expected speed.
    SA's say LAN card has had no errors during this time frame and is
    processing a good number of bytes but nowhere near it's capacity.
    The application has some very good timing points where there is no
    human element in response time, but there is a big "unknown" category
    that is a larger chunk of time than previously. We suspect that is
    machine wait time of some kind.

    We just bounced the instance because someone wanted to try it and after
    being back up for 20 minutes, early indicators are that performance is
    back to normal. We'll see how long that lasts.

    We have seen a few client sessions getting errors that indicate
    connectivity problems (listener not responding, etc) so we wrote a .com
    file that is repeatedly connecting to the database and will run
    overnight and stop if there are any errors.

    Metalink search for SQL*Net waits gives both "tuning advice" and "you
    can't tune much" notes. I strongly suspect some kind of hardware
    failure, but don't know where since everyone involved says everything is
    working fine.

    Environment Notes:
    Server
    8.1.7.3
    Tru64 5.1A (upgrade to A was done a few weeks ago)
    Compaq GS160 with 16 CPU's and 32 GB RAM (RAM is from memory, so that
    may be off)

    Client
    Open VMS version 7.2
    Client is 8.0.5

    Any ideas on a next step for finding out the cause (solution) to this
    drop in performance???

    Help
    Stephen

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: StephenAndert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    Privileged/Confidential information may be contained in this message.
    If you are not the addressee indicated in this message
    (or responsible for delivery of the message to such person),
    you may not copy or deliver this message to anyone.
    In such case, you should destroy this message and kindly notify the sender
    by reply e-mail or by telephone on (61 3) 9612-6999.
    Please advise immediately if you or your employer does not consent to
    Internet e-mail for messages of this kind.
    Opinions, conclusions and other information in this message
    that do not relate to the official business of
    Transurban City Link Ltd
    shall be understood as neither given nor endorsed by it.
    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Mark Richard
    INET: mrichard_at_transurban.com.au

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Stephen Andert at Oct 30, 2002 at 3:03 am
    Anjo,

    They have reported the results of ping times and said that response
    times are "within spec". They have told me that the network card is
    handling "x bytes per second without errors" and that the card is
    capable of many times that throughput. They have swtiched network cards
    8 times last week and "tested" each one without any errors. The reason
    for this testing/card switching was the errors that they were seeing
    that was causing the box to stop responding.

    I am asking our swat team to look into the network stats from the
    response time perspective instead of utilization.

    Thanks
    Stephen
    anjo_at_oraperf.com 10/29/02 06:23PM >>>
    This is called "Blame Storming". Every component works "fine" but
    response time sucks and the problem is some other area. So how do we
    turn "Blame Storming" into "Brain Storming"?

    Check out the network components. One of the problems is that the
    network people look at utilization, instead of response time. They
    will
    find that utilization of certain components may be low (due to some
    problem), and assume that the problem is some where else. Can they
    tell
    how long a packet is on your network connection?

    Anjo.

    -----Original Message-----
    Andert
    Sent: Wednesday, October 30, 2002 1:28 AM
    To: Multiple recipients of list ORACLE-L

    We have experienced a sudden and dramatic decrease in performance
    sometime over the weekend (after Sat but before Monday 4 am). In
    following Gaja's tuning philosophy, I've found that the top 2 waits
    are
    usually (always 2 of the top 3) SQL*Net Message to client/SQL*Net more
    data to client. Everybody swears there have been no changes. SA's
    say
    no harware or kernel changes. AppDev say no code changes. DBA (me)
    says
    no database changes.

    WAN folks say no WAN issues and ping is responding at expected speed.
    SA's say LAN card has had no errors during this time frame and is
    processing a good number of bytes but nowhere near it's capacity.
    The application has some very good timing points where there is no
    human element in response time, but there is a big "unknown" category
    that is a larger chunk of time than previously. We suspect that is
    machine wait time of some kind.

    We just bounced the instance because someone wanted to try it and
    after
    being back up for 20 minutes, early indicators are that performance is
    back to normal. We'll see how long that lasts.

    We have seen a few client sessions getting errors that indicate
    connectivity problems (listener not responding, etc) so we wrote a
    .com
    file that is repeatedly connecting to the database and will run
    overnight and stop if there are any errors.

    Metalink search for SQL*Net waits gives both "tuning advice" and "you
    can't tune much" notes. I strongly suspect some kind of hardware
    failure, but don't know where since everyone involved says everything
    is
    working fine.

    Environment Notes:
    Server
    8.1.7.3
    Tru64 5.1A (upgrade to A was done a few weeks ago)
    Compaq GS160 with 16 CPU's and 32 GB RAM (RAM is from memory, so that
    may be off)

    Client
    Open VMS version 7.2
    Client is 8.0.5

    Any ideas on a next step for finding out the cause (solution) to this
    drop in performance???

    Help
    Stephen

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: StephenAndert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Anjo Kolk
    INET: anjo_at_oraperf.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: stephenandert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • DENNIS WILLIAMS at Oct 30, 2002 at 4:58 am
    Stephen

    Can they tell you what the ping times were before? Some applications
    are more sensitive to slower networks than others, probably because they
    make more network round trips. Even if the ping times are "within spec", if
    they have increased, that might cause an issue. Just a thought for you.

    Dennis Williams
    DBA, 40%OCP

    Lifetouch, Inc.
    dwilliams_at_lifetouch.com

    -----Original Message-----
    Sent: Tuesday, October 29, 2002 9:04 PM
    To: Multiple recipients of list ORACLE-L

    Anjo,

    They have reported the results of ping times and said that response
    times are "within spec". They have told me that the network card is
    handling "x bytes per second without errors" and that the card is
    capable of many times that throughput. They have swtiched network cards
    8 times last week and "tested" each one without any errors. The reason
    for this testing/card switching was the errors that they were seeing
    that was causing the box to stop responding.

    I am asking our swat team to look into the network stats from the
    response time perspective instead of utilization.

    Thanks
    Stephen
    anjo_at_oraperf.com 10/29/02 06:23PM >>>
    This is called "Blame Storming". Every component works "fine" but
    response time sucks and the problem is some other area. So how do we
    turn "Blame Storming" into "Brain Storming"?

    Check out the network components. One of the problems is that the
    network people look at utilization, instead of response time. They
    will
    find that utilization of certain components may be low (due to some
    problem), and assume that the problem is some where else. Can they
    tell
    how long a packet is on your network connection?

    Anjo.

    -----Original Message-----
    Andert
    Sent: Wednesday, October 30, 2002 1:28 AM
    To: Multiple recipients of list ORACLE-L

    We have experienced a sudden and dramatic decrease in performance
    sometime over the weekend (after Sat but before Monday 4 am). In
    following Gaja's tuning philosophy, I've found that the top 2 waits
    are
    usually (always 2 of the top 3) SQL*Net Message to client/SQL*Net more
    data to client. Everybody swears there have been no changes. SA's
    say
    no harware or kernel changes. AppDev say no code changes. DBA (me)
    says
    no database changes.

    WAN folks say no WAN issues and ping is responding at expected speed.
    SA's say LAN card has had no errors during this time frame and is
    processing a good number of bytes but nowhere near it's capacity.
    The application has some very good timing points where there is no
    human element in response time, but there is a big "unknown" category
    that is a larger chunk of time than previously. We suspect that is
    machine wait time of some kind.

    We just bounced the instance because someone wanted to try it and
    after
    being back up for 20 minutes, early indicators are that performance is
    back to normal. We'll see how long that lasts.

    We have seen a few client sessions getting errors that indicate
    connectivity problems (listener not responding, etc) so we wrote a
    .com
    file that is repeatedly connecting to the database and will run
    overnight and stop if there are any errors.

    Metalink search for SQL*Net waits gives both "tuning advice" and "you
    can't tune much" notes. I strongly suspect some kind of hardware
    failure, but don't know where since everyone involved says everything
    is
    working fine.

    Environment Notes:
    Server
    8.1.7.3
    Tru64 5.1A (upgrade to A was done a few weeks ago)
    Compaq GS160 with 16 CPU's and 32 GB RAM (RAM is from memory, so that
    may be off)

    Client
    Open VMS version 7.2
    Client is 8.0.5

    Any ideas on a next step for finding out the cause (solution) to this
    drop in performance???

    Help
    Stephen

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: StephenAndert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Anjo Kolk
    INET: anjo_at_oraperf.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: stephenandert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: DENNIS WILLIAMS
    INET: DWILLIAMS_at_LIFETOUCH.COM

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Jared Still at Oct 30, 2002 at 2:48 pm
    Pings can be misleading.

    There's 7 layers to the OSI model.

    Ping only reaches the third layer, the Network layer.

    The Transport layer is not tested by a ping.

    Ping will tell you that your network is up, but it tells
    you nothing about the state of the server. It's possible
    ( and happens all the time ) that you can successfully
    ping a hung or crashed server.

    There are open source pings available that will test TCP
    as well as IP, but the standard ping doesn't do this.

    Wonder if your network guy knows that?;)

    Jared
    On Tuesday 29 October 2002 19:03, Stephen Andert wrote:
    Anjo,

    They have reported the results of ping times and said that response
    times are "within spec". They have told me that the network card is
    handling "x bytes per second without errors" and that the card is
    capable of many times that throughput. They have swtiched network cards
    8 times last week and "tested" each one without any errors. The reason
    for this testing/card switching was the errors that they were seeing
    that was causing the box to stop responding.

    I am asking our swat team to look into the network stats from the
    response time perspective instead of utilization.

    Thanks
    Stephen
    anjo_at_oraperf.com 10/29/02 06:23PM >>>
    This is called "Blame Storming". Every component works "fine" but
    response time sucks and the problem is some other area. So how do we
    turn "Blame Storming" into "Brain Storming"?

    Check out the network components. One of the problems is that the
    network people look at utilization, instead of response time. They
    will
    find that utilization of certain components may be low (due to some
    problem), and assume that the problem is some where else. Can they
    tell
    how long a packet is on your network connection?

    Anjo.

    -----Original Message-----
    Andert
    Sent: Wednesday, October 30, 2002 1:28 AM
    To: Multiple recipients of list ORACLE-L

    We have experienced a sudden and dramatic decrease in performance
    sometime over the weekend (after Sat but before Monday 4 am). In
    following Gaja's tuning philosophy, I've found that the top 2 waits
    are
    usually (always 2 of the top 3) SQL*Net Message to client/SQL*Net more
    data to client. Everybody swears there have been no changes. SA's
    say
    no harware or kernel changes. AppDev say no code changes. DBA (me)
    says
    no database changes.

    WAN folks say no WAN issues and ping is responding at expected speed.
    SA's say LAN card has had no errors during this time frame and is
    processing a good number of bytes but nowhere near it's capacity.
    The application has some very good timing points where there is no
    human element in response time, but there is a big "unknown" category
    that is a larger chunk of time than previously. We suspect that is
    machine wait time of some kind.

    We just bounced the instance because someone wanted to try it and
    after
    being back up for 20 minutes, early indicators are that performance is
    back to normal. We'll see how long that lasts.

    We have seen a few client sessions getting errors that indicate
    connectivity problems (listener not responding, etc) so we wrote a
    .com
    file that is repeatedly connecting to the database and will run
    overnight and stop if there are any errors.

    Metalink search for SQL*Net waits gives both "tuning advice" and "you
    can't tune much" notes. I strongly suspect some kind of hardware
    failure, but don't know where since everyone involved says everything
    is
    working fine.

    Environment Notes:
    Server
    8.1.7.3
    Tru64 5.1A (upgrade to A was done a few weeks ago)
    Compaq GS160 with 16 CPU's and 32 GB RAM (RAM is from memory, so that
    may be off)

    Client
    Open VMS version 7.2
    Client is 8.0.5

    Any ideas on a next step for finding out the cause (solution) to this
    drop in performance???

    Help
    Stephen
    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Jared Still
    INET: jkstill_at_cybcon.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Stephen Lee at Oct 30, 2002 at 3:14 pm
    For what it's worth, we had the same thing going on here recently and have
    not resolved it. In our case, it is a visual basic app running what is
    essentially a batch job (don't ask me why a batch job was written as a VB
    app; I just work here). The client is a PC, and the database is on Tru64
    (again ... I just work here). Three different PC's were tried. It appears
    that the tcp_nodelay parameter worked on two of them but not the third
    (which, as you might guess, is the production box and the one on which it
    NEEDS to run faster ... Of course!). All the PC's are on the same subnet,
    going through same routers to get to the same database.

    If you get the problem resolved, I will be most interested in your solution.
    Thus far, we have only been able to attribute it either to sunspots or
    something about the W2K OS on the PC ... both of which, as we all know, are
    responsible for a lot of unexplained behavior.

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Lee
    INET: slee_at_dollar.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Stephen Andert at Oct 30, 2002 at 4:39 pm
    Funny you should mention sunspots since that's what some of our people
    are saying along with the phase of the moon. When we find a solution
    (this is business critical, failure is not an option), I'll post a
    summary.

    Thanks to all for the suggestions.

    Stephen
    slee_at_dollar.com 10/30/02 08:14AM >>>
    For what it's worth, we had the same thing going on here recently and
    have
    not resolved it. In our case, it is a visual basic app running what
    is
    essentially a batch job (don't ask me why a batch job was written as a
    VB
    app; I just work here). The client is a PC, and the database is on
    Tru64
    (again ... I just work here). Three different PC's were tried. It
    appears
    that the tcp_nodelay parameter worked on two of them but not the third
    (which, as you might guess, is the production box and the one on which
    it
    NEEDS to run faster ... Of course!). All the PC's are on the same
    subnet,
    going through same routers to get to the same database.

    If you get the problem resolved, I will be most interested in your
    solution.
    Thus far, we have only been able to attribute it either to sunspots or
    something about the W2K OS on the PC ... both of which, as we all know,
    are
    responsible for a lot of unexplained behavior.

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Lee
    INET: slee_at_dollar.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Andert
    INET: StephenAndert_at_firsthealth.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Cary Millsap at Oct 30, 2002 at 4:59 pm
    This sounds like a possible queueing (i.e., load-induced) issue.
    Queueing issues show up as "application works fine sometimes, but really
    slow other times." The impetus for the really slow behavior is when you
    increase concurrent load on the resource in question (here, the net)
    just enough that response times begin to degrade exponentially instead
    of linearly. It's where you're going "up" faster than you're going "out"
    on that hockey-stick-shaped performance curve. (Sorry I can't show
    pictures here in text; they'll all be in the book due out in about
    June.)

    If this is the problem, it's not your network administrator's fault,
    it's your application's fault. Response problems are caused by either or
    both of only two things: excessive call LATENCY, or excessive call
    COUNT. Many analysts focus exclusively on latency, never understanding
    that it's the call COUNT that's the real problem. Excessive call counts
    within individual application programs, when combined with high user
    concurrency, produce loads that drive response times into the
    exponential degradation behavior.

    Count the number of relevant SQL*Net events in your raw SQL trace data,
    and get an idea of which cursor most of these events are associated
    with. For the SQL associated with that cursor, check its tkprof output
    for unnecessarily high db call counts (or use the Hotsos Profiler and do
    it in one step). For example, if your app parses every time it executes,
    it's putting more load than necessary on the network: take your parse
    calls out of your loops. If it executes once for every row that
    inserted, updated, or deleted, then it's putting more load than
    necessary on the network: use array inserts. If it fetches once for ever
    row that's selected, then it's putting more load than necessary on the
    network: use array fetching.

    You don't have to make your application perfect if this is your problem;
    you only have to reduce unnecessary load enough that your total workload
    remains left of that knee in the performance curve. You can often
    eliminate thousands of db calls (and hence network round-trips) by
    focusing your effort upon just a handful of SQL statements that are
    executed with the highest levels of concurrency.

    Cary Millsap
    Hotsos Enterprises, Ltd.
    http://www.hotsos.com

    Upcoming events:

    - Hotsos Clinic, Dec 9-11 Honolulu
    - 2003 Hotsos Symposium on OracleR System Performance, Feb 9-12 Dallas
    - Jonathan Lewis' Optimising Oracle, Nov 19-21 Dallas

    -----Original Message-----
    Lee
    Sent: Wednesday, October 30, 2002 9:15 AM
    To: Multiple recipients of list ORACLE-L

    For what it's worth, we had the same thing going on here recently and
    have
    not resolved it. In our case, it is a visual basic app running what is
    essentially a batch job (don't ask me why a batch job was written as a
    VB
    app; I just work here). The client is a PC, and the database is on
    Tru64
    (again ... I just work here). Three different PC's were tried. It
    appears
    that the tcp_nodelay parameter worked on two of them but not the third
    (which, as you might guess, is the production box and the one on which
    it
    NEEDS to run faster ... Of course!). All the PC's are on the same
    subnet,
    going through same routers to get to the same database.

    If you get the problem resolved, I will be most interested in your
    solution.
    Thus far, we have only been able to attribute it either to sunspots or
    something about the W2K OS on the PC ... both of which, as we all know,
    are
    responsible for a lot of unexplained behavior.

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Lee
    INET: slee_at_dollar.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Cary Millsap
    INET: cary.millsap_at_hotsos.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Jared.Still_at_radisys.com at Oct 30, 2002 at 6:42 pm
    I've seen that very thing happen to an app.

    It was object oriented to the Nth degree. Joins
    were not generally done in the database, rather each
    method in the app retrieved each individual piece of
    data atomically.

    In an hour I think this app created 10 million network
    packets, average size was 200 bytes.

    Since no one was willing to fix the app, we moved it
    to the database server for a 40% performance gain.

    Jared

    "Cary Millsap"
    Sent by: root_at_fatcity.com
    10/30/2002 08:59 AM
    Please respond to ORACLE-L



    To: Multiple recipients of list ORACLE-L
    cc:
    Subject: RE: SQL*Net Message to client/SQL*Net more data to client

    This sounds like a possible queueing (i.e., load-induced) issue.
    Queueing issues show up as "application works fine sometimes, but really
    slow other times." The impetus for the really slow behavior is when you
    increase concurrent load on the resource in question (here, the net)
    just enough that response times begin to degrade exponentially instead
    of linearly. It's where you're going "up" faster than you're going "out"
    on that hockey-stick-shaped performance curve. (Sorry I can't show
    pictures here in text; they'll all be in the book due out in about
    June.)

    If this is the problem, it's not your network administrator's fault,
    it's your application's fault. Response problems are caused by either or
    both of only two things: excessive call LATENCY, or excessive call
    COUNT. Many analysts focus exclusively on latency, never understanding
    that it's the call COUNT that's the real problem. Excessive call counts
    within individual application programs, when combined with high user
    concurrency, produce loads that drive response times into the
    exponential degradation behavior.

    Count the number of relevant SQL*Net events in your raw SQL trace data,
    and get an idea of which cursor most of these events are associated
    with. For the SQL associated with that cursor, check its tkprof output
    for unnecessarily high db call counts (or use the Hotsos Profiler and do
    it in one step). For example, if your app parses every time it executes,
    it's putting more load than necessary on the network: take your parse
    calls out of your loops. If it executes once for every row that
    inserted, updated, or deleted, then it's putting more load than
    necessary on the network: use array inserts. If it fetches once for ever
    row that's selected, then it's putting more load than necessary on the
    network: use array fetching.

    You don't have to make your application perfect if this is your problem;
    you only have to reduce unnecessary load enough that your total workload
    remains left of that knee in the performance curve. You can often
    eliminate thousands of db calls (and hence network round-trips) by
    focusing your effort upon just a handful of SQL statements that are
    executed with the highest levels of concurrency.

    Cary Millsap
    Hotsos Enterprises, Ltd.
    http://www.hotsos.com

    Upcoming events:

    - Hotsos Clinic, Dec 9-11 Honolulu
    - 2003 Hotsos Symposium on OracleR System Performance, Feb 9-12 Dallas
    - Jonathan Lewis' Optimising Oracle, Nov 19-21 Dallas

    -----Original Message-----
    Lee
    Sent: Wednesday, October 30, 2002 9:15 AM
    To: Multiple recipients of list ORACLE-L

    For what it's worth, we had the same thing going on here recently and
    have
    not resolved it. In our case, it is a visual basic app running what is
    essentially a batch job (don't ask me why a batch job was written as a
    VB
    app; I just work here). The client is a PC, and the database is on
    Tru64
    (again ... I just work here). Three different PC's were tried. It
    appears
    that the tcp_nodelay parameter worked on two of them but not the third
    (which, as you might guess, is the production box and the one on which
    it
    NEEDS to run faster ... Of course!). All the PC's are on the same
    subnet,
    going through same routers to get to the same database.

    If you get the problem resolved, I will be most interested in your
    solution.
    Thus far, we have only been able to attribute it either to sunspots or
    something about the W2K OS on the PC ... both of which, as we all know,
    are
    responsible for a lot of unexplained behavior.
    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Lee
    INET: slee_at_dollar.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Cary Millsap
    INET: cary.millsap_at_hotsos.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author:
    INET: Jared.Still_at_radisys.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------

    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
  • Rachel Carmichael at Oct 30, 2002 at 6:42 pm

    on that hockey-stick-shaped performance curve. (Sorry I can't show
    pictures here in text; they'll all be in the book due out in about
    June.)
    I'd like to place my order now please :)

    Cary Millsap wrote:
    This sounds like a possible queueing (i.e., load-induced) issue.
    Queueing issues show up as "application works fine sometimes, but
    really
    slow other times." The impetus for the really slow behavior is when
    you
    increase concurrent load on the resource in question (here, the net)
    just enough that response times begin to degrade exponentially
    instead
    of linearly. It's where you're going "up" faster than you're going
    "out"
    on that hockey-stick-shaped performance curve. (Sorry I can't show
    pictures here in text; they'll all be in the book due out in about
    June.)

    If this is the problem, it's not your network administrator's fault,
    it's your application's fault. Response problems are caused by either
    or
    both of only two things: excessive call LATENCY, or excessive call
    COUNT. Many analysts focus exclusively on latency, never
    understanding
    that it's the call COUNT that's the real problem. Excessive call
    counts
    within individual application programs, when combined with high user
    concurrency, produce loads that drive response times into the
    exponential degradation behavior.

    Count the number of relevant SQL*Net events in your raw SQL trace
    data,
    and get an idea of which cursor most of these events are associated
    with. For the SQL associated with that cursor, check its tkprof
    output
    for unnecessarily high db call counts (or use the Hotsos Profiler and
    do
    it in one step). For example, if your app parses every time it
    executes,
    it's putting more load than necessary on the network: take your parse
    calls out of your loops. If it executes once for every row that
    inserted, updated, or deleted, then it's putting more load than
    necessary on the network: use array inserts. If it fetches once for
    ever
    row that's selected, then it's putting more load than necessary on
    the
    network: use array fetching.

    You don't have to make your application perfect if this is your
    problem;
    you only have to reduce unnecessary load enough that your total
    workload
    remains left of that knee in the performance curve. You can often
    eliminate thousands of db calls (and hence network round-trips) by
    focusing your effort upon just a handful of SQL statements that are
    executed with the highest levels of concurrency.


    Cary Millsap
    Hotsos Enterprises, Ltd.
    http://www.hotsos.com

    Upcoming events:
    - Hotsos Clinic, Dec 9-11 Honolulu
    - 2003 Hotsos Symposium on OracleR System Performance, Feb 9-12
    Dallas
    - Jonathan Lewis' Optimising Oracle, Nov 19-21 Dallas


    -----Original Message-----
    Lee
    Sent: Wednesday, October 30, 2002 9:15 AM
    To: Multiple recipients of list ORACLE-L


    For what it's worth, we had the same thing going on here recently and
    have
    not resolved it. In our case, it is a visual basic app running what
    is
    essentially a batch job (don't ask me why a batch job was written as
    a
    VB
    app; I just work here). The client is a PC, and the database is on
    Tru64
    (again ... I just work here). Three different PC's were tried. It
    appears
    that the tcp_nodelay parameter worked on two of them but not the
    third
    (which, as you might guess, is the production box and the one on
    which
    it
    NEEDS to run faster ... Of course!). All the PC's are on the same
    subnet,
    going through same routers to get to the same database.

    If you get the problem resolved, I will be most interested in your
    solution.
    Thus far, we have only been able to attribute it either to sunspots
    or
    something about the W2K OS on the PC ... both of which, as we all
    know,
    are
    responsible for a lot of unexplained behavior.
    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Stephen Lee
    INET: slee_at_dollar.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Cary Millsap
    INET: cary.millsap_at_hotsos.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).
    Do you Yahoo!?
    HotJobs - Search new jobs daily now
    http://hotjobs.yahoo.com/

    --
    Please see the official ORACLE-L FAQ: http://www.orafaq.com
    --
    Author: Rachel Carmichael
    INET: wisernet100_at_yahoo.com

    Fat City Network Services -- 858-538-5051 http://www.fatcity.com
    San Diego, California -- Mailing list and web hosting services
    ---------------------------------------------------------------------
    To REMOVE yourself from this mailing list, send an E-Mail message
    to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
    the message BODY, include a line containing: UNSUB ORACLE-L
    (or the name of mailing list you want to be removed from). You may
    also send the HELP command for other information (like subscribing).

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouporacle-l @
categoriesoracle
postedOct 30, '02 at 12:28a
activeOct 30, '02 at 6:42p
posts11
users9
websiteoracle.com

People

Translate

site design / logo © 2023 Grokbase