FAQ
Greetings,

This is a HP-UX 11.23 machine with 30 or so 10g & 9i databases on it.

The problem was one volume's space maxing out. Found 4 10.2.0.3 db's
with 2Gb+ MMON trace files with date stamps over 1 month old. Deleted
these files and cleaned house on several others db's. Checking disk
space shows the MMON trace file space was not released. I will look
into note 422954.1 to try to avoid a recurrence, but in the mean time I
would rather not have to shutdown a dozen db's to find the 4 that had
the MMON problem.

Is there any method for finding which databases have these deleted trace
files still open? Is there a way to close out a MMON trace without
shutting down the db?

Thanks,
Mike Hand

Search Discussions

  • Tim Gorman at Nov 14, 2008 at 8:26 pm
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

    Michael,
    The problem is open file descriptors.  On UNIX/Linux, if you delete a
    file with an open file descriptor or "handle" from a running process,
    then the file is deleted, but its contents are not deallocated from the
    file-system until the process(es) holding open file descriptors finally
    close them.  From your question, I can tell you know all this, but I'm
    just restating it....
    At any rate, I've found that performing a "truncating write" to such
    files can do the trick of not disturbing the open file descriptor(s)
    and not deleting the file, but causes space to be deallocated.
    Try something like:
    echo "This file truncated by DBA on `date`" > {file-name}

    where "{file-name}" is the trace file (or alert.log
    file) in question.  The MMON process will still write to the file (and
    maybe you'll muck up any writes it was doing to the file at the time
    you truncated it), but since it has its file descriptor open in "append
    write" mode, it'll start over again from where your write left off.
    Hope this helps...
    Tim Gorman
    consultant - Evergreen Database Technologies, Inc.
    P.O. Box 630791, Highlands Ranch CO 80163-0791
    website = http://www.EvDBT.com/
    email = Tim@EvDBT.com
    mobile = +1-303-885-4526
    fax = +1-303-484-3608
    Yahoo IM = tim_evdbt

    Hand, Michael T wrote:

    Greetings,

    This is a HP-UX 11.23 machine with 30 or so 10g & 9i databases on it.

    The problem was one volume's space maxing out. Found 4 10.2.0.3 db's
    with 2Gb+ MMON trace files with date stamps over 1 month old. Deleted
    these files and cleaned house on several others db's. Checking disk
    space shows the MMON trace file space was not released. I will look
    into note 422954.1 to try to avoid a recurrence, but in the mean time I
    would rather not have to shutdown a dozen db's to find the 4 that had
    the MMON problem.

    Is there any method for finding which databases have these deleted trace
    files still open? Is there a way to close out a MMON trace without
    shutting down the db?

    Thanks,
    Mike Hand
  • Hand, Michael T at Nov 14, 2008 at 8:57 pm
    Tim, Martin,

    Unfortunately the suggestions require knowing the filenames that were
    deleted. I would have to guess what the PID was as I didn't take note
    when I rm'ed them. I think I'm up the creek, but fortunately, this is a
    development system. Will do some more digging.

    Thanks,
    Mike
    -----Original Message-----
    first try
    MB> fuser filename
    >
    TG> echo "This file truncated by DBA on `date`" > {file-name}
  • Tim Gorman at Nov 14, 2008 at 9:04 pm
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

    Ouch!� Didn't see the part where you said that you had already removed
    the files.
    Since its development, how about finding out if the entire instance
    will shutdown if the MMON process is killed?� Or if, like the job-queue
    processes and px-worker processes, they'll just restart themselves?� I
    have the feeling that any process named "xMON" will kill the instance
    if they are killed, but what the heck?� As my teenagers say, "JK!�
    LOL!"...

    Hand, Michael T wrote:

    Tim, Martin,

    Unfortunately the suggestions require knowing the filenames that were
    deleted. I would have to guess what the PID was as I didn't take note
    when I rm'ed them. I think I'm up the creek, but fortunately, this is a
    development system. Will do some more digging.

    Thanks,
    Mike

    -----Original Message-----
    first try

    MB> fuser filename

    TG> echo "This file truncated by DBA on `date`" > {file-name}
    --
    http://www.freelists.org/webpage/oracle-l

    --
    http://www.freelists.org/webpage/oracle-l
  • Andreas Piesk at Nov 14, 2008 at 11:33 pm

    Hand, Michael T schrieb:

    Is there any method for finding which databases have these deleted trace
    files still open? Is there a way to close out a MMON trace without
    shutting down the db?
    do you have 'lsof' on your system? this tool shows all open files, even
    the deleted ones.

    regards,
    -ap
  • Yong Huang at Nov 15, 2008 at 6:19 pm

    Since its development, how about finding out if the entire
    instance will shutdown if the MMON process is killed?
    Killing MMON doesn't crash instance. Before we migrated most our applications from the unstable Itanium boxes running 10.2.0.2, we periodically killed MMON to release memory. See
    http://www.freelists.org/archives/oracle-l/10-2008/msg00145.html

    Yong Huang
  • Tim Gorman at Nov 15, 2008 at 6:44 pm
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

    Thanks, Yong!� Does MMON get restarted automatically after it is
    killed?� Or do we have to restart the instance in order for that to
    happen?

    Yong Huang wrote:

    Since its development, how about finding out if the entire
    instance will shutdown if the MMON process is killed?

    Killing MMON doesn't crash instance. Before we migrated most our applications from the unstable Itanium boxes running 10.2.0.2, we periodically killed MMON to release memory. See
    http://www.freelists.org/archives/oracle-l/10-2008/msg00145.html

    Yong Huang
  • Yong Huang at Nov 15, 2008 at 8:43 pm
    I forgot to say. It will be restarted. Here's my screen (instance name changed):

    $ kill -9 20576
    $ tail -f /home/oracle/oracle/product/10.2.0/db/admin/sidname/bdump/alert_sidname.log
    ...
    Sat Nov 15 14:26:58 2008
    Restarting dead background process MMON
    MMON started with pid=16, OS id=1776
    $ ps -fp 1776

    UID PID PPID C STIME TTY TIME CMD
    oracle 1776 1 0 14:26 ? 00:00:00 ora_mmon_sidname

    pmon probably restarted it. At the time mmon was killed, pmon trace has a line about state object cleanup:
    KSS cleanup: kssdch_stage: possible loop ...

    Yong Huang
    On Sat, 11/15/08, Tim Gorman wrote:

    From: Tim Gorman
    Subject: Re: Huge MMON traces
    To: yong321_at_yahoo.com
    Cc: oracle-l_at_freelists.org
    Date: Saturday, November 15, 2008, 12:44 PM

    Thanks, Yong! Does MMON get restarted automatically after
    it is killed? Or do we have to restart the instance in order for
    that to happen?


    Yong Huang wrote:
    Since its development, how about finding out if the
    entire instance will shutdown if the MMON process is killed?
    Killing MMON doesn't crash instance. Before we migrated
    most our applications from the unstable Itanium boxes
    running 10.2.0.2, we periodically killed MMON to release
    memory. See
    http://www.freelists.org/archives/oracle-l/10-2008/msg00145.html

    Yong Huang


    --
    http://www.freelists.org/webpage/oracle-l
  • Hand, Michael T at Nov 17, 2008 at 4:21 pm
    Thanks all


    Yong's method was successful as show below, Had to go through 10
    databases to find the 3 in question. Volume went from 99% full to 56%.
    I'll know better next time.


    # bdf /ora01
    Filesystem kbytes used avail %used Mounted on
    /dev/vg03/ora01 24576000 21709740 2687817 89% /ora01
    # ps -ef | grep -i mmon_XXXXX
    oracle 14975 1 0 Jul 29 ? 21:41 ora_mmon_XXXXX
    oracle 1179 29733 0 10:48:04 pts/ta 0:00 grep -i mmon_XXXXX
    # kill -9 14975
    # tail -3 alert_XXXXX.log
    Mon Nov 17 10:48:33 2008
    Restarting dead background process MMON
    MMON started with pid=11, OS id=1184
    # bdf /ora01
    Filesystem kbytes used avail %used Mounted on
    /dev/vg03/ora01 24576000 16164345 7886625 67% /ora01

    Glad to learn something new.


    Regards,
    Mike Hand

    From: oracle-l-bounce_at_freelists.org
    On Behalf Of Tim Gorman

    Sent: Saturday, November 15, 2008 1:45 PM
    To: yong321_at_yahoo.com
    Cc: oracle-l_at_freelists.org
    Subject: Re: Huge MMON traces

    Thanks, Yong! Does MMON get restarted automatically after it is

    killed? Or do we have to restart the instance in order for that to
    happen?





    Yong Huang wrote:

    Since its development, how about finding out if
    the entire

    instance will shutdown if the MMON process is
    killed?




    Killing MMON doesn't crash instance. Before we migrated
    most our applications from the unstable Itanium boxes running 10.2.0.2,
    we periodically killed MMON to release memory. See


    http://www.freelists.org/archives/oracle-l/10-2008/msg00145.html


    Yong Huang

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouporacle-l @
categoriesoracle
postedNov 14, '08 at 4:07p
activeNov 17, '08 at 4:21p
posts9
users4
websiteoracle.com

People

Translate

site design / logo © 2022 Grokbase