How do I watch my logs? Where are my logs?

On a BOSH VM, there are two common logs to watch regardless of what BOSH
release is being deployed:

* /var/vcap/bosh/log/current - the agent log - its the log that tells you
your network configuration is screwed and the agent can't communicate with
the other parts of BOSH (director, nats, blobstore)
* /var/vcap/monit/monit/log - the log from monit when its starting/stopping
the server and all the colocated jobs (or singluar job) - its the log that
tells you a job is not starting correctly and monit retries to launch it
over and over

For cf-release, and also any bosh releases I've created, the job's have
their own log location in an easily findable location:

* /var/vcap/sys/log/JOB_NAME/*

Except for cf-release's postgresql job which does have some logs within the
PG database folder /var/vcap/store/postgresql/pg_log/ etc. Pooey to
postgresql.

So, you can go a long way watching logs on a server by running:

$ bosh ssh NAME/INDEX (e.g. bosh ssh cloud_controller/0)
# sudo tail -f -n 200 /var/vcap/sys/log/*/* /var/vcap/bosh/log/current
/var/vcap/monit/monit.log

Perhaps someone could make a bosh plugin to add a streaming log watcher via
SSH. :) You know you want to :)

Nic

--
Dr Nic Williams
http://drnicwilliams.com
cell +1 (415) 860-2185

Search Discussions

  • Mike Heath at Jan 16, 2013 at 11:29 pm
    Excellent tip, thank you. Perhaps you can also answer I monit question I've
    had for a while. If I ssh into a job VM and make a change, how do I tell
    monit to restart the service that I changed? Right now I kill the process
    and wait for monit to bring it back up but I can't help but think there's a
    better way.

    -Mike
    On Wednesday, January 16, 2013 9:12:14 AM UTC-7, Dr Nic Williams wrote:

    How do I watch my logs? Where are my logs?

    On a BOSH VM, there are two common logs to watch regardless of what BOSH
    release is being deployed:

    * /var/vcap/bosh/log/current - the agent log - its the log that tells you
    your network configuration is screwed and the agent can't communicate with
    the other parts of BOSH (director, nats, blobstore)
    * /var/vcap/monit/monit/log - the log from monit when its
    starting/stopping the server and all the colocated jobs (or singluar job) -
    its the log that tells you a job is not starting correctly and monit
    retries to launch it over and over

    For cf-release, and also any bosh releases I've created, the job's have
    their own log location in an easily findable location:

    * /var/vcap/sys/log/JOB_NAME/*

    Except for cf-release's postgresql job which does have some logs within
    the PG database folder /var/vcap/store/postgresql/pg_log/ etc. Pooey to
    postgresql.

    So, you can go a long way watching logs on a server by running:

    $ bosh ssh NAME/INDEX (e.g. bosh ssh cloud_controller/0)
    # sudo tail -f -n 200 /var/vcap/sys/log/*/* /var/vcap/bosh/log/current
    /var/vcap/monit/monit.log

    Perhaps someone could make a bosh plugin to add a streaming log watcher
    via SSH. :) You know you want to :)

    Nic

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 17, 2013 at 12:07 am
    Off the top of my head, the monit binary is at /var/vcap/bosh/bin/monit

    Once you know where it is, you can read any monit
    man/help/tutorials/articles on how to do stuff (e.g.
    http://linux.die.net/man/1/monit)

    Try: /var/vcap/bosh/bin/monit restart all

    Nic

    On Wed, Jan 16, 2013 at 3:29 PM, Mike Heath wrote:

    Excellent tip, thank you. Perhaps you can also answer I monit question
    I've had for a while. If I ssh into a job VM and make a change, how do I
    tell monit to restart the service that I changed? Right now I kill the
    process and wait for monit to bring it back up but I can't help but think
    there's a better way.

    -Mike

    On Wednesday, January 16, 2013 9:12:14 AM UTC-7, Dr Nic Williams wrote:

    How do I watch my logs? Where are my logs?

    On a BOSH VM, there are two common logs to watch regardless of what BOSH
    release is being deployed:

    * /var/vcap/bosh/log/current - the agent log - its the log that tells you
    your network configuration is screwed and the agent can't communicate with
    the other parts of BOSH (director, nats, blobstore)
    * /var/vcap/monit/monit/log - the log from monit when its
    starting/stopping the server and all the colocated jobs (or singluar job) -
    its the log that tells you a job is not starting correctly and monit
    retries to launch it over and over

    For cf-release, and also any bosh releases I've created, the job's have
    their own log location in an easily findable location:

    * /var/vcap/sys/log/JOB_NAME/*

    Except for cf-release's postgresql job which does have some logs within
    the PG database folder /var/vcap/store/postgresql/pg_**log/ etc. Pooey
    to postgresql.

    So, you can go a long way watching logs on a server by running:

    $ bosh ssh NAME/INDEX (e.g. bosh ssh cloud_controller/0)
    # sudo tail -f -n 200 /var/vcap/sys/log/*/* /var/vcap/bosh/log/current
    /var/vcap/monit/monit.log

    Perhaps someone could make a bosh plugin to add a streaming log watcher
    via SSH. :) You know you want to :)

    Nic

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Dr Nic Williams at Jan 17, 2013 at 12:08 am
    That's as root user, so:

    sudo /var/vcap/bosh/bin/monit restart all

    On Wed, Jan 16, 2013 at 4:06 PM, Dr Nic Williams wrote:

    Off the top of my head, the monit binary is at /var/vcap/bosh/bin/monit

    Once you know where it is, you can read any monit
    man/help/tutorials/articles on how to do stuff (e.g.
    http://linux.die.net/man/1/monit)

    Try: /var/vcap/bosh/bin/monit restart all

    Nic


    On Wed, Jan 16, 2013 at 3:29 PM, Mike Heath wrote:

    Excellent tip, thank you. Perhaps you can also answer I monit question
    I've had for a while. If I ssh into a job VM and make a change, how do I
    tell monit to restart the service that I changed? Right now I kill the
    process and wait for monit to bring it back up but I can't help but think
    there's a better way.

    -Mike

    On Wednesday, January 16, 2013 9:12:14 AM UTC-7, Dr Nic Williams wrote:

    How do I watch my logs? Where are my logs?

    On a BOSH VM, there are two common logs to watch regardless of what BOSH
    release is being deployed:

    * /var/vcap/bosh/log/current - the agent log - its the log that tells
    you your network configuration is screwed and the agent can't communicate
    with the other parts of BOSH (director, nats, blobstore)
    * /var/vcap/monit/monit/log - the log from monit when its
    starting/stopping the server and all the colocated jobs (or singluar job) -
    its the log that tells you a job is not starting correctly and monit
    retries to launch it over and over

    For cf-release, and also any bosh releases I've created, the job's have
    their own log location in an easily findable location:

    * /var/vcap/sys/log/JOB_NAME/*

    Except for cf-release's postgresql job which does have some logs within
    the PG database folder /var/vcap/store/postgresql/pg_**log/ etc. Pooey
    to postgresql.

    So, you can go a long way watching logs on a server by running:

    $ bosh ssh NAME/INDEX (e.g. bosh ssh cloud_controller/0)
    # sudo tail -f -n 200 /var/vcap/sys/log/*/* /var/vcap/bosh/log/current
    /var/vcap/monit/monit.log

    Perhaps someone could make a bosh plugin to add a streaming log watcher
    via SSH. :) You know you want to :)

    Nic

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185


    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185
  • Tony Hansmann at Jan 17, 2013 at 12:14 am
    Hi Mike,

    You can do do a 'monit status' to list the names of the managed services
    and then do something like 'monit start dea'.

    Monit is opensource, full docs at:
    http://mmonit.com/monit/documentation/monit.html

    Regarding logs, you can also pull a log bundle via bosh:
    bosh logs dea 1

    bosh help logs
    logs <job> <index> [--agent] [--job] [--only filter1,filter2,...]
    [--all]
    Fetch job or agent logs from a BOSH-managed VM
    --agent fetch agent logs
    --job fetch job logs
    --only filter1,filter2,... only fetch logs that satisfy given filters
    (defined in job spec)
    --all fetch all files in the job or agent log
    directory

    And lastly, you may want more detail when you are doing a 'bosh deploy',
    while a deploy is going, you can open another term and do 'bosh task last
    --debug | less' to get the nitty-gritty detail (including CPI). You can
    safely ctrl-c out without affecting the deploy. You can also get --debug
    detail on completed deploys.

    -Tony
    On Wed, Jan 16, 2013 at 3:29 PM, Mike Heath wrote:

    Excellent tip, thank you. Perhaps you can also answer I monit question
    I've had for a while. If I ssh into a job VM and make a change, how do I
    tell monit to restart the service that I changed? Right now I kill the
    process and wait for monit to bring it back up but I can't help but think
    there's a better way.

    -Mike

    On Wednesday, January 16, 2013 9:12:14 AM UTC-7, Dr Nic Williams wrote:

    How do I watch my logs? Where are my logs?

    On a BOSH VM, there are two common logs to watch regardless of what BOSH
    release is being deployed:

    * /var/vcap/bosh/log/current - the agent log - its the log that tells you
    your network configuration is screwed and the agent can't communicate with
    the other parts of BOSH (director, nats, blobstore)
    * /var/vcap/monit/monit/log - the log from monit when its
    starting/stopping the server and all the colocated jobs (or singluar job) -
    its the log that tells you a job is not starting correctly and monit
    retries to launch it over and over

    For cf-release, and also any bosh releases I've created, the job's have
    their own log location in an easily findable location:

    * /var/vcap/sys/log/JOB_NAME/*

    Except for cf-release's postgresql job which does have some logs within
    the PG database folder /var/vcap/store/postgresql/pg_**log/ etc. Pooey
    to postgresql.

    So, you can go a long way watching logs on a server by running:

    $ bosh ssh NAME/INDEX (e.g. bosh ssh cloud_controller/0)
    # sudo tail -f -n 200 /var/vcap/sys/log/*/* /var/vcap/bosh/log/current
    /var/vcap/monit/monit.log

    Perhaps someone could make a bosh plugin to add a streaming log watcher
    via SSH. :) You know you want to :)

    Nic

    --
    Dr Nic Williams
    http://drnicwilliams.com
    cell +1 (415) 860-2185

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupbosh-users @
postedJan 16, '13 at 4:12p
activeJan 17, '13 at 12:14a
posts5
users3

People

Translate

site design / logo © 2022 Grokbase