FAQ
Up to now we use simple logging to files.


We don't use syslog or an other server based solution.


I am unsure which architecture works for our environment.


Our environment:


  - django based applications
  - a lot of batch/cron jobs (non web gui) processing
  - One linux server runs several systems. Each system has its own virtualenv.
  - servers are running in the intranet of our customers.
  - We have access via VPN, but it is slow. And sometimes down for some minutes.
  - 40MByte logs per day per server.


Why we are unhappy with logging to files:


  - filtering: We don't want to get INFO messages over the VPN.
    Background: Sometimes a there is too much logging and we don't want
    to pull gigabytes over the VPN.


  - Rotating: Rotating files is possible, but somehow cumbersome.


  - Support structured logging of values (json) in the future.


I have not locked at Sentry or Logstash in detail.


Which solution could fit for our environment?


Please ask if you have question!


Regards,
   Thomas G?ttler

Search Discussions

  • Dieter at Sep 10, 2015 at 6:41 am

    Thomas G?ttler <hv@tbz-pariv.de> writes:
    ...
    Why we are unhappy with logging to files:

    - filtering: We don't want to get INFO messages over the VPN.

    You can quite easily control at what level messages are logged with
    the standard Python logging framework. Each handler has a level
    and will ignore messages at a lower level.

    - Rotating: Rotating files is possible, but somehow cumbersome.

    There are standard tools to rotate logfiles.

    - Support structured logging of values (json) in the future.

    Again, the Python logging framework is quite flexible with
    respect to the format of logged messages.

    ...
    Which solution could fit for our environment?

    I work for a customer with a similar environment (he uses "Zope" instead
    of "Django") - and he uses logfiles. The logfiles are automatically
    rotated and there are in the order of half a dozen to a dozen logfiles
    per day.


    When I have to analyse a problem with the help of the logfiles,
    I do not copy them via VPN but do the filtering remotely and only
    copy the filtered portion, if necessary.
  • Thomas Güttler at Sep 11, 2015 at 7:22 am

    Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
    Thomas G?ttler writes:
    ...
    Why we are unhappy with logging to files:

    - filtering: We don't want to get INFO messages over the VPN.
    You can quite easily control at what level messages are logged with
    the standard Python logging framework. Each handler has a level
    and will ignore messages at a lower level.



    I want INFO to be logged and stored on the remote host.
    Therefore I must not filter INFO messages.


    I don't want to pull INFO messages over the VPN.


    Ergo, the filtering at Python level does not help in my use case.
    Or I am missing something.


    And now I have an ugly soup.


    The ugly soup is a text file with not well defined syntax. It looks line
    based. But sometimes there are log messages which span multiple lines ....


    Remember: "Life is too short to (re)write parsers"


    Yes, there are tools to parse that soup. But why create this soup in
    the first place?


    That's why I want to move from file based logging to a different solution.


    Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(





    - Rotating: Rotating files is possible, but somehow cumbersome.
    There are standard tools to rotate logfiles.
    - Support structured logging of values (json) in the future.
    Again, the Python logging framework is quite flexible with
    respect to the format of logged messages.
    ...
    Which solution could fit for our environment?
    I work for a customer with a similar environment (he uses "Zope" instead
    of "Django") - and he uses logfiles. The logfiles are automatically
    rotated and there are in the order of half a dozen to a dozen logfiles
    per day.

    When I have to analyse a problem with the help of the logfiles,
    I do not copy them via VPN but do the filtering remotely and only
    copy the filtered portion, if necessary.

    Good to know that I am not the only one running servers in remote intranets.


    Regards,
       Thomas G?ttler
  • Marco Nawijn at Sep 11, 2015 at 8:17 am

    On Friday, September 11, 2015 at 9:22:42 AM UTC+2, Thomas G?ttler wrote:
    Am Donnerstag, 10. September 2015 08:42:47 UTC+2 schrieb dieter:
    Thomas G?ttler writes:
    ...
    Why we are unhappy with logging to files:

    - filtering: We don't want to get INFO messages over the VPN.
    You can quite easily control at what level messages are logged with
    the standard Python logging framework. Each handler has a level
    and will ignore messages at a lower level.

    I want INFO to be logged and stored on the remote host.
    Therefore I must not filter INFO messages.

    I don't want to pull INFO messages over the VPN.

    Ergo, the filtering at Python level does not help in my use case.
    Or I am missing something.

    And now I have an ugly soup.

    The ugly soup is a text file with not well defined syntax. It looks line
    based. But sometimes there are log messages which span multiple lines ....

    Remember: "Life is too short to (re)write parsers"

    Yes, there are tools to parse that soup. But why create this soup in
    the first place?

    That's why I want to move from file based logging to a different solution.

    Unfortunately there too many choices (graylog, logstash, sentry, ...) :-(


    - Rotating: Rotating files is possible, but somehow cumbersome.
    There are standard tools to rotate logfiles.
    - Support structured logging of values (json) in the future.
    Again, the Python logging framework is quite flexible with
    respect to the format of logged messages.
    ...
    Which solution could fit for our environment?
    I work for a customer with a similar environment (he uses "Zope" instead
    of "Django") - and he uses logfiles. The logfiles are automatically
    rotated and there are in the order of half a dozen to a dozen logfiles
    per day.

    When I have to analyse a problem with the help of the logfiles,
    I do not copy them via VPN but do the filtering remotely and only
    copy the filtered portion, if necessary.
    Good to know that I am not the only one running servers in remote intranets.

    Regards,
    Thomas G?ttler

    So, if logging to json is on the horizon anyway, why don't you create
    something like a MongoDb handler in the standard Python logging framework
    and run a MongoDb server in the client intranet? You could then connect
    over VPN to MongoDb, filter the warning/error messages there on the server
    side and fetch them to your local systems for analysis.


    Marco
  • Jmp at Sep 11, 2015 at 9:02 am

    On 09/11/2015 09:22 AM, Thomas G?ttler wrote:
    I want INFO to be logged and stored on the remote host.
    Therefore I must not filter INFO messages.

    I don't want to pull INFO messages over the VPN.

    Ergo, the filtering at Python level does not help in my use case.
    Or I am missing something.

    Probably,


    Your logger should have


        * a remote host handler
        * and a VPN handler.


    You can set filters and log levels separately for each handler.
    More info here
    https://docs.python.org/2/library/logging.html#handler-objects
    https://docs.python.org/2/howto/logging-cookbook.html#logging-to-multiple-destinations


    Something like (python 2.7)


    import logging


    logCfg = {
          'remote':(
              logging.StreamHandler(),
              logging.Formatter('Remote - %(levelname)s - %(message)s'),
              logging.INFO,
              ),
          'vpn':(
              logging.StreamHandler(),
              logging.Formatter('VPN - %(levelname)s - %(message)s'),
              logging.ERROR,
              ),
    }


    log = logging.getLogger()
    log.setLevel(logging.DEBUG)


    for handler, formatter, level in logCfg.itervalues():
          handler.setFormatter(formatter)
          handler.setLevel(level)
          log.addHandler(handler)


    log.info('This is an info')
    log.error('This is error')




    and the result:


    Remote - INFO - This is an info
    VPN - ERROR - This is error
    Remote - ERROR - This is error




    JM

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedSep 9, '15 at 8:33a
activeSep 11, '15 at 9:02a
posts5
users4
websitepython.org

People

Translate

site design / logo © 2019 Grokbase