One of my clients has been having a recurring problem of the
filesystems on their brokers randomly becoming readonly. Their admin
investigated the errors and determined it was related to this issue
(http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalIdQ306)
with their running Oracle Enterprise Linux on top of VMWare.

I'm a little confounded and how there could even be too high I/O
interaction for the filesystem to keep up. At most, the brokers only
see 400 messages per second and usually average around 180 messages
per second. This is way below the rates I see on a cluster I run on
Rackspace for another client and I've never seen this happen before.

It's pretty annoying because the filesystem goes readonly randomly in
middle of the night and causes rabbitMQ to shutdown (since having /var
read only screws a lot of things up) and whoever is on call gets paged
(usually at 2am). I've suggested moving the brokers to physical boxes
or using Ubuntu instead of Oracle Enterprise Linux, but I feel like
I'm just making the later suggestion out of spite (I don't like Oracle
Enterprise Linux). :)

Any ideas?


Thanks,
James

Search Discussions

  • Marek Majkowski at Nov 1, 2011 at 12:40 pm

    On Thu, Oct 27, 2011 at 12:25, James Carr wrote:
    One of my clients has been having a recurring problem of the
    filesystems on their brokers randomly becoming readonly. Their admin
    investigated the errors and determined it was related to this issue
    (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalIdQ306)
    with their running Oracle Enterprise Linux on top of VMWare.

    I'm a little confounded and how there could even be too high I/O
    interaction for the filesystem to keep up. At most, the brokers only
    see 400 messages per second and usually average around 180 messages
    per second. This is way below the rates I see on a cluster I run on
    Rackspace for another client and I've never seen this happen before.

    It's pretty annoying because the filesystem goes readonly randomly in
    middle of the night and causes rabbitMQ to shutdown (since having /var
    read only screws a lot of things up) and whoever is on call gets paged
    (usually at 2am). I've suggested moving the brokers to physical boxes
    or using Ubuntu instead of Oracle Enterprise Linux, but I feel like
    I'm just making the later suggestion out of spite (I don't like Oracle
    Enterprise Linux). :)

    Any ideas?
    I have no clue how to solve the problem. But I have some
    ideas about possible workarounds:
    a) maybe use ext2? (basically - a fs without journaling)
    b) take a look at `errors=continue` option in `man mount`

    Cheers,
    Marek

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouprabbitmq-discuss @
categoriesrabbitmq
postedOct 27, '11 at 11:25a
activeNov 1, '11 at 12:40p
posts2
users2
websiterabbitmq.com
irc#rabbitmq

2 users in discussion

James Carr: 1 post Marek Majkowski: 1 post

People

Translate

site design / logo © 2021 Grokbase