FAQ
Hey,

Quite new to ActiveMQ. We noticed an issue with replicated LevelDB where the
corrupted leveldb was being replicated between machines.

I had hoped that I would be able to use the configuration parameters for
verifyChecksums or paranoidChecks to help identify this scenario. Am I
missing something here?

To test this we did the following in our cluster of 3 brokers.

Cleared the level db directory on 2 brokers leaving them clean.
Left the corrupt DB (corrupt in that it's causing the exception below when
trying to expire messages) on 1 broker

I set both config parameters above to true but I see the broker failing over
with an IO Exception as below

INFO | jvm 1 | 2013/12/06 05:12:20 | INFO | Stopping
BrokerService[localhost] due to exception, java.io.IOException
INFO | jvm 1 | 2013/12/06 05:12:20 | java.io.IOException
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:543)
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:974)
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1270)
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1194)
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:708)
INFO | jvm 1 | 2013/12/06 05:12:20 | at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:741)

This happens on both servers as the corrupted db has been replicated over.

Having a quick look it doesn't seem like ReadOptions are used for this
function so no verifyChecksums flag is passed in right?

Was hoping that on initialization of the masterLevelDBClient we would be
able to validate the datastore at that point and if it was invalid fall
over?

Is that the expected behavior of those parameters or is it completely
distinct from the replication process as they are used in the non-replicated
leveldb adapter as well?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Any-way-to-protect-from-corruption-being-replicated-in-LevelDB-tp4675294.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Search Discussions

  • Hiram Chirino at Feb 20, 2014 at 1:57 am

    On Mon, Dec 9, 2013 at 11:14 AM, pwalker wrote:
    Having a quick look it doesn't seem like ReadOptions are used for this
    function so no verifyChecksums flag is passed in right?
    Yep.. perhaps we should.
    Was hoping that on initialization of the masterLevelDBClient we would be
    able to validate the datastore at that point and if it was invalid fall
    over?
    It might be impractical to check for all corruption at startup since
    that might significantly delay the startup process.
    Is that the expected behavior of those parameters or is it completely
    distinct from the replication process as they are used in the non-replicated
    leveldb adapter as well?
    The replication bits added to leveldb replicate files at the block
    level and don't really check the integrity of the files it's
    transferring.

    Perhaps if we do detect consistency problem with a master we should
    just stop replication and mark it's data files as being inconsistent
    so that it does not try to become a master in the future. That still
    would require that one of the slaves data files be consistent to be
    able to recover from the failure.



    --
    Hiram Chirino
    Engineering | Red Hat, Inc.
    hchirino@redhat.com | fusesource.com | redhat.com
    skype: hiramchirino | twitter: @hiramchirino

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriesactivemq
postedDec 9, '13 at 4:15p
activeFeb 20, '14 at 1:57a
posts2
users2
websiteactivemq.apache.org

2 users in discussion

Hiram Chirino: 1 post Pwalker: 1 post

People

Translate

site design / logo © 2022 Grokbase