FAQ
Hi,

Recently I saw some strange behavior on one of the nodes of a 3-node
cluster. A while ago I created a table and put some data (about 150M) in it
for testing. A few days ago I started to import full data into that table
using normal cql INSERT statements. As soon as inserting started, one node
started non-stop full GC. The other two nodes were totally fine. I stopped
the inserting process, restarted C* on all the nodes. All nodes are fine.
But once I started inserting again, full GC kicked in on that node within a
minute.The insertion speed is moderate. Again, the other two nodes were
fine. I tried this process a couple of times. Every time the same node
jumped into full GC. I even rebooted all the boxes. I checked system.log
but found no errors or warnings before full GC started.

Finally I deleted and recreated the table. All of sudden the problem went
away. The only thing I can think of is that table was created using STCS.
After I inserted 150M data into it, I switched it to LCS. Then I ran
incremental repair a couple of times. I saw validation and normal
compaction on that table as expected. When I recreated the table, I created
it with LCS.

I don't have the problem any more but just want to share the experience.
Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
7 and Java 8. All boxes have the identical configurations.

Thanks.

Search Discussions

  • DuyHai Doan at Jan 16, 2016 at 11:28 pm
    "As soon as inserting started, one node started non-stop full GC. The other
    two nodes were totally fine"

    Just a guest, how did you insert data ? Did you use Batch statements ?
    On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang wrote:

    Hi,

    Recently I saw some strange behavior on one of the nodes of a 3-node
    cluster. A while ago I created a table and put some data (about 150M) in it
    for testing. A few days ago I started to import full data into that table
    using normal cql INSERT statements. As soon as inserting started, one node
    started non-stop full GC. The other two nodes were totally fine. I stopped
    the inserting process, restarted C* on all the nodes. All nodes are fine.
    But once I started inserting again, full GC kicked in on that node within a
    minute.The insertion speed is moderate. Again, the other two nodes were
    fine. I tried this process a couple of times. Every time the same node
    jumped into full GC. I even rebooted all the boxes. I checked system.log
    but found no errors or warnings before full GC started.

    Finally I deleted and recreated the table. All of sudden the problem went
    away. The only thing I can think of is that table was created using STCS.
    After I inserted 150M data into it, I switched it to LCS. Then I ran
    incremental repair a couple of times. I saw validation and normal
    compaction on that table as expected. When I recreated the table, I created
    it with LCS.

    I don't have the problem any more but just want to share the experience.
    Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
    7 and Java 8. All boxes have the identical configurations.

    Thanks.
  • Kai Wang at Jan 17, 2016 at 10:18 pm
    DuyHai,

    In this case I didn't use batch, just bind a single PreparedStatement and
    execute. Nor did I see any warning/error about batch being too large in the
    log.

    Thanks.
    On Sat, Jan 16, 2016 at 6:27 PM, DuyHai Doan wrote:

    "As soon as inserting started, one node started non-stop full GC. The
    other two nodes were totally fine"

    Just a guest, how did you insert data ? Did you use Batch statements ?
    On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang wrote:

    Hi,

    Recently I saw some strange behavior on one of the nodes of a 3-node
    cluster. A while ago I created a table and put some data (about 150M) in it
    for testing. A few days ago I started to import full data into that table
    using normal cql INSERT statements. As soon as inserting started, one node
    started non-stop full GC. The other two nodes were totally fine. I stopped
    the inserting process, restarted C* on all the nodes. All nodes are fine.
    But once I started inserting again, full GC kicked in on that node within a
    minute.The insertion speed is moderate. Again, the other two nodes were
    fine. I tried this process a couple of times. Every time the same node
    jumped into full GC. I even rebooted all the boxes. I checked system.log
    but found no errors or warnings before full GC started.

    Finally I deleted and recreated the table. All of sudden the problem went
    away. The only thing I can think of is that table was created using STCS.
    After I inserted 150M data into it, I switched it to LCS. Then I ran
    incremental repair a couple of times. I saw validation and normal
    compaction on that table as expected. When I recreated the table, I created
    it with LCS.

    I don't have the problem any more but just want to share the experience.
    Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS
    7 and Java 8. All boxes have the identical configurations.

    Thanks.
  • Xutom at Jan 18, 2016 at 1:26 am
    Hi Kai Wang,
         I also encounter such issue a few days ago. I have 6 nodes, and I found 2 nodes do endless full gc when I export ALL datas from C* using "Select * from table". I remove all datas of the 2 nodes and install Cassandra again, and the problem gone away.



    At 2016-01-18 06:18:46, "Kai Wang" wrote:

    DuyHai,


    In this case I didn't use batch, just bind a single PreparedStatement and execute. Nor did I see any warning/error about batch being too large in the log.


    Thanks.



    On Sat, Jan 16, 2016 at 6:27 PM, DuyHai Doan wrote:

    "As soon as inserting started, one node started non-stop full GC. The other two nodes were totally fine"


    Just a guest, how did you insert data ? Did you use Batch statements ?


    On Sat, Jan 16, 2016 at 10:12 PM, Kai Wang wrote:

    Hi,


    Recently I saw some strange behavior on one of the nodes of a 3-node cluster. A while ago I created a table and put some data (about 150M) in it for testing. A few days ago I started to import full data into that table using normal cql INSERT statements. As soon as inserting started, one node started non-stop full GC. The other two nodes were totally fine. I stopped the inserting process, restarted C* on all the nodes. All nodes are fine. But once I started inserting again, full GC kicked in on that node within a minute.The insertion speed is moderate. Again, the other two nodes were fine. I tried this process a couple of times. Every time the same node jumped into full GC. I even rebooted all the boxes. I checked system.log but found no errors or warnings before full GC started.


    Finally I deleted and recreated the table. All of sudden the problem went away. The only thing I can think of is that table was created using STCS. After I inserted 150M data into it, I switched it to LCS. Then I ran incremental repair a couple of times. I saw validation and normal compaction on that table as expected. When I recreated the table, I created it with LCS.


    I don't have the problem any more but just want to share the experience. Maybe someone has an theory on this? BTW I am running C* 2.2.4 with CentOS 7 and Java 8. All boxes have the identical configurations.



    Thanks.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriescassandra
postedJan 16, '16 at 9:12p
activeJan 18, '16 at 1:26a
posts4
users3
websitecassandra.apache.org
irc#cassandra

3 users in discussion

Kai Wang: 2 posts DuyHai Doan: 1 post Xutom: 1 post

People

Translate

site design / logo © 2022 Grokbase