FAQ
Hi Roshni,
I was just about to summarize. I got it running.
The problem eventually was the firewall. I didn't have the full control
over the server so I couldn't check the iptables rules.
The incoming traffic on 7000 port was rejected on the second node. I've
just checked one way communication from node2 -> node1 using telnet.
It was very helpful to run cassandra in TRACE mode. It displays a lot of
logs but was able to compare the behavior with a working ring.
I've noticed that the second node was sending a Gossip SYN message but was
never getting back the SYN_ACK.

Anyway to everybody that will have a problem like that, check your firewall
settings:)

Thanks for help Roshni.

--
regards,
Jakub Glapa

On Fri, Aug 3, 2012 at 9:41 AM, Roshni Rajagopal wrote:

Hi Jakub,

Were you able to resolve the issue?
For a multi data center setup I do believe some steps are different. You
may need to set Networktopology as your replication strategy rather than
simple strategy, and setup a snitch. And mention the rack/dc configurations
in a config file.
You can refer to the steps for a multi data center installation.

Regards,
Roshni



From: Jakub Glapa <jakub.glapa@gmail.com Reply-To: "user@cassandra.apache.org " <
user@cassandra.apache.org To: "user@cassandra.apache.org " <
user@cassandra.apache.org Subject: Re: Unsuccessful attempt to add a second node to a ring.

I found a similar thread from March :
http://www.mail-archive.com/user@cassandra.apache.org/msg21007.html

For me clearing the data and starting from the beginning didn't help.

It's interesting because on my dev environment I was able to add another
node without any problems.

The only difference is that the second node now is in a different data
center. (but I'm not using any different settings, SimpleSnitch)
7000,9160,7199 ports were open between those 2 nodes.

How else can I check if the communication between those 2 nodes is working?
In the logs I see that:
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642
OutboundTcpConnection.java (line 206) attempting to connect to
NODE1/node1.ip

So I assume that the communication is somehow established?


--
regards,
Jakub Glapa


On Wed, Aug 1, 2012 at 11:36 AM, Jakub Glapa wrote:
yes it's the same



--
regards,
pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 11:24 AM, Roshni Rajagopal <
roshni.rajagopal@wal-mart.com wrote:
Ok, sorry it may not be required,
I was thinking of a configuration I had done on my local laptop, where I
had aliased my IP address.
In that case the directories and jmx port needed to be different.

Cluster name is same right?


From: Jakub Glapa <jakub.glapa@gmail.com ><mailto:jakub.glapa@gmail.com Reply-To: "user@cassandra.apache.org ><mailto:user@cassandra.apache.org " <
user@cassandra.apache.org user@cassandra.apache.org To: "user@cassandra.apache.org user@cassandra.apache.org " <
user@cassandra.apache.org user@cassandra.apache.org Subject: Re: Unsuccessful attempt to add a second node to a ring.

Hi Roshni,
no they are the same, my changes in cassandra.yaml were only in the
listen_address, rpc_address, seeds and initial_token field.
The rest is exactly the same as on node1.

That's how the file looks on node2:



cluster_name: 'Test Cluster'
initial_token: 85070591730234615865843651857942052864
hinted_handoff_enabled: true
hinted_handoff_throttle_delay_in_ms: 1
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partitioner: org.apache.cassandra.dht.RandomPartitioner
data_file_directories:
- /data/servers/cassandra_sbe_edtool/cassandra_data/data
commitlog_directory:
/data/servers/cassandra_sbe_edtool/cassandra_data/commitlog
saved_caches_directory:
/data/servers/cassandra_sbe_edtool/cassandra_data/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "NODE1"
flush_largest_memtables_at: 0.75
reduce_cache_sizes_at: 0.85
reduce_cache_capacity_to: 0.6
concurrent_reads: 32
concurrent_writes: 32
memtable_flush_queue_size: 4
sliced_buffer_size_in_kb: 64
storage_port: 7000
ssl_storage_port: 7001
listen_address: NODE2
rpc_address: NODE2
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
incremental_backups: false
snapshot_before_compaction: false
column_index_size_in_kb: 64
in_memory_compaction_limit_in_mb: 64
multithreaded_compaction: false
compaction_throughput_mb_per_sec: 16
compaction_preheat_key_cache: true
rpc_timeout_in_ms: 10000
endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
index_interval: 128
encryption_options:
internode_encryption: none
keystore: conf/.keystore
keystore_password: cassandra
truststore: conf/.truststore
truststore_password: cassandra




--
regards,
pozdrawiam,
Jakub Glapa


On Wed, Aug 1, 2012 at 10:29 AM, Roshni Rajagopal <
roshni.rajagopal@wal-mart.com > Roshni.Rajagopal@wal-mart.com>>> wrote:
Jakub,

Have you set the
Data, commitlog, saved cache directories to different ones in each yaml
file for each node?

Regards,
Roshni


From: Jakub Glapa <jakub.glapa@gmail.com ><mailto:jakub.glapa@gmail.com jakub.glapa@gmail.com jakub.glapa@gmail.com Reply-To: "user@cassandra.apache.org ><mailto:user@cassandra.apache.org >><mailto:user@cassandra.apache.org ><mailto:user@cassandra.apache.org " <
user@cassandra.apache.org user@cassandra.apache.org user@cassandra.apache.org user@cassandra.apache.org To: "user@cassandra.apache.org user@cassandra.apache.org user@cassandra.apache.org user@cassandra.apache.org " <
user@cassandra.apache.org user@cassandra.apache.org user@cassandra.apache.org user@cassandra.apache.org Subject: Unsuccessful attempt to add a second node to a ring.

Hi Everybody!

I'm trying to add a second node to an already operating one node cluster.

Some specs:
- cassandra 1.0.7
- both nodes have a routable listen_address and rpc_address.
- Ports are open: (from node2) telnet node1 7000 is successful
- Seeds parameter on node2 points to node 1.

[node1] nodetool -h localhost ring
Address DC Rack Status State Load
Owns Token
node1.ip datacenter1 rack1 Up Normal 74.33 KB
100.00% 0

- initial token on node2 was specified

I see something like that in the logs on node2:

DEBUG [main] 2012-07-31 13:50:38,640 CollationController.java (line 76)
collectTimeOrderedData
INFO [main] 2012-07-31 13:50:38,641 StorageService.java (line 667)
JOINING: waiting for ring and schema information
DEBUG [WRITE-NODE1/node1.ip] 2012-07-31 13:50:39,642
OutboundTcpConnection.java (line 206) attempting to connect to
NODE1/node1.ip
DEBUG [ScheduledTasks:1] 2012-07-31 13:50:40,639 LoadBroadcaster.java
(line 86) Disseminating load info ...
INFO [main] 2012-07-31 13:51:08,641 StorageService.java (line 667)
JOINING: schema complete, ready to bootstrap
DEBUG [main] 2012-07-31 13:51:08,642 StorageService.java (line 554) ...
got ring + schema info
INFO [main] 2012-07-31 13:51:08,642 StorageService.java (line 667)
JOINING: getting bootstrap token
DEBUG [main] 2012-07-31 13:51:08,644 BootStrapper.java (line 138) token
manually specified as 85070591730234615865843651857942052864
DEBUG [main] 2012-07-31 13:51:08,645 Table.java (line 387) applying
mutation of row 4c


but it doesn't join the ring:

[node2] nodetool -h localhost ring
Address DC Rack Status State Load
Owns Token
node2.ip datacenter1 rack1 Up Normal 13.49 KB 100.00%
85070591730234615865843651857942052864



I'm attaching the full log from node2 startup in debug mode.



PS.
When I didn't specified the initial token on node2 I ended up with
exception like that:
"Exception encountered during startup: No other nodes seen! Unable to
bootstrap.If you intended to start a single-node cluster, you should make
sure your broadcast_address (or listen_address) is listed as a seed.
Otherwise, you need to determine why the seed being contacted has no
knowledge of the rest of the cluster. Usually, this can be solved by
giving all nodes the same seed list."


I'm not sure how to proceed now. I found a couple of posts with problems
like that but they weren't very useful.

--
regards,
Jakub Glapa

This email and any files transmitted with it are confidential and intended
solely for the individual or entity to whom they are addressed. If you have
received this email in error destroy it immediately. *** Walmart
Confidential ***

This email and any files transmitted with it are confidential and intended
solely for the individual or entity to whom they are addressed. If you have
received this email in error destroy it immediately. *** Walmart
Confidential ***


This email and any files transmitted with it are confidential and intended
solely for the individual or entity to whom they are addressed. If you have
received this email in error destroy it immediately. *** Walmart
Confidential ***

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 8 of 8 | next ›
Discussion Overview
groupuser @
categoriescassandra
postedAug 1, '12 at 8:47a
activeAug 3, '12 at 1:38p
posts8
users2
websitecassandra.apache.org
irc#cassandra

2 users in discussion

Jakub Glapa: 5 posts Roshni Rajagopal: 3 posts

People

Translate

site design / logo © 2021 Grokbase