FAQ

On Thursday, May 30, 2013 1:51:58 AM UTC+1, Christian Hedegaard wrote:

We use a load balancer because our infrastructure is inside AWS/EC2 and
it makes a lot of things easier and gives us certain visibility that we
would normally lack.



But, I did go ahead and open up our ACL’s and security groups to allow our
log server to talk ES directly with the ES cluster, and I brought up some
new nodes. I’d like to use EC2 hinting, which is documented in the
elasticsearch.yml that graylog comes with, but it requires an EC2 plugin.
Is there any way to install this plugin for the embedded ES node?
You can start graylog2-server with aws-java-sdk-1.3.26.jar and
elasticsearch-cloud-aws-1.10.0.jar on the Java classpath, and then set the
following in /etc/graylog2-elasticsearch.yml:

plugin.mandatory: cloud-aws
cloud.aws.region: 'us-east-1'
cloud.node.auto_attributes: true
discovery:
     type: ec2
# Comma-separated list of security groups
discovery.ec2.groups: 'ElasticSearch'

This is what we have done and it is working for discovery of our cluster in
EC2. You should adjust the region and security groups to match your own
context, of course.

Otherwise I have to give it a hostname to hint on, which could go away at
some point. This is another reason we like to use load balancers.



*From:* gray...@googlegroups.com <javascript:> [mailto:
gray...@googlegroups.com <javascript:>] *On Behalf Of *Kay Röpke
*Sent:* Saturday, May 25, 2013 4:47 PM
*To:* gray...@googlegroups.com <javascript:>
*Subject:* Re: [graylog2] RE: Can't find ES cluster.



Hi!

Don't use a load balancer address for the ES node connection.
You only need the address for discovery, afterwards the graylog2 server
knows all the nodes in the cluster through the ES gossip protocol.
Load balancing only really works for stateless protocols, so just bite the
bullet and give graylog2 a couple of your ES node addresses. Don't worry
about nodes shutting down once its running, the cluster state will be
properly propagated. Graylog2 only needs one address for discovery to work.

Not sure why you want to use a load balancer here, I can't imagine you
have so many ES nodes joining and leaving the cluster all the time.
For actual workload the load balancer will buy you nothing, that's handled
by elasticsearch's protocol.

Best,
-k

On May 25, 2013 8:50 PM, "Christian Hedegaard" wrote:

Does no one have any information about this?



*From:* gray...@googlegroups.com <javascript:> [mailto:
gray...@googlegroups.com <javascript:>] *On Behalf Of *Christian Hedegaard
*Sent:* Thursday, May 23, 2013 11:45 AM
*To:* gray...@googlegroups.com <javascript:>
*Subject:* [graylog2] RE: Can't find ES cluster.



After a little digging, I found a method using the elasticsearch.yml to
put in a host for unicast discover. So I put in the host and port of our ES
load balancer which sits in front of the ES cluster. Here are the new
errors:



2013-05-23 18:44:16,546 WARN :
org.elasticsearch.discovery.zen.ping.unicast - [graylog2-server] failed to
send ping to [[#zen_unicast_1#][inet[
internal-elasticsearch-log-20-vpc01-1114059026.eu-west-1.elb.amazonaws.com/10.220.9.175:9200]<http://internal-elasticsearch-log-20-vpc01-1114059026.eu-west-1.elb.amazonaws.com/10.220.9.175:9200%5D>
]]

org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[
internal-elasticsearch-log-20-vpc01-1114059026.eu-west-1.elb.amazonaws.com/10.220.9.175:9200]][discovery/zen/unicast<http://internal-elasticsearch-log-20-vpc01-1114059026.eu-west-1.elb.amazonaws.com/10.220.9.175:9200%5D%5D%5Bdiscovery/zen/unicast>]
request_id [9] timed out after [3750ms]

at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:679)





*From:* gray...@googlegroups.com <javascript:> [
mailto:...@googlegroups.com <javascript:>] *On Behalf Of *Christian
Hedegaard
*Sent:* Thursday, May 23, 2013 11:26 AM
*To:* gray...@googlegroups.com <javascript:>
*Subject:* [graylog2] Can't find ES cluster.





I’ve installed a new machine with graylog2 0.12.0-rc.1 but am getting this
error in the log when starting the server:



2013-05-23 18:10:24,465 WARN : org.elasticsearch.discovery -
[graylog2-server] waited for 30s and no initial state was set by the
discovery

Exception in thread "main"
org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

at
org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(TransportMasterNodeOperationAction.java:169)

at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:371)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:679)



We have our ES cluster behind a load balancer and use chef to provision
all of our machines. Normally we do a load-balancer search and just give
that to the graylog configs. However, it looks like in the new versions
with the embedded ES, that it relies on the ES search.



Is there an option to just point the graylog2.conf at an ES host directly,
so that I can give it the load balancer name?



We use ES in other ways in our infrastructure and being able to follow our
own convention is pretty important to us. Plus, this setup works fine with
.9.6rc-1

--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to graylog2+u...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to graylog2+u...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to graylog2+u...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.



--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to graylog2+u...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 6 of 7 | next ›
Discussion Overview
groupgraylog2 @
categoriesrubyonrails, mongodb
postedMay 23, '13 at 6:25p
activeJul 30, '13 at 8:28a
posts7
users4
websitegraylog2.org
irc#graylog2

People

Translate

site design / logo © 2022 Grokbase