Grokbase Groups Flume user

Search Discussions

376 discussions - 1,707 posts

  • Hi folks, I just posted to the Apache blog on how to do performance tuning with Flume. I plan on following it up with a post about using the Flume monitoring capabilities while tuning. Feedback is ...
    Mike PercyMike Percy
    Jan 11, 2013 at 8:14 pm
    Jan 11, 2013 at 10:27 pm
  • Hi, How can I write custom serializer to write event body and header to HDFS now I am getting only log messages which are written on HDFS. Timestamp and other information is not coming. Regards, ...
    Chhaya VishwakarmaChhaya Vishwakarma
    Jan 11, 2013 at 9:24 am
    Jan 11, 2013 at 7:37 pm
  • I am trying to write custom HTTP Source handler for creating events based on XML input. I know I can look at the JSONHandler that is provided. Are there any other resources\documentation that I can ...
    Vikram KulkarniVikram Kulkarni
    Jan 11, 2013 at 9:21 am
    Jan 11, 2013 at 9:21 am
  • Hi, Do you have any preference or recommendation regarding the java version and flavor you recommend for flume 1.3? Oracle/Sun or Open JRE. Java 6 or Java 7? Thanks, ~Rahul.
    Rahul RavindranRahul Ravindran
    Jan 10, 2013 at 10:15 pm
    Jan 11, 2013 at 7:22 am
  • hi all, I didn't find the relationship between AvroSink and other types of RpcClient, including LoadBalancingRpcClient. In my opinion, user can set the specified RpcClient type from AvroSink with ...
    Denny YeDenny Ye
    Jan 10, 2013 at 6:56 am
    Jan 10, 2013 at 7:34 pm
  • hi, I am using Flume log4j appender to write log events to HDFS but it contains some junk value and I am not able to see anything other than log message no timestamp. Here is my configuration ...
    Chhaya VishwakarmaChhaya Vishwakarma
    Jan 9, 2013 at 10:03 am
    Jan 9, 2013 at 5:08 pm
  • Hi folks, I am using flume to sink into hbase. I had the set up working yesterday, but when I restarted my flume agent, it for some reason cannot dump to hbase anymore (too bad I don't have a ...
    Xu (Simon) ChenXu (Simon) Chen
    Jan 9, 2013 at 4:20 am
    Jan 9, 2013 at 2:37 pm
  • Can some one explain the importance of the following 1) Batch Size 1.a) When configured by client code using the flume-core-sdk , to send events to flume avro source. 1.b) When set as a parameter on ...
    Bhaskar V. KarambelkarBhaskar V. Karambelkar
    Jan 8, 2013 at 6:46 pm
    Jan 11, 2013 at 5:04 pm
  • Hi all, Which hbase sink keeps the event header, and how to configure it to do so? The SimpleHbaseEventSerializer would certainly discard the header ...
    Xu (Simon) ChenXu (Simon) Chen
    Jan 7, 2013 at 4:34 pm
    Jan 7, 2013 at 4:43 pm
  • Hi, I encountered with a problem while executing "mvn clean install -DskipTests": [INFO] [site:site {execution: flume-site}] [INFO] Parent project loaded from repository: org.apache:apache:pom:9 ...
    Jan 5, 2013 at 10:55 am
    Jan 9, 2013 at 7:09 am
  • Hello, I write a customize source of flume to read data from memory cache (Redis). And it results the CPU usage of redis is too high. So I want to decrease the trigger frequency of the source. So ...
    Jan 5, 2013 at 2:48 am
    Jan 5, 2013 at 2:48 am
  • Hi All, I submitted a patch for exec source, please take a look.
    Azuryy YuAzuryy Yu
    Jan 5, 2013 at 1:07 am
    Jan 7, 2013 at 6:07 am
  • Are eventserializers thread safe ? I assume so, coz the OutputStream is stored as a class level variable. but just wanted to double check. thanks
    Bhaskar V. KarambelkarBhaskar V. Karambelkar
    Jan 4, 2013 at 6:03 pm
    Jan 4, 2013 at 6:03 pm
  • Hi, I am using flume (old gen) as an extension to an existant syslog system and would like to use the timestamp of the syslog message as the timestamp of the flume event. I guess the timestamp is ...
    Bertrand DechouxBertrand Dechoux
    Jan 4, 2013 at 2:22 pm
    Jan 4, 2013 at 4:37 pm
  • Hi all, Is there any guidance that demonstrates a way to start an agent programmatically? I want to embed an agent within a client program. Thanks!
    Jan 4, 2013 at 5:32 am
    Jan 5, 2013 at 5:39 am
  • I see , thanks very much! 2013/1/1 Hari Shreedharan <<span class="m_body_email_addr" title="7f32581cd4791e891bb15a7ed8bce929"</span
    Jan 4, 2013 at 5:19 am
    Jan 4, 2013 at 5:19 am
  • Myself and a colleague will be doing a presentation/usecase on flume at "Hadoop Conference Japan 2013 winter", trying to get it more awareness here. Is it reasonable to reuse the logo within the ...
    Juhani ConnollyJuhani Connolly
    Jan 4, 2013 at 4:28 am
    Jan 4, 2013 at 7:57 am
  • Hi, Is there any additional management/monitoring abilities or anything else for flume which is available via Cloudera Manager? Thanks, ~Rahul.
    Rahul RavindranRahul Ravindran
    Jan 4, 2013 at 12:04 am
    Jan 4, 2013 at 4:05 pm
  • The Apache Flume team is pleased to announce the release of Flume version 1.3.1. Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large ...
    Hari ShreedharanHari Shreedharan
    Jan 3, 2013 at 12:03 am
    Jan 4, 2013 at 10:53 pm
  • Dear all, In my project, I use flume 1.3 to collect log. But when I want to use HBaseSync to store data into HBASE. I got the following error message. 12/12/28 17:45:55 INFO zookeeper.ClientCnxn ...
    Dec 28, 2012 at 9:48 am
    Jan 4, 2013 at 10:31 am
  • Hi, It was possible to have source and sinks on different machines using Flume 0.9x. Is it possible to have source and sinks on different machines in Flume-NG (1.3.0)? I want to use tail source on ...
    Abhijeet PathakAbhijeet Pathak
    Dec 28, 2012 at 6:37 am
    Dec 28, 2012 at 8:10 am
  • Hi, I want to use flume in windows 2003 64bit system. Could I to run flume agent? thanks -- Venorp Zhang Sent with Sparrow (
    Venorp ChangVenorp Chang
    Dec 28, 2012 at 5:46 am
    Dec 31, 2012 at 3:47 am
  • hi, Flume load config file every 30seconds, so it stop all compoents, then start all compoents. if I use exec source with tail -F, does there lost data or duplicate data becauseof restart all ...
    Azuryy YuAzuryy Yu
    Dec 27, 2012 at 11:56 pm
    Dec 28, 2012 at 1:05 am
  • This is kind of a generic HDFS question, but it does relate to flume, so hopefully someone can provide feedback. I have a flume configuration that sinks to HDFS using timestamp headers. I would like ...
    Paul ChavezPaul Chavez
    Dec 27, 2012 at 6:02 pm
    Dec 27, 2012 at 9:18 pm
  • Hi there are some typos on the in the section: Configuring a multi agent flow the first tier agent: agent_foo, but the second agent also named agent_foo? I ...
    Azuryy YuAzuryy Yu
    Dec 27, 2012 at 3:16 am
    Dec 27, 2012 at 3:16 am
  • [Moving the conversation to <span class="m_body_email_addr" title="66a62f9bf370cfd01785f877dba2a17d"</span as this is not a CDH specific question] Flume allows writing events to ...
    Arvind PrabhakarArvind Prabhakar
    Dec 26, 2012 at 9:30 pm
    Dec 26, 2012 at 9:30 pm
  • HI, I am working on the POC of Cassandra flume integration. For that am using Cassandra sink plugin from *Github (flume-ng Cassandra sink plugin).* *And * *Flume-NG version-1.2.0* *Apache Cassandra ...
    Priyanka jainPriyanka jain
    Dec 26, 2012 at 5:39 pm
    Dec 27, 2012 at 6:54 am
  • I have an application running on multiple EC2 instances. I need to aggregate the logs generated in those instances to a central location. i.e., the logs generated on the EC2 instances automatically ...
    Dec 26, 2012 at 1:08 pm
    Dec 26, 2012 at 3:28 pm
  • Hi I was thinking of 2 possible approaches for this : Approach 1. Deduplication at destination- Using spooling dir source -file channel - hdfs sink combination ...
    Jagadish BihaniJagadish Bihani
    Dec 24, 2012 at 7:49 am
    Dec 24, 2012 at 7:49 am
  • just had a thought... before I turn this script up and make a mess of things I figured I'd ask the group... I'm running FLUME 1.3 running using FILE_ROLL at the sink.... the 'live in use' files are ...
    Cochran, DavidCochran, David
    Dec 21, 2012 at 8:28 pm
    Dec 21, 2012 at 8:47 pm
  • Hi, Is Flume 1.3 part of CDH4? Is Flume 1.3 part of any debian repo for installation? I have the link for which gives me the tar file. However, this does not ...
    Rahul RavindranRahul Ravindran
    Dec 19, 2012 at 5:35 pm
    Dec 21, 2012 at 4:17 am
  • Hi, I noticed that when my flume master is down for a quite while. My flume agent is retrying to connect to the master, i can tell that from the flume logs as below; 2012-12-19 11:11:42,126 WARN ...
    Shengjie MinShengjie Min
    Dec 19, 2012 at 11:35 am
    Dec 19, 2012 at 12:21 pm
  • Hi, I've working setup of flume-ng agent (1 source [tail], 1 channel[jdbc], 1 sink [AsyncHbase]) on one server. However, I am not able to find documentation on how to configure failover sinks for ...
    Abhijeet PathakAbhijeet Pathak
    Dec 19, 2012 at 8:24 am
    Dec 20, 2012 at 7:49 am
  • Hi, I'm have a flume-ng source listening on port 45454, and I attempt to use the following python script to send an event to the source, but receive the netty exception below in the flume logs. Has ...
    John MichaelsJohn Michaels
    Dec 18, 2012 at 9:13 pm
    Dec 18, 2012 at 9:26 pm
  • Hi, We’ve run into a strange problem regarding NFS and File Channel performance while evaluating the new version of Flume. We had no issues with the previous version (1.2.0). Our configuration looks ...
    Rakos, RudolfRakos, Rudolf
    Dec 18, 2012 at 4:08 pm
    Dec 19, 2012 at 4:09 pm
  • On the wiki,, the basic instructions say that I can use either the maven-eclipse ...
    DeCarlo, ThomDeCarlo, Thom
    Dec 18, 2012 at 1:55 pm
    Dec 18, 2012 at 4:13 pm
  • Please help me on the mail below! I am stuck Thanks and Regards, Shouvanik Haldar | Cloud SME Pool | Mobile:+91-9830017568 From: Haldar, Shouvanik Sent: Tuesday, December 18, 2012 5:59 PM To: '<span ...
    Shouvanik HaldarShouvanik Haldar
    Dec 18, 2012 at 1:25 pm
    Dec 19, 2012 at 7:00 am
  • I am always getting this annoying error Unable to map logical node 'test_agnt_nd1' to physical node 'ip-10-40-222-77.ec2.internal' I have the following configuration in a script file. Please help! ...
    Shouvanik HaldarShouvanik Haldar
    Dec 18, 2012 at 1:13 pm
    Dec 18, 2012 at 2:42 pm
  • Hello,every one: Unfortunately, I encountered trouble,but I think it may be easy.I use the hbase as a sink,but it cannot write data to hbase,I see the source,and find that the payloadColumn must be ...
    Dec 18, 2012 at 10:29 am
    Dec 18, 2012 at 12:29 pm
  • Hi, I started flume. It says, logging at "logging to /var/log/flume/flume-flume-node-ip-10-83-35-173.log". But when I fire the command "tail -f logging to ...
    Shouvanik HaldarShouvanik Haldar
    Dec 17, 2012 at 10:15 am
    Dec 17, 2012 at 12:54 pm
  • To add to the below information, My motive is to start a flow where a single node acts as a flume agent and sends data to a collector(which is the same node). Is it possible? If yes, how do I set it ...
    Shouvanik HaldarShouvanik Haldar
    Dec 17, 2012 at 7:54 am
    Dec 17, 2012 at 3:19 pm
  • [cid:<span class="m_body_email_addr" title="c3951cf6d0e086df830669795d75263d" image001.png@01cddb1c.cb7126d0</span ] Please help! This message is for the designated recipient only and may contain ...
    Shouvanik HaldarShouvanik Haldar
    Dec 15, 2012 at 6:05 pm
    Dec 17, 2012 at 7:46 am
  • Hello All, I'm also a new user to Flume and was hoping someone could point me in the right direction or tell me what silly little piece I'm missing from the puzzle. I apologize if this has been ...
    Cormier, ChristopherCormier, Christopher
    Dec 14, 2012 at 8:48 pm
    Dec 14, 2012 at 9:09 pm
  • Flummers, Loved working with Flume 1.2 - very easy and simple configuration, it was a pleasure to work with. Managed to "tail -F" logs from unix server and into a hdfs cluster. The problem started ...
    Kartashov, AndyKartashov, Andy
    Dec 14, 2012 at 2:48 pm
    Dec 20, 2012 at 5:44 pm
  • I haven't been able to find any recent comments about a logback appender for Flume. There is a log4j appender, as noted in [FLUME-27], but the similar request that specifically mentions logback ...
    DeCarlo, ThomDeCarlo, Thom
    Dec 14, 2012 at 2:11 pm
    Dec 14, 2012 at 2:43 pm
  • 2012-12-14 04:56:09,857 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: open attempt 6 failed, backoff (60000ms): Failed to open thrift event sink to ...
    Shouvanik HaldarShouvanik Haldar
    Dec 14, 2012 at 10:01 am
    Dec 14, 2012 at 12:14 pm
  • My aim is to collect data as and when it appears inside a log file. For that I am using the below configuration exec config node1 default-Flow 'tail("/var/log/flume/test1.log",startFromEnd="true")' ...
    Shouvanik HaldarShouvanik Haldar
    Dec 14, 2012 at 3:45 am
    Dec 14, 2012 at 3:45 am
  • Flummers, I successfully ran flume-ng 1.2 on two boxes running Unix and would now like to try to connect to one box running Windows. The installation instructions on Cloudera site are too ambiguous ...
    Kartashov, AndyKartashov, Andy
    Dec 13, 2012 at 4:53 pm
    Dec 13, 2012 at 10:12 pm
  • Say I have multi-hop flow, and lets say the last one stores its data in HDFS using the HDFS sink. In the last agent, as in every agent, there are the source-channel-sink trio, my question is: why do ...
    Guy PelegGuy Peleg
    Dec 13, 2012 at 7:35 am
    Dec 13, 2012 at 8:26 am
  • Hi, the reliable delivery of the events" But if I have a multi-hope flow, and the first agent crashes before the message received from the client was stored on the channel then the message is lost ...
    Guy PelegGuy Peleg
    Dec 13, 2012 at 5:42 am
    Dec 13, 2012 at 6:21 am
Group Navigation
period‹ prev | Latest | first ›
Group Overview
groupuser @

Top users

Brock Noland: 212 posts Hari Shreedharan: 108 posts Alexander Alten-Lorenz: 93 posts Mike Percy: 56 posts Juhani Connolly: 47 posts Mohit Anchlia: 46 posts Mohammad Tariq: 45 posts mardan Khan: 41 posts Jagadish Bihani: 39 posts Prabhu k: 35 posts Rahul Ravindran: 31 posts Nitin Pawar: 27 posts Shouvanik Haldar: 27 posts Denny Ye: 23 posts Chris Neal: 23 posts Dan Young: 22 posts Raymond Ng: 21 posts Patrick Wendell: 21 posts Bhaskar V. Karambelkar: 20 posts Will McQueen: 19 posts
show more