FAQ

Search Discussions

78 discussions - 266 posts

  • Hi all, I have an Exec Source running a tail -F on a log4J-generated log file that gets rolled once a day. It seems that when log4J rolls the file to the new date, the hdfs sink ends up with a .tmp ...
    Chris NealChris Neal
    Aug 29, 2012 at 2:18 pm
    Nov 6, 2012 at 8:05 pm
  • Hi All, Whatever I have tuned parameters of hdfs sink, It can't get higher performance over than 20MB per minutes. Is that normal? I think it is weird. How can I improve it Regards Ruihong Shi ...
    Shara ShiShara Shi
    Aug 27, 2012 at 9:28 am
    Aug 28, 2012 at 11:48 am
  • Hi I have downloaded the tarball of latest flume-ng 1.2.0. I have JAVA_HOME properly set. To begin with I have followed the instructions in " https://cwiki.apache.org/FLUME/getting-started.html" as ...
    Jagadish BihaniJagadish Bihani
    Aug 8, 2012 at 7:45 am
    Aug 14, 2012 at 1:11 pm
  • Hey guys, I'm sure this is not a new question, but I haven't found an answer in my searches. I'm curious why there is as of yet no Tail Source with NG? It seems one of the most common use cases for ...
    Chris NealChris Neal
    Aug 29, 2012 at 4:18 pm
    Sep 5, 2012 at 4:47 pm
  • I have 3 HDFS sinks all writing to the same namenode, but different paths. e.g. sink1 = hdfs://namenode/path1 sink2 = hdfs://namenode/path2 etc. When flume is shutdown (kill <flume-pid ), the file ...
    Bhaskar V. KarambelkarBhaskar V. Karambelkar
    Aug 23, 2012 at 10:24 pm
    Aug 29, 2012 at 9:30 am
  • I was wondering if I pass random data to an event's header, can the HDFSSink write it to HDFS? I know it can use the headers to split the data into different paths, but what about writing the data to ...
    David CapwellDavid Capwell
    Aug 21, 2012 at 2:36 am
    Aug 24, 2012 at 2:39 am
  • Hi all. The User Guide talks about the various types of Sink Processors, but doesn't say whether they can be aggregated together. A Failover Processor that moves between 1..n sinks is great, as is a ...
    Chris NealChris Neal
    Aug 17, 2012 at 4:00 pm
    Sep 6, 2012 at 2:35 pm
  • Hi Users, I have followed the below link * http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html * for sample text file to HDFS sink using tail source. and i have executed ...
    Prabhu kPrabhu k
    Aug 22, 2012 at 9:44 am
    Aug 23, 2012 at 9:37 am
  • Hi all. I have a very typical configuration: Application logs to log4J file. FlumeNG avro-client watches the file and sends events into FlumeNG Agent Tier 1 FlumeNG Agent Tier 1: AvroSource to ...
    Chris NealChris Neal
    Aug 13, 2012 at 5:44 pm
    Aug 15, 2012 at 2:13 pm
  • The sink group document doesn't mention anything about how to hook up sink groups to the rest of the config in order to work. e.g. under normal circumstances one channel is linked with one sink. But ...
    Bhaskar V. KarambelkarBhaskar V. Karambelkar
    Aug 23, 2012 at 4:29 pm
    Aug 23, 2012 at 10:55 pm
  • Hi, I'm using flume version Flume 0.9.4-cdh3u4 I'm using flume master webpage to configure the following to move data into hdfs. host : syslogTcp(5140) | agentSink("hadoop1.liaisondevqa.local",35862) ...
    Sandeep Reddy PSandeep Reddy P
    Aug 13, 2012 at 3:13 pm
    Aug 17, 2012 at 3:42 pm
  • Hello everybody: I was using the flume,but I set that a file is a event. But the event was always fail to send. Because I want to know the flume's size . Thank you very much! My Name Yanzhi Liu
    Yanzhi.liuYanzhi.liu
    Aug 28, 2012 at 2:54 am
    Aug 28, 2012 at 2:49 pm
  • I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do I need to write some custom interceptor? Could you please point me to an example? Currently my AvroClient looks like this ...
    Mohit AnchliaMohit Anchlia
    Aug 22, 2012 at 12:53 am
    Aug 23, 2012 at 6:34 pm
  • Hi, I'm using flume-ng to write data from log file to HDFS but unable to write. I got the following exceptions in Name node logs 2012-08-16 13:57:42,560 INFO ...
    Sandeep Reddy PSandeep Reddy P
    Aug 16, 2012 at 6:21 pm
    Aug 16, 2012 at 8:56 pm
  • Hi, I'm trying to use the snappy codec but keep getting "native snappy library not available" errors. I'm using CDH4 but replaced the flume 1.1 JARs that are included with that distribution with ...
    Eran KutnerEran Kutner
    Aug 3, 2012 at 2:34 pm
    Aug 3, 2012 at 6:55 pm
  • Folks, I was follow this link Hadoop at Twitter (part 1): Splittable LZO Compression to integration LZO in Hadoop2.0, but seems Flume-ng lzo compress not work. My flume-ng configuratioin file is: cat ...
    Kevin LeeKevin Lee
    Aug 28, 2012 at 6:57 am
    Aug 29, 2012 at 1:14 am
  • Hi Folks, I'm completely new to flume, but was hoping to use it as a way to collect logs on a couple web servers and funnel them to an s3 sink in Avro. I've found a tutorial for doing this on Flume ...
    igor Finkelshteynigor Finkelshteyn
    Aug 27, 2012 at 10:39 pm
    Aug 28, 2012 at 9:42 pm
  • Hi All, My query is related to the flume’s sink processor for the failover. I implemented the failover data flow to transfer the event to downstream in case of agent failure. I have configured the ...
    Ashutosh(오픈플랫폼개발팀)Ashutosh(오픈플랫폼개발팀)
    Aug 14, 2012 at 9:08 am
    Aug 16, 2012 at 9:44 am
  • I am prototyping flume ng for real time sink to hbase and hdfs for clickstream. I was wondering if Flume provides something that can read from HBase too. That is API request comes to our web server ...
    Mohit AnchliaMohit Anchlia
    Aug 4, 2012 at 9:34 pm
    Aug 5, 2012 at 1:25 am
  • hi all, I posted performance of MemoryChannel last week. That's normal throughput in most environment. Therefore, the performance result of FileChannel is below expectation with same environments and ...
    Denny YeDenny Ye
    Aug 3, 2012 at 3:02 am
    Aug 3, 2012 at 5:50 am
  • HI, Im getting errros 2012-08-02 16:58:50,065 INFO source.AvroSource: Avro source seqGenSrc started. 2012-08-02 16:59:02,463 INFO ipc.NettyServer: [id: 0x45cbda0a, /localhost= / localhost 41414 ...
    JPJP
    Aug 3, 2012 at 3:26 am
    Aug 3, 2012 at 5:37 am
  • Hi, I have added the attached CustomAvroSerializer class to send avro source to the hdfs sink. after im trying to compile im getting the following exception. Even, If i try to compile parent also im ...
    JPJP
    Aug 1, 2012 at 5:44 pm
    Aug 2, 2012 at 6:11 am
  • Let me see if I can explain what I'd like to accomplish, and yes I"m certain I'm just not grasping something here and making a mess of it all, and mayhaps ya'll can point me straight... Simple.. grab ...
    Cochran, David M (Contractor)Cochran, David M (Contractor)
    Aug 23, 2012 at 2:09 pm
    Sep 5, 2012 at 7:15 pm
  • Tue Aug 7 00:00:00 2012 User-Name = "xxxxxxxx" NAS-Port = xxxxxxxx NAS-IP-Address = xxxxxxxx Framed-IP-Address = xxxxxxxx Filter-Id = " xxxxxxxx " Class = " xxxxxxxx " NAS-Identifier = " xxxxxxxx " ...
    Manu Moncy KManu Moncy K
    Aug 30, 2012 at 10:19 am
    Aug 30, 2012 at 1:51 pm
  • I have rollover defined either to roll every 5G or 1+ hr but doesn't seem to be working. Could you please suggest if I got the conf incorrectly configured? foo.sinks.hdfsSink.hdfs.filePrefix = web ...
    Mohit AnchliaMohit Anchlia
    Aug 25, 2012 at 12:43 am
    Aug 27, 2012 at 1:29 am
  • Abhishek, Moving this to user@flume lists, as it is Flume specific. P.s. Please do not cross post to multiple lists, it does not guarantee you a faster response nor is mailing to a *-dev list ...
    Harsh JHarsh J
    Aug 12, 2012 at 1:16 pm
    Aug 13, 2012 at 12:50 am
  • FLUME-1163 (https://issues.apache.org/jira/browse/FLUME-1163) states that this problem has been fixed in Flume 1.2.0. However, my tests indicate that the fix works sometimes, but does not work most ...
    Yongcheng LiYongcheng Li
    Aug 3, 2012 at 2:16 pm
    Aug 3, 2012 at 5:59 pm
  • hello people: I am doing the custom source.But I was found a problem about Flume Developer Guide.At the Guide has a explain that is "public class BarSource extends AbstractSource implements ...
    荆棘鸟荆棘鸟
    Aug 2, 2012 at 5:38 am
    Aug 2, 2012 at 9:58 pm
  • Good morning. I'm trying to test Flume for use for our companies log collection. These are application transaction logs (not syslogs) so I need the agents channel data to be fully recoverable. I have ...
    Steve JohnsonSteve Johnson
    Aug 29, 2012 at 3:38 pm
    Aug 29, 2012 at 4:18 pm
  • Hi All , Can I use Failover Sink Processor and Load balancing Sink Processor together? How can I configure it? Regards Ruihong Shi
    Shara ShiShara Shi
    Aug 27, 2012 at 8:21 am
    Aug 27, 2012 at 9:29 am
  • Hello everybody: I hava a problem about every event. The flume wheather is supporting the transmission of compressed event. And if it is supporting ,I should be to however configuration the file ...
    Yanzhi.liuYanzhi.liu
    Aug 24, 2012 at 7:33 am
    Aug 24, 2012 at 10:38 am
  • Hi! I have, what I would imagine, is a pretty common use case -- a bunch of S3 buckets with access logging enabled. S3 is nice in a sense that the logging for the bucket all goes back to a bucket of ...
    Roman ShaposhnikRoman Shaposhnik
    Aug 24, 2012 at 12:06 am
    Aug 24, 2012 at 12:50 am
  • Hey All! I'm writing here because we're facing a problem at our flume-ng installation (version 1.2.0). We're collecting logs from Exec Sources to File Channel and HDFS Sinks (in a load balanced sink ...
    KARASZI IstvanKARASZI Istvan
    Aug 23, 2012 at 7:30 am
    Aug 23, 2012 at 8:06 am
  • Greetings, I'm currently storing log data using flume - hdfs and was looking for a way to alter the log data from the normal string format to a json representation (and possibly in the future store ...
    João AscensoJoão Ascenso
    Aug 20, 2012 at 4:04 pm
    Aug 20, 2012 at 4:24 pm
  • Hi All, We are developing a tool which embeds Flume. I have some questions about the planned / upcoming features of Flume. I understand that it is possible to create fan out configurations using ...
    Rakos, RudolfRakos, Rudolf
    Aug 16, 2012 at 12:31 pm
    Aug 17, 2012 at 9:51 am
  • I am looking at flume for data ingestion on AWS. I was wondering how people handle dynamic hostnames in flume? I have created my own ConfigurationProvider and looking to mimic FailoverSinkProcessor ...
    David CapwellDavid Capwell
    Aug 13, 2012 at 6:33 pm
    Aug 15, 2012 at 12:28 am
  • I want to write a custom interceptor to modify the text of an event (not just headers). Can this be done? Is there any documentation on how to go about doing it? Thanks, Mark Stern
    Stern, MarkStern, Mark
    Aug 9, 2012 at 10:21 am
    Aug 13, 2012 at 12:37 am
  • Hey All, For any in the Dallas area interested in Flume-NG, I'll be talking at the Dallas HUG about flume tonight: http://www.dfwbigdata.org/events/73702252/?eventId=73702252&action=detail Thursday, ...
    Patrick WendellPatrick Wendell
    Aug 9, 2012 at 6:21 am
    Aug 9, 2012 at 8:16 am
  • hello: I have a error for the flume configuration. My configuration is: agent_foo.sources = tailsource-1 tailsource agent_foo.channels = memoryChannel-1 memoryChannel agent_foo.sinks = Sink-1 Sink ...
    Yanzhi.liuYanzhi.liu
    Aug 8, 2012 at 11:14 am
    Aug 8, 2012 at 11:33 am
  • Hi, I'm trying to write events to HDFS using Flume 1.2.0 and I have a couple of questions. Firstly, about the reliability semantics of the HdfsEventSink. My number one requirement is reliability, ...
    バーチャル クリストファーバーチャル クリストファー
    Aug 2, 2012 at 2:07 am
    Aug 3, 2012 at 8:46 am
  • Trying to understand how flume sink invocation works. I see HBaseSink.java has start and stop, is this object created once or is this object created for every event call to sink? Also, who calls ...
    Mohit AnchliaMohit Anchlia
    Aug 2, 2012 at 7:24 pm
    Aug 3, 2012 at 6:40 am
  • I downloaded the latest release (1.2.0) of flume and tried to get it sending data to HDFS; for some reason it isn't adding hdfs as a sink. Here is my properties file: agent.sources = seqGenSrc ...
    David CapwellDavid Capwell
    Aug 2, 2012 at 11:19 pm
    Aug 3, 2012 at 2:30 am
  • Hi gurus, I'm trying to create my own hbase sink implementing HbaseEventSerializer. I am struggling with the correct dependencies within my pom file.I am getting the following: [WARNING] The POM for ...
    George FormanGeorge Forman
    Aug 2, 2012 at 3:42 pm
    Aug 2, 2012 at 6:11 pm
  • Hi, i have some trouble setting up the HDFS sink in Flume-NG (CDH3U4, 1.1.0): Here's my sink configuration: agent.sinks.hdfsSinkSMP.type = hdfs agent.sinks.hdfsSinkSMP.channel = memoryChannel ...
    Christian SchroerChristian Schroer
    Aug 1, 2012 at 12:42 pm
    Aug 2, 2012 at 2:00 am
  • Hi, what is the max size about one event in the flume agent? is one event one line(or one log)? when the flume transfer the data, could it compress in the process? Best Regards Malone 2012-08-31
    Yingnan.maYingnan.ma
    Aug 31, 2012 at 8:04 am
    Aug 31, 2012 at 10:49 am
  • Hello, I was wondering if there is any sort of message routing available in pre-NG version of Flume? Concretely, I have a stream of messages belonging to different users and I want to route some ...
    Otis GospodneticOtis Gospodnetic
    Aug 29, 2012 at 10:39 am
    Aug 29, 2012 at 8:54 pm
  • Hello everyone: My flume Clusters was always successed. But when I use the source as long as and mine source-flume's log was appeared these. I want to know what it is meaning and whether it is ...
    Yanzhi.liuYanzhi.liu
    Aug 28, 2012 at 9:53 am
    Aug 28, 2012 at 2:47 pm
  • I fetch flume source code from github, and checkout flume-1.3.0 branch to build.The following fault is errite, it seems that the flume-ng-configuration package is missing.Deos anyone meet the same ...
    YuXinYuXin
    Aug 28, 2012 at 3:25 am
    Aug 28, 2012 at 3:39 am
  • I have 5 threads writing json blob to hdfs but what I see is application hanging and when I do the thread dump on the application I see threads waiting on epollWait: "New I/O client worker #5-1" ...
    Mohit AnchliaMohit Anchlia
    Aug 22, 2012 at 7:54 pm
    Aug 22, 2012 at 8:28 pm
  • Hi All, We are developing a tool which embeds Flume. I have some questions about configuring Flume. The Avro sink emits error level log messages every few seconds when it loses connection to the next ...
    Rakos, RudolfRakos, Rudolf
    Aug 16, 2012 at 12:35 pm
    Aug 16, 2012 at 6:09 pm
Group Navigation
period‹ prev | Aug 2012 | next ›
Group Overview
groupuser @
categoriesflume
discussions78
posts266
users61
websiteflume.apache.org
irc#flume

61 users for August 2012

Hari Shreedharan: 21 posts Mohit Anchlia: 17 posts Chris Neal: 14 posts Denny Ye: 14 posts Patrick Wendell: 14 posts Brock Noland: 13 posts Bhaskar V. Karambelkar: 11 posts JP: 11 posts Mike Percy: 10 posts Ashutosh(오픈플랫폼개발팀): 8 posts Yanzhi.liu: 8 posts Alo alt: 7 posts David Capwell: 7 posts Sandeep Reddy P: 7 posts Shara Shi: 7 posts Stern, Mark: 6 posts Juhani Connolly: 5 posts バーチャル クリストファー: 4 posts Eran Kutner: 4 posts Jagadish Bihani: 4 posts
show more