Hi,
I am running a Flume agent using Cloudera Manager and the agent is not able
to roll the data on the basis of specified configurations. Rather it always
rolls data arbitrarily in small small files (1KB). I have tested the same
configuration in local mode and it is working fine but the same config
doesn't work using the manager. Following is the config for HDFS sink:
ataplatform.sinks.sink1.hdfs.round = true
dataplatform.sinks.sink1.hdfs.roundValue = 1
dataplatform.sinks.sink1.hdfs.roundUnit = minute
dataplatform.sinks.sink1.hdfs.rollInterval = 60
dataplatform.sinks.sink1.hdfs.rollSize = 0
dataplatform.sinks.sink1.hdfs.rollCount = 0
dataplatform.sinks.sink1.hdfs.batchSize = 100
dataplatform.sinks.sink1.hdfs.txnEventMax = 40000
dataplatform.sinks.sink1.hdfs.fileType = DataStream
dataplatform.sinks.sink1.hdfs.maxOpenFiles = 50
dataplatform.sinks.sink1.hdfs.appendTimeout = 10000
dataplatform.sinks.sink1.hdfs.callTimeout = 10000
dataplatform.sinks.sink1.hdfs.threadsPoolSize = 100
dataplatform.sinks.sink1.hdfs.rollTimerPoolSize = 1
Please let me know me if I am doing anything wrong which is causing this
issue.
Thanks
Amar
