[Hadoop-common-dev] [jira] Created: (HADOOP-3719) Chukwa

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated HADOOP-3719:
-------------------------------

Attachment: chukwa_08.pdf

Chukwa is designed to collect monitoring data (especially log files), and get the data into HDFS as quickly as possible. Data is initially collected by a Local Agent running on each machine being monitored. This Local Agent has a pluggable architecture, allowing many different adaptors to be used, each of which produces a particular stream of data. Local Agents send their data via HTTP to Collectors, which write out data into "sink files" in HDFS.

Map-reduce jobs run periodically to analyze these sink files, and to drain their contents into structured storage.

Chukwa provides a natural solution to the log collection problem, posed in HADOOP-2206. Once we have Chukwa working at scale, we intend to produce some patches to Hadoop to trigger log collection appropriately.

We expect this work to ultimately be complementary to HADOOP-3585, the failure analysis system. We want to collect similar data, and our framework is flexible enough to accommodate the proposed structure there, with only modest code changes on each side.

The attached document introduces Chukwa, and describes the data collection architecture. We do not present our analytics and visualization in detail in this document. We intend to describe them in a second document in the near future.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612463#action_12612463 ]

Enis Soztutar commented on HADOOP-3719:
---------------------------------------

Wow, that's just what I needed.
When can we expect the patch ?

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612733#action_12612733 ]

Mac Yang commented on HADOOP-3719:
----------------------------------

Very happy to see interest in this work.

We are planning on doing the initial check-in in a week or so. Hopefully that will give folks better idea on what we are trying to do and also serves as the starting point for collaboration.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613390#action_12613390 ]

Pete Wyckoff commented on HADOOP-3719:
--------------------------------------

Do the "sink files" need to be in HDFS or is this pluggable as well so I could write to other filesystems, e.g., NFS?

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613398#action_12613398 ]

Ari Rabkin commented on HADOOP-3719:
------------------------------------

Pete --
Yes, the sink file writers are pluggable. In fact, our current writer uses the Hadoop FileSystem class, so I believe that if you pass a local path that points at NFS, it'll "just work". We haven't tested that, though.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613658#action_12613658 ]

Doug Cutting commented on HADOOP-3719:
--------------------------------------

How will this be integrated with Hadoop? As a contrib module?

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613660#action_12613660 ]

Jerome Boulon commented on HADOOP-3719:
---------------------------------------

Yes, we're planning to add Chukwa to hadoop Tree as a contrib module within
the next few days.
/Jerome

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated HADOOP-3719:
-------------------------------

Attachment: chukwa-0.0.1.tgz

Initial chukwa release.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614903#action_12614903 ]

Owen O'Malley commented on HADOOP-3719:
---------------------------------------

Please include license files parallel to the included jars that are not Apache projects.

Please make the tarball relative to $HADOOP_HOME.

Make sure you don't have any empty directories (or others that shouldn't be checked in).

Please run the release audit tool over the submission to make sure that your source files all have copyright notices.
I notice that inputtools/mdl/DBSummaryLoader.java and JobLogDataLoader.java do not.

Please remove the source code for org.openflashcart.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Attachment: chukwa-patch-0.0.1.tgz

- Removed openflashchart from source and compiled into library file.
- Added licenses for library files.
- Added licenses for all java source files.
- compress tarball relative to HADOOP_HOME
- Ran RAT audit tool successfully.
- Removed empty directory.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Attachment: (was: chukwa-patch-0.0.1.tgz)

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Status: Patch Available (was: Open)

Polish patch to meet patching requirement.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Attachment: chukwa-patch-0.0.1.tgz

Polish patch to meet requirement.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620480#action_12620480 ]

Hadoop QA commented on HADOOP-3719:
-----------------------------------

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12387671/chukwa-patch-0.0.1.tgz
against trunk revision 683448.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

-1 patch. The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3026/console

This message is automatically generated.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-3719:
----------------------------------

Status: Open (was: Patch Available)

The current patch is still missing license files for a lot of the jar files and it includes LGPL libraries, which can't be included. It would probably help to have a README in the lib directory that lists the jar files and which project they are from and their license.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Attachment: (was: chukwa-patch-0.0.1.tgz)

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Status: Patch Available (was: Open)

Removed LGPL libraries, and include README and LICENSE file for all dependent libraries.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-3719:
------------------------------

Attachment: chukwa-patch-0.0.1.tar.gz

Removed all LGPL dependent libraries, and include readme and license files for all dependent libraries.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-3719:
----------------------------------

Resolution: Fixed
Fix Version/s: 0.19.0
Assignee: Ari Rabkin
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)

I just committed this. Thanks, guys!

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated HADOOP-3719:
-------------------------------

Attachment: (was: chukwa-0.0.1.tgz)

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624780#action_12624780 ]

Hudson commented on HADOOP-3719:
--------------------------------

Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-3719:
------------------------------------

Release Note: Introduced Chukwa data collection and analysis framework.

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Chansler updated HADOOP-3719:
------------------------------------

Component/s: contrib/chukwa

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643034#action_12643034 ]

Alex Loddengaard commented on HADOOP-3719:
------------------------------------------

Ari, you and Jerome, in an email thread a week or so back, mentioned that you were planning on releasing a second Chukwa patch. Any updates here?

[ https://issues.apache.org/jira/browse/HADOOP-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643038#action_12643038 ]

Jerome Boulon commented on HADOOP-3719:
---------------------------------------

Hi Alex,
If you search for "chukwa" against the Jira website you'll see a list of
patches that we want to commit, but we're depending on external Apache
committers to get them committed.

/Jerome.