FAQ
Hi folks

I notice that the .proto files are not present in the built tarball. This presents a problem to those of us working on 3rd party tools that need to talk to Hadoop tools such as the resource manager. It means that anyone wanting to build our tools has to install an svn checkout of the code as opposed to simply installing the tarball.

Is there any reason -not- to include the .proto files in the tarball for distribution? It would help a great deal.

Thanks
Ralph

Search Discussions

  • Eli Collins at Mar 25, 2012 at 1:39 am
    Good idea, no reason we shouldn't, the build probably wasn't updated to
    include the who we added then. File a jira?
    On Saturday, March 24, 2012, Ralph Castain wrote:
    Hi folks

    I notice that the .proto files are not present in the built tarball. This
    presents a problem to those of us working on 3rd party tools that need to
    talk to Hadoop tools such as the resource manager. It means that anyone
    wanting to build our tools has to install an svn checkout of the code as
    opposed to simply installing the tarball.
    Is there any reason -not- to include the .proto files in the tarball for
    distribution? It would help a great deal.
    Thanks
    Ralph
  • Robert Evans at Mar 26, 2012 at 2:30 pm
    I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.

    --Bobby Evans

    On 3/24/12 8:38 PM, "Eli Collins" wrote:

    Good idea, no reason we shouldn't, the build probably wasn't updated to
    include the who we added then. File a jira?
    On Saturday, March 24, 2012, Ralph Castain wrote:
    Hi folks

    I notice that the .proto files are not present in the built tarball. This
    presents a problem to those of us working on 3rd party tools that need to
    talk to Hadoop tools such as the resource manager. It means that anyone
    wanting to build our tools has to install an svn checkout of the code as
    opposed to simply installing the tarball.
    Is there any reason -not- to include the .proto files in the tarball for
    distribution? It would help a great deal.
    Thanks
    Ralph
  • Ralph Castain at Mar 26, 2012 at 10:56 pm
    Perhaps it would help if I outline the use case. I have a Java client that needs to launch a non-Java application manager. Obviously, the client talks to the RM using the Java APIs. The AM is passed all the info it needs to launch the actual application in its cmd line and environ, so no client to AM comm is required. Each client will launch it's own dedicated AM, so no cross-talk is possible.

    The AM needs to talk to the RM and NM to get an allocation and launch. Hence the proto requirement. I have not encountered any issue in that comm so far, provided I can build the AM against the proto files,

    Hence the jira. Hope that helps clarify the request.

    Sent from my iPad
    On Mar 26, 2012, at 8:30 AM, Robert Evans wrote:

    I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.

    --Bobby Evans

    On 3/24/12 8:38 PM, "Eli Collins" wrote:

    Good idea, no reason we shouldn't, the build probably wasn't updated to
    include the who we added then. File a jira?
    On Saturday, March 24, 2012, Ralph Castain wrote:
    Hi folks

    I notice that the .proto files are not present in the built tarball. This
    presents a problem to those of us working on 3rd party tools that need to
    talk to Hadoop tools such as the resource manager. It means that anyone
    wanting to build our tools has to install an svn checkout of the code as
    opposed to simply installing the tarball.
    Is there any reason -not- to include the .proto files in the tarball for
    distribution? It would help a great deal.
    Thanks
    Ralph
  • Arun C Murthy at Mar 27, 2012 at 12:00 am
    Is your AM written in Java or C?
    On Mar 26, 2012, at 3:55 PM, Ralph Castain wrote:

    Perhaps it would help if I outline the use case. I have a Java client that needs to launch a non-Java application manager. Obviously, the client talks to the RM using the Java APIs. The AM is passed all the info it needs to launch the actual application in its cmd line and environ, so no client to AM comm is required. Each client will launch it's own dedicated AM, so no cross-talk is possible.

    The AM needs to talk to the RM and NM to get an allocation and launch. Hence the proto requirement. I have not encountered any issue in that comm so far, provided I can build the AM against the proto files,

    Hence the jira. Hope that helps clarify the request.

    Sent from my iPad
    On Mar 26, 2012, at 8:30 AM, Robert Evans wrote:

    I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.

    --Bobby Evans

    On 3/24/12 8:38 PM, "Eli Collins" wrote:

    Good idea, no reason we shouldn't, the build probably wasn't updated to
    include the who we added then. File a jira?
    On Saturday, March 24, 2012, Ralph Castain wrote:
    Hi folks

    I notice that the .proto files are not present in the built tarball. This
    presents a problem to those of us working on 3rd party tools that need to
    talk to Hadoop tools such as the resource manager. It means that anyone
    wanting to build our tools has to install an svn checkout of the code as
    opposed to simply installing the tarball.
    Is there any reason -not- to include the .proto files in the tarball for
    distribution? It would help a great deal.
    Thanks
    Ralph
    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/
  • Ralph Castain at Mar 27, 2012 at 12:36 am
    Has to be in C to support MPI wireup. We wrote a little lib that provides the required messaging support using protobuf-c, which works fine but protobuf-c needs to compile the proto files into something C can understand.

    Sent from my iPad
    On Mar 26, 2012, at 6:00 PM, Arun C Murthy wrote:

    Is your AM written in Java or C?
    On Mar 26, 2012, at 3:55 PM, Ralph Castain wrote:

    Perhaps it would help if I outline the use case. I have a Java client that needs to launch a non-Java application manager. Obviously, the client talks to the RM using the Java APIs. The AM is passed all the info it needs to launch the actual application in its cmd line and environ, so no client to AM comm is required. Each client will launch it's own dedicated AM, so no cross-talk is possible.

    The AM needs to talk to the RM and NM to get an allocation and launch. Hence the proto requirement. I have not encountered any issue in that comm so far, provided I can build the AM against the proto files,

    Hence the jira. Hope that helps clarify the request.

    Sent from my iPad
    On Mar 26, 2012, at 8:30 AM, Robert Evans wrote:

    I responded in the JIRA for this. Because we wrap proto in Hadoop RPC right now those .proto files are not going to do very many people a lot of good, unless they have a client that can also communicate over a simple form of Hadoop RPC. I think it would be good to move to a pure PB RPC implementation, but that involves security changes and a lot of other things so it is not a small undertaking.

    --Bobby Evans

    On 3/24/12 8:38 PM, "Eli Collins" wrote:

    Good idea, no reason we shouldn't, the build probably wasn't updated to
    include the who we added then. File a jira?
    On Saturday, March 24, 2012, Ralph Castain wrote:
    Hi folks

    I notice that the .proto files are not present in the built tarball. This
    presents a problem to those of us working on 3rd party tools that need to
    talk to Hadoop tools such as the resource manager. It means that anyone
    wanting to build our tools has to install an svn checkout of the code as
    opposed to simply installing the tarball.
    Is there any reason -not- to include the .proto files in the tarball for
    distribution? It would help a great deal.
    Thanks
    Ralph
    --
    Arun C. Murthy
    Hortonworks Inc.
    http://hortonworks.com/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 25, '12 at 12:36a
activeMar 27, '12 at 12:36a
posts6
users5
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase