[Hadoop-common-dev] [jira] Created: (HADOOP-4998) Implement a native OS runtime for Hadoop

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662243#action_12662243 ]

Vinod K V commented on HADOOP-4998:
-----------------------------------

+1.

Of late, there have been many instances in which Mapred, particualrly the TT side, started to depend on platform specific features. I guess the handling of these can be made much easier with such a native runtime. Some of them I can immediately think of are memory management of tasks by TT(HADOOP-3581), starting tasks via job control(HADOOP-2721), starting tasks as the job submitting user(HADOOP-4490). But I am not very sure as to which of these and how much of it can be helped by a native runtime.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662245#action_12662245 ]

Doug Cutting commented on HADOOP-4998:
--------------------------------------

Arun> It would be useful to implement a JNI-based runtime [ ... ]

What do you mean by 'useful'? Is this about performance?

Arun> This would allow us to stop relying on exec'ing bash [ ... ]

It would only allow this on platforms where the native library is built, right? So, unless we intend to support this on Windows, Solaris and MacOS, we still have to maintain the shell-based implementations too.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662272#action_12662272 ]

Arun C Murthy commented on HADOOP-4998:
---------------------------------------

Yes to both.

We would need to maintain a shell-based implementation for platforms which do not the native implementations.

Having a JNI-based runtime would let us access runtime information in a significantly more performant manner. For e.g. HADOOP-4656 could be implemented by directly calling a native method which fetches groups via posix apis. The absence of which forces us to cache the output of 'groups' shell command.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662438#action_12662438 ]

Doug Cutting commented on HADOOP-4998:
--------------------------------------

So, since performance is the reason, do we have a benchmark that shows this as significant? If not, we'll need one, right?

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662469#action_12662469 ]

Hong Tang commented on HADOOP-4998:
-----------------------------------

@Doug

It seems to be common sense that process invocation is orders of magnitude more expensive than function calls (or system calls).

Are you asking whether those invocations take a big portion of execution time in the real application?

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662483#action_12662483 ]

Doug Cutting commented on HADOOP-4998:
--------------------------------------
Yes, precisely.

If HADOOP-4656 is the motivating case, then, after HADOOP-4656, the namenode might become nearly unusably slow without native code, which would be a significant negative change. So, for HADOOP-4656, we might instead change the default to, e.g., read /etc/group rather than exec a command.

In general, it's best to design things so that we do not require native code for decent performance.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662495#action_12662495 ]

Raghu Angadi commented on HADOOP-4998:
--------------------------------------

Another use case:

If we have access to poll(), we would just us that for Hadoop's blocking IO on non-blocking sockets rather than Sun's epoll based implementation and avoid requiring extra 3 fds for each thread that is blocked.. this simplifies the implementation as well.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662500#action_12662500 ]

Doug Cutting commented on HADOOP-4998:
--------------------------------------
Should each of these be a separate issue?

We currently have an optional libhadoop. If we want to add more to it, I can see proceeding in one of a few ways:
- adding more optional optimizations there, on a case-by-case basis; or
- replace all shell access with native code, replacing the reliance on bash with reliance on a native library.

But adding a few more optional optimizations doesn't seem like a single coherent issue and would better be addressed by more specific jiras, no?

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662507#action_12662507 ]

Raghu Angadi commented on HADOOP-4998:
--------------------------------------

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662506#action_12662506 ]

Doug Cutting commented on HADOOP-4998:
--------------------------------------

Another thing to consider is building on http://apr.apache.org/. This seems to have most of what we'd want, and we could ship with pre-built versions for linux, windows, etc, since APR includes these. Then we might get rid of the bash & cygwin requirements. If we want to go this way it would be good to do an inventory of all the places we use bash and see how many APR might replace.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662930#action_12662930 ]

Steve Loughran commented on HADOOP-4998:
----------------------------------------

It's worth remembering that once you go to JNI, you are more at risk from memory leaks, pointer problems, race conditions and other C/C++ coding issues that can affect long-lived programs. The benchmarks would also need to track process memory consumption to make sure that switching to a JNI wrapper didn't cause the memory use of the app to grow, and it would be handy to have an option in production systems to switch from JNI to shell calls to see if it makes any observed problems go away.

This isn't a -1 to a JNI, just a warning that there can often be a price.

Incidentally, in past experiments of mine, a JNI call takes about 600 PII clock cycles cycles round trip; this is a lot less than starting a process, but not entirely free.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663068#action_12663068 ]

Allen Wittenauer commented on HADOOP-4998:
------------------------------------------

FWIW, we have already seen a condition where the secondary name node required more memory to operate than the primary name node to do a shell out due to a requirement to run whoami or id or whatever due to DFSClient being in the code path. (As it was explained to me and/or as I understood it). From an ops perspective, this is highly suspect.... ;)

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663825#action_12663825 ]

Marco Nicosia commented on HADOOP-4998:
---------------------------------------

I am really close to raising this to a blocker for Hadoop 0.18.3, BUT, I'm worried this isn't the right ticket. The symptom we're struggling with is that when a primary or secondary namenode wants to fork to run bash, it requires double memory than what it's currently using.

I don't care if the solution is JNI or some IPC to some other process to run shell processes, but it's an inconvenient issue to work around. Is this the right ticket to take on fixing that, or should we create a different JIRA?

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663836#action_12663836 ]

Arun C Murthy commented on HADOOP-4998:
---------------------------------------

Marco, we could use HADOOP-4656 in conjuction with this jira?

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663883#action_12663883 ]

Doug Cutting commented on HADOOP-4998:
--------------------------------------

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664094#action_12664094 ]

Steve Loughran commented on HADOOP-4998:
----------------------------------------

I agree; the memory use for an exec() is odd and needs to be looked at.

Incidentally, one thing we are fond of doing is exec-ing a long-lived shell rather than running a shell script. This lets us use SSH to connect to nearby hosts, or to boost from being untrusted to root, but it may also have memory consumption benefits. You just need to keep that single shell connection open and issue commands down its IO streams.

[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664229#action_12664229 ]

Koji Noguchi commented on HADOOP-4998:
--------------------------------------

bq. Running a shell should use vfork and exec, and shouldn't double the memory use, should it?

Created separate Jira. HADOOP-5059.