FAQ
Hi Ben,

Hive's UDFs would run wherever the query's structure needs it to. For
instance, a transformational query such as SELECT UPPER(X) FROM Y;
would run the UDF UPPER in map-end cause there's no grouping or
reducing required of the query.

The UDFs also run within the same task JVM, or within the same task
JVM parent, that runs the record readers, etc.. So they are bound by
the virtual memory limits (mapred.child.ulimit in MR1) applied on the
task JVMs, and cannot go beyond that.

There should be no caveats to use a UDF, as long as you trust/know its
implementation. Bad UDFs (such as those that may store KVs in memory
without control causing OOMEs, etc.) would cause job failures but only
within the queries that use them.
On Tue, May 13, 2014 at 11:57 PM, Benjamin Kim wrote:
I would like to know how Hive UDF's behave when run on a node of the
cluster. I have seen UDF's mainly run in the reduce phase of a job. Do they
always run during this phase? I would like to know what kind of memory
footprint do they use. Can they usurp memory from other services?

Just want to know if there are any caveats to using UDF's.

Thanks,
Ben

--

---
You received this message because you are subscribed to the Google Groups
"CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.


--
Harsh J

--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/d/optout.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 4 | next ›
Discussion Overview
groupcdh-user @
categorieshadoop
postedMay 13, '14 at 6:27p
activeMay 15, '14 at 2:20a
posts4
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2021 Grokbase