FAQ
Hello, Nathan et al,

Is it possible to specify hadoop configuration parameters (e.g.
mapred.map.tasks etc.) for each _query_? with-job-conf does the job, but if
I have nested queries, they sometimes need different configuration.

Thanks,
Artem.

Search Discussions

  • Nathan Marz at Jan 8, 2012 at 6:00 am
    This is currently not possible, as Cascading does not provide that
    flexibility. The only way to do this right now is to break up the query
    into multiple stages with the intermediate outputs sent to temporary paths.
    cascalog-checkpoint helps a lot with doing stuff like that (see
    http://sritchie.github.com/2011/11/15/introducing-cascalogcontrib.html for
    more details).

    I opened up an issue for this since it would be useful:
    https://github.com/nathanmarz/cascalog/issues/43

    On Sat, Jan 7, 2012 at 1:05 AM, Artem Boytsov wrote:

    Hello, Nathan et al,

    Is it possible to specify hadoop configuration parameters (e.g.
    mapred.map.tasks etc.) for each _query_? with-job-conf does the job, but if
    I have nested queries, they sometimes need different configuration.

    Thanks,
    Artem.


    --
    Twitter: @nathanmarz
    http://nathanmarz.com

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedJan 7, '12 at 9:06a
activeJan 8, '12 at 6:00a
posts2
users2
websiteclojure.org
irc#clojure

2 users in discussion

Nathan Marz: 1 post Artem Boytsov: 1 post

People

Translate

site design / logo © 2022 Grokbase