FAQ
This is hopefully easily answered: I'm bundling some helper functions into
a namespace.
One of the benefits I'd like to provide (via this namespace) is to give
*JOB-CONF* a
default list of useful properties, but I can't figure out how to do this
elegantly: I don't
think I can call (with-job-conf) inside my custom namespace, because the
binding
would not be exposed. I guess I could just override *JOB-CONF* directly,
but that
seems ugly. Is there a recommend way to achieve this? ie. by including:

(:use my.custom.namespace)

one could avoid having to explicitly call (with-job-conf) from one's own
code, in order
to gain those specific properties.

Thanks!

Search Discussions

  • Nathanmarz at Oct 30, 2011 at 12:10 am
    We have a feature planned where Cascalog will look for a "job-
    conf.clj" file on the classpath and use the information in there as
    the default JOB-CONF. For now, you should either use with-job-conf or
    do an alter-var-root on the *JOB-CONF* var.

    -Nathan

    On Oct 26, 2:50 pm, R Daneel wrote:
    This is hopefully easily answered: I'm bundling some helper functions into
    a namespace.
    One of the benefits I'd like to provide (via this namespace) is to give
    *JOB-CONF* a
    default list of useful properties, but I can't figure out how to do this
    elegantly: I don't
    think I can call (with-job-conf) inside my custom namespace, because the
    binding
    would not be exposed.  I guess I could just override *JOB-CONF* directly,
    but that
    seems ugly.  Is there a recommend way to achieve this?  ie. by including:

    (:use my.custom.namespace)

    one could avoid having to explicitly call (with-job-conf) from one's own
    code, in order
    to gain those specific properties.

    Thanks!
  • R Daneel at Oct 30, 2011 at 5:43 pm
    That's good news! Looks like alter-var-root is what I need in the meantime.

    Thanks again :)
  • R Daneel at Oct 31, 2011 at 12:43 am
    Yeah, that works great! Just for completeness, for anyone still as
    clojure-challenged
    as me, here's my working example:

    (def job-conf-default { "hadoop.tmp.dir" "/tmp"

    "mapreduce.job.complete.cancel.delegation.tokens" false
    "mapred.map.tasks.speculative.execution"
    true

    "mapred.reduce.tasks.speculative.execution" true })
    (alter-var-root (var r/*JOB-CONF*) (constantly job-conf-default))

    and of course I can override those, or add more properties via
    (with-job-conf).

    Thanks again! (The job-conf.clj solution will be even cooler though!)

    Mike
  • R Daneel at Oct 31, 2011 at 8:05 pm
    Nathan: I'm trying to anticipate your job-conf.clj solution: do you have
    any (firm enough)
    specs for it yet? (eg. the format of job-conf.clj, or where you'll look
    for it?) I'm currently
    looking for ~/.job-conf.clj and ${CWD}/job-conf.clj, and then letting the
    latter override the
    former (and both override some site-specific defaults to workaround the
    kerberos issue).
    And I'm just putting a naked hash-map into job-conf.clj. If you can
    recommend any change
    that is more likely to make this compatible with whatever finally launches
    in cascalog, that
    would be very useful :) Here's my (probably very
    inefficient/non-idiomatic!) code (below).

    Thanks again!

    Mike


    ;; Define some necessary default job conf properties.

    (def ^{:private true} default-job-conf
    {"hadoop.tmp.dir" "/tmp"
    "mapreduce.job.complete.cancel.delegation.tokens" false

    ;; Read any additional properties from user's CWD ("./job-conf.clj")
    ;; and user's homedir ("~/.job-conf.clj").

    (defn ^{:private true} read-job-conf [path]
    (binding [*read-eval* false]
    (try
    (with-open [r (io/reader path)]
    (read (PushbackReader. r)))
    (catch java.io.FileNotFoundException e))))

    (defn ^{:private true} system-property [family prop]
    (System/getProperty (str family "." prop)))

    (defn ^{:private true} user-property [prop]
    (system-property "user" prop))

    (def ^{:private true} home-job-conf
    (read-job-conf (str (user-property "home") "/.job-conf.clj")))

    (def ^{:private true} cwd-job-conf
    (read-job-conf "job-conf.clj"))

    ;; Merge the default and user job conf properties: the CWD properties
    ;; override the homedir properties, and both override the defaults.

    (def ^{:private true} final-job-conf
    (conj default-job-conf home-job-conf cwd-job-conf))

    ;; Re-bind the (empty) *JOB-CONF* hash to the new defaults. These
    ;; defaults will be automatically included within any subsequent call
    ;; to (with-job-conf).

    (alter-var-root (var r/*JOB-CONF*) (constantly final-job-conf))
  • R Daneel at Oct 31, 2011 at 8:17 pm
    Oops: just re-read your previous message and noticed the crucial "on the
    classpath" :)
  • Nathanmarz at Oct 31, 2011 at 8:31 pm
    Yup, it will just look for job-conf.clj on the classpath. The format
    of the conf will be a regular Clojure map.

    On Oct 31, 1:17 pm, R Daneel wrote:
    Oops: just re-read your previous message and noticed the crucial "on the
    classpath" :)
  • R Daneel at Nov 1, 2011 at 4:38 am
    Stupid question: will it look inside jars on the classpath?
  • Sam Ritchie at Nov 1, 2011 at 5:24 am
    Hey, that's a good question -- multiple job-conf.clj files on the classpath
    might clash. Let me prototype this in the next day or two and report back
    to you all with a first draft.
    On Mon, Oct 31, 2011 at 9:38 PM, R Daneel wrote:

    Stupid question: will it look inside jars on the classpath?



    --
    Sam Ritchie, Twitter Inc
    703.662.1337
    @sritchie09

    (Too brief? Here's why! http://emailcharter.org)
  • R Daneel at Nov 1, 2011 at 5:38 am
    Great!! :)
Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedOct 26, '11 at 9:50p
activeNov 1, '11 at 5:38a
posts10
users3
websiteclojure.org
irc#clojure

People

Translate

site design / logo © 2022 Grokbase