FAQ
Hi all -- one thing we have been doing a lot of is instead of using the
query macro we dynamically build out predicates and use construct. The
reason is that often we want to make sure certain expensive predicates
don't get evaluated if the predicates 'out var' isn't requested.

so for example -- the follow should not print out "minus" since we don't
actually need it.

(use 'cascalog.api)
(def src [[1 2] [3 4] [5 8]])

(defn my-add [a b] (do (println "adding") (+ a b)))

(defn my-minus [a b] (do (println "minus") (- a b)))

(?<- (stdout)
      [!add]
      (src :> !a !b)
      (my-add !a !b :> !add)
      (my-minus !a !b :> !minus))

print outs:

adding
minus
adding
minus
adding
minus


RESULTS
-----------------------
3
7
13
-----------------------

anyone else have to deal with this?

--
You received this message because you are subscribed to the Google Groups "cascalog-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Search Discussions

  • Andy Xue at Apr 23, 2014 at 12:03 am
    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
       [output-fields {:keys [op input output] :as pred}]
       (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
       [output-fields raw-predicates]
       (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
       [output-fields raw-predicates]
       (let [[options predicates] (opts/extract-options raw-predicates)
             expanded (mapcat expand-outvars predicates)
             filtered (filter-predicates-by-output-fields output-fields
    expanded)]
         (validate-predicates! filtered options)
         (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using the
    query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we don't
    actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Soren Macbeth at Apr 23, 2014 at 12:08 am
    do all the tests still pass with that in place?

    On Tue, Apr 22, 2014 at 5:03 PM, Andy Xue wrote:

    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
    [output-fields {:keys [op input output] :as pred}]
    (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
    [output-fields raw-predicates]
    (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
    [output-fields raw-predicates]
    (let [[options predicates] (opts/extract-options raw-predicates)
    expanded (mapcat expand-outvars predicates)
    filtered (filter-predicates-by-output-fields output-fields
    expanded)]
    (validate-predicates! filtered options)
    (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using the
    query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we don't
    actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google Groups
    "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    http://about.me/soren

    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Soren Macbeth at Apr 23, 2014 at 12:13 am
    I think that this sort of optimization has been bucket with:

    "Cross query optimization: push constants and filters down into subqueries
    when possible"

    Sam,
    If you could refresh my memory about what and how should happen, perhaps we
    can start taking stabs at.

    On Tue, Apr 22, 2014 at 5:08 PM, Soren Macbeth wrote:

    do all the tests still pass with that in place?

    On Tue, Apr 22, 2014 at 5:03 PM, Andy Xue wrote:

    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
    [output-fields {:keys [op input output] :as pred}]
    (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
    [output-fields raw-predicates]
    (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
    [output-fields raw-predicates]
    (let [[options predicates] (opts/extract-options raw-predicates)
    expanded (mapcat expand-outvars predicates)
    filtered (filter-predicates-by-output-fields output-fields
    expanded)]
    (validate-predicates! filtered options)
    (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using the
    query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we don't
    actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google Groups
    "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    http://about.me/soren


    --
    http://about.me/soren

    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andy Xue at Apr 23, 2014 at 12:19 am
    ha no, i just typed that out as a description of the logic. not sure if
    this is the best place to put it.
    On Tuesday, April 22, 2014 8:08:06 PM UTC-4, Soren Macbeth wrote:

    do all the tests still pass with that in place?


    On Tue, Apr 22, 2014 at 5:03 PM, Andy Xue <and...@lumoslabs.com<javascript:>
    wrote:
    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
    [output-fields {:keys [op input output] :as pred}]
    (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
    [output-fields raw-predicates]
    (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
    [output-fields raw-predicates]
    (let [[options predicates] (opts/extract-options raw-predicates)
    expanded (mapcat expand-outvars predicates)
    filtered (filter-predicates-by-output-fields output-fields
    expanded)]
    (validate-predicates! filtered options)
    (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using the
    query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we don't
    actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google Groups
    "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to cascalog-use...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/d/optout.


    --
    http://about.me/soren
    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andy Xue at Apr 23, 2014 at 7:38 pm
    ok -- played around with this for a little bit, and I think there were a
    few usages of cascalog that I haven't really encountered, and those tests
    fail. interesting!

      .. cascalog.logic.parse/build-rule ...
              generators (concat (grouped Generator)
                                 (grouped GeneratorSet))
    - operations (concat (grouped Operation)
    + nessecary-fields (->> (concat (grouped Operation)
    + (grouped FilterOperation)
    + (grouped Aggregator))
    + (mapcat :input)
    + (concat fields)
    + (concat (:sort options)))
    + pruned-operations (filter #(clojure.set/subset? (set (:output %))
    (set nessecary-fields)) (grouped Operation))
    + operations (concat pruned-operations
                                 (grouped FilterOperation))
              aggs (grouped Aggregator)
              tails (concat (initial-tails generators operations)

    one test in particular that fails ...

    (defmapfn mk-one
       "Returns 1 for any input."
       [& tuple] 1)

    (deftest test-no-input
       (let [nums [[1] [2] [3]]]
         ...
         (test?<- [[1 1] [1 2] [1 3]
                   [2 1] [2 2] [2 3]
                   [3 1] [3 2] [3 3]]
                  [?n ?n3]
                  (nums ?n)
                  (mk-one :> ?n2)
                  (nums ?n3))))

    I am still not really clear what (mk-one :> ?n2) is doing/why it is
    necessary. I think before running this test I would have thought that
    predicate isn't needed.


    On Tuesday, April 22, 2014 8:19:00 PM UTC-4, Andy Xue wrote:

    ha no, i just typed that out as a description of the logic. not sure if
    this is the best place to put it.
    On Tuesday, April 22, 2014 8:08:06 PM UTC-4, Soren Macbeth wrote:

    do all the tests still pass with that in place?

    On Tue, Apr 22, 2014 at 5:03 PM, Andy Xue wrote:

    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
    [output-fields {:keys [op input output] :as pred}]
    (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
    [output-fields raw-predicates]
    (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
    [output-fields raw-predicates]
    (let [[options predicates] (opts/extract-options raw-predicates)
    expanded (mapcat expand-outvars predicates)
    filtered (filter-predicates-by-output-fields output-fields
    expanded)]
    (validate-predicates! filtered options)
    (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using the
    query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we
    don't actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google
    Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to cascalog-use...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    http://about.me/soren
    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andy Xue at Apr 23, 2014 at 10:22 pm
    ok -- i got something to pass tests. just had to keep honing down the
    definition of what an unnecessary predicate really is.

    what do ya'll think? how would i go about requesting a PR?

    --- a/cascalog-core/src/clj/cascalog/logic/parse.clj
    +++ b/cascalog-core/src/clj/cascalog/logic/parse.clj
    @@ -642,6 +642,32 @@ This won't work in distributed mode because of the
    ->Record functions."
              (assoc :available-fields fields)
              (assoc :ground? (v/fully-ground? fields)))))

    +(defn prune-operations
    + "Remove any operations whose output is not used; ie, output does not
    intersect with the set
    + of fields that is the query's out-fields, generators (joins), inputs of
    other operations,
    + and :sort option. Additionally, don't try to prune operations if a no
    input operator exists"
    + [fields grouped options]
    + (let [check-input-operations (concat (grouped Operation)
    + (grouped FilterOperation)
    + (grouped Aggregator))]
    + (if (some #(empty? (:input %)) check-input-operations)
    + (concat (grouped Operation)
    + (grouped FilterOperation))
    + (let [generators (concat (grouped Generator)
    + (grouped GeneratorSet))
    + nessecary-fields (->> check-input-operations
    + (mapcat :input)
    + (concat fields)
    + (concat (mapcat :fields generators))
    + (concat (:sort options)))
    + pruned-operations (filter #(seq
    + (clojure.set/intersection
    + (set (:output %))
    + (set nessecary-fields)))
    + (grouped Operation))]
    + (concat pruned-operations
    + (grouped FilterOperation))))))
    +
      (defn build-rule
        [{:keys [fields predicates options] :as input}]
        (let [[nodes expanded] (s/separate #(tail? (:op %)) predicates)
    @@ -650,8 +676,7 @@ This won't work in distributed mode because of the
    ->Record functions."
                           (group-by type))
              generators (concat (grouped Generator)
                                 (grouped GeneratorSet))
    - operations (concat (grouped Operation)
    - (grouped FilterOperation))
    + operations (prune-operations fields grouped options)
              aggs (grouped Aggregator)
              tails (concat (initial-tails generators operations)
                                 (map (fn [{:keys [op output]}]
    On Wednesday, April 23, 2014 3:38:48 PM UTC-4, Andy Xue wrote:

    ok -- played around with this for a little bit, and I think there were a
    few usages of cascalog that I haven't really encountered, and those tests
    fail. interesting!

    .. cascalog.logic.parse/build-rule ...
    generators (concat (grouped Generator)
    (grouped GeneratorSet))
    - operations (concat (grouped Operation)
    + nessecary-fields (->> (concat (grouped Operation)
    + (grouped FilterOperation)
    + (grouped Aggregator))
    + (mapcat :input)
    + (concat fields)
    + (concat (:sort options)))
    + pruned-operations (filter #(clojure.set/subset? (set (:output %))
    (set nessecary-fields)) (grouped Operation))
    + operations (concat pruned-operations
    (grouped FilterOperation))
    aggs (grouped Aggregator)
    tails (concat (initial-tails generators operations)

    one test in particular that fails ...

    (defmapfn mk-one
    "Returns 1 for any input."
    [& tuple] 1)

    (deftest test-no-input
    (let [nums [[1] [2] [3]]]
    ...
    (test?<- [[1 1] [1 2] [1 3]
    [2 1] [2 2] [2 3]
    [3 1] [3 2] [3 3]]
    [?n ?n3]
    (nums ?n)
    (mk-one :> ?n2)
    (nums ?n3))))

    I am still not really clear what (mk-one :> ?n2) is doing/why it is
    necessary. I think before running this test I would have thought that
    predicate isn't needed.


    On Tuesday, April 22, 2014 8:19:00 PM UTC-4, Andy Xue wrote:

    ha no, i just typed that out as a description of the logic. not sure if
    this is the best place to put it.
    On Tuesday, April 22, 2014 8:08:06 PM UTC-4, Soren Macbeth wrote:

    do all the tests still pass with that in place?

    On Tue, Apr 22, 2014 at 5:03 PM, Andy Xue wrote:

    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
    [output-fields {:keys [op input output] :as pred}]
    (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
    [output-fields raw-predicates]
    (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
    [output-fields raw-predicates]
    (let [[options predicates] (opts/extract-options raw-predicates)
    expanded (mapcat expand-outvars predicates)
    filtered (filter-predicates-by-output-fields output-fields
    expanded)]
    (validate-predicates! filtered options)
    (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using
    the query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we
    don't actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google
    Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to cascalog-use...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    http://about.me/soren
    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.
  • Andy Xue at Apr 23, 2014 at 11:37 pm
    oops, totally forgot about fork

    https://github.com/nathanmarz/cascalog/pull/247
    On Wednesday, April 23, 2014 6:22:36 PM UTC-4, Andy Xue wrote:

    ok -- i got something to pass tests. just had to keep honing down the
    definition of what an unnecessary predicate really is.

    what do ya'll think? how would i go about requesting a PR?

    --- a/cascalog-core/src/clj/cascalog/logic/parse.clj
    +++ b/cascalog-core/src/clj/cascalog/logic/parse.clj
    @@ -642,6 +642,32 @@ This won't work in distributed mode because of the
    ->Record functions."
    (assoc :available-fields fields)
    (assoc :ground? (v/fully-ground? fields)))))

    +(defn prune-operations
    + "Remove any operations whose output is not used; ie, output does not
    intersect with the set
    + of fields that is the query's out-fields, generators (joins), inputs of
    other operations,
    + and :sort option. Additionally, don't try to prune operations if a no
    input operator exists"
    + [fields grouped options]
    + (let [check-input-operations (concat (grouped Operation)
    + (grouped FilterOperation)
    + (grouped Aggregator))]
    + (if (some #(empty? (:input %)) check-input-operations)
    + (concat (grouped Operation)
    + (grouped FilterOperation))
    + (let [generators (concat (grouped Generator)
    + (grouped GeneratorSet))
    + nessecary-fields (->> check-input-operations
    + (mapcat :input)
    + (concat fields)
    + (concat (mapcat :fields generators))
    + (concat (:sort options)))
    + pruned-operations (filter #(seq
    + (clojure.set/intersection
    + (set (:output %))
    + (set nessecary-fields)))
    + (grouped Operation))]
    + (concat pruned-operations
    + (grouped FilterOperation))))))
    +
    (defn build-rule
    [{:keys [fields predicates options] :as input}]
    (let [[nodes expanded] (s/separate #(tail? (:op %)) predicates)
    @@ -650,8 +676,7 @@ This won't work in distributed mode because of the
    ->Record functions."
    (group-by type))
    generators (concat (grouped Generator)
    (grouped GeneratorSet))
    - operations (concat (grouped Operation)
    - (grouped FilterOperation))
    + operations (prune-operations fields grouped options)
    aggs (grouped Aggregator)
    tails (concat (initial-tails generators operations)
    (map (fn [{:keys [op output]}]
    On Wednesday, April 23, 2014 3:38:48 PM UTC-4, Andy Xue wrote:

    ok -- played around with this for a little bit, and I think there were a
    few usages of cascalog that I haven't really encountered, and those tests
    fail. interesting!

    .. cascalog.logic.parse/build-rule ...
    generators (concat (grouped Generator)
    (grouped GeneratorSet))
    - operations (concat (grouped Operation)
    + nessecary-fields (->> (concat (grouped Operation)
    + (grouped FilterOperation)
    + (grouped Aggregator))
    + (mapcat :input)
    + (concat fields)
    + (concat (:sort options)))
    + pruned-operations (filter #(clojure.set/subset? (set (:output
    %)) (set nessecary-fields)) (grouped Operation))
    + operations (concat pruned-operations
    (grouped FilterOperation))
    aggs (grouped Aggregator)
    tails (concat (initial-tails generators operations)

    one test in particular that fails ...

    (defmapfn mk-one
    "Returns 1 for any input."
    [& tuple] 1)

    (deftest test-no-input
    (let [nums [[1] [2] [3]]]
    ...
    (test?<- [[1 1] [1 2] [1 3]
    [2 1] [2 2] [2 3]
    [3 1] [3 2] [3 3]]
    [?n ?n3]
    (nums ?n)
    (mk-one :> ?n2)
    (nums ?n3))))

    I am still not really clear what (mk-one :> ?n2) is doing/why it is
    necessary. I think before running this test I would have thought that
    predicate isn't needed.


    On Tuesday, April 22, 2014 8:19:00 PM UTC-4, Andy Xue wrote:

    ha no, i just typed that out as a description of the logic. not sure if
    this is the best place to put it.
    On Tuesday, April 22, 2014 8:08:06 PM UTC-4, Soren Macbeth wrote:

    do all the tests still pass with that in place?

    On Tue, Apr 22, 2014 at 5:03 PM, Andy Xue wrote:

    what do you guys think of adding a filtering step in build-query?

    (defn filter-predicate-by-output-fields
    [output-fields {:keys [op input output] :as pred}]
    (clojure.set/subset? (set output) (set output-fields))

    (defn filter-predicates-by-output-fields
    [output-fields raw-predicates]
    (fitler (partial filter-predicate-by-output-fields output-fields)
    raw-predicates)

    (defn build-query
    [output-fields raw-predicates]
    (let [[options predicates] (opts/extract-options raw-predicates)
    expanded (mapcat expand-outvars predicates)
    filtered (filter-predicates-by-output-fields output-fields
    expanded)]
    (validate-predicates! filtered options)
    (p/RawSubquery. output-fields filtered options)))


    On Tuesday, April 22, 2014 6:44:09 PM UTC-4, Andy Xue wrote:

    Hi all -- one thing we have been doing a lot of is instead of using
    the query macro we dynamically build out predicates and use construct. The
    reason is that often we want to make sure certain expensive predicates
    don't get evaluated if the predicates 'out var' isn't requested.

    so for example -- the follow should not print out "minus" since we
    don't actually need it.

    (use 'cascalog.api)
    (def src [[1 2] [3 4] [5 8]])

    (defn my-add [a b] (do (println "adding") (+ a b)))

    (defn my-minus [a b] (do (println "minus") (- a b)))

    (?<- (stdout)
    [!add]
    (src :> !a !b)
    (my-add !a !b :> !add)
    (my-minus !a !b :> !minus))

    print outs:

    adding
    minus
    adding
    minus
    adding
    minus


    RESULTS
    -----------------------
    3
    7
    13
    -----------------------

    anyone else have to deal with this?
    --
    You received this message because you are subscribed to the Google
    Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to cascalog-use...@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.


    --
    http://about.me/soren
    --
    You received this message because you are subscribed to the Google Groups "cascalog-user" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to cascalog-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/d/optout.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcascalog-user @
categoriesclojure, hadoop
postedApr 22, '14 at 10:44p
activeApr 23, '14 at 11:37p
posts8
users2
websiteclojure.org
irc#clojure

2 users in discussion

Andy Xue: 6 posts Soren Macbeth: 2 posts

People

Translate

site design / logo © 2021 Grokbase