Grokbase › Groups › Hadoop › impala-user › March 2013
FAQ
It's very likely that you ran out of memory. To verify, please rerun
this and simultaneously watch the memory consumption of the impalad
process.
On Fri, Mar 15, 2013 at 2:38 AM, FU Tianyuan wrote:
Hi all,

I launch TPC-H on my pseudo-cluster(CentOS 6.3 64bit, 3GB ram) with 1GB
dataset generated by DBGEN.
I can execute all the querys in tpc-h successfully, but when I query q9, the
process seems blocked and impala-server will shut down at last.
these are the log files:

****************************************************
13/03/15 16:52:03 INFO service.Frontend: analyze query select nation,
o_year, sum(amount) as sum_profit
from
(
select
n_name as nation, year(o_orderdate) as o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
from
orders o join
(select l_extendedprice, l_discount, l_quantity, l_orderkey, n_name,
ps_supplycost
from part p join
(select l_extendedprice, l_discount, l_quantity, l_partkey, l_orderkey,
n_name, ps_supplycost
from partsupp ps join
(select l_suppkey, l_extendedprice, l_discount, l_quantity, l_partkey,
l_orderkey, n_name
from
(select s_suppkey, n_name
from nation n join supplier s on n.n_nationkey = s.s_nationkey
) s1 join lineitem l on s1.s_suppkey = l.l_suppkey
) l1 on ps.ps_suppkey = l1.l_suppkey and ps.ps_partkey = l1.l_partkey
) l2 on p.p_name like '%green%' and p.p_partkey = l2.l_partkey
) l3 on o.o_orderkey = l3.l_orderkey
)profit
group by nation, o_year
order by nation, o_year desc
LIMIT 2147483647
13/03/15 16:52:04 INFO analysis.AggregateInfo: agg info:
AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=nation, id=39}
SlotRef{tblName=null, col=o_year, id=40}),
aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
(SlotRef{tblName=null, col=amount, id=41})}),
agg_tuple=TupleDescriptor{id=11, tbl=null, byte_size=0,
is_materialized=true, slots=[SlotDescriptor{id=42, col=null, type=STRING,
materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=43, col=null, type=INT,
materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=44, col=null, type=DOUBLE,
materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
col=nation, id=39}:SlotRef{tblName=null, col=null, id=42}
SlotRef{tblName=null, col=o_year, id=40}:SlotRef{tblName=null, col=null,
id=43} AggregateExpr{op=SUM, isStar=false, isDistinct=false,
(SlotRef{tblName=null, col=amount, id=41})}:SlotRef{tblName=null, col=null,
id=44})}
mergeAggInfo:
AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=42}
SlotRef{tblName=null, col=null, id=43}),
aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
(SlotRef{tblName=null, col=null, id=44})}), agg_tuple=TupleDescriptor{id=11,
tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=42,
col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=43,
col=null, type=INT, materialized=false, byteSize=0, byteOffset=-1,
nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=44,
col=null, type=DOUBLE, materialized=false, byteSize=0, byteOffset=-1,
nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
smap=smap(SlotRef{tblName=null, col=nation, id=39}:SlotRef{tblName=null,
col=null, id=42} SlotRef{tblName=null, col=o_year,
id=40}:SlotRef{tblName=null, col=null, id=43} AggregateExpr{op=SUM,
isStar=false, isDistinct=false, (SlotRef{tblName=null, col=amount,
id=41})}:SlotRef{tblName=null, col=null, id=44})}
13/03/15 16:52:04 INFO service.Frontend: create plan
13/03/15 16:52:04 INFO planner.Planner: create single-node plan
13/03/15 16:52:04 INFO planner.Planner: create plan fragments
13/03/15 16:52:04 INFO planner.Planner: finalize plan fragments
13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
supplier
13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
lineitem
13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
nation
13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
partsupp
13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
part
13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
orders
13/03/15 16:52:04 INFO service.Frontend: get scan range locations
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
PartitionBlockMetadata{#blocks=3, #filenames=1, totalStringLen=59}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
PartitionBlockMetadata{#blocks=3, #filenames=1, totalStringLen=59}
13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
CacheStats{hitCount=4, missCount=31, loadSuccessCount=31,
loadExceptionCount=0, totalLoadTime=1198211478, evictionCount=0}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=55}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=55}
13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
CacheStats{hitCount=4, missCount=32, loadSuccessCount=32,
loadExceptionCount=0, totalLoadTime=1206078498, evictionCount=0}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
PartitionBlockMetadata{#blocks=2, #filenames=1, totalStringLen=63}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
PartitionBlockMetadata{#blocks=2, #filenames=1, totalStringLen=63}
13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
CacheStats{hitCount=4, missCount=33, loadSuccessCount=33,
loadExceptionCount=0, totalLoadTime=1215564320, evictionCount=0}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=59}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=59}
13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
CacheStats{hitCount=4, missCount=34, loadSuccessCount=34,
loadExceptionCount=0, totalLoadTime=1224243116, evictionCount=0}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
PartitionBlockMetadata{#blocks=12, #filenames=1, totalStringLen=63}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
PartitionBlockMetadata{#blocks=12, #filenames=1, totalStringLen=63}
13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
CacheStats{hitCount=4, missCount=35, loadSuccessCount=35,
loadExceptionCount=0, totalLoadTime=1236160423, evictionCount=0}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=63}
13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=63}
13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
CacheStats{hitCount=4, missCount=36, loadSuccessCount=36,
loadExceptionCount=0, totalLoadTime=1246694017, evictionCount=0}
13/03/15 16:52:04 INFO service.Frontend: create result set metadata
13/03/15 16:52:04 INFO service.JniFrontend: Plan Fragment 0
UNPARTITIONED
TOP-N
ORDER BY: <slot 42> ASC, <slot 43> DESC
LIMIT: 2147483647
TUPLE IDS: 11
AGGREGATE
OUTPUT: SUM(<slot 44>)
GROUP BY: <slot 42>, <slot 43>
TUPLE IDS: 11
EXCHANGE (18)
TUPLE IDS: 11

Plan Fragment 1
RANDOM
STREAM DATA SINK
EXCHANGE ID: 18
UNPARTITIONED

AGGREGATE
OUTPUT: SUM(l_extendedprice * 1.0 - l_discount - ps_supplycost *
l_quantity)
GROUP BY: n_name, year(o_orderdate)
TUPLE IDS: 11
HASH JOIN
JOIN OP: INNER JOIN
HASH PREDICATES:
o.o_orderkey = l_orderkey
TUPLE IDS: 0 1 2 3 4 6
SCAN HDFS table=default.orders #partitions=1 size=163.99MB (0)
TUPLE IDS: 0
EXCHANGE (17)
TUPLE IDS: 1 2 3 4 6

Plan Fragment 2
RANDOM
STREAM DATA SINK
EXCHANGE ID: 17
UNPARTITIONED

HASH JOIN
JOIN OP: INNER JOIN
HASH PREDICATES:
p.p_partkey = l_partkey
TUPLE IDS: 1 2 3 4 6
SCAN HDFS table=default.part #partitions=1 size=23.02MB (1) compact
PREDICATES: p.p_name LIKE '%green%'
TUPLE IDS: 1
EXCHANGE (16)
TUPLE IDS: 2 3 4 6

Plan Fragment 3
RANDOM
STREAM DATA SINK
EXCHANGE ID: 16
UNPARTITIONED

HASH JOIN
JOIN OP: INNER JOIN
HASH PREDICATES:
ps.ps_suppkey = l_suppkey
ps.ps_partkey = l_partkey
TUPLE IDS: 2 3 4 6
SCAN HDFS table=default.partsupp #partitions=1 size=113.47MB (2)
compact
TUPLE IDS: 2
EXCHANGE (15)
TUPLE IDS: 3 4 6

Plan Fragment 4
RANDOM
STREAM DATA SINK
EXCHANGE ID: 15
UNPARTITIONED

HASH JOIN
JOIN OP: INNER JOIN
HASH PREDICATES:
s_suppkey = l.l_suppkey
TUPLE IDS: 3 4 6
HASH JOIN
JOIN OP: INNER JOIN
HASH PREDICATES:
n.n_nationkey = s.s_nationkey
TUPLE IDS: 3 4
SCAN HDFS table=default.nation #partitions=1 size=2.17KB (3)
compact
TUPLE IDS: 3
EXCHANGE (13)
TUPLE IDS: 4
EXCHANGE (14)
TUPLE IDS: 6

Plan Fragment 5
RANDOM
STREAM DATA SINK
EXCHANGE ID: 14
UNPARTITIONED

SCAN HDFS table=default.lineitem #partitions=1 size=724.66MB (6) compact
TUPLE IDS: 6

Plan Fragment 6
RANDOM
STREAM DATA SINK
EXCHANGE ID: 13
UNPARTITIONED

SCAN HDFS table=default.supplier #partitions=1 size=1.34MB (4) compact
TUPLE IDS: 4

#
# A fatal error has been detected by the Java Runtime Environment:
[thread 140571726599936 also had an error]#

# [thread 140571759159040 also had an error]
SIGSEGV (0xb) at pc=0x0000003107c69e73, pid=7127, tid=140571747579648
#
# JRE version: 6.0_43-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode
linux-amd64 compressed oops)
# Problematic frame:
# C [libstdc++.so.6+0x69e73]
std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x13
#
# An error report file with more information is saved as:
# /var/run/impala/hs_err_pid7127.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
****************************************************

Is this because my RAM is not enough or some reason else ? Who can tell me
what's wrong with this? Thank you!

Best,
Tianyuan

Search Discussions

  • Marcel Kornacker at Mar 18, 2013 at 2:20 am ⇧
    I should have pointed out: version 0.7 will let you set per-process
    and per-query memory limits, which will prevent a query like this from
    crashing the server.
    On Sun, Mar 17, 2013 at 7:19 PM, Marcel Kornacker wrote:
    It's very likely that you ran out of memory. To verify, please rerun
    this and simultaneously watch the memory consumption of the impalad
    process.
    On Fri, Mar 15, 2013 at 2:38 AM, FU Tianyuan wrote:
    Hi all,

    I launch TPC-H on my pseudo-cluster(CentOS 6.3 64bit, 3GB ram) with 1GB
    dataset generated by DBGEN.
    I can execute all the querys in tpc-h successfully, but when I query q9, the
    process seems blocked and impala-server will shut down at last.
    these are the log files:

    ****************************************************
    13/03/15 16:52:03 INFO service.Frontend: analyze query select nation,
    o_year, sum(amount) as sum_profit
    from
    (
    select
    n_name as nation, year(o_orderdate) as o_year,
    l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
    from
    orders o join
    (select l_extendedprice, l_discount, l_quantity, l_orderkey, n_name,
    ps_supplycost
    from part p join
    (select l_extendedprice, l_discount, l_quantity, l_partkey, l_orderkey,
    n_name, ps_supplycost
    from partsupp ps join
    (select l_suppkey, l_extendedprice, l_discount, l_quantity, l_partkey,
    l_orderkey, n_name
    from
    (select s_suppkey, n_name
    from nation n join supplier s on n.n_nationkey = s.s_nationkey
    ) s1 join lineitem l on s1.s_suppkey = l.l_suppkey
    ) l1 on ps.ps_suppkey = l1.l_suppkey and ps.ps_partkey = l1.l_partkey
    ) l2 on p.p_name like '%green%' and p.p_partkey = l2.l_partkey
    ) l3 on o.o_orderkey = l3.l_orderkey
    )profit
    group by nation, o_year
    order by nation, o_year desc
    LIMIT 2147483647
    13/03/15 16:52:04 INFO analysis.AggregateInfo: agg info:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=nation, id=39}
    SlotRef{tblName=null, col=o_year, id=40}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=amount, id=41})}),
    agg_tuple=TupleDescriptor{id=11, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=42, col=null, type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=43, col=null, type=INT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=44, col=null, type=DOUBLE,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
    col=nation, id=39}:SlotRef{tblName=null, col=null, id=42}
    SlotRef{tblName=null, col=o_year, id=40}:SlotRef{tblName=null, col=null,
    id=43} AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=amount, id=41})}:SlotRef{tblName=null, col=null,
    id=44})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=42}
    SlotRef{tblName=null, col=null, id=43}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=44})}), agg_tuple=TupleDescriptor{id=11,
    tbl=null, byte_size=0, is_materialized=true, slots=[SlotDescriptor{id=42,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=43,
    col=null, type=INT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=44,
    col=null, type=DOUBLE, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=nation, id=39}:SlotRef{tblName=null,
    col=null, id=42} SlotRef{tblName=null, col=o_year,
    id=40}:SlotRef{tblName=null, col=null, id=43} AggregateExpr{op=SUM,
    isStar=false, isDistinct=false, (SlotRef{tblName=null, col=amount,
    id=41})}:SlotRef{tblName=null, col=null, id=44})}
    13/03/15 16:52:04 INFO service.Frontend: create plan
    13/03/15 16:52:04 INFO planner.Planner: create single-node plan
    13/03/15 16:52:04 INFO planner.Planner: create plan fragments
    13/03/15 16:52:04 INFO planner.Planner: finalize plan fragments
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    supplier
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    lineitem
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    nation
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    partsupp
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    part
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    orders
    13/03/15 16:52:04 INFO service.Frontend: get scan range locations
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=3, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=3, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=31, loadSuccessCount=31,
    loadExceptionCount=0, totalLoadTime=1198211478, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=55}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=55}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=32, loadSuccessCount=32,
    loadExceptionCount=0, totalLoadTime=1206078498, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=2, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=2, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=33, loadSuccessCount=33,
    loadExceptionCount=0, totalLoadTime=1215564320, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=34, loadSuccessCount=34,
    loadExceptionCount=0, totalLoadTime=1224243116, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=12, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=12, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=35, loadSuccessCount=35,
    loadExceptionCount=0, totalLoadTime=1236160423, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=36, loadSuccessCount=36,
    loadExceptionCount=0, totalLoadTime=1246694017, evictionCount=0}
    13/03/15 16:52:04 INFO service.Frontend: create result set metadata
    13/03/15 16:52:04 INFO service.JniFrontend: Plan Fragment 0
    UNPARTITIONED
    TOP-N
    ORDER BY: <slot 42> ASC, <slot 43> DESC
    LIMIT: 2147483647
    TUPLE IDS: 11
    AGGREGATE
    OUTPUT: SUM(<slot 44>)
    GROUP BY: <slot 42>, <slot 43>
    TUPLE IDS: 11
    EXCHANGE (18)
    TUPLE IDS: 11

    Plan Fragment 1
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 18
    UNPARTITIONED

    AGGREGATE
    OUTPUT: SUM(l_extendedprice * 1.0 - l_discount - ps_supplycost *
    l_quantity)
    GROUP BY: n_name, year(o_orderdate)
    TUPLE IDS: 11
    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    o.o_orderkey = l_orderkey
    TUPLE IDS: 0 1 2 3 4 6
    SCAN HDFS table=default.orders #partitions=1 size=163.99MB (0)
    TUPLE IDS: 0
    EXCHANGE (17)
    TUPLE IDS: 1 2 3 4 6

    Plan Fragment 2
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 17
    UNPARTITIONED

    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    p.p_partkey = l_partkey
    TUPLE IDS: 1 2 3 4 6
    SCAN HDFS table=default.part #partitions=1 size=23.02MB (1) compact
    PREDICATES: p.p_name LIKE '%green%'
    TUPLE IDS: 1
    EXCHANGE (16)
    TUPLE IDS: 2 3 4 6

    Plan Fragment 3
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 16
    UNPARTITIONED

    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    ps.ps_suppkey = l_suppkey
    ps.ps_partkey = l_partkey
    TUPLE IDS: 2 3 4 6
    SCAN HDFS table=default.partsupp #partitions=1 size=113.47MB (2)
    compact
    TUPLE IDS: 2
    EXCHANGE (15)
    TUPLE IDS: 3 4 6

    Plan Fragment 4
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 15
    UNPARTITIONED

    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    s_suppkey = l.l_suppkey
    TUPLE IDS: 3 4 6
    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    n.n_nationkey = s.s_nationkey
    TUPLE IDS: 3 4
    SCAN HDFS table=default.nation #partitions=1 size=2.17KB (3)
    compact
    TUPLE IDS: 3
    EXCHANGE (13)
    TUPLE IDS: 4
    EXCHANGE (14)
    TUPLE IDS: 6

    Plan Fragment 5
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 14
    UNPARTITIONED

    SCAN HDFS table=default.lineitem #partitions=1 size=724.66MB (6) compact
    TUPLE IDS: 6

    Plan Fragment 6
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 13
    UNPARTITIONED

    SCAN HDFS table=default.supplier #partitions=1 size=1.34MB (4) compact
    TUPLE IDS: 4

    #
    # A fatal error has been detected by the Java Runtime Environment:
    [thread 140571726599936 also had an error]#

    # [thread 140571759159040 also had an error]
    SIGSEGV (0xb) at pc=0x0000003107c69e73, pid=7127, tid=140571747579648
    #
    # JRE version: 6.0_43-b01
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode
    linux-amd64 compressed oops)
    # Problematic frame:
    # C [libstdc++.so.6+0x69e73]
    std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x13
    #
    # An error report file with more information is saved as:
    # /var/run/impala/hs_err_pid7127.log
    #
    # If you would like to submit a bug report, please visit:
    # http://java.sun.com/webapps/bugreport/crash.jsp
    #
    ****************************************************

    Is this because my RAM is not enough or some reason else ? Who can tell me
    what's wrong with this? Thank you!

    Best,
    Tianyuan
  • Alan Choi at Mar 18, 2013 at 5:42 pm ⇧
    Hi,

    It's also worth looking at the join order. For best results:

    - Join the biggest table first.
    - Join subsequent tables according to which table has the most selective
    filter. Joining the table with the most selective filter results in the
    fewest number of rows are being returned.

    https://ccp.cloudera.com/display/IMPALA10BETADOC/Language+Reference

    Thanks,
    Alan

    On Sun, Mar 17, 2013 at 7:37 PM, FU Tianyuan wrote:

    OK, I will try it again.
    Thank you Marcel.


    2013/3/18 Marcel Kornacker <marcel@cloudera.com>
    It's very likely that you ran out of memory. To verify, please rerun
    this and simultaneously watch the memory consumption of the impalad
    process.
    On Fri, Mar 15, 2013 at 2:38 AM, FU Tianyuan wrote:
    Hi all,

    I launch TPC-H on my pseudo-cluster(CentOS 6.3 64bit, 3GB ram) with 1GB
    dataset generated by DBGEN.
    I can execute all the querys in tpc-h successfully, but when I query q9, the
    process seems blocked and impala-server will shut down at last.
    these are the log files:

    ****************************************************
    13/03/15 16:52:03 INFO service.Frontend: analyze query select nation,
    o_year, sum(amount) as sum_profit
    from
    (
    select
    n_name as nation, year(o_orderdate) as o_year,
    l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
    from
    orders o join
    (select l_extendedprice, l_discount, l_quantity, l_orderkey, n_name,
    ps_supplycost
    from part p join
    (select l_extendedprice, l_discount, l_quantity, l_partkey, l_orderkey,
    n_name, ps_supplycost
    from partsupp ps join
    (select l_suppkey, l_extendedprice, l_discount, l_quantity, l_partkey,
    l_orderkey, n_name
    from
    (select s_suppkey, n_name
    from nation n join supplier s on n.n_nationkey = s.s_nationkey
    ) s1 join lineitem l on s1.s_suppkey = l.l_suppkey
    ) l1 on ps.ps_suppkey = l1.l_suppkey and ps.ps_partkey = l1.l_partkey
    ) l2 on p.p_name like '%green%' and p.p_partkey = l2.l_partkey
    ) l3 on o.o_orderkey = l3.l_orderkey
    )profit
    group by nation, o_year
    order by nation, o_year desc
    LIMIT 2147483647
    13/03/15 16:52:04 INFO analysis.AggregateInfo: agg info:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=nation, id=39}
    SlotRef{tblName=null, col=o_year, id=40}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=amount, id=41})}),
    agg_tuple=TupleDescriptor{id=11, tbl=null, byte_size=0,
    is_materialized=true, slots=[SlotDescriptor{id=42, col=null,
    type=STRING,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=43, col=null, type=INT,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}, SlotDescriptor{id=44, col=null,
    type=DOUBLE,
    materialized=false, byteSize=0, byteOffset=-1, nullIndicatorByte=0,
    nullIndicatorBit=0, slotIdx=0}]}, smap=smap(SlotRef{tblName=null,
    col=nation, id=39}:SlotRef{tblName=null, col=null, id=42}
    SlotRef{tblName=null, col=o_year, id=40}:SlotRef{tblName=null, col=null,
    id=43} AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=amount, id=41})}:SlotRef{tblName=null, col=null,
    id=44})}
    mergeAggInfo:
    AggregateInfo{grouping_exprs=(SlotRef{tblName=null, col=null, id=42}
    SlotRef{tblName=null, col=null, id=43}),
    aggregate_exprs=(AggregateExpr{op=SUM, isStar=false, isDistinct=false,
    (SlotRef{tblName=null, col=null, id=44})}),
    agg_tuple=TupleDescriptor{id=11,
    tbl=null, byte_size=0, is_materialized=true,
    slots=[SlotDescriptor{id=42,
    col=null, type=STRING, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0},
    SlotDescriptor{id=43,
    col=null, type=INT, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0},
    SlotDescriptor{id=44,
    col=null, type=DOUBLE, materialized=false, byteSize=0, byteOffset=-1,
    nullIndicatorByte=0, nullIndicatorBit=0, slotIdx=0}]},
    smap=smap(SlotRef{tblName=null, col=nation, id=39}:SlotRef{tblName=null,
    col=null, id=42} SlotRef{tblName=null, col=o_year,
    id=40}:SlotRef{tblName=null, col=null, id=43} AggregateExpr{op=SUM,
    isStar=false, isDistinct=false, (SlotRef{tblName=null, col=amount,
    id=41})}:SlotRef{tblName=null, col=null, id=44})}
    13/03/15 16:52:04 INFO service.Frontend: create plan
    13/03/15 16:52:04 INFO planner.Planner: create single-node plan
    13/03/15 16:52:04 INFO planner.Planner: create plan fragments
    13/03/15 16:52:04 INFO planner.Planner: finalize plan fragments
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    supplier
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    lineitem
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    nation
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    partsupp
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    part
    13/03/15 16:52:04 INFO planner.HdfsScanNode: collecting partitions for table
    orders
    13/03/15 16:52:04 INFO service.Frontend: get scan range locations
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=3, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=3, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=31, loadSuccessCount=31,
    loadExceptionCount=0, totalLoadTime=1198211478, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=55}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=55}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=32, loadSuccessCount=32,
    loadExceptionCount=0, totalLoadTime=1206078498, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=2, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=2, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=33, loadSuccessCount=33,
    loadExceptionCount=0, totalLoadTime=1215564320, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=59}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=34, loadSuccessCount=34,
    loadExceptionCount=0, totalLoadTime=1224243116, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=12, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=12, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=35, loadSuccessCount=35,
    loadExceptionCount=0, totalLoadTime=1236160423, evictionCount=0}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded partiton
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: loaded disk ids for
    PartitionBlockMetadata{#blocks=1, #filenames=1, totalStringLen=63}
    13/03/15 16:52:04 INFO catalog.HdfsTable: block metadata cache:
    CacheStats{hitCount=4, missCount=36, loadSuccessCount=36,
    loadExceptionCount=0, totalLoadTime=1246694017, evictionCount=0}
    13/03/15 16:52:04 INFO service.Frontend: create result set metadata
    13/03/15 16:52:04 INFO service.JniFrontend: Plan Fragment 0
    UNPARTITIONED
    TOP-N
    ORDER BY: <slot 42> ASC, <slot 43> DESC
    LIMIT: 2147483647
    TUPLE IDS: 11
    AGGREGATE
    OUTPUT: SUM(<slot 44>)
    GROUP BY: <slot 42>, <slot 43>
    TUPLE IDS: 11
    EXCHANGE (18)
    TUPLE IDS: 11

    Plan Fragment 1
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 18
    UNPARTITIONED

    AGGREGATE
    OUTPUT: SUM(l_extendedprice * 1.0 - l_discount - ps_supplycost *
    l_quantity)
    GROUP BY: n_name, year(o_orderdate)
    TUPLE IDS: 11
    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    o.o_orderkey = l_orderkey
    TUPLE IDS: 0 1 2 3 4 6
    SCAN HDFS table=default.orders #partitions=1 size=163.99MB (0)
    TUPLE IDS: 0
    EXCHANGE (17)
    TUPLE IDS: 1 2 3 4 6

    Plan Fragment 2
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 17
    UNPARTITIONED

    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    p.p_partkey = l_partkey
    TUPLE IDS: 1 2 3 4 6
    SCAN HDFS table=default.part #partitions=1 size=23.02MB (1) compact
    PREDICATES: p.p_name LIKE '%green%'
    TUPLE IDS: 1
    EXCHANGE (16)
    TUPLE IDS: 2 3 4 6

    Plan Fragment 3
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 16
    UNPARTITIONED

    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    ps.ps_suppkey = l_suppkey
    ps.ps_partkey = l_partkey
    TUPLE IDS: 2 3 4 6
    SCAN HDFS table=default.partsupp #partitions=1 size=113.47MB (2)
    compact
    TUPLE IDS: 2
    EXCHANGE (15)
    TUPLE IDS: 3 4 6

    Plan Fragment 4
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 15
    UNPARTITIONED

    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    s_suppkey = l.l_suppkey
    TUPLE IDS: 3 4 6
    HASH JOIN
    JOIN OP: INNER JOIN
    HASH PREDICATES:
    n.n_nationkey = s.s_nationkey
    TUPLE IDS: 3 4
    SCAN HDFS table=default.nation #partitions=1 size=2.17KB (3)
    compact
    TUPLE IDS: 3
    EXCHANGE (13)
    TUPLE IDS: 4
    EXCHANGE (14)
    TUPLE IDS: 6

    Plan Fragment 5
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 14
    UNPARTITIONED

    SCAN HDFS table=default.lineitem #partitions=1 size=724.66MB (6) compact
    TUPLE IDS: 6

    Plan Fragment 6
    RANDOM
    STREAM DATA SINK
    EXCHANGE ID: 13
    UNPARTITIONED

    SCAN HDFS table=default.supplier #partitions=1 size=1.34MB (4) compact
    TUPLE IDS: 4

    #
    # A fatal error has been detected by the Java Runtime Environment:
    [thread 140571726599936 also had an error]#

    # [thread 140571759159040 also had an error]
    SIGSEGV (0xb) at pc=0x0000003107c69e73, pid=7127, tid=140571747579648
    #
    # JRE version: 6.0_43-b01
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode
    linux-amd64 compressed oops)
    # Problematic frame:
    # C [libstdc++.so.6+0x69e73]
    std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x13
    #
    # An error report file with more information is saved as:
    # /var/run/impala/hs_err_pid7127.log
    #
    # If you would like to submit a bug report, please visit:
    # http://java.sun.com/webapps/bugreport/crash.jsp
    #
    ****************************************************

    Is this because my RAM is not enough or some reason else ? Who can tell me
    what's wrong with this? Thank you!

    Best,
    Tianyuan

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimpala-user @
categorieshadoop
postedMar 18, '13 at 2:19a
activeMar 18, '13 at 5:42p
posts3
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Marcel Kornacker: 2 posts Alan Choi: 1 post

Content

People

Support

Translate

site design / logo © 2022 Grokbase