Grokbase Groups Hive user May 2011
FAQ
Hi all,

I want to custom the delimiter of the table in a row.
Like my data format is '1&&&&2&&&&4‘, and how could I create a table (int,
int, int)

Thanks.

--
dujinhang

Search Discussions

  • Bejoy_ks at May 25, 2011 at 12:21 pm
    Hi Jinhang
    I don't think hive supports multi character delimiters. The hassle free option here would be to preprocess the data using mapreduce to replace the multi character delimiter with another permissible one that suits your data.
    Regards
    Bejoy K S

    -----Original Message-----
    From: jinhang du <dujinhang@gmail.com>
    Date: Wed, 25 May 2011 19:56:16
    To: <user@hive.apache.org>
    Reply-To: user@hive.apache.org
    Subject: Hive create table

    Hi all,

    I want to custom the delimiter of the table in a row.
    Like my data format is '1&&&&2&&&&4‘, and how could I create a table (int,
    int, int)

    Thanks.

    --
    dujinhang
  • Valentina kroshilina at May 25, 2011 at 1:58 pm
    you can use something like this:

    CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
    str1 int,
    str2 int,
    str3 int
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    With SERDEPROPERTIES (
    "input.regex"="(\\d+)&&&&(\\d+)&&&&(\\d+)"
    )
    LOCATION '/path1;

    2011/5/25 jinhang du <dujinhang@gmail.com>
    Hi all,

    I want to custom the delimiter of the table in a row.
    Like my data format is '1&&&&2&&&&4‘, and how could I create a table (int,
    int, int)

    Thanks.

    --
    dujinhang


    --
    valentina kroshilina
  • Jinhang du at May 26, 2011 at 1:51 am
    Thanks for your information. It works.

    在 2011年5月25日 下午9:57,valentina kroshilina <kroshilina@gmail.com>写道:
    you can use something like this:

    CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
    str1 int,
    str2 int,
    str3 int
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    With SERDEPROPERTIES (
    "input.regex"="(\\d+)&&&&(\\d+)&&&&(\\d+)"
    )
    LOCATION '/path1;


    2011/5/25 jinhang du <dujinhang@gmail.com>
    Hi all,

    I want to custom the delimiter of the table in a row.
    Like my data format is '1&&&&2&&&&4‘, and how could I create a table (int,
    int, int)

    Thanks.

    --
    dujinhang


    --
    valentina kroshilina


    --
    dujinhang
  • Jinhang du at May 27, 2011 at 12:07 pm
    Are there any documents to help me understand the meaning and usage of
    "input.regex"?

    I create table as your introduction. And "select * from table1" returns the
    right answer.
    However, "select table1.str1 from table1" gains the following exception.

    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201104251139_2123, Tracking URL =
    http://hadoop-12.apexlab.org:50030/jobdetails.jsp?jobid=job_201104251139_2123
    Kill Command = /usr/lib/hadoop/bin/hadoop job
    -Dmapred.job.tracker=hadoop-12:9001 -kill job_201104251139_2123
    2011-05-27 20:04:41,618 Stage-1 map = 0%, reduce = 0%
    2011-05-27 20:05:05,413 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104251139_2123 with errors
    FAILED: Execution Error, return code 2 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    Can you help me? thanks.



    在 2011年5月25日 下午9:57,valentina kroshilina <kroshilina@gmail.com>写道:
    you can use something like this:

    CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
    str1 int,
    str2 int,
    str3 int
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    With SERDEPROPERTIES (
    "input.regex"="(\\d+)&&&&(\\d+)&&&&(\\d+)"
    )
    LOCATION '/path1;


    2011/5/25 jinhang du <dujinhang@gmail.com>
    Hi all,

    I want to custom the delimiter of the table in a row.
    Like my data format is '1&&&&2&&&&4‘, and how could I create a table (int,
    int, int)

    Thanks.

    --
    dujinhang


    --
    valentina kroshilina


    --
    dujinhang
  • Bangbig at May 27, 2011 at 3:21 pm
    Check the log file, you can find more information about the error. The log file could be found from the URL given below.


    At 2011-05-27 20:07:17,"jinhang du" wrote:
    Are there any documents to help me understand the meaning and usage of "input.regex"?


    I create table as your introduction. And "select * from table1" returns the right answer.
    However, "select table1.str1 from table1" gains the following exception.


    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201104251139_2123, Tracking URL =http://hadoop-12.apexlab.org:50030/jobdetails.jsp?jobid=job_201104251139_2123
    Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=hadoop-12:9001 -kill job_201104251139_2123
    2011-05-27 20:04:41,618 Stage-1 map = 0%, reduce = 0%
    2011-05-27 20:05:05,413 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104251139_2123 with errors
    FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

    Can you help me? thanks.





    在 2011年5月25日 下午9:57,valentina kroshilina<kroshilina@gmail.com>写道:
    you can use something like this:

    CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
    str1 int,
    str2 int,
    str3 int
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    With SERDEPROPERTIES (
    "input.regex"="(\\d+)&&&&(\\d+)&&&&(\\d+)"
    )
    LOCATION '/path1;



    2011/5/25 jinhang du<dujinhang@gmail.com>
    Hi all,


    I want to custom the delimiter of the table in a row.
    Like my data format is '1&&&&2&&&&4‘, and how could I create a table (int, int, int)


    Thanks.

    --
    dujinhang





    --
    valentina kroshilina




    --
    dujinhang
  • Jinhang du at May 30, 2011 at 4:51 am
    Thanks for all your help.
    I fix my problem by editing the hive-site.xml.

    <property>
    <name>hive.aux.jars.path</name>
    <value>file:///usr/lib/hive/lib/hive-contrib-0.7.0-CDH3B4.jar</value>
    <description>These JAR file are available to all users for all
    jobs</description>
    </property>

    Now, I want to understand the "input.regex" . Is this Java regular
    expression?
    Thanks.


    在 2011年5月27日 下午11:21,bangbig <lizhonglianggood@163.com>写道:
    Check the log file, you can find more information about the error. The log
    file could be found from the URL given below.


    At 2011-05-27 20:07:17,"jinhang du" wrote:

    Are there any documents to help me understand the meaning and usage of
    "input.regex"?

    I create table as your introduction. And "select * from table1" returns the
    right answer.
    However, "select table1.str1 from table1" gains the following exception.

    Total MapReduce jobs = 1
    Launching Job 1 out of 1
    Number of reduce tasks is set to 0 since there's no reduce operator
    Starting Job = job_201104251139_2123, Tracking URL =
    http://hadoop-12.apexlab.org:50030/jobdetails.jsp?jobid=job_201104251139_2123
    Kill Command = /usr/lib/hadoop/bin/hadoop job
    -Dmapred.job.tracker=hadoop-12:9001 -kill job_201104251139_2123
    2011-05-27 20:04:41,618 Stage-1 map = 0%, reduce = 0%
    2011-05-27 20:05:05,413 Stage-1 map = 100%, reduce = 100%
    Ended Job = job_201104251139_2123 with errors
    FAILED: Execution Error, return code 2 from
    org.apache.hadoop.hive.ql.exec.MapRedTask

    Can you help me? thanks.



    在 2011年5月25日 下午9:57,valentina kroshilina <kroshilina@gmail.com>写道:
    you can use something like this:

    CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
    str1 int,
    str2 int,
    str3 int
    )
    ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
    With SERDEPROPERTIES (
    "input.regex"="(\\d+)&&&&(\\d+)&&&&(\\d+)"
    )
    LOCATION '/path1;


    2011/5/25 jinhang du <dujinhang@gmail.com>
    Hi all,

    I want to custom the delimiter of the table in a row.
    Like my data format is '1&&&&2&&&&4‘, and how could I create a table
    (int, int, int)

    Thanks.

    --
    dujinhang


    --
    valentina kroshilina


    --
    dujinhang



    --
    dujinhang

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedMay 25, '11 at 11:56a
activeMay 30, '11 at 4:51a
posts7
users4
websitehive.apache.org

People

Translate

site design / logo © 2022 Grokbase