FAQ
If I have a line like the following:

<2010-02-09 18:00:16.123 UTC>:[48394803]:<MDS-CS_MDS1>:<DEBUG>:<LAYER =
EP2P, EVENT = Receiving, DEVICEPIN = 2032acb14, GMETAG = -1966209606, TYPE =
22, METHOD = onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
EXP_TIMEOUT(S) = 3600, SIZE = 7312>

What would be the best way to store it into a table like this:

ts STRING "2010-02-09 18:00:16.123 UTC"
epochtime INT "345093824" // <-- I'm not sure how to do this
column either
requestId INT 48394803
component STRING "MDS-CS_MDS1"
log_level STRING "DEBUG"
properties STRING "LAYER = EP2P, EVENT = Receiving, DEVICEPIN =
2032acb14, GMETAG = -1966209606, TYPE = 22, METHOD =
onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
EXP_TIMEOUT(S) = 3600, SIZE = 7312"

Thanks,

Daniel

Search Discussions

  • Carl Steinbach at Feb 25, 2010 at 6:50 pm
    Hi Daniel,

    You can use the RegexSerDe to extract the fields embedded in the text. Try
    looking at the examples in
    contrib/src/test/queries/clientpositive/serde_regex.q

    Carl
    On Thu, Feb 25, 2010 at 8:44 AM, Daniel Joanes wrote:

    If I have a line like the following:

    <2010-02-09 18:00:16.123 UTC>:[48394803]:<MDS-CS_MDS1>:<DEBUG>:<LAYER =
    EP2P, EVENT = Receiving, DEVICEPIN = 2032acb14, GMETAG = -1966209606, TYPE =
    22, METHOD = onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
    EXP_TIMEOUT(S) = 3600, SIZE = 7312>

    What would be the best way to store it into a table like this:

    ts STRING "2010-02-09 18:00:16.123 UTC"
    epochtime INT "345093824" // <-- I'm not sure how to do this
    column either
    requestId INT 48394803
    component STRING "MDS-CS_MDS1"
    log_level STRING "DEBUG"
    properties STRING "LAYER = EP2P, EVENT = Receiving, DEVICEPIN =
    2032acb14, GMETAG = -1966209606, TYPE = 22, METHOD =
    onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
    EXP_TIMEOUT(S) = 3600, SIZE = 7312"

    Thanks,

    Daniel
  • Daniel Joanes at Feb 25, 2010 at 7:05 pm
    Awesome, that worked. From what I can tell the columns in my table have to
    be strings.. how would I use other data types?
    On Thu, Feb 25, 2010 at 1:49 PM, Carl Steinbach wrote:

    Hi Daniel,

    You can use the RegexSerDe to extract the fields embedded in the text. Try
    looking at the examples in
    contrib/src/test/queries/clientpositive/serde_regex.q

    Carl

    On Thu, Feb 25, 2010 at 8:44 AM, Daniel Joanes wrote:

    If I have a line like the following:

    <2010-02-09 18:00:16.123 UTC>:[48394803]:<MDS-CS_MDS1>:<DEBUG>:<LAYER =
    EP2P, EVENT = Receiving, DEVICEPIN = 2032acb14, GMETAG = -1966209606, TYPE =
    22, METHOD = onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
    EXP_TIMEOUT(S) = 3600, SIZE = 7312>

    What would be the best way to store it into a table like this:

    ts STRING "2010-02-09 18:00:16.123 UTC"
    epochtime INT "345093824" // <-- I'm not sure how to do this
    column either
    requestId INT 48394803
    component STRING "MDS-CS_MDS1"
    log_level STRING "DEBUG"
    properties STRING "LAYER = EP2P, EVENT = Receiving, DEVICEPIN =
    2032acb14, GMETAG = -1966209606, TYPE = 22, METHOD =
    onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
    EXP_TIMEOUT(S) = 3600, SIZE = 7312"

    Thanks,

    Daniel
  • Carl Steinbach at Feb 25, 2010 at 7:15 pm
    You can do a type conversion using the CAST UDF (while streaming the data
    from one table to another). See the documentation here:
    http://wiki.apache.org/hadoop/Hive/LanguageManual/UDF#Type_Conversion_Functions

    Carl
    On Thu, Feb 25, 2010 at 11:02 AM, Daniel Joanes wrote:

    Awesome, that worked. From what I can tell the columns in my table have to
    be strings.. how would I use other data types?

    On Thu, Feb 25, 2010 at 1:49 PM, Carl Steinbach wrote:

    Hi Daniel,

    You can use the RegexSerDe to extract the fields embedded in the text. Try
    looking at the examples in
    contrib/src/test/queries/clientpositive/serde_regex.q

    Carl

    On Thu, Feb 25, 2010 at 8:44 AM, Daniel Joanes wrote:

    If I have a line like the following:

    <2010-02-09 18:00:16.123 UTC>:[48394803]:<MDS-CS_MDS1>:<DEBUG>:<LAYER =
    EP2P, EVENT = Receiving, DEVICEPIN = 2032acb14, GMETAG = -1966209606, TYPE =
    22, METHOD = onInEp2p, DESTINATION = 24a69edf, CONFIRM = true,
    EXP_TIMEOUT(S) = 3600, SIZE = 7312>

    What would be the best way to store it into a table like this:

    ts STRING "2010-02-09 18:00:16.123 UTC"
    epochtime INT "345093824" // <-- I'm not sure how to do this
    column either
    requestId INT 48394803
    component STRING "MDS-CS_MDS1"
    log_level STRING "DEBUG"
    properties STRING "LAYER = EP2P, EVENT = Receiving, DEVICEPIN =
    2032acb14, GMETAG = -1966209606, TYPE = 22, METHOD =
    onInEp2p, DESTINATION = 24a69edf, CONFIRM =
    true, EXP_TIMEOUT(S) = 3600, SIZE = 7312"

    Thanks,

    Daniel

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedFeb 25, '10 at 4:45p
activeFeb 25, '10 at 7:15p
posts4
users2
websitehive.apache.org

2 users in discussion

Daniel Joanes: 2 posts Carl Steinbach: 2 posts

People

Translate

site design / logo © 2021 Grokbase