Grokbase Groups Hive user August 2009
FAQ
Hi,

I built a UDF function to decode the URLs from the log files. When I run
the script in Hive, i get the following error message. Is there something I
should be catching beside the NULL values?


Thanks,
-ray

java.lang.RuntimeException: Unable to execute method public
org.apache.hadoop.io.Text
com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
arguments {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
of size 1:null
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.Child.main(Child.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
execute method public org.apache.hadoop.io.Text
com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
arguments {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
of size 1:null
at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
... 3 more



Below is the UDF class;
package com.test.norm;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
import org.apache.log4j.Logger;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;


public final class URLDecode extends UDF {



private static final Logger LOG = Logger.getLogger(URLDecode.class);


public Text evaluate(final Text s) {
if (s == null) {
LOG.error("input is null");
return null;
}


String url_orig = s.toString();

if (url_orig == null || url_orig.isEmpty()) {
LOG.error("orig url is null");
return null;
}

String url_norm = null;

try {
url_norm = URLDecoder.decode(url_orig, "UTF-8");
// url_norm = URLDecoder.decode(url_norm, "UTF-8");
} catch (Exception exception) {
LOG.error("wrong url code :::" + exception.getMessage());
return null;
}
if (url_norm == null) {
LOG.error("cant decode from " + s);
return null;
}
Text to_value = new Text();
to_value.set(url_norm);
return to_value;
}
}

Search Discussions

  • Zheng Shao at Aug 21, 2009 at 5:48 am
    http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034

    Can you change the following several lines?

    e.printStackTrace();
    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString()
    + ":" + e.getMessage());

    to:

    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString(), e);


    Some exception were thrown but unfortunately e.getMessage() returns
    "null". With this change, we will know what exactly happened.

    Zheng

    On Thu, Aug 20, 2009 at 4:29 PM, Ray Duongwrote:
    Hi,

    I built a UDF function to decode the URLs from the log files.  When I run
    the script in Hive, i get the following error message.  Is there something I
    should be catching beside the NULL values?


    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)

    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)

    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more


    Below is the UDF class;
    package com.test.norm;

    import org.apache.hadoop.hive.ql.exec.UDF;
    import org.apache.hadoop.io.Text;
    import org.apache.log4j.Logger;
    import java.io.UnsupportedEncodingException;
    import java.net.URLDecoder;


    public final class URLDecode extends UDF {



    private static final Logger LOG = Logger.getLogger(URLDecode.class);


    public Text evaluate(final Text s) {
    if (s == null) {
    LOG.error("input is null");
    return null;
    }


    String url_orig = s.toString();

    if (url_orig == null || url_orig.isEmpty()) {
    LOG.error("orig url is null");
    return null;
    }

    String url_norm = null;

    try {
    url_norm = URLDecoder.decode(url_orig, "UTF-8");
    //      url_norm = URLDecoder.decode(url_norm, "UTF-8");
    } catch (Exception exception) {
    LOG.error("wrong url code :::" + exception.getMessage());
    return null;
    }
    if (url_norm == null) {
    LOG.error("cant decode from " + s);
    return null;
    }
    Text to_value = new Text();
    to_value.set(url_norm);
    return to_value;
    }
    }




    --
    Yours,
    Zheng
  • Ray Duong at Aug 21, 2009 at 2:48 pm
    Hi Zheng,

    I made the change the FunctionRegistry.java file and rebuild the Hive code
    base. After re-running the UDF, I got the following exceptions for illegal
    Hex characters in escape (%) pattern. So in my UDF function, I have try and
    exception catcher, shouldn't the value return null on the exception?

    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode
    with arguments {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode with
    arguments {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"
    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)


    ... 15 more

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode
    with arguments {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode
    with arguments {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"
    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
    ... 15 more


    On Thu, Aug 20, 2009 at 10:48 PM, Zheng Shao wrote:


    http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034

    Can you change the following several lines?

    e.printStackTrace();
    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString()
    + ":" + e.getMessage());

    to:

    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString(), e);


    Some exception were thrown but unfortunately e.getMessage() returns
    "null". With this change, we will know what exactly happened.

    Zheng

    On Thu, Aug 20, 2009 at 4:29 PM, Ray Duongwrote:
    Hi,

    I built a UDF function to decode the URLs from the log files. When I run
    the script in Hive, i get the following error message. Is there
    something I
    should be catching beside the NULL values?


    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)
    ... 3 more


    Below is the UDF class;
    package com.test.norm;

    import org.apache.hadoop.hive.ql.exec.UDF;
    import org.apache.hadoop.io.Text;
    import org.apache.log4j.Logger;
    import java.io.UnsupportedEncodingException;
    import java.net.URLDecoder;


    public final class URLDecode extends UDF {



    private static final Logger LOG =
    Logger.getLogger(URLDecode.class);

    public Text evaluate(final Text s) {
    if (s == null) {
    LOG.error("input is null");
    return null;
    }


    String url_orig = s.toString();

    if (url_orig == null || url_orig.isEmpty()) {
    LOG.error("orig url is null");
    return null;
    }

    String url_norm = null;

    try {
    url_norm = URLDecoder.decode(url_orig, "UTF-8");
    // url_norm = URLDecoder.decode(url_norm, "UTF-8");
    } catch (Exception exception) {
    LOG.error("wrong url code :::" + exception.getMessage());
    return null;
    }
    if (url_norm == null) {
    LOG.error("cant decode from " + s);
    return null;
    }
    Text to_value = new Text();
    to_value.set(url_norm);
    return to_value;
    }
    }




    --
    Yours,
    Zheng
  • Zheng Shao at Aug 25, 2009 at 8:13 am
    Not sure if you figured this out or not, but the stack trace clearly shows
    there is a problem at the URLDecode.java:25.
    Maybe you didn't redeploy the jar after changing the code.


    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at *com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)*
    ... 15 more

    Zheng


    On Fri, Aug 21, 2009 at 7:48 AM, Ray Duongwrote:
    Hi Zheng,

    I made the change the FunctionRegistry.java file and rebuild the Hive code
    base. After re-running the UDF, I got the following exceptions for illegal
    Hex characters in escape (%) pattern. So in my UDF function, I have try and
    exception catcher, shouldn't the value return null on the exception?

    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)


    ... 15 more

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
    ... 15 more


    On Thu, Aug 20, 2009 at 10:48 PM, Zheng Shao wrote:

    http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034
    Can you change the following several lines?

    e.printStackTrace();
    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString()
    + ":" + e.getMessage());

    to:

    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString(), e);


    Some exception were thrown but unfortunately e.getMessage() returns
    "null". With this change, we will know what exactly happened.

    Zheng

    On Thu, Aug 20, 2009 at 4:29 PM, Ray Duongwrote:
    Hi,

    I built a UDF function to decode the URLs from the log files. When I
    run
    the script in Hive, i get the following error message. Is there
    something I
    should be catching beside the NULL values?


    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more


    Below is the UDF class;
    package com.test.norm;

    import org.apache.hadoop.hive.ql.exec.UDF;
    import org.apache.hadoop.io.Text;
    import org.apache.log4j.Logger;
    import java.io.UnsupportedEncodingException;
    import java.net.URLDecoder;


    public final class URLDecode extends UDF {



    private static final Logger LOG =
    Logger.getLogger(URLDecode.class);


    public Text evaluate(final Text s) {
    if (s == null) {
    LOG.error("input is null");
    return null;
    }


    String url_orig = s.toString();

    if (url_orig == null || url_orig.isEmpty()) {
    LOG.error("orig url is null");
    return null;
    }

    String url_norm = null;

    try {
    url_norm = URLDecoder.decode(url_orig, "UTF-8");
    // url_norm = URLDecoder.decode(url_norm, "UTF-8");
    } catch (Exception exception) {
    LOG.error("wrong url code :::" +
    exception.getMessage());
    return null;
    }
    if (url_norm == null) {
    LOG.error("cant decode from " + s);
    return null;
    }
    Text to_value = new Text();
    to_value.set(url_norm);
    return to_value;
    }
    }




    --
    Yours,
    Zheng


    --
    Yours,
    Zheng
  • Edward Capriolo at Aug 25, 2009 at 2:37 pm

    On Tue, Aug 25, 2009 at 4:13 AM, Zheng Shaowrote:
    Not sure if you figured this out or not, but the stack trace clearly shows
    there is a problem at the URLDecode.java:25.
    Maybe you didn't redeploy the jar after changing the code.


    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
    ... 15 more

    Zheng


    On Fri, Aug 21, 2009 at 7:48 AM, Ray Duongwrote:
    Hi Zheng,

    I made the change the FunctionRegistry.java file and rebuild the Hive code
    base.  After re-running the UDF, I got the following exceptions for
    illegal
    Hex characters in escape (%) pattern.  So in my UDF function, I have try
    and
    exception catcher, shouldn't the value return null on the exception?

    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text)  on object
    com.sharethis.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode
    with
    arguments

    {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text)  on object
    com.test.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode with
    arguments

    {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at

    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at

    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at

    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)

    at

    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at

    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)

    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at

    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)
    at

    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)


    ... 15 more

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text)  on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode
    with
    arguments

    {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text)  on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode
    with
    arguments

    {http%3A%2F%2Fwww.swapalease.com%2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at

    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at

    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at

    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)

    at

    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at

    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)

    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at

    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)
    at

    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
    ... 15 more


    On Thu, Aug 20, 2009 at 10:48 PM, Zheng Shao wrote:



    http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034

    Can you change the following several lines?

    e.printStackTrace();
    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString()
    + ":" + e.getMessage());

    to:

    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString(), e);


    Some exception were thrown but unfortunately e.getMessage() returns
    "null". With this change, we will know what exactly happened.

    Zheng

    On Thu, Aug 20, 2009 at 4:29 PM, Ray Duongwrote:
    Hi,

    I built a UDF function to decode the URLs from the log files.  When I
    run
    the script in Hive, i get the following error message.  Is there
    something I
    should be catching beside the NULL values?


    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text)  on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments


    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text)  on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode with
    arguments


    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at


    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at


    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at


    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)

    at


    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at


    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)

    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at

    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more


    Below is the UDF class;
    package com.test.norm;

    import org.apache.hadoop.hive.ql.exec.UDF;
    import org.apache.hadoop.io.Text;
    import org.apache.log4j.Logger;
    import java.io.UnsupportedEncodingException;
    import java.net.URLDecoder;


    public final class URLDecode extends UDF {



    private static final Logger LOG =
    Logger.getLogger(URLDecode.class);


    public Text evaluate(final Text s) {
    if (s == null) {
    LOG.error("input is null");
    return null;
    }


    String url_orig = s.toString();

    if (url_orig == null || url_orig.isEmpty()) {
    LOG.error("orig url is null");
    return null;
    }

    String url_norm = null;

    try {
    url_norm = URLDecoder.decode(url_orig, "UTF-8");
    //      url_norm = URLDecoder.decode(url_norm, "UTF-8");
    } catch (Exception exception) {
    LOG.error("wrong url code :::" +
    exception.getMessage());
    return null;
    }
    if (url_norm == null) {
    LOG.error("cant decode from " + s);
    return null;
    }
    Text to_value = new Text();
    to_value.set(url_norm);
    return to_value;
    }
    }




    --
    Yours,
    Zheng


    --
    Yours,
    Zheng
    Also you could just use 'add jar' and 'create temporary function' to
    add your UDF. This will save you the time of rebuilding/redeploying
    hive with each change.

    Edward
  • Ray Duong at Aug 25, 2009 at 4:55 pm
    Hi Zheng,

    I was able resolved the problem by putting a isEmpty() along with the null
    string condition checker. This seems to resolve the issue.

    Thanks,
    -ray
    On Tue, Aug 25, 2009 at 1:13 AM, Zheng Shao wrote:

    Not sure if you figured this out or not, but the stack trace clearly shows
    there is a problem at the URLDecode.java:25.
    Maybe you didn't redeploy the jar after changing the code.


    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at *com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)*
    ... 15 more

    Zheng



    On Fri, Aug 21, 2009 at 7:48 AM, Ray Duongwrote:
    Hi Zheng,

    I made the change the FunctionRegistry.java file and rebuild the Hive code
    base. After re-running the UDF, I got the following exceptions for illegal
    Hex characters in escape (%) pattern. So in my UDF function, I have try and
    exception catcher, shouldn't the value return null on the exception?

    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@6f9ec4ed of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)


    ... 15 more

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.sharethis.norm.URLDecode@6a5eb489 of class com.test.norm.URLDecode with
    arguments
    {http%3A%2F%2Fwww.swapalease.com
    %2FVehicleSearch%2FVehicleDtlL.asp%3Fyear1%3D1995%26year2%3D2010%26make%3D%2525%26model%3D%2525%26category%3Dsuvs%26state%3DOR%26zipcode%3D97301%26zipDistance%3D100%26begPayment%3D0%26endPayment%3D7500%26begPrice%3D%26endPrice%3D%26BegMonthsRemaining%3D0%26EndMonthsRemaining%3D73%26BegMilesPerMo%3D0%26EndMilesPerMo%3D20000%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DL%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D535792:org.apache.hadoop.io.Text}
    of size 1

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:454)
    ... 11 more
    Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex
    characters in escape (%) pattern - For input string: "&I"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at com.sharethis.norm.URLDecode.evaluate(URLDecode.java:25)
    ... 15 more


    On Thu, Aug 20, 2009 at 10:48 PM, Zheng Shao wrote:

    http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java?revision=806034
    Can you change the following several lines?

    e.printStackTrace();
    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString()
    + ":" + e.getMessage());

    to:

    throw new HiveException("Unable to execute method " + m + " "
    + " on object " + thisObjectString
    + " with arguments " + argumentString.toString(), e);


    Some exception were thrown but unfortunately e.getMessage() returns
    "null". With this change, we will know what exactly happened.

    Zheng

    On Thu, Aug 20, 2009 at 4:29 PM, Ray Duongwrote:
    Hi,

    I built a UDF function to decode the URLs from the log files. When I
    run
    the script in Hive, i get the following error message. Is there
    something I
    should be catching beside the NULL values?


    Thanks,
    -ray

    java.lang.RuntimeException: Unable to execute method public
    org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode
    with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:182)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.Child.main(Child.java:158)

    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
    execute method public org.apache.hadoop.io.Text
    com.test.norm.URLDecode.evaluate(org.apache.hadoop.io.Text) on object
    com.test.norm.URLDecode@27b03c1a of class com.test.norm.URLDecode
    with
    arguments
    {http%3A%2F%2Fswapalease.com%2Fvehiclesearch%2FVehicleDtlF.asp%3Fyear1%3D2007%26year2%3D2010%26make%3DLexus%26model%3DRX%2B350%26category%3D%2525%26state%3D%26zipcode%3D43017%26zipDistance%3D100%26begPayment%3D%26endPayment%3D%26begPrice%3D0%26endPrice%3D250000%26BegMonthsRemaining%3D%26EndMonthsRemaining%3D%26BegMilesPerMo%3D%26EndMilesPerMo%3D%26recent_list%3D%26forSale%3D%26Country%3DUS%26ListingType%3DF%26ExtColor%3D%25%26IntColor%3D%25%26Engine%3D%25%26Transmission%3D%25%26showIncentive%3D%26showNewLease%3D%26leaseCompany%3D0%26showOnlyDealer%3D%26DUID%3D%26sort%3D3%26page%3D1%26ResultsPerPage%3D10%26vehicleID%3D545503:org.apache.hadoop.io.Text}
    of size 1:null

    at
    org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:478)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:131)
    at
    org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluator.java:99)
    at
    org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:72)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:42)
    at
    org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:486)
    at
    org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:297)
    at
    org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:165)

    ... 3 more


    Below is the UDF class;
    package com.test.norm;

    import org.apache.hadoop.hive.ql.exec.UDF;
    import org.apache.hadoop.io.Text;
    import org.apache.log4j.Logger;
    import java.io.UnsupportedEncodingException;
    import java.net.URLDecoder;


    public final class URLDecode extends UDF {



    private static final Logger LOG =
    Logger.getLogger(URLDecode.class);


    public Text evaluate(final Text s) {
    if (s == null) {
    LOG.error("input is null");
    return null;
    }


    String url_orig = s.toString();

    if (url_orig == null || url_orig.isEmpty()) {
    LOG.error("orig url is null");
    return null;
    }

    String url_norm = null;

    try {
    url_norm = URLDecoder.decode(url_orig, "UTF-8");
    // url_norm = URLDecoder.decode(url_norm, "UTF-8");
    } catch (Exception exception) {
    LOG.error("wrong url code :::" +
    exception.getMessage());
    return null;
    }
    if (url_norm == null) {
    LOG.error("cant decode from " + s);
    return null;
    }
    Text to_value = new Text();
    to_value.set(url_norm);
    return to_value;
    }
    }




    --
    Yours,
    Zheng


    --
    Yours,
    Zheng

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categorieshive, hadoop
postedAug 20, '09 at 11:29p
activeAug 25, '09 at 4:55p
posts6
users3
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase