Grokbase Groups Hive dev July 2011
FAQ
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1135/
-----------------------------------------------------------

(Updated 2011-07-28 19:17:59.868991)


Review request for hive.


Changes
-------

Decided to store timestamps as UTC on disk and add offsets (localize) if user needs through UDF. This simplifies use cases (compared to MySQL) where users from different time zones querying the same data would receive different results.

Will file a follow up JIRA ticket to add table level timezone defaults. This would allow a user to import existing data that is already offset from UTC and write new data to the table with that offset.

Summary:

-Added config var for default timezone
-Added GenericUDF for converting from UTC to localized timezone (either default, or supplied from list of Java timezones)
Ex: utc_to_localized_timestamp(<timestamp>) OR utc_to_localized_timestamp(<timestamp>, "America/Chicago")
-Fixed DateDiff bug I introduced


Summary
-------

Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization.
Supports SQL style jdbc timestamps of the format with nanosecond precision
YYYY-MM-DD HH:MM:SS[.fff...]


This addresses bug HIVE-2272.
https://issues.apache.org/jira/browse/HIVE-2272


Diffs (updated)
-----

trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1151571
trunk/conf/hive-default.xml 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 1151571
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java PRE-CREATION
trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtcToLocalizedTimestamp.java PRE-CREATION
trunk/ql/src/test/queries/clientnegative/invalid_t_create3.q 1151571
trunk/ql/src/test/queries/clientpositive/timestamp_1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_2.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_3.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_comparison.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/timestamp_udf.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/invalid_create_tbl1.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_alter1.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_alter2.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_create1.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_create2.q.out 1151571
trunk/ql/src/test/results/clientnegative/invalid_t_transform.q.out 1151571
trunk/ql/src/test/results/clientnegative/wrong_column_type.q.out 1151571
trunk/ql/src/test/results/clientpositive/show_functions.q.out 1151571
trunk/ql/src/test/results/clientpositive/timestamp_1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_2.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_3.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_comparison.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/timestamp_udf.q.out PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampObjectInspector.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestamp.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1151571
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampObjectInspector.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampObjectInspector.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampObjectInspector.java PRE-CREATION
trunk/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 1151571

Diff: https://reviews.apache.org/r/1135/diff


Testing
-------

Wrote some unit tests for string/binary serialization. Also ran against test data and looked consistent


Thanks,

Franklin

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 5 of 12 | next ›
Discussion Overview
groupdev @
categorieshive, hadoop
postedJul 15, '11 at 1:46a
activeAug 10, '11 at 11:53p
posts12
users3
websitehive.apache.org

People

Translate

site design / logo © 2021 Grokbase