Grokbase Groups Lucene dev May 2011
FAQ
NumericField should be stored in binary format in index (matching Solr's format)
--------------------------------------------------------------------------------

Key: LUCENE-3065
URL: https://issues.apache.org/jira/browse/LUCENE-3065
Project: Lucene - Java
Issue Type: Bug
Components: Index
Reporter: Michael McCandless
Priority: Minor
Fix For: 3.2, 4.0


(Spinoff of LUCENE-3001)

Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972

We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.

A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Search Discussions

  • Uwe Schindler (JIRA) at May 3, 2011 at 6:13 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028354#comment-13028354 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Ideally this could be done with the schema-like approach of one of the GSoC projects?

    We already discussed about that: We can use the FieldsReader/FieldsWriter type flag (which currently says, binary/text and compressed (unused now)) in the index file format to mark a field as NumericField. In that case, Document.getField() would return the NumericField instance.

    For Lucene backwards we should still support creating "text-only" fields.

    The new binary format would also be compatible with solr, as on getField, Solr would get a NumericField and can decide using instanceof what to do. Old Solr indexes without the NumericField marker flag would return as byte[], in which case, solr would do the decoding.

    For storing on index side, Solr could move to NumericField completely (I dont like the current approach using NumericTokenStream and to/fromInternal wrappers around conventional Field).
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at May 3, 2011 at 7:59 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Michael McCandless updated LUCENE-3065:
    ---------------------------------------

    Attachment: LUCENE-3065.patch

    Patch against 3.x.

    I moved the to/from byte[] methods from Solr's TrieField into Lucene's
    NumericUtils, and fixed FieldsWriter/Reader to use free bits in the
    field's flags to know if the field is Numeric, and which type.

    I added a random test case to verify we now get the right NumericField
    back, when we stored NumericField during indexing.

    Old indices are handled fine (you'll get a String-ified Field back like
    you did before).

    Spookily, nothing failed in Solr... I assume there's somewhere in Solr
    that must now be fixed to handle the fact that a field can come back
    as NumericField? Anyone know where...?
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 3, 2011 at 8:05 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028395#comment-13028395 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    {quote}
    Spookily, nothing failed in Solr... I assume there's somewhere in Solr
    that must now be fixed to handle the fact that a field can come back
    as NumericField? Anyone know where...?
    {quote}

    Thats easy to understand: Solr does not use NumericField at all. It produces a NumericTokenStream and indexes it like any other analyzer. The byte[] field is indexed as a separate Field with only store=true and binary.

    This is what I wanted to say with my last comment.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 3, 2011 at 8:09 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028397#comment-13028397 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Aboutthe patch: Maybe the byte[] returning methods in NumericUtils should use BytesRef and reuse that for storing (applies to trunk)?
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at May 3, 2011 at 8:09 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028398#comment-13028398 ]

    Michael McCandless commented on LUCENE-3065:
    --------------------------------------------

    {quote}
    Thats easy to understand: Solr does not use NumericField at all. It produces a NumericTokenStream and indexes it like any other analyzer. The byte[] field is indexed as a separate Field with only store=true and binary.

    This is what I wanted to say with my last comment.
    {quote}
    Ahhhh, OK. So, not spooky.

    We should eventually fix that; shouldn't Solr just use NumericField instead of doing this encode/decode itself? Is there some reason...?
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Ryan McKinley (JIRA) at May 3, 2011 at 8:15 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028399#comment-13028399 ]

    Ryan McKinley commented on LUCENE-3065:
    ---------------------------------------

    bq. Is there some reason...?

    Solr did its own encoding/decoding so that it could store a binary field -- with this patch, that is not necessary anymore.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at May 3, 2011 at 8:19 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028404#comment-13028404 ]

    Michael McCandless commented on LUCENE-3065:
    --------------------------------------------

    Uwe: I agree, I'll use BytesRef in trunk.

    Ryan: OK. Should we try to fix that w/ this issue? If so, can you take a crack at it? Thanks. Or, we can postpone... not necessary for this initial cutover.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Ryan McKinley (JIRA) at May 3, 2011 at 8:25 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028408#comment-13028408 ]

    Ryan McKinley commented on LUCENE-3065:
    ---------------------------------------

    bq. If so, can you take a crack at it? Thanks. Or, we can postpone... not necessary for this initial cutover.

    I'll take a crack at it... but I don't think its necessary in the first pass

    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Yonik Seeley (JIRA) at May 3, 2011 at 8:37 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028412#comment-13028412 ]

    Yonik Seeley commented on LUCENE-3065:
    --------------------------------------

    bq. I'll take a crack at it... but I don't think its necessary in the first pass

    Should we try to accept both (binary or numeric field coming back) so this isn't a needless index format break, or is there another lucene index format break in the cards soon anyway?
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 3, 2011 at 8:46 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028416#comment-13028416 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Mike: One thing about the bitmask and the 4 values. There is also an issue open to extend NumericField by byte and short. Maybe we should reserve 3 bits instead of 2 for the numeric field type - so 0x70 instead of 0x30 as mask? I just want to reseve this one extra bit, so we dont need to do any dumb masks and values later, if we extend.

    About the index format change:
    As described above, for Solr it's not a problem. New fields are always indexed using NumericField. On the query side, when Document.getField is called, it could simply check the return value with instanceof. If the getter returns not a NumericField, Solr knows that it's binary and can decode manually. This would safe backwards.

    Else its no break at all if we support both stored field formats during indexing somehow (in Lucene its string, returning a String Field or new binary NumericField). The index format itsself does not change generally (no need to bump version numbers, as we only use unused bits?)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 3, 2011 at 9:51 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028454#comment-13028454 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    There is still a problem - first the good news:

    - If user calls Document.get(field), the returned string is as before, so there is no break at all. The reason is the implementation of NumericField.stringValue(), it returns what the user is used to from 3.0
    - If a user calls getFieldable(field) all is fine, too. The only change is that it not could return NumericField. If the user simply calls stringValue() all is identical to 3.0

    Problems start with:

    - If user calls Document.getField(name) it returns Field (internally it casts the getFieldable()) result to Field. But NumericField does not subclass Field -> ClassCastException.

    How to handle this?

    - Maybe change those methods to return AbstractField, but thats a binary break and users will complain, because not everything works as expected
    - Make NumericField subclass Field (and Field is unfinalized) - thats a bad idea, because Field has too many methods / members that are out of scope
    - Deprecate Document.getField() and make it internally do an instanceof check, if it gets NumericField transform to a backwards-compatible Field? - This method is already broken. If you request Lazy field loading it also throws ClassCastEx (e.g. LUCENE-609).

    Not sure how to proceed. Else the patch looks fine. I think simply ignoring LazyField loading is fine, as numeric fields are a maximum of 8 bytes.... Else we would need LazyNumericField :(
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 3, 2011 at 10:03 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028454#comment-13028454 ]

    Uwe Schindler edited comment on LUCENE-3065 at 5/3/11 10:01 PM:
    ----------------------------------------------------------------

    There is still a problem - first the good news:

    - If user calls Document.get(field), the returned string is as before, so there is no break at all. The reason is the implementation of NumericField.stringValue(), it returns what the user is used to from 3.0
    - If a user calls getFieldable(field) all is fine, too. The only change is that it could return NumericField now. If the user simply calls stringValue() all is identical to 3.0

    Problems start with:

    - If user calls Document.getField(name) it returns Field (internally it casts the getFieldable()) result to Field. But NumericField does not subclass Field -> ClassCastException.

    How to handle this?

    - Maybe change those methods to return AbstractField, but thats a binary break and users will complain, because not everything works as expected
    - Make NumericField subclass Field (and Field is unfinalized) - thats a bad idea, because Field has too many methods / members that are out of scope
    - Deprecate Document.getField() and make it internally do an instanceof check, if it gets NumericField transform to a backwards-compatible Field? - This method is already broken. If you request Lazy field loading it also throws ClassCastEx (e.g. LUCENE-609).

    Not sure how to proceed. Else the patch looks fine. I think simply ignoring LazyField loading is fine, as numeric fields are a maximum of 8 bytes.... Else we would need LazyNumericField :(

    was (Author: thetaphi):
    There is still a problem - first the good news:

    - If user calls Document.get(field), the returned string is as before, so there is no break at all. The reason is the implementation of NumericField.stringValue(), it returns what the user is used to from 3.0
    - If a user calls getFieldable(field) all is fine, too. The only change is that it not could return NumericField. If the user simply calls stringValue() all is identical to 3.0

    Problems start with:

    - If user calls Document.getField(name) it returns Field (internally it casts the getFieldable()) result to Field. But NumericField does not subclass Field -> ClassCastException.

    How to handle this?

    - Maybe change those methods to return AbstractField, but thats a binary break and users will complain, because not everything works as expected
    - Make NumericField subclass Field (and Field is unfinalized) - thats a bad idea, because Field has too many methods / members that are out of scope
    - Deprecate Document.getField() and make it internally do an instanceof check, if it gets NumericField transform to a backwards-compatible Field? - This method is already broken. If you request Lazy field loading it also throws ClassCastEx (e.g. LUCENE-609).

    Not sure how to proceed. Else the patch looks fine. I think simply ignoring LazyField loading is fine, as numeric fields are a maximum of 8 bytes.... Else we would need LazyNumericField :(
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at May 3, 2011 at 10:35 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028483#comment-13028483 ]

    Michael McCandless commented on LUCENE-3065:
    --------------------------------------------

    Ugh! Field/Fieldable/AbstractField strikes again.... hmm not sure what to do.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Chris Male (JIRA) at May 4, 2011 at 12:34 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028526#comment-13028526 ]

    Chris Male commented on LUCENE-3065:
    ------------------------------------

    The Field/Fieldable/AbstractField problem is what I've been addressing in LUCENE-2310. There I took the step of making NumericField extend Field, with a series of unsupported fields. This seemed easiest to do particularly with FieldType in mind. I then deprecated all the Fieldable methods in Document.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Yonik Seeley (JIRA) at May 4, 2011 at 1:07 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028540#comment-13028540 ]

    Yonik Seeley commented on LUCENE-3065:
    --------------------------------------

    bq. I then deprecated all the Fieldable methods in Document.

    Hmmm, I thought Fieldable was a step forward. The Field class is the worst of the bunch!
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Chris Male (JIRA) at May 4, 2011 at 3:08 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028570#comment-13028570 ]

    Chris Male commented on LUCENE-3065:
    ------------------------------------

    Yeah there is an element of truth to that except I'm not convinced we need to have such a complicated hierarchy (although I've since been thinking about field definitions coming from different sources, so maybe an interface is best). But yes, Field is a mess and I've been trying to clean that out too.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 9:18 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028677#comment-13028677 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Mike:
    I reviewed the patch again: You are currently using 3 bits already. 1 bit is solely for detecting numerics, the other two are the type.

    In my opinion, to check if its a numeric field, use a MASK of 3 bits and check for !=0. As soon as any bit in this mask is set, its numeric. The actual numeric fields have values !=0:

    {code}
    private static final int _NUMERIC_BIT_SHIFT = 3;
    static final byte FIELD_IS_NUMERIC_MASK = 0x07 << _NUMERIC_BIT_SHIFT;

    static final byte FIELD_IS_NUMERIC_INT = 1 << _NUMERIC_BIT_SHIFT;
    static final byte FIELD_IS_NUMERIC_LONG = 2 << _NUMERIC_BIT_SHIFT;
    static final byte FIELD_IS_NUMERIC_FLOAT = 3 << _NUMERIC_BIT_SHIFT;
    static final byte FIELD_IS_NUMERIC_DOUBLE = 4 << _NUMERIC_BIT_SHIFT;
    // unused: static final byte FIELD_IS_NUMERIC_SHORT = 5 << _NUMERIC_BIT_SHIFT;
    // unused: static final byte FIELD_IS_NUMERIC_BYTE = 6 << _NUMERIC_BIT_SHIFT;
    // and we have still one more over :-) 7 << _NUMERIC_BIT_SHIFT

    // check if field is numeric:
    if ((bits & FIELD_IS_NUMERIC_MASK) != 0) {}

    // parse type:
    switch (bits & FIELD_IS_NUMERIC_MASK) {
    case FIELD_IS_NUMERIC_INT: ...
    }
    {code}
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 9:22 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028680#comment-13028680 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    This gives us more freedom in future, as we are limit to completely 8 bits, 3 are already used - this only adds 3 more not 4.

    By the way, for performance reasons all constants should be declared as int not byte, as the byte read from index is already an int.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 9:50 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    Here the patch with my changes
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at May 4, 2011 at 9:54 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028693#comment-13028693 ]

    Michael McCandless commented on LUCENE-3065:
    --------------------------------------------

    Patch looks great Uwe! Except we need to resolve this Field/Fieldable/AbstractField. Probably we should go and finish LUCENE-2310...
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:37 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    More refactoring:
    - Now NumericFields also reproduce the indexed/omitNorms/omitTF settings - only precStep cannot be reproduced
    - Cut over to int instead of byte, this removes lots of casting in FieldsReader
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:43 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    New patch, previous one had a leftover unused constant from Mike's patch.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:43 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: (was: LUCENE-3065.patch)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 11:06 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    This patch adds some refactoring because FieldSelectorResult is an enum since 3.0, so the (slow) queue of id-statements can be replaced by a fast switch.

    Also some minor comments and a missing & 0xFF when casting byte to int.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 11:08 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028707#comment-13028707 ]

    Uwe Schindler edited comment on LUCENE-3065 at 5/4/11 11:06 AM:
    ----------------------------------------------------------------

    This patch adds some refactoring because FieldSelectorResult is an enum since 3.0, so the (slow) queue of if-statements can be replaced by a fast switch.

    Also some minor comments and a missing & 0xFF when casting byte to int.

    was (Author: thetaphi):
    This patch adds some refactoring because FieldSelectorResult is an enum since 3.0, so the (slow) queue of id-statements can be replaced by a fast switch.

    Also some minor comments and a missing & 0xFF when casting byte to int.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 11:51 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    Next iteration:

    Reverted changes in Solr (they should come later), Lucene instead uses natively IndexInput and IndexOutput to write/read ints and longs.

    Solr's changes are completely unrelated.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 11:55 am
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028718#comment-13028718 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Just to note: We also need to change the Forrest index format documentation!
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 12:09 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler reassigned LUCENE-3065:
    -------------------------------------

    Assignee: Uwe Schindler
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 12:23 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    Moved test to TestFieldsReader
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Michael McCandless (JIRA) at May 4, 2011 at 12:35 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028733#comment-13028733 ]

    Michael McCandless commented on LUCENE-3065:
    --------------------------------------------

    Patch looks great Uwe!

    I think we should deprecate Document.getField? And advertise in CHANGES that this is an [intentional] BW break, ie, you can no longer .getField if it's a NumericField (you'll hit CCE, just like you already do for lazy fields)? I think that's the lesser evil here?
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 2:46 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    I added some javadocs to Document class:
    - getField() / getFields() is deprecated [we may change this in ]

    Some thoughts:
    - maybe we should make getField()/getFields() simply return null or does not include the Field into the returned array, if its not instanceof Field? We can add that to documentation, that lazy loaded and numerical fields are not returned.
    - I would also like to add a method Document.getNumericValue(s), that returns Number[] or Number like the NumericField one. Like above getField() it can return null/empty array if the field name has no numeric Fields?

    The CHANGES entry may also be extended, currently it under "bugs" - we shold move.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 2:46 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028764#comment-13028764 ]

    Uwe Schindler edited comment on LUCENE-3065 at 5/4/11 2:44 PM:
    ---------------------------------------------------------------

    I added some javadocs to Document class:
    - getField() / getFields() is deprecated [we may change this in LUCENE-2310]

    Some thoughts:
    - maybe we should make getField()/getFields() simply return null or does not include the Field into the returned array, if its not instanceof Field? We can add that to documentation, that lazy loaded and numerical fields are not returned.
    - I would also like to add a method Document.getNumericValue(s), that returns Number[] or Number like the NumericField one. Like above getField() it can return null/empty array if the field name has no numeric Fields?

    The CHANGES entry may also be extended, currently it under "bugs" - we shold move.

    was (Author: thetaphi):
    I added some javadocs to Document class:
    - getField() / getFields() is deprecated [we may change this in ]

    Some thoughts:
    - maybe we should make getField()/getFields() simply return null or does not include the Field into the returned array, if its not instanceof Field? We can add that to documentation, that lazy loaded and numerical fields are not returned.
    - I would also like to add a method Document.getNumericValue(s), that returns Number[] or Number like the NumericField one. Like above getField() it can return null/empty array if the field name has no numeric Fields?

    The CHANGES entry may also be extended, currently it under "bugs" - we shold move.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 7:06 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065-solr-only.patch

    Here a first step in cutover of Solr to NumericField. Most tests work, except:
    - TestDistributedSearch, fails with a strange date problem - I have no idea what goes wrong
    - TestMoreLikeThis: fails because the returned documents are different than expected. The reason for this is simple: As TrieField's underlying Lucene fields now are NumericField, stringValue() returns something (in contrast, solr's old fields returned null because they were binary). This confuses maybe MoreLikeThis (needs maybe fixed in Lucene, I havent looked into the code). Maybe we should simply exclude those fields or fix the test (I prefer latter one, because the numerics should also taken into account).

    The following changes had to be made:
    - Cut over all places in Solr where Field instead of abstract Fieldable is used to Fieldable. This affects some leftover parts in various components (calling Document.getField instead of Document.getFieldable), but mainly SchemaField/FieldType: createField() now returns Fieldable
    - TrieDateField code duplication was removed, all methods delegate to a wrapped TrieField. There was also an inconsitency between TrieField and TrieDateField's toExternal(). This was fixed to work correct (the date format was wrong, now it uses dateField.toExternal())

    If somebody could help with the rest of the solr stuff and maybe test test test! Yonik? Ryan? There may be some itches not covered by tests.

    Thanks for help from Solr specialists (I am definitely not one, I am more afraid of the code than I can help)!!!
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065-solr-only.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:20 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Comment: was deleted

    (was: Here a first step in cutover of Solr to NumericField. Most tests work, except:
    - TestDistributedSearch, fails with a strange date problem - I have no idea what goes wrong
    - TestMoreLikeThis: fails because the returned documents are different than expected. The reason for this is simple: As TrieField's underlying Lucene fields now are NumericField, stringValue() returns something (in contrast, solr's old fields returned null because they were binary). This confuses maybe MoreLikeThis (needs maybe fixed in Lucene, I havent looked into the code). Maybe we should simply exclude those fields or fix the test (I prefer latter one, because the numerics should also taken into account).

    The following changes had to be made:
    - Cut over all places in Solr where Field instead of abstract Fieldable is used to Fieldable. This affects some leftover parts in various components (calling Document.getField instead of Document.getFieldable), but mainly SchemaField/FieldType: createField() now returns Fieldable
    - TrieDateField code duplication was removed, all methods delegate to a wrapped TrieField. There was also an inconsitency between TrieField and TrieDateField's toExternal(). This was fixed to work correct (the date format was wrong, now it uses dateField.toExternal())

    If somebody could help with the rest of the solr stuff and maybe test test test! Yonik? Ryan? There may be some itches not covered by tests.

    Thanks for help from Solr specialists (I am definitely not one, I am more afraid of the code than I can help)!!!)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:20 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029011#comment-13029011 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    I started a new issue in Solr for the changes there: SOLR-2497
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:20 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: (was: LUCENE-3065-solr-only.patch)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Bug
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 4, 2011 at 10:28 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Issue Type: Improvement (was: Bug)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:11 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: SOLR-2497.patch

    MoreLikeThis problem solved, it was as I said. The test included a TrieInt field into the "similarity fields", so it was used to calculate similarity. As with previous Solr the TrieField was invisible to MLT this had no effect.
    By the way: There is a commented out part with explicitely the MLT field, but I dont understand it. It seems that it was never understood/supported.
    Now, all numeric fields should work with MLT.

    Now only the TestDistributedSearch is still failing with a strange date failure. I'll dig.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, SOLR-2497.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:13 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: (was: SOLR-2497.patch)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:13 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Comment: was deleted

    (was: Ideally this could be done with the schema-like approach of one of the GSoC projects?

    We already discussed about that: We can use the FieldsReader/FieldsWriter type flag (which currently says, binary/text and compressed (unused now)) in the index file format to mark a field as NumericField. In that case, Document.getField() would return the NumericField instance.

    For Lucene backwards we should still support creating "text-only" fields.

    The new binary format would also be compatible with solr, as on getField, Solr would get a NumericField and can decide using instanceof what to do. Old Solr indexes without the NumericField marker flag would return as byte[], in which case, solr would do the decoding.

    For storing on index side, Solr could move to NumericField completely (I dont like the current approach using NumericTokenStream and to/fromInternal wrappers around conventional Field).)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler at May 5, 2011 at 4:16 pm
    Sorry, I did not want to delete this one, my huper duper browser gots totally confused and disturbed...

    -----
    Uwe Schindler
    H.-H.-Meier-Allee 63, D-28213 Bremen
    http://www.thetaphi.de
    eMail: uwe@thetaphi.de
    -----Original Message-----
    From: Uwe Schindler (JIRA)
    Sent: Thursday, May 05, 2011 6:13 PM
    To: dev@lucene.apache.org
    Subject: [jira] [Updated] (LUCENE-3065) NumericField should be stored in
    binary format in index (matching Solr's format)


    [ https://issues.apache.org/jira/browse/LUCENE-
    3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Comment: was deleted

    (was: Ideally this could be done with the schema-like approach of one of the
    GSoC projects?

    We already discussed about that: We can use the FieldsReader/FieldsWriter
    type flag (which currently says, binary/text and compressed (unused now))
    in the index file format to mark a field as NumericField. In that case,
    Document.getField() would return the NumericField instance.

    For Lucene backwards we should still support creating "text-only" fields.

    The new binary format would also be compatible with solr, as on getField,
    Solr would get a NumericField and can decide using instanceof what to do.
    Old Solr indexes without the NumericField marker flag would return as
    byte[], in which case, solr would do the decoding.

    For storing on index side, Solr could move to NumericField completely (I dont
    like the current approach using NumericTokenStream and to/fromInternal
    wrappers around conventional Field).)
    NumericField should be stored in binary format in index (matching
    Solr's format)
    ----------------------------------------------------------------------
    ----------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch,
    LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch,
    LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a
    NumericField, and so at IndexReader time you get back an "ordinary"
    Field and your number has turned into a string. See
    https://issues.apache.org/jira/browse/LUCENE-
    1701?focusedCommentId=127
    21972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment- tab
    panel#comment-12721972 We have spare bits already in stored fields,
    so, we should use one to record that the field is numeric, and then encode
    the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a
    NumericField back when loading your document.

    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
    commands, e-mail: dev-help@lucene.apache.org


    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:13 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Comment: was deleted

    (was: Patch against 3.x.

    I moved the to/from byte[] methods from Solr's TrieField into Lucene's
    NumericUtils, and fixed FieldsWriter/Reader to use free bits in the
    field's flags to know if the field is Numeric, and which type.

    I added a random test case to verify we now get the right NumericField
    back, when we stored NumericField during indexing.

    Old indices are handled fine (you'll get a String-ified Field back like
    you did before).

    Spookily, nothing failed in Solr... I assume there's somewhere in Solr
    that must now be fixed to handle the fact that a field can come back
    as NumericField? Anyone know where...?)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:19 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029410#comment-13029410 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Sorry my browser or JIRA deleted wrong comments, so I removed one from me and one from Mike :( - Sorry.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:19 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Comment: was deleted

    (was: MoreLikeThis problem solved, it was as I said. The test included a TrieInt field into the "similarity fields", so it was used to calculate similarity. As with previous Solr the TrieField was invisible to MLT this had no effect.
    By the way: There is a commented out part with explicitely the MLT field, but I dont understand it. It seems that it was never understood/supported.
    Now, all numeric fields should work with MLT.

    Now only the TestDistributedSearch is still failing with a strange date failure. I'll dig.)
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:21 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029412#comment-13029412 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Patch against 3.x.

    I moved the to/from byte[] methods from Solr's TrieField into Lucene's NumericUtils, and fixed FieldsWriter/Reader to use free bits in the field's flags to know if the field is Numeric, and which type.

    I added a random test case to verify we now get the right NumericField back, when we stored NumericField during indexing.

    Old indices are handled fine (you'll get a String-ified Field back like you did before).

    Spookily, nothing failed in Solr... I assume there's somewhere in Solr that must now be fixed to handle the fact that a field can come back as NumericField? Anyone know where...?)

    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:23 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029412#comment-13029412 ]

    Uwe Schindler edited comment on LUCENE-3065 at 5/5/11 4:22 PM:
    ---------------------------------------------------------------

    Revert of deletion of Mike's first comment (sorry)

    {quote}
    Patch against 3.x.

    I moved the to/from byte[] methods from Solr's TrieField into Lucene's NumericUtils, and fixed FieldsWriter/Reader to use free bits in the field's flags to know if the field is Numeric, and which type.

    I added a random test case to verify we now get the right NumericField back, when we stored NumericField during indexing.

    Old indices are handled fine (you'll get a String-ified Field back like you did before).

    Spookily, nothing failed in Solr... I assume there's somewhere in Solr that must now be fixed to handle the fact that a field can come back as NumericField? Anyone know where...?)
    {quote}

    was (Author: thetaphi):
    Patch against 3.x.

    I moved the to/from byte[] methods from Solr's TrieField into Lucene's NumericUtils, and fixed FieldsWriter/Reader to use free bits in the field's flags to know if the field is Numeric, and which type.

    I added a random test case to verify we now get the right NumericField back, when we stored NumericField during indexing.

    Old indices are handled fine (you'll get a String-ified Field back like you did before).

    Spookily, nothing failed in Solr... I assume there's somewhere in Solr that must now be fixed to handle the fact that a field can come back as NumericField? Anyone know where...?)

    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Earwin Burrfoot (JIRA) at May 5, 2011 at 4:35 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029421#comment-13029421 ]

    Earwin Burrfoot commented on LUCENE-3065:
    -----------------------------------------

    It's sad NumericFields are hardbaked into index format.

    Eg - I have some fields that are similar to Numeric in that they are 'stringified' binary structures, and they can't become first-class in the same manner as Numeric.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 4:43 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029427#comment-13029427 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    Earwin: The long-term plan for flexible indexing is to make also stored fields flexible. For now its not possible, so NumericFields are handled separately. In the future, this might be a stored fields codec.
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 11:07 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Uwe Schindler updated LUCENE-3065:
    ----------------------------------

    Attachment: LUCENE-3065.patch

    Updated patch with some improvements:
    - NumericField now lazy inits the NumericTokenStream only when tokenStreamValue() is caled for the first time. This speeds up stored fields reading, as the TokenStream is generally not needed in that case.
    - I currently dont like the instanceof chains in FieldsWriter and this lazy init code. Maybe NumericField and NumericTokenStream should define an enum type for the value so you can call NumericField.getValueType() - does anybody have a better idea?
    - Improved JavaDocs for NumericField to reflect the new stored fields format
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org
  • Uwe Schindler (JIRA) at May 5, 2011 at 11:49 pm
    [ https://issues.apache.org/jira/browse/LUCENE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029657#comment-13029657 ]

    Uwe Schindler commented on LUCENE-3065:
    ---------------------------------------

    I just add this TODO here:
    *Don't forget to add a new 3.1 index format to TestBackwardsCompatibility!*
    NumericField should be stored in binary format in index (matching Solr's format)
    --------------------------------------------------------------------------------

    Key: LUCENE-3065
    URL: https://issues.apache.org/jira/browse/LUCENE-3065
    Project: Lucene - Java
    Issue Type: Improvement
    Components: Index
    Reporter: Michael McCandless
    Assignee: Uwe Schindler
    Priority: Minor
    Fix For: 3.2, 4.0

    Attachments: LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch, LUCENE-3065.patch


    (Spinoff of LUCENE-3001)
    Today when writing stored fields we don't record that the field was a NumericField, and so at IndexReader time you get back an "ordinary" Field and your number has turned into a string. See https://issues.apache.org/jira/browse/LUCENE-1701?focusedCommentId=12721972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12721972
    We have spare bits already in stored fields, so, we should use one to record that the field is numeric, and then encode the numeric field in Solr's more-compact binary format.
    A nice side-effect is we fix the long standing issue that you don't get a NumericField back when loading your document.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
    For additional commands, e-mail: dev-help@lucene.apache.org

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categorieslucene
postedMay 3, '11 at 5:41p
activeMay 7, '11 at 1:16p
posts59
users2
websitelucene.apache.org

People

Translate

site design / logo © 2021 Grokbase