FAQ
Allow configurable deletion policy
----------------------------------

Key: SOLR-617
URL: https://issues.apache.org/jira/browse/SOLR-617
Project: Solr
Issue Type: New Feature
Components: search, update
Affects Versions: 1.3
Reporter: Noble Paul
Priority: Minor


Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .

I propose the configuration go into the <mainIndex> section

sample configuration
{code:xml|title=solrconfig.xml}
<mainIndex>
<!-- configure deletion policy here-->
<deletionPolicy>
<!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
the last IndexWriter/IndexReader using this commit point is closed -->
<keepOptimizedOnly>true</keepOptimizedOnly>
<!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
<maxCommitsToKeep></maxCommitsToKeep>
<!-- max age of a stored commit-->
<maxCommitAge></maxCommitAge>
</deletionPolicy>

</mainIndex>
{code}




--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Shalin Shekhar Mangar (JIRA) at Jul 20, 2008 at 8:07 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar reassigned SOLR-617:
    ------------------------------------------

    Assignee: Shalin Shekhar Mangar
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.3
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor

    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Aug 25, 2008 at 12:52 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-617:
    --------------------------------

    Attachment: solr-617.patch

    This patch adds support for the configuration described in the issue description.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Aug 28, 2008 at 2:13 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626572#action_12626572 ]

    Shalin Shekhar Mangar commented on SOLR-617:
    --------------------------------------------

    Thanks for the patch Akshay!

    I think we should allow a user to specify his custom IndexDeletionPolicy too. We can provide a default implementation with all the options specified in the issue description. So I propose that we have the following syntax:

    {code:xml}
    <deletionPolicy class="com.MyDeletionPolicy" />
    {code}

    The default implementation will be SolrIndexDeletionPolicy which can be configured through a NamedList. Any custom deletion policy will be initialized with a NamedList if it implements NamedListInitializedPlugin.
    {code:xml}
    <!-- configure deletion policy here-->
    <deletionPolicy class="solr.SolrIndexDeletionPolicy">
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <bool name="keepOptimizedOnly">true</bool>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <int name="maxCommitsToKeep"></int>
    <!-- max age of a stored commit-->
    <str name="maxCommitAge"></str>
    </deletionPolicy>
    {code}

    To facilitate replication, we can have a wrapper over the IndexDeletionPolicy which can provide us the features needed for replication (SOLR-561). We need access to a list of non-deleted IndexCommit instances, a way to lookup IndexCommit given a version as well as the latest commit point.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Sep 1, 2008 at 6:19 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-617:
    --------------------------------

    Attachment: 617.patch

    Attached is a patch with SolrDeletionPolicy class as the default deletion policy, with configuration as in Shalin's comment above, except that <str> is used for all the parameters in the named list. IndexDeletionPolicyWrapper class over IndexDeletionPolicy to enable access to commit details. Currently <maxCommitAge> named list parameter is not considered in the implementation of SolrDeletionPolicy.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 2, 2008 at 12:00 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626572#action_12626572 ]

    shalinmangar edited comment on SOLR-617 at 9/2/08 4:59 AM:
    --------------------------------------------------------------------

    Thanks for the patch Akshay!

    I think we should allow a user to specify his custom IndexDeletionPolicy too. We can provide a default implementation with all the options specified in the issue description. So I propose that we have the following syntax:

    {code:xml}
    <deletionPolicy class="com.MyDeletionPolicy" />
    {code}

    The default implementation will be SolrIndexDeletionPolicy which can be configured through a NamedList. Any custom deletion policy will be initialized with a NamedList if it implements NamedListInitializedPlugin.
    {code:xml}
    <!-- configure deletion policy here-->
    <deletionPolicy class="solr.SolrIndexDeletionPolicy">
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <str name="keepOptimizedOnly">true</str>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <str name="maxCommitsToKeep"></str>
    <!-- max age of a stored commit-->
    <str name="maxCommitAge"></str>
    </deletionPolicy>
    {code}

    To facilitate replication, we can have a wrapper over the IndexDeletionPolicy which can provide us the features needed for replication (SOLR-561). We need access to a list of non-deleted IndexCommit instances, a way to lookup IndexCommit given a version as well as the latest commit point.

    was (Author: shalinmangar):
    Thanks for the patch Akshay!

    I think we should allow a user to specify his custom IndexDeletionPolicy too. We can provide a default implementation with all the options specified in the issue description. So I propose that we have the following syntax:

    {code:xml}
    <deletionPolicy class="com.MyDeletionPolicy" />
    {code}

    The default implementation will be SolrIndexDeletionPolicy which can be configured through a NamedList. Any custom deletion policy will be initialized with a NamedList if it implements NamedListInitializedPlugin.
    {code:xml}
    <!-- configure deletion policy here-->
    <deletionPolicy class="solr.SolrIndexDeletionPolicy">
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <bool name="keepOptimizedOnly">true</bool>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <int name="maxCommitsToKeep"></int>
    <!-- max age of a stored commit-->
    <str name="maxCommitAge"></str>
    </deletionPolicy>
    {code}

    To facilitate replication, we can have a wrapper over the IndexDeletionPolicy which can provide us the features needed for replication (SOLR-561). We need access to a list of non-deleted IndexCommit instances, a way to lookup IndexCommit given a version as well as the latest commit point.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 2, 2008 at 12:00 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627640#action_12627640 ]

    Shalin Shekhar Mangar commented on SOLR-617:
    --------------------------------------------

    This is looking great. Thanks!

    We need a few tests for this. With the recent changes in Lucene trunk, we can get rid of the wrapper over IndexCommit. We need to support maxCommitAge too so that users who do a lot commits can reliably replicate without storing too many generations.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Yonik Seeley (JIRA) at Sep 2, 2008 at 5:18 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627750#action_12627750 ]

    Yonik Seeley commented on SOLR-617:
    -----------------------------------

    I think the deletion policy should be able to support true reservation... see the prototype patch I put together in https://issues.apache.org/jira/secure/attachment/12383728/deletion_policy.patch

    Lucene has added more capabilities since then, so we should be able to simplify it.

    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 2, 2008 at 6:14 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627770#action_12627770 ]

    Shalin Shekhar Mangar commented on SOLR-617:
    --------------------------------------------

    We have two options:
    # Do not reserve a commit point -- If it gets deleted due to a newer commit, let an in-flight replication fail so that the slave re-polls and gets a fresher commit point.
    # Let an in-flight replication reserve a commit point -- The slave would start another replication immediately after the previous one because the master now has a newer commit point that what it had just pulled.

    I'm more in favor of the first approach. Here, the onus of keeping a commit point for reliable replication will fall on the user supplying configuration according to his commit frequency (the maxAge condition will be handy).

    Also, wouldn't implementing a reserve method limit us to only the default SolrDeletionPolicy?
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Yonik Seeley (JIRA) at Sep 3, 2008 at 1:08 am
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627881#action_12627881 ]

    Yonik Seeley commented on SOLR-617:
    -----------------------------------

    #1 and #2 can both lead to starvation. I think the default should finish grabbing an index even if a newer version is available.

    bq. Here, the onus of keeping a commit point for reliable replication will fall on the user supplying configuration

    If we can make it such that it just works, regardless of network speed, index size, etc, then I think we should. A reservation mechanism would easily enable this.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Noble Paul (JIRA) at Sep 3, 2008 at 4:50 am
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627900#action_12627900 ]

    Noble Paul commented on SOLR-617:
    ---------------------------------

    The IndexDeletionPolicyWrapper should be able to support the reserve feature even if the user provided IndexDeletionPolicy does not do it because it is wrapping the IndexCommit object.
    the config can be as follows
    {code}
    <deletionPolicy class="solr.SolrIndexDeletionPolicy" >
    //this value will be honoured by the wrapper itself irrespective of the underlying implementation
    <str name="reserve">10</str>
    </deletionPolicy>
    {code}

    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Sep 8, 2008 at 1:39 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-617:
    --------------------------------

    Attachment: solr-617.patch

    Patch synced with trunk.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Sep 23, 2008 at 7:55 am
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-617:
    --------------------------------

    Attachment: solr-617.patch

    Patch with test cases and in sync with trunk.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Sep 23, 2008 at 8:15 am
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633649#action_12633649 ]

    akshay edited comment on SOLR-617 at 9/23/08 1:13 AM:
    --------------------------------------------------------------

    This patch has:
    1. maxCommitAge configuration support
    2. Reservation mechanism is added in the ReplicationHandler configuration. Code for the same is in the latest patch of https://issues.apache.org/jira/browse/SOLR-561
    3. Test cases.

    was (Author: akshay):
    Patch with test cases and in sync with trunk.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 24, 2008 at 7:04 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634258#action_12634258 ]

    Shalin Shekhar Mangar commented on SOLR-617:
    --------------------------------------------

    The patch looks good. I think this covers all the features we wanted to have.

    A few minor nitpicks
    # The latestCommit and maxCommitAgeMillis variables in SolrDeletionPolicy are assigned but never used.
    # The additional logging in onInit and onCommit in SolrDeletionPolicy can be removed -- the same message is logged in FINE and INFO both
    # The defaults in solrconfig.xml should mimic the previous behavior i.e. keep only the last commit point
    # Javadocs on IndexDeletionPolicyWrapper#setReserveDuration will be helpful
    # IndexDeletionPolicyWrapper#getCommits should return the generic version of the Map
    # IndexDeletionPolicyWrapper#getConfiguredDeletionPolicy can be called getWrappedDeletionPolicy or just getDeletionPolicy
    # Slight mistake in the logging, the info message should be in the same block
    {code}
    if(keepOptimizedOnly){
    if(!commit.isOptimized())
    commit.delete();
    log.info("Marking unoptimized index "+getId(commit)+ " for deletion.");
    }
    {code}
    # The getter methods in SolrDeletionPolicy should be named more appropriately e.g. isKeepOptimizedOnly, getMaxCommitAge, getMaxCommitsToKeep

    The tests look good. Thanks for all the work, Akshay :)
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 24, 2008 at 7:36 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634264#action_12634264 ]

    Shalin Shekhar Mangar commented on SOLR-617:
    --------------------------------------------

    Also, we need to use slf4j instead of the JDK Logger in the patch.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Sep 26, 2008 at 7:14 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-617:
    --------------------------------

    Attachment: solr-617.patch

    Patch with changes suggested by Shalin and logging using slf4j.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 26, 2008 at 9:18 pm
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-617:
    ---------------------------------------

    Attachment: solr-617.patch

    Updated with more javadocs and comments in solrconfig.xml

    I think this is ready to go in. I'll commit this in a day or two if there are no objection.
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Sep 29, 2008 at 3:39 am
    [ https://issues.apache.org/jira/browse/SOLR-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar resolved SOLR-617.
    ----------------------------------------

    Resolution: Fixed

    Committed revision 699975.

    Thanks Yonik, Noble and Akshay!
    Allow configurable deletion policy
    ----------------------------------

    Key: SOLR-617
    URL: https://issues.apache.org/jira/browse/SOLR-617
    Project: Solr
    Issue Type: New Feature
    Components: search, update
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Priority: Minor
    Fix For: 1.4

    Attachments: 617.patch, solr-617.patch, solr-617.patch, solr-617.patch, solr-617.patch, solr-617.patch


    Lucene API provides means to configure deletion policy. Solr should be able to expose it through configuration in solrconfig.xml. Moreover the new replication (SOLR-561) strategy is going to rely on this .
    I propose the configuration go into the <mainIndex> section
    sample configuration
    {code:xml|title=solrconfig.xml}
    <mainIndex>
    <!-- configure deletion policy here-->
    <deletionPolicy>
    <!-- Store only the commits with optimize.Non optimized commits will get deleted by lucene when
    the last IndexWriter/IndexReader using this commit point is closed -->
    <keepOptimizedOnly>true</keepOptimizedOnly>
    <!--Maximum no: of commit points stored . Older ones will be cleaned when they go out of scope-->
    <maxCommitsToKeep></maxCommitsToKeep>
    <!-- max age of a stored commit-->
    <maxCommitAge></maxCommitAge>
    </deletionPolicy>

    </mainIndex>
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupsolr-dev @
categorieslucene
postedJul 7, '08 at 6:19a
activeSep 29, '08 at 3:39a
posts19
users1
websitelucene.apache.org...

1 user in discussion

Shalin Shekhar Mangar (JIRA): 19 posts

People

Translate

site design / logo © 2019 Grokbase