FAQ
Allow Solr to load index from arbitrary directory in dataDir and Commit point
-----------------------------------------------------------------------------

Key: SOLR-658
URL: https://issues.apache.org/jira/browse/SOLR-658
Project: Solr
Issue Type: Improvement
Affects Versions: 1.3
Reporter: Noble Paul


This is a requirement for java based Solr replication

Usecase for arbitrary index directory:
if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.

Usecase for arbitrary commitpoint :
Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.





--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Noble Paul (JIRA) at Jul 25, 2008 at 8:36 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616801#action_12616801 ]

    Noble Paul commented on SOLR-658:
    ---------------------------------

    Implementation

    * keep a file index.properties in the data dir
    * Have an entry index=<new.index> in that file
    * This file may also keep _version_
    * When a new indexsearcher/writer is loaded, read this property and try to load the index from that folder
    * if it is absent , default to the hardcoded value for index and latest commitpoint


    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.3
    Reporter: Noble Paul

    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Jul 25, 2008 at 8:42 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar reassigned SOLR-658:
    ------------------------------------------

    Assignee: Shalin Shekhar Mangar
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.3
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar

    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Aug 27, 2008 at 6:52 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-658:
    ---------------------------------------

    Attachment: SOLR-658.patch

    This is cut out of the SOLR-561 patch supports loading index from an arbitrary directory.

    *Changes*
    # A new method SolrCore#getNewIndexDir() is introduced which tries to read the latest indexDir from index.properties file. If that file is not present the default value (dataDir + "index/")
    # SolrIndexSearcher now stores the path (indexDir) on which it is opened and has a getter for it.
    # When SolrCore#getIndexDir() is called, it gives the current searcher's index directory, failing which the default value is given
    # SolrIndexSearcher is always created with getNewIndexDir() and UpdateHandler also uses getNewIndexDir() to open IndexWriter instances.

    TODO:
    * Add a test
    * Add feature for loading arbitrary commit point.
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Sep 24, 2008 at 9:52 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-658:
    --------------------------------

    Attachment: SOLR-658.patch

    Patch in sync with trunk and with a test case (loading arbitrary commit point feature not supported in this patch).
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Akshay K. Ukey (JIRA) at Oct 6, 2008 at 11:14 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Akshay K. Ukey updated SOLR-658:
    --------------------------------

    Attachment: SOLR-658.patch

    Patch in sync with the trunk.
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Oct 8, 2008 at 8:24 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-658:
    ---------------------------------------

    Attachment: SOLR-658.patch

    Thanks Akshay.

    Updated patch which calls getNewIndexDir before calling IndexReader#reopen so that if the new index directory is different from the old index directory, we always create a new SolrIndexSearcher with the new index directory.

    I'd like to commit this in the next two or three days if there are no objections.
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Oct 11, 2008 at 9:07 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-658:
    ---------------------------------------

    Attachment: SOLR-658.patch

    Updated with a bug fix:
    {code}
    if (result != null && result.trim().length() > 0) {
    File tmp = new File(dataDir + s);
    if (tmp.exists() && tmp.isDirectory())
    result = dataDir + s;
    }
    {code}
    should be:
    {code}
    if (s != null && s.trim().length() > 0) {
    File tmp = new File(dataDir + s);
    if (tmp.exists() && tmp.isDirectory())
    result = dataDir + s;
    }
    {code}

    I'll commit shortly.
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Oct 13, 2008 at 9:07 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-658:
    ---------------------------------------

    Attachment: SOLR-658.patch

    Instead of comparing path strings, we should compare the corresponding File objects to handle relative and absolute paths correctly.

    Patch to cover the above case.
    Allow Solr to load index from arbitrary directory in dataDir and Commit point
    -----------------------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Oct 13, 2008 at 9:09 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-658:
    ---------------------------------------

    Description:
    This is a requirement for java based Solr replication

    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.



    was:
    This is a requirement for java based Solr replication

    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.

    Usecase for arbitrary commitpoint :
    Replication can also provide rollback feature . The rollback should be able to mention a comitpoint /generation so that rollback is possible.





    Summary: Allow Solr to load index from arbitrary directory in dataDir (was: Allow Solr to load index from arbitrary directory in dataDir and Commit point)

    Removing reference to rollbacks and commit points which is being handled in SOLR-670
    Allow Solr to load index from arbitrary directory in dataDir
    ------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Oct 13, 2008 at 9:15 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar resolved SOLR-658.
    ----------------------------------------

    Resolution: Fixed

    Committed revision 703981.

    Thanks Noble and Akshay!
    Allow Solr to load index from arbitrary directory in dataDir
    ------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Yonik Seeley (JIRA) at Jan 30, 2009 at 8:27 pm
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Yonik Seeley reopened SOLR-658:
    -------------------------------


    This causes reopen() to never be used on Windows because the following condition comes up false:
    {code}
    if(new File(getIndexDir()).equals(new File(newIndexDir))) {
    {code}



    Allow Solr to load index from arbitrary directory in dataDir
    ------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Feb 3, 2009 at 8:04 pm
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670088#action_12670088 ]

    Shalin Shekhar Mangar commented on SOLR-658:
    --------------------------------------------

    Copying over from the solr-dev thread on failing tests:

    The first problem is that File.equals compares only the path and not the absolute path. A work around is to compare absolute path ourselves. But a bigger problem is with the canonical paths where long directory names is uppercased and shortened into 8 character names (e.g. "C:\Documents and Settings" becomes "C:\DOCUME~1").

    The test fails because we use java.io.tmpdir which defaults to user's home directory (shortened and canonicalized) on windows and comparison on this path fails. What I'm not able to figure out yet is why does Slave Jetty, running on this canonical path, returns the full path of the index directory.

    Slave's SolrCore.getIndexDir gives:
    C:\Documents and Settings\shalinsmangar\Local Settings\Temp\org.apache.solr.handler.TestReplicationHandler$SolrInstance-1233681533000master\data\index

    The value written by TestReplicationHandler is:
    C:\DOCUME~1\SHALIN~1\LOCALS~1\Temp\org.apache.solr.handler.TestReplicationHandler$SolrInstance-1233681533000master\data\index

    Allow Solr to load index from arbitrary directory in dataDir
    ------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Mar 27, 2009 at 9:51 pm
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar updated SOLR-658:
    ---------------------------------------

    Attachment: SOLR-658-reopen-windows-fix.patch

    I should read javadocs more. This patch compares the index directories using their canonical paths This fixes the problem on windows.
    Allow Solr to load index from arbitrary directory in dataDir
    ------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658-reopen-windows-fix.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Shalin Shekhar Mangar (JIRA) at Mar 29, 2009 at 6:23 am
    [ https://issues.apache.org/jira/browse/SOLR-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Shalin Shekhar Mangar resolved SOLR-658.
    ----------------------------------------

    Resolution: Fixed

    Committed revision 759641.
    Allow Solr to load index from arbitrary directory in dataDir
    ------------------------------------------------------------

    Key: SOLR-658
    URL: https://issues.apache.org/jira/browse/SOLR-658
    Project: Solr
    Issue Type: Improvement
    Affects Versions: 1.4
    Reporter: Noble Paul
    Assignee: Shalin Shekhar Mangar
    Fix For: 1.4

    Attachments: SOLR-658-reopen-windows-fix.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch, SOLR-658.patch


    This is a requirement for java based Solr replication
    Usecase for arbitrary index directory:
    if the slave has a corrupted index and the filesystem does not allow overwriting files in use (NTFS) replication will fail. The solution is to copy the index from master to an alternate directory on slave and load indexreader/indexwriter from this alternate directory.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupsolr-dev @
categorieslucene
postedJul 25, '08 at 8:31a
activeMar 29, '09 at 6:23a
posts15
users1
websitelucene.apache.org...

1 user in discussion

Shalin Shekhar Mangar (JIRA): 15 posts

People

Translate

site design / logo © 2019 Grokbase