Storing special characters in Lucene

Hola Juan,
If your source code is encoded as Latin-1, then it will likely appear to you to be the correct character (depending on the editor/viewer you're using and its configuration), but Java may not properly convert it to Unicode, depending on the encoding it expects your source code to be in (see the -encoding option to javac - if you don't specify it, then the platform default encoding is used). You could test whether this is the problem by instead trying:

String content = "ni\u00F1os";
...
Make sure that the data you give to Lucene is encoded properly, and then what you get back should also be.

Please try the suggestion I gave you above ("ni\u00F1os"). If you still have the same problem, you may have found a bug - please report back what you find.

Steve

Hi Steve
I just gave my example that string, ant it correctly wrote it on the screen
as an ñ, but storing and retrieving it yielded the same results, that is,
the character gets lost in translation.

Here's a unit test:
import junit.framework.TestCase;
import org.apache.lucene.analysis.snowball.SnowballAnalyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.search.Hits;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.store.RAMDirectory;


public class SpanishTest extends TestCase {

public void testSpanish() throws Exception {
RAMDirectory directory = new RAMDirectory();
String content = "niños";
IndexWriter writer = new IndexWriter(directory, new
StandardAnalyzer(), true);
Document document = new Document();
document.add(new Field("name", content, Field.Store.YES,
Field.Index.TOKENIZED));
SnowballAnalyzer snowballAnalyzer = new
SnowballAnalyzer("Spanish");
writer.addDocument(document, snowballAnalyzer);
writer.close();

IndexSearcher searcher = new IndexSearcher(directory);
QueryParser parser = new QueryParser("name", snowballAnalyzer);
Query query = parser.parse(content);
System.out.println("Query: " + query);
Hits hits = searcher.search(query);
assertTrue("hits Size: " + hits.length() + " is not: " + 1,
hits.length() == 1);
Document theDoc = hits.doc(0);
String nombre = theDoc.get("name");
System.out.println("Nombre: " + nombre);
}
}


When I run this in IntelliJ, I get:

Query: name:niñ
Nombre: niños

Process finished with exit code 0


Are you by chance indexing XML?


--------------------------
Grant Ingersoll
http://www.lucidimagination.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ








---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

You are right, it does work. I'll look into my example to see where the
difference is.
Indirectly, yes

It was, after all an XML issue, the servlets creating the content that was
being indexed were not sending UTF but the XML declaration stated the code
WAS UTF, so it really was not a Lucene issue after all. Thanks for all the
help.

On Thu, Aug 21, 2008 at 6:18 PM, Juan Pablo Morales
wrote: