|
|
Xapian 1.0.6 released
By Olly Betts at Mar 25, 2008, 4:28 pm UTC
I've uploaded Xapian 1.0.6, which as usual you can download from: http://www.xapian.org/download.php Some highlights: This release adds the ability to perform single-ended value range checks. The stemmers have been sped up (by around 2.5%). xapian-check and xapian-compact now handle user metadata... More...
I've uploaded Xapian 1.0.6, which as usual you can download from:
http://www.xapian.org/download.php
Some highlights:
This release adds the ability to perform single-ended value range checks.
The stemmers have been sped up (by around 2.5%).
xapian-check and xapian-compact now handle user metadata better.
We now update the base files atomically to avoid problems with reading
processes finding partially written ones.
Lazily created tables now get the correct revision.
OS/2 support has been integrated.
omindex now supports indexing DjVu files.
Three new OmegaScript commands have been added: $addfilter, $lower, $upper.
The Python module now always has the extension which Python expects, which
fixes a problem on Mac OS X.
For an broader overview see:
http://wiki.xapian.org/ReleaseOverview/1.0.6
The full lists of user-visible changes are linked to from there, and are
also available via the "[news]" links on the download page.
As always, if you encounter problems, please report them here, or to
the bugtracker.
Search::Xapian 1.0.6.0 and Debian and Ubuntu packages should follow
later this week.
Cheers,
Olly
6 Replies
|
|
|
Website outage
By Richard Boulton at Mar 25, 2008, 11:44 am UTC
The xapian website http://xapian.org/ (and the wiki and bugzilla) are currently offline. This is due to problems with connectivity to the machine hosting them. This affects only the main website, the wiki and bugzilla. We hope to get the main website back up shortly; it is likely that the wiki and... More...
The xapian website http://xapian.org/ (and the wiki and bugzilla) are
currently offline. This is due to problems with connectivity to the
machine hosting them.
This affects only the main website, the wiki and bugzilla. We hope to
get the main website back up shortly; it is likely that the wiki and
bugzilla will take a little longer.
Tarball downloads are still available as usual, from
http://oligarchy.co.uk/xapian/1.0.6/ (However, Debian and Ubuntu
packages are hosted on the main xapian website, and as such are
currently inaccessible.)
SVN access (both native SVN and the web browsable interface) should
still be working, and the mailing lists should also be working as usual.
0 Replies
|
|
|
[typo] Release of Typo 5.0.2
By Frederic de Villamil at Mar 25, 2008, 06:02 am UTC
Hello, We've released this morning Typo 5.0.2. This is a bugfix release = replacing Typo 5.0 and Typo 5.0.1 It fixes 2 critical bugs : =96 Typo suicidal tendencies when clearing the cache. =96 A Rails bug that prevents sidebars to be saved under sqlite and pgsql It also adds some improvements to... More...
Hello,
We've released this morning Typo 5.0.2. This is a bugfix release =
replacing Typo 5.0 and Typo 5.0.1
It fixes 2 critical bugs :
=96 Typo suicidal tendencies when clearing the cache.
=96 A Rails bug that prevents sidebars to be saved under sqlite and pgsql
It also adds some improvements to the back office.
Cheers,
Fr=E9d=E9ric
14 Replies
|
|
|
Feature request: Ligthen pressure on backup
By Jesper Krogh at Mar 24, 2008, 11:19 pm UTC
Hi. This i a small feature request for Xapian. Currently I have a xapian-database with >5m records, the files fills around 124GB in the Xapian catalog. With a few "quite large" files: # du -sh * 0 flintlock 4.0K iamflint 1000K position.baseA 63G position.DB 716K postlist.baseA 624K postlist.baseB... More...
Hi.
This i a small feature request for Xapian. Currently I have a
xapian-database with >5m records, the files fills around 124GB in the
Xapian catalog. With a few "quite large" files:
# du -sh *
0 flintlock
4.0K iamflint
1000K position.baseA
63G position.DB
716K postlist.baseA
624K postlist.baseB
45G postlist.DB
8.0K record.baseA
385M record.DB
240K termlist.baseA
15G termlist.DB
12K value.baseB
696M value.DB
(And it is my impression that I have a quite small record.DB-file)
The idea comes from PostgreSQL's filesystem layout, it has a (probably
historic) filesize of 2GB, but it helps the backup significantly.
This layout, gives some "challenges" to backup systems since the daily
incremental runs basically now has to backup the complete set => 124GB
even if only a single new document has been merged.
The suggesting would be to split the files in several smaller files. I
know that the algorithms for searching the binary trees probably would
be a bit more complex, but it could result in that changes only touches
a subset of the files, thus letting the backup proceed easier.
Another solution could be to let Xapian query several databases and
"merge" the result. Then I could make a new database each day and merge
once a week (or another timepattern that would fit the purpose).
Other suggestions are welcome.
Thanks.
Jesper
3 Replies
|
|
|
how to compile java-swig under linux.
By jarrod roberson at Mar 24, 2008, 6:59 pm UTC
ok, I am moving my development environment for the java portion of my application to linux (Centos 5 to be specific). I have found I can still do my python development on Windows. At least Komodo and Intellij IDEA run on both windows and linux! :-) I lost my Mac due to a job change :-(. Anyway,... More...
ok, I am moving my development environment for the java portion of my
application to linux (Centos 5 to be specific).
I have found I can still do my python development on Windows. At least
Komodo and Intellij IDEA run on both windows and linux! :-) I lost my Mac
due to a job change :-(.
Anyway, long story short, I am trying to get the java-swig bindings to build
and am having no luck.
xapian-core 1.0.5 builds fine, I got the python bindings builidng fine as
well. Just as plain and simple as it was on OSX.
This is what I am getting as output.
[[email protected: j...@localhost] java-swig]$ make
make all-am
make[1]: Entering directory `/home/jhr/xapian-bindings-1.0.5/java-swig'
/usr/lib/jvm/java-1.6.0-sun-1.6.0.04/bin/javac -classpath .:. -d . Auto.java
./XapianJNI.java:192: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void Enquire_setSortByKey__SWIG_0(long jarg1,
Enquire jarg1_, long jarg2, Sorter jarg2_, boolean jarg3);
^
./XapianJNI.java:193: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void Enquire_setSortByKey__SWIG_1(long jarg1,
Enquire jarg1_, long jarg2, Sorter jarg2_);
^
./XapianJNI.java:194: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void
Enquire_setSortByKeyThenRelevance__SWIG_0(long jarg1, Enquire jarg1_, long
jarg2, Sorter jarg2_, boolean jarg3);
^
./XapianJNI.java:195: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void
Enquire_setSortByKeyThenRelevance__SWIG_1(long jarg1, Enquire jarg1_, long
jarg2, Sorter jarg2_);
^
./XapianJNI.java:196: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void
Enquire_setSortByRelevanceThenKey__SWIG_0(long jarg1, Enquire jarg1_, long
jarg2, Sorter jarg2_, boolean jarg3);
^
./XapianJNI.java:197: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void
Enquire_setSortByRelevanceThenKey__SWIG_1(long jarg1, Enquire jarg1_, long
jarg2, Sorter jarg2_);
^
./XapianJNI.java:457: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native String Sorter_apply(long jarg1, Sorter jarg1_,
long jarg2, Document jarg2_);
^
./XapianJNI.java:460: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void Sorter_director_connect(Sorter obj, long
cptr, boolean mem_own, boolean weak_global);
^
./XapianJNI.java:461: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public final static native void Sorter_change_ownership(Sorter obj, long
cptr, boolean take_or_release);
^
./XapianJNI.java:463: cannot find symbol
symbol : class MultiValueSorter
location: class XapianJNI
public final static native String MultiValueSorter_apply(long jarg1,
MultiValueSorter jarg1_, long jarg2, Document jarg2_);
^
./XapianJNI.java:464: cannot find symbol
symbol : class MultiValueSorter
location: class XapianJNI
public final static native void MultiValueSorter_add__SWIG_0(long jarg1,
MultiValueSorter jarg1_, long jarg2, boolean jarg3);
^
./XapianJNI.java:465: cannot find symbol
symbol : class MultiValueSorter
location: class XapianJNI
public final static native void MultiValueSorter_add__SWIG_1(long jarg1,
MultiValueSorter jarg1_, long jarg2);
^
./XapianJNI.java:492: cannot find symbol
symbol : class Sorter
location: class XapianJNI
public static String SwigDirector_Sorter_apply(Sorter self, long doc) {
^
./Enquire.java:99: cannot find symbol
symbol : class Sorter
location: class Enquire
public void setSortByKey(Sorter sorter, boolean ascending) {
^
./Enquire.java:103: cannot find symbol
symbol : class Sorter
location: class Enquire
public void setSortByKey(Sorter sorter) {
^
./Enquire.java:107: cannot find symbol
symbol : class Sorter
location: class Enquire
public void setSortByKeyThenRelevance(Sorter sorter, boolean ascending) {
^
./Enquire.java:111: cannot find symbol
symbol : class Sorter
location: class Enquire
public void setSortByKeyThenRelevance(Sorter sorter) {
^
./Enquire.java:115: cannot find symbol
symbol : class Sorter
location: class Enquire
public void setSortByRelevanceThenKey(Sorter sorter, boolean ascending) {
^
./Enquire.java:119: cannot find symbol
symbol : class Sorter
location: class Enquire
public void setSortByRelevanceThenKey(Sorter sorter) {
^
./DateValueRangeProcessor.java:39: DateValueRangeProcessor(long,boolean) is
already defined in DateValueRangeProcessor
public DateValueRangeProcessor(long valno_, boolean prefer_mdy_) {
^
./Enquire.java:100: cannot find symbol
symbol : variable Sorter
location: class Enquire
XapianJNI.Enquire_setSortByKey__SWIG_0(swigCPtr, this,
Sorter.getCPtr(sorter),
sorter, ascending);
^
./Enquire.java:104: cannot find symbol
symbol : variable Sorter
location: class Enquire
XapianJNI.Enquire_setSortByKey__SWIG_1(swigCPtr, this,
Sorter.getCPtr(sorter),
sorter);
^
./Enquire.java:108: cannot find symbol
symbol : variable Sorter
location: class Enquire
XapianJNI.Enquire_setSortByKeyThenRelevance__SWIG_0(swigCPtr, this,
Sorter.getCPtr(sorter), sorter, ascending);
^
./Enquire.java:112: cannot find symbol
symbol : variable Sorter
location: class Enquire
XapianJNI.Enquire_setSortByKeyThenRelevance__SWIG_1(swigCPtr, this,
Sorter.getCPtr(sorter), sorter);
^
./Enquire.java:116: cannot find symbol
symbol : variable Sorter
location: class Enquire
XapianJNI.Enquire_setSortByRelevanceThenKey__SWIG_0(swigCPtr, this,
Sorter.getCPtr(sorter), sorter, ascending);
^
./Enquire.java:120: cannot find symbol
symbol : variable Sorter
location: class Enquire
XapianJNI.Enquire_setSortByRelevanceThenKey__SWIG_1(swigCPtr, this,
Sorter.getCPtr(sorter), sorter);
^
26 errors
make[1]: *** [Auto.class] Error 1
make[1]: Leaving directory `/home/jhr/xapian-bindings-1.0.5/java-swig'
make: *** [all] Error 2
[[email protected: j...@localhost] java-swig]$
4 Replies
|
|
|
[typo] How to write typo plugins
By Ulf Moehring at Mar 24, 2008, 01:18 am UTC
Hi all, I wrote a quick & dirty image gallery plugin for my Typo blog a regular Rails plugin, which means I'm overwriting a few files of the original typo install. So ... I'm wondering if there is a better way to hook up new views to the administration interface or extend existing models. Do you... More...
Hi all,
I wrote a quick & dirty image gallery plugin for my Typo blog
(5.0.2). It seems to run all right but I wrote it like I would write
a regular Rails plugin, which means I'm overwriting a few files of
the original typo install.
So ... I'm wondering if there is a better way to hook up new views to
the administration interface or extend existing models. Do you guys
have any suggestions?
Thanks,
Ulf
0 Replies
|
|
|
[typo] Ticket #1210 - content state model flawed
By Benjamin Jackson at Mar 20, 2008, 3:09 pm UTC
Hi all, Just wanted to email the list as my ticket has not been addressed and I'm finding more holes in the state model... just looked at the publication_pending state, and found this: def change_published_state(content, published) content[:published] = published if published &&... More...
Hi all,
Just wanted to email the list as my ticket has not been addressed and I'm
finding more holes in the state model... just looked at the
publication_pending state, and found this:
def change_published_state(content, published)
content[:published] = published
if published && content.published_at <= Time.now
content.state = JustPublished.instance
end
end
def set_published_at(content, new_time)
content[:published_at] = new_time
Trigger.remove(content, :trigger_method => 'publish!')
if new_time.nil?
content.state = Draft.instance
elsif new_time <= Time.now
content.state = JustPublished.instance
end
end
Unless I'm mistaken, these are called when published and pubished_at are set
in the model, right?
In the metaweblog service, the new post and edit post functions set
published and published_at respectively as:
article.published = publish
article.author = username
article.published_at = struct['dateCreated'].to_time.getlocal rescue
Time.now
article.published = publish
article.author = username
article.published_at = struct['dateCreated'].to_time.getlocal unless
struct['dateCreated'].blank?
So in other words, as a client of the API I have to know that unless I set
dateCreated to the future when creating a post, and then never set it at all
when editing a post, my post will always be marked for publication even
though I'm explicitly sending published=false.
Please let me know if I'm off here, but I'd love to see this addressed as my
ticket has been stale for 2-3 weeks. Thanks,
Ben
1 Reply
|
|
|
UTF-8 Corruption
By Colin Bell at Mar 20, 2008, 2:24 pm UTC
Hi All I was wondering if anyone every came across a problem I seem to be having. I'm indexing in text files using some basic code written in C+ +. The text files may or may not be in UTF-8, ISO 8859-1 or possibly Question is, does Xapian convert none UTF-8 characters when it stores the document. I... More...
Hi All
I was wondering if anyone every came across a problem I seem to be
having. I'm indexing in text files using some basic code written in C+
+. The text files may or may not be in UTF-8, ISO 8859-1 or possibly
(but very rarely) even some other format - I have no way of knowing.
Question is, does Xapian convert none UTF-8 characters when it stores
the document. I think I read that UTF-8 is the default encoding for
Xapian, which is exactly what I am after.
The reason I'm asking is that I am getting some seriously corrupted
characters in the index. When they are displayed on Tomcat I get a
"sun.io.MalformedInputException" when trying to display the search
results. I have set the pages charset to UTF-8 and apparently this
error is thrown when when the streamreader detects characters that are
not proper UTF-8 characters.
I know my query may seem naive,but I would really appreciate any
insight you may be willing to offer on this.
Many thanks
Colin
4 Replies
|
|
|
[typo] Theme downloads...
By Timothy Freund at Mar 20, 2008, 04:19 am UTC
Hey Everyone -- I migrated to a new server, and the Typo theme repository gracefully. I'm in the process of restoring it, but the upload is taking a while. (My DSL upload speed leaves much room for improvement). I hope to get it back up and running before I call it a night, but if it takes a... More...
Hey Everyone --
I migrated to a new server, and the Typo theme repository
(http://proofread.digital-achievement.com) didn't move all that
gracefully. I'm in the process of restoring it, but the upload is
taking a while. (My DSL upload speed leaves much room for improvement).
I hope to get it back up and running before I call it a night, but if it
takes a *really* long time, it won't be until tomorrow night that things
are back to normal.
Just a reminder, if anyone wants to submit patches or new themes, I'm
easy to please: just have an interest in the work and send me the output
of "htpasswd -ns USERNAME". The username 'tim' is taken, but you're
free to pick anything else. ;-)
Thanks for your patience,
Tim
0 Replies
|
|
|
Re: Document snippet generation
By Olly Betts at Mar 19, 2008, 9:05 pm UTC
Xapian doesn't currently have such a feature. There's a wishlist bug open for it, which links to some previous list discussion: http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=211 Cheers, Olly More...
On Tue, Jan 15, 2008 at 12:02:04AM +0100, Ycrux wrote:
> I'm looking for a good way to get a good "snippet text"
> for a personal search engine based Xapian when showing the
> result.
Xapian doesn't currently have such a feature. There's a wishlist bug
open for it, which links to some previous list discussion:
http://www.xapian.org/cgi-bin/bugzilla/show_bug.cgi?id=211
Cheers,
Olly
14 Replies
|
|
|
Re: Xapian performance testing
By Arjen van der Meijden at Mar 19, 2008, 7:31 pm UTC
We finally found time to upgrade our 0.9.8 to 1.0.5 and reindex the whole database. The results so far are quite good. When taking the daily average of our forum search result page it went down from about 0.55 seconds to about 0.38 seconds. Looking at the log-files, the slow query log file (queries... More...
On 5-11-2007 14:40, Olly Betts wrote:
> BTW, I have implemented the hoisting of the positional information
> checking part of NEAR and PHRASE, so that the "AND" inside can be
> merged with other AND and FILTER operations. This gave a big
> performance boost to the slow queries (~50% saving in time just
> from this one change) and a good boost to the other queries (~25%
> saving from just this change).
>
> This optimisation and all the earlier ones are in 1.0.4, so once you
> upgrade to that, it would be interesting to see what the slow query log
> looks like with these new optimisations in place.
We finally found time to upgrade our 0.9.8 to 1.0.5 and reindex the
whole database. The results so far are quite good.
When taking the daily average of our forum search result page it went
down from about 0.55 seconds to about 0.38 seconds.
Looking at the log-files, the slow query log file (queries taking more
than 2 seconds) dramatically reduced in size. Prior to the update we had
3190 and 3112 lines in a week and now in the latest week it had only 863
lines.
The slowest queries seem to be the single-term phrases with a single
character attached to a common word like 'd-link', 's-video' and
variants on that with only a few additional terms. As expected, I don't
see any version numbers anymore in the slow query log.
Best regards,
Arjen van der Meijden
Tweakers.net
3 Replies
|
|
|
scriptindex + zombie processes
By Jim Spath at Mar 19, 2008, 2:13 pm UTC
Last night we experienced a problem with scriptindex where it spawned many (80 or so) zombie processes. This same problem occurred earlier this month as well, and also a couple of times in January. It actually spawned hundreds of these processes during these previous occurrences. Has anyone else... More...
Last night we experienced a problem with scriptindex where it spawned
many (80 or so) zombie processes. This same problem occurred earlier
this month as well, and also a couple of times in January. It actually
spawned hundreds of these processes during these previous occurrences.
Has anyone else experienced anything like this?
scriptindex has generally been quite well behaved for us, we run it
almost constantly, so we're not quite sure what triggers the problem.
Thanks!
Jim
0 Replies
|
|
|
[typo] Typo Integration with existing application
By Tom Cocca at Mar 19, 2008, 1:19 pm UTC
Hello, I am thinking about using Typo as a blog for my existing site. My question is, in my current rails site I already have an authentication system, acts_as_authenticated, is there anyway to integrate this login with Typo so that users do not have to log into the blog separately? Has anyone... More...
Hello,
I am thinking about using Typo as a blog for my existing site. My
question is, in my current rails site I already have an authentication
system, acts_as_authenticated, is there anyway to integrate this login
with Typo so that users do not have to log into the blog separately?
Has anyone installed the type of a system before? Also, with
acts_as_authenticated I am also using the authorization plugin to give
roles to users.
Any ideas? Thank you all for your time and I apologize if you have
answered this before but my searches didn't yield any results in this topic.
Thanks,
~ Tom Cocca
2 Replies
|
|
|
simple frequency list
By s|s at Mar 18, 2008, 04:22 am UTC
I wrote a python script for finding frequency of terms in database. Thought of sharing since it could would come handy for introspecting database. Possibly it could be included as an example to python bindings.. regards supreet More...
I wrote a python script for finding frequency of terms in database.
Thought of sharing since it could would come handy for introspecting
database. Possibly it could be included as an example to python
bindings..
regards
supreet
1 Reply 
|
|
|
Spell Checking
By Martin Hearn at Mar 18, 2008, 04:03 am UTC
Hi I wonder if someone could answer these two questions about spelling correction: 1) Is there any way you can get more than one spelling suggestion from get_spelling_suggestion(); 2) Can anyone confirm that any suggestion given won't necessarily return any results. Many thanks for a great product!... More...
Hi
I wonder if someone could answer these two questions about spelling
correction:
1) Is there any way you can get more than one spelling suggestion from
get_spelling_suggestion();
2) Can anyone confirm that any suggestion given won't necessarily
return any results.
Many thanks for a great product!
Martin
5 Replies
|
|
|
Enquire set_cutoff problem
By Donato Di Leo at Mar 18, 2008, 02:56 am UTC
while testing Xapian we found out that the function set_cutoff doesn't work when called from php on calling set_cutoff this error occurs: Fatal error: No matching function for overloaded 'Enquire_set_cutoff' in /srv/www/htdocs/xapian.php on line 702 I tried with one and two parameters but the error... More...
while testing Xapian we found out that the function set_cutoff doesn't
work when called from php
on calling set_cutoff this error occurs:
Fatal error: No matching function for overloaded 'Enquire_set_cutoff' in
/srv/www/htdocs/xapian.php on line 702
I tried with one and two parameters but the error is always the same.
I saw the function definition in xapian.php and I think I'm using it in
a good way
I tried:
$enquire->set_cutoff($threshold);
and
$enquire->set_cutoff($threshold,0.0);
and
$enquire->set_cutoff(70,0.0);
and
$enquire->set_cutoff(70);
where of course $enquire is well defined and in any case the error is
the same.
of course the other functions like $enquire->set_sort_by_value() work
fine.
I'm stopped waiting for a solution for this very important problem.
Thx
Donato
7 Replies
|
|
|
Memory consume issue
By chun yu at Mar 13, 2008, 07:43 am UTC
Hi, all When there are many files to index, the Omindex continues to consume the memory and never to release till it stops which cause too much memory is occupied. check the source code: follwing codes are found. // FIXME: this should be done by checking memory usage, not the number of // changes.... More...
Hi, all When there are many files to index, the Omindex continues to consume the memory and never to release till it stops which cause too much memory is occupied. check the source code: follwing codes are found. // FIXME: this should be done by checking memory usage, not the number of // changes. // We could also look at: // * mod_plists.size() // * doclens.size() // * freq_deltas.size() // // cout << "+++ mod_plists.size() " << mod_plists.size() << // ", doclens.size() " << doclens.size() << // ", freq_deltas.size() " << freq_deltas.size() << endl; if (++change_count >= flush_threshold) { flush_postlist_changes(); if (!transaction_active()) apply(); }Looks the author also noticed the memory usage.Could you please tell me the progress of the issue's fixing? Thanks a lot!
_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
0 Replies
|
|
|
how can i use stopwords?
By Wojewsky, Sascha, Springer BauMedien at Mar 12, 2008, 09:35 am UTC
Hi, I do not understand the stopword function... I've set the termgenerator like this: $self->{'Stemmer'} = new Search::Xapian::Stem(german2); $self->{'Stopper'} = new Search::Xapian::SimpleStopper(); $self->{'TermGenerator'} = new Search::Xapian::TermGenerator;... More...
Hi,
I do not understand the stopword function...
I've set the termgenerator like this:
$self->{'Stemmer'} = new Search::Xapian::Stem(german2);
$self->{'Stopper'} = new Search::Xapian::SimpleStopper();
$self->{'TermGenerator'} = new Search::Xapian::TermGenerator;
$self->{'TermGenerator'}->set_stemmer( $self->{'Stemmer'} );
$self->{'TermGenerator'}->set_stopper( $self->{'Stopper'} );
I've thought that xapian now exclude the stopwords automatic by the stemming-language.
Could someone explain the stopword function?
Thanks.
Greetings
Sascha
1 Reply
|
|
 | |
