Serious Problem with Zend_Search: Searches aren't working

[Brendan W. McAdams]

Having serious problems with Zend_search and I don't even know where
to begin.

The code we have worked previously, but now our index has grown to
about 3000 documents and at some point things just stopped functioning.

If I dump $index->terms() it appears correct: terms I'm looking for
appear associated with the proper documents.

I've tried an $index->optimize() as well but it seems to have not
helped things.

When I do searches, it's just flat out weird: the documents returned
don't contain the terms searched for (i've tried a variety of ways of
searching including simple one-word searches). The returned
documents are so far off base as to confuse me.

I'm lost at this point, and looking for some guidance towards what to
do to solve this problem ASAP.

[Shahar Evron]

Hi,

You should try opening your index with Luke http://www.getopt.org/luke/
and performing the same search - check if you get the same results. It
will be interesting to see if Luke provides the same results (meaning
something is wrong with the index) or different one (perhaps something
is wrong with the search engine).

Shahar.

Post by Brendan W. McAdams
Having serious problems with Zend_search and I don't even know where
to begin.
The code we have worked previously, but now our index has grown to
about 3000 documents and at some point things just stopped functioning.
If I dump $index->terms() it appears correct: terms I'm looking for
appear associated with the proper documents.
I've tried an $index->optimize() as well but it seems to have not
helped things.
When I do searches, it's just flat out weird: the documents returned
don't contain the terms searched for (i've tried a variety of ways of
searching including simple one-word searches). The returned
documents are so far off base as to confuse me.
I'm lost at this point, and looking for some guidance towards what to
do to solve this problem ASAP.

[Brendan W. McAdams]

Yup.

I pulled down Luke last night, and scarily enough it found the same
problem. It seems the hash table internal to the database was
corrupted... Dumping TERMS pointed specific words (e.g. words that
were very esoteric and only existed in a single document) at the
right place. But SEARCHING on said term would bring back 15
documents that were completely unrelated and did not contain any of
the words being searched for (be it said esoteric word or a more
general search).

Fortunately do to the nature of our data (Web page content which is
stored elsewhere) I was able to rebuild our database. The rebuild
worked and produced a proper database, which searches correctly.

I believe we were using 0.92 yesterday; I upgraded to 0.93 when
things started looking weird (still got bad reads, but then again,
LUKE couldn't read it either). The rebuild happened with 0.93 and
seems safe.

Looking back at the changelog I do see some changes related to index
corruption with optimisation in 0.93, although we had not run an
optimize before the corruption occurred (Is there an auto-corruption
threshhold?).

My concern obviously is that over the next few days as this index
continues to grow, we may see corruption again. Any thoughts?

[Brendan W. McAdams]

Sorry. Freud would love me this morning. I meant an auto-
optimisation threshhold - e.g. at a certain size doe the index try to
optimize itself, and is it possible the 0.92 optimization bug
triggered (And are we sure it's fixed in 0.93)?

(Is there an auto-corruption threshhold?)

[Shahar Evron]

There is an auto-optimization process, see
http://framework.zend.com/manual/en/zend.search.index-creation.html#zend.search.index-creation.optimization

I am not sure if ZF-915 or any of the other fixed bugs are related -
JIRA seems to be down again for some reason. In any case I didn't write
the code so I'm not too knowledgeable about this ;)

Shahar.

Post by Brendan W. McAdams
Sorry. Freud would love me this morning. I meant an
auto-optimisation threshhold - e.g. at a certain size doe the index
try to optimize itself, and is it possible the 0.92 optimization bug
triggered (And are we sure it's fixed in 0.93)?

(Is there an auto-corruption threshhold?)

[Brendan W. McAdams]

Seems to have happened again, only about a day after we re-created
our index.

This is obviously a bad bad bad thing... anyoen got tips for
recovering the index and/or keeping it from happening again?

Post by Shahar Evron
There is an auto-optimization process, see
http://framework.zend.com/manual/en/zend.search.index-
creation.html#zend.search.index-creation.optimization
I am not sure if ZF-915 or any of the other fixed bugs are related -
JIRA seems to be down again for some reason. In any case I didn't write
the code so I'm not too knowledgeable about this ;)
Shahar.

Post by Brendan W. McAdams
Sorry. Freud would love me this morning. I meant an
auto-optimisation threshhold - e.g. at a certain size doe the index
try to optimize itself, and is it possible the 0.92 optimization bug
triggered (And are we sure it's fixed in 0.93)?

(Is there an auto-corruption threshhold?)

[Brendan W. McAdams]

Well, I found a core cause of the issue...

It seems the index was trying to auto-optimise and exceeding
maximum_execution_time.

When I set that to 0, and optimized, I saw my index start to look
more...sane. It's definitely still off, but there's a clear bug:

If max_execution_time hits, while the index is being optimized, it
severely corrupts the database.

Running an Optimize seems to help, but not completely recover the db.

This may not be a fixable bug...

But everyone should be aware.

My recommendation would be some kind of warning generated if
Zend_Search_lucene sees a ridiculously low (E.g. default)
max_execution_time.

[Alexander Veremyev]

Hi,

Great thanks for your research!
But I am not sure the problem is in the termination during optimization...

Index optimization (or auto-optimization) is actually segment merging
procedure. It just takes several (or all) index segments and merges them
into new one. When it's done, new segments list is generated. It's
written into 'segments.new' file and then it's renamed ti 'segments'.
New merged segments are removed from the list and new is added into it.

New segments list generation is surrounded by exclusive index lock and
segments switching is atomic filesystem operation.

So script termination can't corrupt index.


All segments are completely independent peaces of data.
Segment names generation is auto-incremental atomic operation.
So optimization/auto-optimization shouldn't corrupt index...


PS Zend_Search_Lucene uses flock() for locking. It doesn't work on NFS
and some others network file systems.
Which OS/filesystem do you use? Could you check, that flock()
(http://www.php.net/manual/en/function.flock.php) works correctly?


With best regards,
Alexander Veremyev.

Post by Brendan W. McAdams
Well, I found a core cause of the issue...
It seems the index was trying to auto-optimise and exceeding
maximum_execution_time.
When I set that to 0, and optimized, I saw my index start to look
If max_execution_time hits, while the index is being optimized, it
severely corrupts the database.
Running an Optimize seems to help, but not completely recover the db.
This may not be a fixable bug...
But everyone should be aware.
My recommendation would be some kind of warning generated if
Zend_Search_lucene sees a ridiculously low (E.g. default)
max_execution_time.