The short answer is no. Search consists of data collection, creation of an index and ranking. Traditional data collection is done by crawlers following links. Obviously that’s not optimal for real-time search unless you treat oft-changing sites differ…

There doesn’t seem to be much written about this yet so I would be happy to hear your or others opinions.

Cheers!

I used pages from the Norwegian wikipedia and while I’ve never seen the reduce_over_flow error, it quickly becomes dead slow. I started with 10000 articles and the map reduce seemed to get almost stuck at a little above 300 tasks (out of 10000). I stopped it and tested with fewer documents, but I didn’t find the pain point, but it’s somewhere between 100 and 10000, based on the point it got stuck earlier I guess it’s around 300.

It might be that the reduce_over_flow you see is a hint that you need to handle rereduce.

Change the reduce function to

function(keys, values, rereduce) {
var docs = [];
if (rereduce) {
for (var i = 0; i < values.length; ++i) {
for (var j = 0; j < values[i].length; ++j) docs.push(values[i][j]);
}
return docs;
}
for (var i = 0; i < values.length; ++i) docs.push(values[i]);
return docs;
}

and handle rereduce.

http://friendpaste.com/2nQ841lJ7O8aP4iPBeRPgc

Apologies if I have misread something.