Comments for The Knut Hellan Blog http://knuthellan.com General geekyness and starting a company Sat, 21 Nov 2009 22:27:58 +0000 http://wordpress.com/ hourly 1 Comment on The CouchDB indexer – lightweight search engine in hours by knuthellan http://knuthellan.com/2009/07/09/the-couchdb-indexer-lightweight-search-engine-in-hours/#comment-33 knuthellan Sat, 21 Nov 2009 22:27:58 +0000 http://knuthellan.com/?p=185#comment-33 Just have to say that as of CouchDB 0.10, this approach will probably not work. That being said, it's still pretty easy to do an indexer in CouchDB, but you either have to do more work in the mapper or summarize outside of CouchDB. While this takes some elegance out of the solution, I believe it will be more viable for larger data sets. Just have to say that as of CouchDB 0.10, this approach will probably not work. That being said, it’s still pretty easy to do an indexer in CouchDB, but you either have to do more work in the mapper or summarize outside of CouchDB. While this takes some elegance out of the solution, I believe it will be more viable for larger data sets.

]]> Comment on The CouchDB indexer – lightweight search engine in hours by Scott http://knuthellan.com/2009/07/09/the-couchdb-indexer-lightweight-search-engine-in-hours/#comment-29 Scott Tue, 18 Aug 2009 17:09:28 +0000 http://knuthellan.com/?p=185#comment-29 If you are interested, I wrote about my own experiences using CouchDB to compute TF-IDF: http://www.elusivesnark.com/2009/08/computing-tf-idf-with-couchdb-and.html There doesn't seem to be much written about this yet so I would be happy to hear your or others opinions. Cheers! If you are interested, I wrote about my own experiences using CouchDB to compute TF-IDF: http://www.elusivesnark.com/2009/08/computing-tf-idf-with-couchdb-and.html

There doesn’t seem to be much written about this yet so I would be happy to hear your or others opinions.

Cheers!

]]> Comment on The CouchDB indexer – lightweight search engine in hours by knuthellan http://knuthellan.com/2009/07/09/the-couchdb-indexer-lightweight-search-engine-in-hours/#comment-27 knuthellan Fri, 14 Aug 2009 13:45:19 +0000 http://knuthellan.com/?p=185#comment-27 I've tried to torture CouchDB with this particular map reduce the last couple of days and it seems like it's too much for it at least in the standard configuration. I used pages from the Norwegian wikipedia and while I've never seen the reduce_over_flow error, it quickly becomes dead slow. I started with 10000 articles and the map reduce seemed to get almost stuck at a little above 300 tasks (out of 10000). I stopped it and tested with fewer documents, but I didn't find the pain point, but it's somewhere between 100 and 10000, based on the point it got stuck earlier I guess it's around 300. It might be that the reduce_over_flow you see is a hint that you need to handle rereduce. Change the reduce function to function(keys, values, rereduce) { var docs = []; if (rereduce) { for (var i = 0; i < values.length; ++i) { for (var j = 0; j < values[i].length; ++j) docs.push(values[i][j]); } return docs; } for (var i = 0; i < values.length; ++i) docs.push(values[i]); return docs; } and handle rereduce. I’ve tried to torture CouchDB with this particular map reduce the last couple of days and it seems like it’s too much for it at least in the standard configuration.

I used pages from the Norwegian wikipedia and while I’ve never seen the reduce_over_flow error, it quickly becomes dead slow. I started with 10000 articles and the map reduce seemed to get almost stuck at a little above 300 tasks (out of 10000). I stopped it and tested with fewer documents, but I didn’t find the pain point, but it’s somewhere between 100 and 10000, based on the point it got stuck earlier I guess it’s around 300.

It might be that the reduce_over_flow you see is a hint that you need to handle rereduce.

Change the reduce function to

function(keys, values, rereduce) {
var docs = [];
if (rereduce) {
for (var i = 0; i < values.length; ++i) {
for (var j = 0; j < values[i].length; ++j) docs.push(values[i][j]);
}
return docs;
}
for (var i = 0; i < values.length; ++i) docs.push(values[i]);
return docs;
}

and handle rereduce.

]]> Comment on The CouchDB indexer – lightweight search engine in hours by Scott http://knuthellan.com/2009/07/09/the-couchdb-indexer-lightweight-search-engine-in-hours/#comment-26 Scott Tue, 11 Aug 2009 20:08:28 +0000 http://knuthellan.com/?p=185#comment-26 It could also be a case of my own experience :) I have only been playing with couchdb for 2 days now. I pasted a small example to friendpaste: http://friendpaste.com/2nQ841lJ7O8aP4iPBeRPgc Apologies if I have misread something. It could also be a case of my own experience :) I have only been playing with couchdb for 2 days now. I pasted a small example to friendpaste:

http://friendpaste.com/2nQ841lJ7O8aP4iPBeRPgc

Apologies if I have misread something.

]]> Comment on The CouchDB indexer – lightweight search engine in hours by knuthellan http://knuthellan.com/2009/07/09/the-couchdb-indexer-lightweight-search-engine-in-hours/#comment-25 knuthellan Tue, 11 Aug 2009 18:39:35 +0000 http://knuthellan.com/?p=185#comment-25 I only ran this with about 100 documents for a demo application showing off something else, but I will definitely run a test as well. If couchdb map reduce in general breaks down at 3000 documents that means it needs fixing. Could of course be a combined factor of document length and number of documents, but 3000 documents is still surprisingly low and should probably be much higher. I only ran this with about 100 documents for a demo application showing off something else, but I will definitely run a test as well. If couchdb map reduce in general breaks down at 3000 documents that means it needs fixing. Could of course be a combined factor of document length and number of documents, but 3000 documents is still surprisingly low and should probably be much higher.

]]> Comment on The CouchDB indexer – lightweight search engine in hours by Scott http://knuthellan.com/2009/07/09/the-couchdb-indexer-lightweight-search-engine-in-hours/#comment-24 Scott Tue, 11 Aug 2009 17:25:02 +0000 http://knuthellan.com/?p=185#comment-24 Did you test this approach on a non-trivial database? I tried following your example on a database with ~3000 documents and get reduce_over_flow errors. Did you test this approach on a non-trivial database? I tried following your example on a database with ~3000 documents and get reduce_over_flow errors.

]]> Comment on Lifestyle design by Mary http://knuthellan.com/2009/06/19/lifestyle-design/#comment-19 Mary Wed, 24 Jun 2009 03:44:29 +0000 http://knuthellan.com/?p=142#comment-19 Pretty cool post. I just found your blog and wanted to say that I've really enjoyed browsing your posts. Anyway I'll be subscribing to your feed and I hope you post again soon! Pretty cool post. I just found your blog and wanted to say
that I’ve really enjoyed browsing your posts. Anyway
I’ll be subscribing to your feed and I hope you post again soon!

]]> Comment on Lifestyle design by Jeremy http://knuthellan.com/2009/06/19/lifestyle-design/#comment-17 Jeremy Fri, 19 Jun 2009 13:25:31 +0000 http://knuthellan.com/?p=142#comment-17 Knut, Great post and I definitely do think that it what I was talking about on StartupStudent. I agree some people's idea of Lifestyle Design (such as Ferriss's) don't fit my desires either. Many people think that in order to believe in the lifestyle design concept you have to want to live in some distant land. I believe it's much more about living the life that you want to live before you retire. Thanks for the links! Jeremy Lattimore http://www.refocusingtechnology.com Knut,

Great post and I definitely do think that it what I was talking about on StartupStudent. I agree some people’s idea of Lifestyle Design (such as Ferriss’s) don’t fit my desires either. Many people think that in order to believe in the lifestyle design concept you have to want to live in some distant land. I believe it’s much more about living the life that you want to live before you retire.

Thanks for the links!

Jeremy Lattimore
http://www.refocusingtechnology.com

]]> Comment on Camping with CouchDB by Going Camping with CouchDB On OS X Tiger | Tim Gittos .com http://knuthellan.com/2009/03/08/camping-with-couchdb/#comment-15 Going Camping with CouchDB On OS X Tiger | Tim Gittos .com Fri, 12 Jun 2009 20:13:12 +0000 http://knuthellan.wordpress.com/?p=23#comment-15 [...] found this great article about getting Camping talking to CouchDB, and it seemed easy enough, so I decided to try it. Since [...] [...] found this great article about getting Camping talking to CouchDB, and it seemed easy enough, so I decided to try it. Since [...]

]]> Comment on First Week by knuthellan http://knuthellan.com/2009/04/15/first-week/#comment-13 knuthellan Fri, 01 May 2009 22:32:04 +0000 http://knuthellan.wordpress.com/?p=51#comment-13 I'll give you some information when we are ready for a little less stealth. It would be really cool to have you write about what we are doing :) I’ll give you some information when we are ready for a little less stealth. It would be really cool to have you write about what we are doing :)

]]>