[Django]-Search engine solution for Django that actually works?

I have developed some Django applications with xapian support too. The biggest of them has a xapian database with an index of 8G storing 2.4M documents (including forum posts, wiki entries, planet entries and blog entries) – still growing.

Overall I am quite happy with xapian. It performs extremely well and is easy to use. The only thing I don’t like is that xapian won’t work with mod_wsgi (except of the global mode) because of a deadlock. So you are forced to use fastcgi (or connect to xapian-tcpsrv or write your own service).

I recommend you, to use the xapian-bindings directly. Xapian nowadays offers quite a lot of useful helpers (TermGenerator, QueryParser etc), which makes both the indexing and the querying simple. In fact, there is nothing I can imaging which would justify an additional library. In my opinion they are all more complicated and don’t allow you to index efficiently.

The only thing you need, is some understanding of the way how xapian is working. (What are terms? What are values? What is stemming and where should I use it? and so on). You can find all those topics on the xapian website, and as soon as you understand those concepts, dealing with xapian will become easy.

Also, the xapian API is extremly stable. I’ve started using it a long time before the 1.0 release and never had any problems with API changes or version conflicts. The only thing which has changed is that all those helpers (query parser, tokenizer, etc.) I have once written for my Django project are now useless, because similar classes have made their way into the xapian core.

So, to summarize, just give the direct usage of xapian-bindings a try.

I can vouch for Django-Haystack with the Xapian backend (In the interest of full disclosure, I am the author of the xapian-haystack backend) in a real life, production environment. We currently use Haystack/Xapian on several sites, the largest of which has more than 20,000 registered users and a Xapian database with 20,000+ documents containing more than 143,000 unique terms for a total size of ~141mb.

As for not being able to get any combination of Haystack and the Xapian backend running, I’ll admit that I was not as diligent as I should have been with my tagging and so there is some confusion with the versions. You should, however, be able to use the current master of both codebases without any issue. If this is not case, I’d be more than happy to assist with problems. You’ll need to be a little bit more specific about the issue though. Simply saying “it did not work” is not enough information.

Daniel and I both do our best to respond to any issues opened on Github within a timely manner. Also, we’re both usually available on the #haystack IRC channel during the day and the django-haystack Google Group.

Versions used:

• Haystack 1.0BETA with Xapian-Haystack 1.1.0BETA
• Haystack 1.0.1FINAL with Xapian-Haystack 1.1.3BETA

Most of the sites we’ve deployed with Haystack have been running Ubuntu 8.04 LTS with Xapian 1.0.5

Short answer: No.

We bailed and went with a Google Custom Search. Although the site has over 10,000 possible page views, we keep the sitemap feed down to the main 4,000 pages or so and it costs $250/year, which is about 2 hours of my time. The customer is happy and he feels comfortable with the results.

I’d love to see someone come up with a good FOSS solution, but in a commercial situation the TCO has got to make economic sense.

The details you requested.

• exact Linux distribution, release version – Ubuntu 9.04 & 9.10
• exact release version of Haystack (or equivalent) – Haystack 1.0 as well as master
• release version of search backend – The Solr & Whoosh backends included with Haystack
• exact release version of the search engine – Solr 1.3, Solr 1.4 & Whoosh 0.3.15
• publicly (!) available documentation how to set up all components exactly in the way that your installation was set up such that the minimal requirements above are met.
  • http://docs.haystacksearch.org/dev/installing_search_engines.html#solr (or #whoosh)

Beyond this, it’s the standard configuration bits from the tutorial, plus any additional overrides from (which I can’t link to, thanks Stack Overflow) as needed.

As the maintainer of Haystack, I’m actively running all of the above previous setups. The smallest Haystack installation (Haystack 1.0 + Whoosh) is ~600 documents. A slightly larger one (Haystack master + Solr 1.4) is ~4000 documents. The largest deployment I’m aware of (Haystack master + Solr 1.4) is ~3 million documents.

I generally try to avoid Stack Overflow, so don’t be surprised if you see nothing further from me. The mailing list is the best place for support, but given your responses thus far, I’m sure you’d rather just trash me here.

I (and my colleagues) have successfully used Haystack to achieve a fairly good search functionality.

It is easy to start with haystack and whoosh backend; and change to the Apache-Solr backend when performance of whoosh is not acceptable.

We really got to get around to write a detailed post about it with links to the projects where it works.

For now I can suggest you to have a look at this search: http://www.webdevjobshq.com/search/?q=rails implemented using Haystack with Apache-Solr backend. Or this: http://www.govbuddy.com/search/?q=Roy

Have you considered Sphinx? What are you using as you data store? It has a MySQL engine that works terrific. I think it meet most of your requirements except I’m not exactly certain how nicely it can be tied into Django-ORM.

I’m heavily considering using Sphinx in one of my own Django Apps to improve performance on an auto-suggest field that does a prefix and infix search on a corpus of 3.5 million records. But I haven’t got around to implementing it yet, so I can’t speak to Django+Sphinx integration. My only Sphinx experience is with the MySQL Engine and directly querying MySQL.

I use Djapian. It was quite simple to install and works great. There is an actual tutorial that covers basic use-cases and shows entire integration process.

Yes, it has some ambiguities but issue tracker is open and authors rapidly fixes bugs and add features.