Apache Solr for Drupal 6 Integration - Phase I - Research

Background Our company hosts several websites that presently have large and ever-growing databases. Queries have become more cumbersome and Drupal Core search has gotten slow. We feel that scalability is a two part endeavor. 1) Create and archival system so as to limit database growth and 2) implement Solr as a mechanism to aid in efficient data processing for the lifetime of the applications. The scope of this project is solely concerned with fleshing the second part - 2) Solr integration. In general we’re looking for someone to take a highly consultative approach to our ideas. We are Drupal developers, but we are not experts in Solr, and we are very (very) busy. We are happy to hear many differing ideas around how to best make this integration happen. Please see the subsequent sections for further information on what our desired outcomes are. Our websites are presently running Drupal 5, but we are beginning the upgrade process to get them ready for Drupal 6. Phase II of this project will be assigned separately, but will ultimately implement the findings in this research project, part I. Attached is the Word version of this description. It is far more legible. Timeline The research should be completed within two weeks of assignment. A) Desired Architecture Please comment upon each point herein and make recommendations for how to proceed with the work involved. Your recommendations need not agree with the ideas below. Solr as a standalone server Almost every webpage will be hitting Solr for data. We estimate that a dedicated server will be best. Solr running on Tomcat or Jetty. Why? Solr version? Tuning Solr to allow for it to utilize the majority of server resources Solr acting as a service to the afore-mentioned websites Inter-application security - no website may access anything other than their Solr core/index Secured to the outside world - only our servers should be able to reach the Solr server. All our servers are presently in the same data center and can be privately networked. Ideally we’d be able to whitelist by IP so that our developers could call the Solr server directly (optional) B) Desired Solr Features, Questions, & Considerations Please comment upon each point herein and make recommendations for how to proceed with the work involved. Your recommendations need not agree with the ideas below. What is the query language? Boolean? NOT-logic? NOT-logic with facets? Advanced boolean such as proximity, wildcard, stemming? Provide documentation Field-specific (as compared to taxonomy-specific) searching (date ranges, authors, nodetype-specific fields) Faceted search Can this include NOT logic? Autocomplete as you type (spell check? Suggestions?) File indexing Solr-provided contextual extract Really only need to search nodes at present. No need to support user or other searches. Suggestions for rich features that we may be missing C) Drupal Considerations & Constraints Drupal modules MUST be Drupal 6 MUST work with Drupal’s access rights scheme We do not run views or CCK. Most content types are very simple, but can vary from website to website What are the appropriate modules required to give us the features listed above We’d like to get by with the fewest modules possible No multi-lingual requirements Deliverables A detailed Word document explaining and expanding on ALL the points above in parts A, B, and C. It should include: A comprehensive look at each bullet point Your ideas and opinions Issues and constraints As many screenshots and visuals as is possible