Is the Lucene query language hack proof

Obviously it cannot be used to trash the index or crack card numbers, passwords etc. (unless one is stupid enough to put card numbers or passwords in the index).

Is it possible to bring down the server with excessively complex searches?

I suppose what I really need to know is can I pass a user-entered Lucene query directly to the search engine without sanitization and be safe from malice.

Answers


It is impossible to modify the index from the input of a query parser. However, there are several things that could hurt a search server running Lucene:

  • A high value for the number of top results to collect

Lucene puts hits in a priority queue to order them (which is implemented with a backing array of the size of the priority queue). So running a request which fetches the results from offset 99 999 900 to offset 100 000 000 will make the server allocate a few hundred of megabytes for this priority queue. Running several queries of this kind in parallel is likely to make the server run out of memory.

  • Sorting on arbitrary fields

Sorting on a field requires the field cache of this field to be loaded. In addition to taking a lot of time, this operation will use a lot of memory (especially on text fields with a lot of large distinct values), and this memory will not be reclaimed until the index reader for which this cache has been loaded is not used anymore.

  • Term dictionary intensive queries

Some queries are more expensive than other ones. To prevent query execution from taking too long, Lucene already has some guards against too complex queries: by default, a BooleanQuery cannot have more than 1024 clauses.

Other queries such as wildcard queries and fuzzy queries are very expensive too.

To prevent your users from hurting your search service, you should decide what they are allowed to do and what they are not. For example, Twitter (which uses Lucene for its search backend) used to limit queries to a few clauses in order to be certain to provide the response in reasonable time. (This question Twitter api - search too complex? talks about this limitation)


As far as I know, there are no major vulnerabilities that you need to worry about. Depending on the query parser you are using, you may want to do some simple sanitization.

  • Limit the length of the query string
  • Check for characters that you don't want to support. For example, +, -, [, ], *
  • If you let the user pick the number of results returned (e.g. 10, 20, 50), then make sure they can't use a really large value.

Need Your Help

What is the ultimate program to make a drawing of a database model?

drawing data-modeling

One of the first things I do when I'm on a new project is design a database model. To visualize the model I use a 7 year old version of Smartdraw. Maybe it's time for something new. What is the ult...

Merge lines between two patterns using sed

sed grep

I have an output file that looks like this: