Date post: | 19-Dec-2014 |
Category: |
Technology |
Upload: | mark-harwood |
View: | 1,246 times |
Download: | 3 times |
Lucene and Bloom-Filtered SegmentsPerformance improvements to be gained from “knowing what we don’t know”
Mark Harwood
Benefits
2 x speed up on primary-key lookups
Small speed-up on general text searches (1.06 x )
Optimised memory overhead
Minimal impact on indexing speeds
Minimal extra disk space
ApproachOne appropriately sized Bitset is held per segment,
per Bloom-filtered field
e.g. 4 segments x 2 filtered fields = 8 bitsets
Segment 1
Segment 2
Segment 3
Segment 4
URL 000010001000000101000001
PKey 0010000001001001000001
URL 000010001000001
PKey 001000000100001
URL 000000001
PKey 001000001
URL 000000001
PKey 001000001
Fail-fast searches: modified TermInfosReaderint hash=searchTerm.hashcode();
int bitIndex=hash%bitsetSize;
if(!bitset.contains(bitIndex)) return false;
//term might be in index – continue as normal search
Segment 1
URL 000010001000000101000001
PKey 0010000001001001000001
Is most effective on fields with many low doc-frequency terms or scenarios where query terms often don’t exist in the index.
An unset bit guarantees the term is missing from the segment and a search can be avoided.
Memory efficiencyBitset sizes are automatically tuned according to:
1. the volume of terms in the segment
2. desired saturation settings (more sparse=more accurate)
Segment 1
Segment 2
Segment 3
Segment 4
URL 000010001000000101000001
PKey 0010000001001001000001
URL 000010001000001
PKey 001000000100001
URL 000000001
PKey 001000001
URL 000010000
PKey 001000001
Indexing: a modified TermInfosWriter
00000000000000000000000000000000001000000000000000000000000000010000000
000000000000001000000000010000000
Term writes are gathered in a large bitset
The final flush operation consolidates information in the big bitset into a suitably compact bitset for storage on disk based on how many set bits were accumulated.This re-mapping saves disk space and the RAM required when servicing queries
NotesSee JIRA LUCENE-4069 for patch to Lucene 3.6
Core modifications pass existing 3.6 Junit tests (but without exercising any Bloom filtering logic).
Benchmarks contrasting Bloom-filtered indexes with non-filtered are here: http://goo.gl/X7QqU
TODOsCurrently relies on a field naming convention to introduce a Bloom filter to the index (use “_blm” on end of indexed field name when writing)How to properly declare need for Bloom filter?Changes to IndexWriterConfig? A new Fieldable/FieldInfo setting? Dare I invoke the “schema” word?Where to expose tuning settings e.g. saturation preferences?Can we give some up-front hints to TermInfosWriter about segment size being written so initial choice of BitSet size can be reduced?Formal Junit tests required to exercise Bloom-filtered indexes – no false negatives. Can this be covered as part of existing random testing frameworks which exercise various index config options?