Be Lazy & ScaleFull-Text Tagging Billions Of Messages

reverse mapping checking getaddrinfo for xxxxx [xxx.xxx.xxx.xxx] failed - POSSIBLE BREAK-IN ATTEMPT!

pam_unix(sshd:session): session opened for user xxxxxx by (uid=0)

reverse mapping checking getaddrinfo for xxxxx [xxx.xxx.xxx.xxx] failed - POSSIBLE BREAK-IN ATTEMPT!

pam_unix(sshd:session): session opened for user xxxxxx by (uid=0)

reverse mapping checking getaddrinfo for xxxxx failed

"reverse mapping"Phrase Query

failedTerm Query

Boolean Query AND, OR, NOT

"reverse mapping" AND failed

PercolatorTraditionally you design documents based on your data, store them into an index, and then define queries via the search API in order to retrieve these documents. The percolator works in the opposite direction. First you store queries into an index and then, via the percolate API, you define documents in order to retrieve these queries.https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html

reverse mapping checking getaddrinfo for xxxxx [xxx.xxx.xxx.xxx] failed - POSSIBLE BREAK-IN ATTEMPT!

reverse mapping checking getaddrinfo for xxxxx [xxx.xxx.xxx.xxx] failed - POSSIBLE BREAK-IN ATTEMPT!

"possible break-in attempt!"

"session opened"

/0-10

/173

$$$ $$$

Perco. Queries Index

Register Queries

In-Memory Index

Reverse mapping ...

Reverse mapping...

Perco. Req. Reversemapping ...

Perco. Resp.

ExecuteEachQuery

105s

160Tags

500000Runs

Use Query A Priori Knowledge via

Appropriate and Optimized Data-Structures

Early Termination

"reverse mapping"

pam_unix(sshd:session): session opened for user xxxxxx by (uid=0)

AND

failed reverse ?

[1, 2]"reverse mapping"

failed

Query Term Index

reverse --> 1mapping --> 2

failed --> 0

Query Clauses Rewritten Clauses

0

Register Queries

Query Term Index

failed --> 0reverse --> 1mapping --> 2

Reverse mapping checking getaddrinfo for xxxxx failed.Raw Message

[1, 2, -1, -1, -1, -1, 0]Message Rewritten in Query Space

0 --> true1 --> true2 --> true

Query Term Presence Bitset

Prepare Log

0failed

Simple Lookup

[1, 2, -1, -1, -1, -1, 0]Message Rewritten in Query Space

0 --> true1 --> true2 --> true

Query Term Presence Bitset

Term Query

[1, 2]"reverse mapping"

1. Quick Check / Early Termination

2. Actual Check~ contains

[1, 2, -1, -1, -1, -1, 0]Message Rewritten in Query Space

0 --> true1 --> true2 --> true

Query Term Presence Bitset

Phrase Query

AND/OR

Boolean Query

105s

160Tags

500000Runs

7.3s

x14.4Faster

8.8s

x22.2Faster

195s

320Tags500000Runs