core.search¤
Classes¤
BM25PlusCSR(
corpus: list[list[int]],
word2vec,
k1: float = 1.7,
b: float = 0.3,
delta: float = 0.65,
)
BM25+ with CSR inverted index: - doc_ids / tfs stored in contiguous arrays - indptr for token → posting list slicing - fully vectorized scoring
Bases: IntEnum
Search methods available
Attributes¤
class-attribute
instance-attribute
¤
Vector-based similarity on document centroid in embedding space
class-attribute
instance-attribute
¤
BM25+ keywords statistics on normalized and stemmed content
class-attribute
instance-attribute
¤
Combination of AI and FUZZY aggregated by Reciprocal Rank Fusion.
Indexer(
db: sqlite3.Connection,
name: str,
word2vec: WordEmbedding,
strip_collocations: bool = False,
principal_components: int = 1,
)
Search engine based on word similarity.
| PARAMETER | DESCRIPTION |
|---|---|
db
|
Opened SQLite database containing at least a
TYPE:
|
name
|
name under which the model will be saved for la ter reuse.
TYPE:
|
word2vec
|
the instance of word embedding model.
TYPE:
|
strip_collocations
|
remove the matrix of collocations in documents, which is the list of word tokens represented by their index in the word2vec dictionnary. It is used for core.search.Indexer.find_query_pattern, which is optional and significatively slower (but not significatively better), so if you don’t plan on using it, removing collocations saves some RAM and I/O.
TYPE:
|
principal_components
|
number of principal components to compute and remove from the index dataset. This helps to make queries more selective and specific in the presence of boilerplate text and formatting language in the sampling.
TYPE:
|
NOTE
The class is optimized to run online, on server: load fast when spawning a new server-side worker, use RAM sparingly.
Attributes¤
instance-attribute
¤
sql: str = ''
Cache the previous SQL filtering conditions
instance-attribute
¤
word2vec: WordEmbedding = word2vec
Word embedding model (Word2Vec or FastText), via the WordEmbedding interface
instance-attribute
¤
Store the list of document tokens encoded by their index number in the Word2Vec vocabulary. Unknown tokens are discarded. This gives a symbolic and more compact representation of tokens collocations in documents (32 bits/token).
Documents are on the first axis.
instance-attribute
¤
ranker: BM25PlusCSR = BM25PlusCSR(
corpus_token_indices, self.word2vec, k1=1.8, b=0.4, delta=0.8
)
BM25+ CSR ranker (TF-IDF).
instance-attribute
¤
Principal component(s) of the dataset vectors (normalized)
instance-attribute
¤
Store the list of document-wise vector embeddings, where the vector represents the normalized centroid of tokens vectors contained the document. Documents are on the first axis.
Methods:¤
init_stats_table(db: sqlite3.Connection)
Create or migrate the plain-SQLite stats table.
Scalar stats use item = ''. Grouped stats use name for the metric
family and item for the domain/category/language key.
init_categories_table(db: sqlite3.Connection)
Create a queryable catalog of all page categories.
init_pages_search_indexes(db: sqlite3.Connection)
Create or migrate all persistent indexes needed by the search layer.
search_rowid is an explicit INTEGER column we assign (0, 1, 2 …)
to every page in ORDER BY url order at build time. Because it is
a real column value, VACUUM cannot renumber it — unlike SQLite’s
implicit rowid for tables with a TEXT primary key.
The covering index idx_pages_search_rowid_category is the key
performance fix for filter_contents: SQLite can answer the entire
candidate-filter query from the compact index without ever touching the
main pages rows (which are large due to stored embeddings / content).
Benchmark: 500-candidate category filter 439 ms → < 2 ms.
save_search_stats(db: sqlite3.Connection, stats: dict)
Store cheap display and diagnostic metadata in plain SQLite rows.
save_categories_index(db: sqlite3.Connection, category_counts: dict[str, int])
Store all existing non-empty categories and their page counts.
build_stats(db: sqlite3.Connection) -> dict
Compute index metadata while the database is writable/prepared.
staticmethod
¤
Store arrays without the .npy wrapper used by SQLite converters.
Raw contiguous blobs hydrate faster with np.frombuffer() at runtime.
verify_db_integrity(db: sqlite3.Connection, full: bool = False)
Raise RuntimeError if the DB has changed since this Indexer was built.
| PARAMETER | DESCRIPTION |
|---|---|
full
|
TYPE:
|
Called automatically by :meth:load.
Remove the principal component of the dataset to the vector. This helps removing stopwords, webpage boilerplates (menu, sidebars), formatting language and SEO junk, and makes cosine similarity between query and documents more specific.
Taken from A simple but tough-to-beat baseline for sentence embeddings, Sanjeev Arora, Yingyu Liang, Tengyu Ma. https://openreview.net/pdf?id=SyK00v5xx
| PARAMETER | DESCRIPTION |
|---|---|
vector
|
can be a single vector (1D) or a document-wise stack of vectors (2D). We always consider the embedding vector to be on the last axis, document-wise vectors should be vertically stacked. |
Returns: normalized vector
filter_contents(
db: sqlite3.Connection,
sql_query: str = "",
sql_params: list[str] | None = None,
candidate_indices: np.ndarray | list[int] | None = None,
) -> list[int]
Filter pages by an arbitrary SQL predicate, returning search_rowid integers.
With candidate_indices, the predicate is evaluated only over that
small set via WHERE search_rowid IN (...). This avoids a full table
scan and, combined with the covering index
idx_pages_search_rowid_category, avoids touching the main
pages rows entirely for category/url filters — the most common case.
Without candidate_indices, the full table is scanned (used for
building candidate sets outside of :meth:rank).
classmethod
¤
load(name: str, db: sqlite3.Connection)
Load an existing trained model by its name from the ../models folder.
tokenize_query(
query: str,
language: str | None = None,
meta_tokens: bool = True,
n_grams: bool = True,
) -> list[str]
Tokenize a query string, returning only tokens known to our vocabulary.
find_query_pattern(
indexed_query: np.ndarray[np.int32],
documents: list[tuple[int, str, float]],
fast: bool = False,
) -> list[tuple[int, str, float]]
The rankers methods treat documents as continuous bag of words (CBOW). As such, they are good for topic extraction (aboutness), but they do not care about words colocations and ordering, therefore they loose syntactical meaning.
This method adds an additional layer of detection using convolution filters that will detect word sequences, direct or reversed, and correct the similarity factor set by the other ranking methods using that collocation factor.
Its major drawback is to be 100 to 500 times slower than the other rankers, due to 2D convolutions, which means it needs to run on a subset of the search index, after previous methods were tried, to refine a previous ranking.
| PARAMETER | DESCRIPTION |
|---|---|
indexed_query
|
the search query tokens translated into their integer indices in the Word2Vec vocabulary. Use core.nlp.Word2Vec.tokens_to_indices to convert the tokenized query. |
documents
|
a symbolic list of documents, as a |
fast
|
if
TYPE:
|
References
Text Matching as Image Recognition, Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. (2016). https://arxiv.org/pdf/1602.06359.pdf
rank_ai(
tokens: list[str],
fast: bool = False,
clip: bool = False,
coverage: float = 0.2,
use_sif: bool = True,
sif_smoothing: float = 0.001,
) -> np.ndarray
Cosine-similarity ranking against document centroid vectors.
| PARAMETER | DESCRIPTION |
|---|---|
tokens
|
tokenised query (output of |
fast
|
use a single dot-product against the aggregate query vector instead of the per-token dual-embedding loop.
TYPE:
|
clip
|
clamp scores to [0, 1]. Off by default: clipping saturates the SIF-weighted cosine and destroys rank resolution, which is fatal for the rank-based RRF fusion. Only enable when blending raw scores.
TYPE:
|
coverage
|
if cluster data has been loaded, restrict the matmul to
the nearest clusters that together cover at least this
fraction of the corpus. Documents outside the selected
clusters keep a score of 0; RRF fusion with BM25+ in
TYPE:
|
rrf(
ranks_1: np.ndarray,
ranks_2: np.ndarray,
coeff: float = 60,
weight_1: float = 1.0,
weight_2: float = 1.0,
) -> np.ndarray
Reciprocal Rank Fusion
Aggregate 2 sets of page rankings obtained from different semantic geometries and weighted differently.
Reference
Reciprocal rank fusion outperforms condorcet and individual rank learning methods, Gordon V. Cormack, Charles L A Clarke, Stefan Buettcher. https://dl.acm.org/doi/10.1145/1571941.1572114
| PARAMETER | DESCRIPTION |
|---|---|
ranks_1
|
0-based ranks from the first ranker (best = 0). |
ranks_2
|
0-based ranks from the second ranker (best = 0). |
coeff
|
RRF smoothing constant
TYPE:
|
weight_1
|
vote weight of
TYPE:
|
weight_2
|
vote weight of
TYPE:
|
rank(
db: sqlite3.Connection,
tokens: list[str],
method: search_methods,
n_results: int = 500,
fine_search: bool = False,
sql_query: str = "",
sql_params: list[str] = [],
ai_weight: float = 0.33,
) -> list[tuple[int, str, float]]
Apply a label on a post based on the trained model.
| PARAMETER | DESCRIPTION |
|---|---|
db
|
the SQLite database holding the indexed set of document. This database must absolutely be up-to-date with the one used to instanciate this class, regarding row ordering of documents, otherwise rowid mismatches are to be expected between fuzzy, AI and regex searches. tokens: the tokenized query.
TYPE:
|
method
|
TYPE:
|
n_results
|
number of results to retain
TYPE:
|
fine_search
|
optionally refine the search using a 2D interaction matrix. See [1]
TYPE:
|
sql_query
|
SQL query to narrow-down the search, for example
TYPE:
|
sql_params
|
the SQL parameters such that: where eachsql_params item is matched in the sql_query by a ?. For example:
sql_params = ['google.com'] will filter all URLs from Google.
|
ai_weight
|
vote weight of the AI (embedding) ranker in the weighted RRF
fusion with BM25+ (which keeps weight 1.0), for
TYPE:
|
Note
Both SQL search into the database and Python filtering into the index are supported, and can be combined. The local index is a partial copy of the database and is already a Python object, so it will be faster to filter if you only need to parse the copied data to filter in/out.
| RETURNS | DESCRIPTION |
|---|---|
list
|
the list of best-matching results as (rank, url, similarity) tuples. |
get_related(
tokens: list[str],
n: int = 15,
k: int = 5,
use_sif: bool = True,
sif_smoothing: float = 0.001,
) -> list
Get the n closest keywords from the query.
get_clusters(db: sqlite3.Connection)
Find document latent topics modelled as clusters of document centroids.
Writes to the database
clusterstable — one row per cluster: label (PK), human-legible keyword labels, centroid BLOB, and max cosine radius so callers can gauge cluster tightness.pages.cluster— integer FK intoclusters.labelfor each page.searchtable — three new BLOB columns (cluster_labels_raw,cluster_centroids_raw,cluster_centroids_shape) that mirror the pattern used forvectors_rawso the data loads at the same speed on startup without touchingpagesat all.
Sets on self (immediately usable without reloading):
self.cluster_centroids: (K, D) float32 — one centroid per cluster.
self.cluster_doc_indices: dict[int, np.ndarray[int32]] — maps each
cluster label to its member row indices in
self.vectors / self.ranker (positions = search_rowid).