serena.memories.memory_reference_analysis

serena.memories.memory_reference_analysis#

Source code: serena/memories/memory_reference_analysis.py

Memory reference matching, similarity scoring, and integrity reporting.

This module contains the pure helpers and data types used by serena.project.MemoriesManager to (a) detect and rank references between memories, and (b) report referential integrity issues. The MemoryReferenceAnalyzer class drives the validation and autofix workflows against a live MemoriesManager instance.

Kept separate from the manager so the matching heuristics can be evolved and tested in isolation from filesystem and lifecycle concerns.

MEMORY_REF_PREFIX: str = 'mem:'#: Reference prefix used inside memory bodies to point at another memory.

NAME_SIMILARITY_THRESHOLD: float = 0.55#: Minimum compute_name_similarity() score for a candidate to be reported.

HIGH_CONFIDENCE_NAME_LENGTH: int = 10#: Length threshold above which a flat memory name is considered “name-shaped” (i.e. unlikely to coincide with ordinary prose).

SHORT_NAME_FLOOR: int = 3#: Below this basename length, only an exact token match keeps similarity above 0.

FUZZY_BARE_TOKEN_JACCARD_FLOOR: float = 0.6#: For fuzzy bare-text matching, the bare token and the candidate existing name must share at least this fraction of their tokenized name parts. Containment / SequenceMatcher signals alone over-match — e.g. a generic prose word like repository is a substring of serena_repository_structure but their token Jaccard is only 1/3, so we reject it at this floor.

BASENAME_JACCARD_FLOOR: float = 0.34#: In compute_name_similarity() case 2 (basenames differ), when the two names also carry differing topic prefixes, demand at least one strong basename signal: token Jaccard >= this floor, basename containment, or typo-level seq ratio. Without this gate, names like frontend/x-subtleties and backend/y-subtleties get matched purely on the shared trailing token via the SequenceMatcher signal.

BASENAME_TYPO_SEQ_FLOOR: float = 0.75#: Companion to BASENAME_JACCARD_FLOOR: a basename seq ratio above this is treated as typo-level edit distance and bypasses the Jaccard/containment requirement.

MAX_STALE_REFERENCE_CANDIDATES: int = 3#: Hard cap on candidates proposed per stale reference. Beyond a small number the list stops aiding disambiguation and becomes noise.

WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES: frozenset[str] = frozenset({'core'})#: Names that coincide with common English words and are filtered out of the unmarked- reference scan to avoid false positives.

NAME_CHAR_CLASS: str = '[A-Za-z0-9_\\-/]'#: Character class delimiting a memory name. Used to anchor regex matches so we never consume a partial name (kept in sync with the boundary rule in MemoriesManager.rename_references_to_memory()).

normalize_for_similarity(name)[source]#

Parameters:: name (str) – a memory name
Returns:: a lowercased copy of name with version/legacy-style trailing suffixes stripped (e.g. "auth_v2" -> "auth"); used as the canonical form for similarity scoring.
Return type:: str

tokenize_name(name)[source]#

Parameters:: name (str) – a memory name
Returns:: the set of lowercase tokens extracted by splitting name on /, _, - and at camelCase boundaries; empty tokens are dropped.
Return type:: set[str]

compute_name_similarity(a, b)[source]#

Computes a similarity score in [0, 1] between two memory names. Examples (with default threshold 0.55):

auth/login        == auth/login          -> 1.00   (exact)
auth_v1           ~  auth_v2             -> 1.00   (version suffix normalized)
login             ~  auth/login          -> 1.00   (flat <-> topic move)
auth/login        ~  auth/v2/login       -> 0.75   (shared prefix token + basename)
auth/login        ~  security/login      -> 0.50   (basename only; disjoint topics)
auth              ~  authentication      -> 0.64   (substring containment)
foo               ~  for                 -> 0.00   (short-name floor)
frontend/x-subtleties ~ backend/y-subtleties -> 0.00 (different topics + only a
                                                     shared generic trailing token)

Parameters:

a (str) – first memory name
b (str) – second memory name

Returns:

the similarity score

Return type:

float

find_stale_reference_candidates(missing_name, existing_names, threshold=None)[source]#

Parameters:

missing_name (str) – the unresolved reference target (without the mem: prefix).
existing_names (list[str]) – the names of all currently existing memories.
threshold (float | None) – optional override for the minimum similarity required; defaults to NAME_SIMILARITY_THRESHOLD.

Returns:

existing memory names whose similarity to missing_name meets or exceeds threshold, sorted in descending order of similarity (ties broken alphabetically) and capped at MAX_STALE_REFERENCE_CANDIDATES entries to keep the output focused on the most plausible matches.

Return type:

list[str]

iter_referenced_names_in_content(content)[source]#

Parameters:: content (str) – arbitrary memory content
Returns:: an iterator over the memory names appearing as mem:NAME references in content; duplicate occurrences are yielded once each.
Return type:: Iterator[str]

find_bare_occurrences(content, name)[source]#

Parameters:

content (str) – arbitrary memory content
name (str) – a memory name to look for

Returns:

the count of bare (i.e. not already mem:-prefixed) word-boundary-anchored occurrences of name in content. The match must not be preceded by mem: and must not be embedded within a longer memory-name-like run of characters.

Return type:

int

add_bare_occurrences_prefix(content, name)[source]#

Parameters:

content (str) – arbitrary memory content
name (str) – the memory name whose bare occurrences should be rewritten

Returns:

(new_content, n_replacements) after prefixing each bare occurrence of name with mem: (using the same boundary rule as find_bare_occurrences()).

Return type:

tuple[str, int]

iter_long_bare_tokens(content)[source]#

Parameters:: content (str) – arbitrary memory content
Returns:: an iterator over (token, count) pairs for each distinct name-shaped token in content whose length meets HIGH_CONFIDENCE_NAME_LENGTH and which is not preceded by mem: (those are already valid references). The boundaries follow NAME_CHAR_CLASS, so embedded substrings of longer name-character runs are not matched.
Return type:: Iterator[tuple[str, int]]

is_self_reference(source_memory, suspected_name)[source]#

Returns:

True iff suspected_name would refer to source_memory itself (including the case where suspected_name equals the basename of a topic-path source memory).

Parameters:

source_memory (str)
suspected_name (str)

Return type:

bool

class StaleReference(source_memory, referenced_name, candidates, source_is_read_only)[source]#

Bases: object

A mem:NAME reference whose target memory does not exist.

Variables:

source_memory – the name of the memory whose content contains the broken reference.
referenced_name – the name following mem: that did not resolve to an existing memory.
candidates – existing memory names proposed as likely intended targets, ranked by decreasing similarity. May be empty if no candidate exceeded the similarity threshold.
source_is_read_only – whether the source memory is read-only.

Parameters:

source_memory (str)
referenced_name (str)
candidates (list[str])
source_is_read_only (bool)

class UnmarkedReferenceWarning( source_memory, suspected_name, occurrences, is_high_confidence, source_is_read_only, actual_token='', )[source]#

Bases: object

A bare occurrence in a memory’s content that looks like a forgotten reference to an existing memory.

Two flavours of finding share this class:

exact match — the bare text equals an existing memory name verbatim (actual_token either equals suspected_name or is left empty).
fuzzy near-miss — the bare text does not equal any existing memory name, but a long, distinctive token in the body similarity-matches a high-confidence existing memory name (actual_token is the bare text actually found, distinct from suspected_name). Such findings are reported but not rewritten by MemoryReferenceAnalyzer.auto_prefix_bare_references(), since they would require substring substitution rather than a prefix addition.

Variables:

source_memory – the name of the memory whose content contains the bare occurrence.
suspected_name – the existing memory name proposed as the intended target.
occurrences – the number of occurrences of actual_token in the source memory’s content.
is_high_confidence – True when suspected_name contains a / separator or exceeds the configured length threshold; such names are unlikely to coincide with ordinary prose. False otherwise (a low-confidence warning).
source_is_read_only – whether the source memory is read-only.
actual_token – the bare text actually found in the source memory’s content. Defaults to "", which is taken to mean suspected_name (the exact-match case). When non-empty and different from suspected_name, this is a fuzzy near-miss.

Parameters:

source_memory (str)
suspected_name (str)
occurrences (int)
is_high_confidence (bool)
source_is_read_only (bool)
actual_token (str)

property is_exact_match: bool#

Returns:: True iff the bare text in the body equals suspected_name (i.e. not a fuzzy near-miss).

class AutofixedReference(source_memory, referenced_name, n_replacements)[source]#

Bases: object

A bare occurrence rewritten to include the mem: prefix.

Variables:

source_memory – the name of the memory whose content was modified.
referenced_name – the memory name whose bare occurrences were prefixed.
n_replacements – the number of bare occurrences replaced in the source memory.

Parameters:

source_memory (str)
referenced_name (str)
n_replacements (int)

class ReferentialIntegrityReport( stale_references=<factory>, high_confidence_unmarked_memories=<factory>, low_confidence_unmarked_memories=<factory>, )[source]#

Bases: object

Outcome of MemoryReferenceAnalyzer.validate_referential_integrity().

Variables:

stale_references – mem:NAME references whose target memory does not exist.
high_confidence_unmarked_memories – bare references whose suspected target name is unlikely to be coincidental prose (topic-path or sufficiently long).
low_confidence_unmarked_memories – bare references whose suspected target name could plausibly appear in ordinary prose (short, flat names).

Parameters:

stale_references (list[StaleReference])
high_confidence_unmarked_memories (list[UnmarkedReferenceWarning])
low_confidence_unmarked_memories (list[UnmarkedReferenceWarning])

is_clean()[source]#

Returns:: True iff no stale references and no warnings of any confidence level were found.
Return type:: bool

format()[source]#

Returns:: a human-readable rendering suitable for CLI display.
Return type:: str

class AutofixReport( autofixed=<factory>, dry_run=False, skipped_read_only=<factory>, skipped_flat=<factory>, skipped_global=<factory>, skipped_fuzzy=<factory>, )[source]#

Bases: object

Outcome of MemoryReferenceAnalyzer.auto_prefix_bare_references().

Variables:

autofixed – per-(source, target) records of bare references that were rewritten. When the call was a dry run, these records describe what would have been written; the files themselves are unchanged.
dry_run – True if the run was a preview that did not write any files.
skipped_read_only – warnings whose source memory was read-only and therefore not modified (only populated when include_read_only is False).
skipped_flat – warnings skipped because their suspected target had no / separator and was not long enough to be high-confidence (only populated when include_flat_names is False).
skipped_global – warnings skipped because their source memory was global and include_global was False.
skipped_fuzzy – warnings whose bare text in the source memory differs from the suspected target (fuzzy near-misses). These require substring substitution rather than a prefix addition and are never autofixed; surface them to the user for manual review instead.

Parameters:

autofixed (list[AutofixedReference])
dry_run (bool)
skipped_read_only (list[UnmarkedReferenceWarning])
skipped_flat (list[UnmarkedReferenceWarning])
skipped_global (list[UnmarkedReferenceWarning])
skipped_fuzzy (list[UnmarkedReferenceWarning])

format()[source]#

Returns:: a human-readable rendering suitable for CLI display.
Return type:: str

class MemoryReferenceAnalyzer(manager)[source]#

Bases: object

Drives validation and autofix workflows against a MemoriesManager.

Composition rather than inheritance — the analyzer only needs the manager’s memory enumeration, loading, saving, and is-global helpers, and is otherwise independent of project-management concerns.

Parameters:: manager (MemoryManager)

validate_referential_integrity( include_unmarked=True, include_fuzzy_matching=True, )[source]#

Scans every (non-ignored) memory’s content for referential integrity issues.

The scan covers both project-local and global memories. Three kinds of finding are produced, each independently gated:

stale references — occurrences of mem:NAME where NAME does not resolve to an existing memory. For each, a list of similarly-named existing memories is proposed as candidate intended targets (see compute_name_similarity()), capped at MAX_STALE_REFERENCE_CANDIDATES. Always reported.
exact unmarked-reference warnings — bare occurrences of an existing memory name that appear without the mem: prefix. Split into high-confidence (the suspected name contains a / or exceeds HIGH_CONFIDENCE_NAME_LENGTH) and low-confidence groups. Candidate names whose basename is in WORDS_TO_IGNORE_AS_MEMORY_NAME_CANDIDATES are skipped. Gated by include_unmarked.
fuzzy near-miss warnings — long, distinctive bare tokens in a memory body that do not match an existing name exactly but similarity-match a high-confidence existing name AND share at least FUZZY_BARE_TOKEN_JACCARD_FLOOR of their tokenized name parts. Always reported as high-confidence; the actual bare text is preserved on UnmarkedReferenceWarning.actual_token. Gated by include_unmarked and include_fuzzy_matching — fuzzy matches are noisy and only meaningful when unmarked-reference checking is enabled.

Self-references (a memory’s content mentioning its own name or basename) and empty memories are skipped silently. This method has no side effects.

Parameters:

include_unmarked (bool) – if False, skip both the exact unmarked scan and the fuzzy near-miss scan; only stale references are reported.
include_fuzzy_matching (bool) – if False, skip the fuzzy near-miss scan even when unmarked-reference checking is otherwise enabled. Has no effect when include_unmarked is False.

Returns:

a ReferentialIntegrityReport summarizing all findings.

Return type:

ReferentialIntegrityReport

auto_prefix_bare_references( include_flat_names=False, include_read_only=False, include_global=False, dry_run=False, )[source]#

Rewrites exact bare occurrences of existing memory names by adding the mem: prefix.

Warning

This is a heuristic, file-mutating operation (unless dry_run is True). A bare word that happens to coincide with a memory name will be rewritten as a reference, even if it was intended as ordinary prose. Pass dry_run=True to preview the rewrites before applying them.

Scope is intentionally narrower than what validate_referential_integrity() reports:

Only exact bare occurrences are rewritten — i.e. the bare text in the source body must equal an existing memory name verbatim. Fuzzy near-miss findings (where the actual token differs from the suspected target) require substring substitution rather than a prefix addition and are routed into AutofixReport.skipped_fuzzy for manual review.
By default the rewrite is restricted to high-confidence findings only — those whose suspected target name contains a / separator or exceeds the configured length threshold — and skips global memories and read-only memories. The defaults intentionally err toward false negatives over false positives.

Parameters:

include_flat_names (bool) – if True, also rewrite low-confidence findings (flat, short memory names). Increases recall but markedly raises false-positive risk.
include_read_only (bool) – if True, also rewrite occurrences inside read-only memories. Use with care, as read-only memories are typically considered authoritative.
include_global (bool) – if True, also rewrite occurrences inside global memories. Modifying a global memory affects every project that consumes it.
dry_run (bool) – if True, the report describes the rewrites that would be applied but no files are modified.

Returns:

an AutofixReport describing every replacement (made or planned) and every warning that was deliberately skipped.

Return type:

AutofixReport

serena.memories.memory_reference_analysis

Contents

serena.memories.memory_reference_analysis#