fix: Centralize PDF processing and generate separate images per chunk
Previously, pymupdf4llm.to_markdown() was called twice - once in PyMuPDFProcessor during indexing and again in PDFHighlighter during visualization. Different image path lengths caused different character offsets, leading to highlighted pages not matching their chunks. Also fixed issue where all chunks on the same page showed all highlights instead of just their own highlight. Now restores original page contents between chunks using xref stream caching. Changes: - Add PDFHighlighter class requiring pre-computed page_boundaries and full_text from document processor (no fallback extraction) - Pass pre-computed data from processor to highlighter - Extract page-relative portion of chunk text for cross-page chunks - Add bounding box highlighting using text anchor search - Run highlight generation in parallel with embedding/BM25 - Cache and restore page contents to isolate highlights per chunk Results: Highlighting success rate improved from 51% to 95% (121/128). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -102,7 +102,7 @@ class PyMuPDFProcessor(DocumentProcessor):
|
||||
await progress_callback(0, 100, "Processing PDF in background thread")
|
||||
|
||||
# Run CPU-bound PDF processing in thread pool to avoid blocking event loop
|
||||
result = await anyio.to_thread.run_sync(
|
||||
result = await anyio.to_thread.run_sync( # type: ignore[attr-defined]
|
||||
self._process_sync,
|
||||
content,
|
||||
filename,
|
||||
@@ -240,7 +240,7 @@ class PyMuPDFProcessor(DocumentProcessor):
|
||||
# Basic document info
|
||||
metadata["page_count"] = doc.page_count
|
||||
metadata["format"] = "PDF 1." + str(
|
||||
doc.pdf_version() if hasattr(doc, "pdf_version") else "?"
|
||||
doc.pdf_version() if hasattr(doc, "pdf_version") else "?" # type: ignore[call-non-callable]
|
||||
)
|
||||
|
||||
if filename:
|
||||
|
||||
Reference in New Issue
Block a user