Compare commits
17 commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aa3fc5ebc5 | ||
|
|
495f50989d | ||
|
|
ff0d0c572c | ||
|
|
608098d378 | ||
|
|
afac75153e | ||
|
|
936971b7f8 | ||
|
|
d365ca28e1 | ||
|
|
de435fec00 | ||
|
|
5fb9b8631d | ||
|
|
deaa105be2 | ||
|
|
d9062d9a33 | ||
|
|
8e8bd31fc8 | ||
|
|
edf777e8f7 | ||
|
|
efb9946694 | ||
|
|
30dbe0ca68 | ||
|
|
4342b284ee | ||
|
|
cd2a67e4b7 |
34 changed files with 4926 additions and 600 deletions
5
.gitignore
vendored
5
.gitignore
vendored
|
|
@ -2,7 +2,7 @@ clippings.json
|
|||
zotero-index.json
|
||||
matches.json
|
||||
match-overrides.generated.json
|
||||
!match-overrides.json
|
||||
match-overrides.json
|
||||
import-plan*.json
|
||||
*.import-plan.json
|
||||
__pycache__/
|
||||
|
|
@ -15,3 +15,6 @@ plugin-summary.json
|
|||
plugin-progress.json
|
||||
plugin-config.json
|
||||
kindle-import-manager.html
|
||||
MEMORY.md
|
||||
docs/announcement*.md
|
||||
docs/mismatch-review.md
|
||||
|
|
|
|||
63
README.md
63
README.md
|
|
@ -1,34 +1,57 @@
|
|||
> [!CAUTION]
|
||||
> **This project has been heavily vibecoded.**
|
||||
|
||||
# Kanzi — Kindle annotations to Zotero
|
||||
# Kanzi — `K`indle `an`notations to `Z`otero `I`mporter
|
||||
|
||||
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero, no terminal needed. Formerly *Kindle Zotero Importer*.
|
||||
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero. Once the required host tools are installed, imports run from the manager without CLI commands. Formerly *Kindle Zotero Importer*.
|
||||
|
||||
<p align="center" style="margin: 18px 0 24px;">
|
||||
<p align="center">
|
||||
|
||||
<a href="https://github.com/sponsors/UtkuBilenDemir"><img src="https://img.shields.io/badge/please%20give%20me%20money+-red?style=for-the-badge" alt="please give me money+"></a>
|
||||
<a href="https://github.com/sponsors/UtkuBilenDemir"><img src="https://img.shields.io/badge/please%20give%20me%20money+-111111?style=flat-square" alt="please give me money+"></a>
|
||||
|
||||
</p>
|
||||
|
||||
# Install
|
||||
|
||||
1. Download `kindle-zotero-importer.xpi` from [Releases](../../releases) (latest `0.6.14`, or `0.6.13-beta` for preview).
|
||||
1. Download `kanzi.xpi` from [Releases](https://github.com/UtkuBilenDemir/kanzi/releases) (latest `0.7.0`).
|
||||
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
|
||||
3. `Tools` → `Kanzi…` to open the manager.
|
||||
|
||||
Works with Zotero 7 to 10, macOS, Windows and Linux, PDF and EPUB.
|
||||
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. Kanzi detects standard Python installations automatically on macOS, Linux and Windows.
|
||||
|
||||
## Windows
|
||||
|
||||
Open PowerShell and install the host tools:
|
||||
|
||||
```powershell
|
||||
winget install --id Python.Python.3.13 --exact
|
||||
winget install --id oschwartz10612.Poppler --exact
|
||||
winget install --id QPDF.QPDF --exact # Optional: handles copy-protected PDFs
|
||||
```
|
||||
|
||||
Fully quit and reopen Zotero after installation so it receives the updated `PATH`, then install `kanzi.xpi` and open `Tools` → `Kanzi…`. Kanzi checks standard per-user and system Python locations as well as `PATH`; you should not need to enter an executable path manually. EPUB-only imports require Python but not Poppler.
|
||||
|
||||
To verify the installation in a new PowerShell window:
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
pdftotext -v
|
||||
pdftohtml -v
|
||||
pdfinfo -v
|
||||
qpdf --version # Optional
|
||||
```
|
||||
|
||||
# Use
|
||||
|
||||
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
|
||||
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
||||
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
|
||||
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
||||
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
||||
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
||||
2. **Review before writing.** New highlights are grouped by title and selected by default. Untick individual highlights or whole titles, then press `Integrate selected`. Skipped highlights return on the next run. `Cancel` and closing the manager write nothing to Zotero and do not mark pending highlights as integrated.
|
||||
3. **Check Integrated.** After integration, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
||||
4. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows one expandable row for that title with up to three suggestions `citekey · title (score%)`. Expand it to inspect individual highlights and positioning problems. Click `Use` on the right candidate, enter a citation key, Zotero key or item ID under `Use Custom`, or choose `Ignore Title`. Tables can be sorted and resized.
|
||||
5. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new, changed, deferred or removed highlights are processed. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
||||
6. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
||||
7. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
||||
|
||||
Your choices are saved in `match-overrides.json` in the project folder. Back it up, share it, or delete an entry to undo a mapping.
|
||||
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
|
||||
|
||||
# Coloured highlights
|
||||
|
||||
|
|
@ -62,15 +85,15 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
|
|||
|
||||
`Tools` → `Kanzi…` opens the manager.
|
||||
|
||||
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
|
||||
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. After positioning, the pending review lets you select exactly what is written to Zotero.
|
||||
|
||||
**Integrated.** The last `import-plan.final.json` annotations. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right, 400 shown, sortable and resizable.
|
||||
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
|
||||
|
||||
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
|
||||
**Conflicts.** One row per unresolved Kindle title, combining title-match and attachment/position statuses. Expand a row to inspect its highlights and positioning problems. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. Columns are sortable and resizable.
|
||||
|
||||
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
|
||||
|
||||
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, `Share anonymised annotations` (on by default, private, hashed only), and `Highlight colours` as above. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
|
||||
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, annotation sharing, and `Highlight colours` as above. Sharing is off until you explicitly opt in and can be disabled here at any time. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
|
||||
|
||||
**Artifacts.** `Mismatch review`, `Persistent overrides`, `Generated suggestions`, `Positioned plan`, `Final writer plan` and `Plugin summary`. Each has `Open` and `Reveal`.
|
||||
|
||||
|
|
@ -79,19 +102,19 @@ Incremental is default. Only new, changed or colour-changed highlights are repos
|
|||
# For developers / CLI
|
||||
|
||||
```sh
|
||||
python scripts/build_plugin.py # → dist/kindle-zotero-importer.xpi
|
||||
python scripts/build_plugin.py # → dist/kanzi.xpi, including the Python runtime
|
||||
python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --pretty
|
||||
python -m kindle_zotero_importer run ... --full # ignore incremental and reprocess all
|
||||
```
|
||||
|
||||
PDF positioning uses Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
|
||||
PDF positioning uses host-installed Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
|
||||
|
||||
# Releases
|
||||
|
||||
Stable releases are on `Releases` with `kindle-zotero-importer.xpi` and `updates.json` attached. Pre-releases (`beta`) are marked `Pre-release` on GitHub. Zotero auto-updates from `releases/latest/download/updates.json`.
|
||||
Stable releases are on `Releases` with `kanzi.xpi` and `updates.json` attached. Pre-releases (`beta`) are marked `Pre-release` on GitHub. Zotero auto-updates from `releases/latest/download/updates.json`.
|
||||
|
||||
# Support
|
||||
|
||||
Issues and pull requests welcome at [Issues](../../issues).
|
||||
|
||||
Anonymised sharing is on by default to help future development. It sends only citation key, hashed highlight, colour, has comment and note, added on and integrated at. It never sends raw highlight text. Opt out any time in `Manager` → `Settings` → uncheck `Share anonymised annotations`.
|
||||
Annotation sharing is off until you explicitly opt in. If enabled, it sends a random pseudonymous installation ID, citation key and publication metadata (including DOI/ISBN when available), raw highlight text and comments, normalised/fuzzy hashes, colour, and dates to `annotation.utkubilen.de`. The collector does not receive your Zotero account identity, but free text can contain identifying information and therefore is not anonymous. Change this any time in `Manager` → `Settings`.
|
||||
|
|
|
|||
357
collector/app.py
Normal file
357
collector/app.py
Normal file
|
|
@ -0,0 +1,357 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import sqlite3
|
||||
import unicodedata
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
|
||||
|
||||
DB = Path(os.environ.get("KANZI_ANNOTATION_DB", Path(__file__).parent / "annotations.db"))
|
||||
MAX_PAGE_SIZE = 200
|
||||
MAX_REQUEST_BYTES = 2_000_000
|
||||
MAX_BATCH_SIZE = 5_000
|
||||
|
||||
app = FastAPI(title="Kanzi Annotation Collector")
|
||||
|
||||
|
||||
def _normalise(text: str) -> str:
|
||||
value = unicodedata.normalize("NFKC", text or "").lower()
|
||||
value = re.sub(r"-\s*\n\s*", "", value)
|
||||
value = re.sub(r"[\u2010-\u2015\u2018\u2019\u201c\u201d]", "'", value)
|
||||
value = re.sub(r"[^a-z0-9]+", " ", value)
|
||||
return re.sub(r"\s+", " ", value).strip()
|
||||
|
||||
|
||||
def _hash_norm(text: str) -> str:
|
||||
return hashlib.sha256(_normalise(text).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def _fuzzy_hash(text: str) -> str:
|
||||
norm = _normalise(text)
|
||||
if len(norm) < 3:
|
||||
return _hash_norm(text)
|
||||
trigrams = {norm[index : index + 3] for index in range(len(norm) - 2)}
|
||||
words = sorted(norm.split())
|
||||
word_trigrams = {
|
||||
" ".join(words[index : index + 3])
|
||||
for index in range(max(1, len(words) - 2))
|
||||
}
|
||||
return hashlib.sha256(" ".join(sorted(trigrams | word_trigrams)).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def _connect() -> sqlite3.Connection:
|
||||
DB.parent.mkdir(parents=True, exist_ok=True)
|
||||
connection = sqlite3.connect(DB)
|
||||
connection.row_factory = sqlite3.Row
|
||||
connection.execute("PRAGMA busy_timeout = 5000")
|
||||
connection.execute("PRAGMA journal_mode = WAL")
|
||||
return connection
|
||||
|
||||
|
||||
def init_db() -> None:
|
||||
with _connect() as connection:
|
||||
connection.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS highlights (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
anon_id TEXT,
|
||||
citation_key TEXT,
|
||||
doi TEXT,
|
||||
isbn TEXT,
|
||||
issn TEXT,
|
||||
publication_title TEXT,
|
||||
clipping_id_hash TEXT,
|
||||
norm_hash TEXT,
|
||||
fuzzy_hash TEXT,
|
||||
text TEXT,
|
||||
color TEXT,
|
||||
has_comment INTEGER,
|
||||
comment TEXT,
|
||||
comment_hash TEXT,
|
||||
clipping_added_on_iso TEXT,
|
||||
integrated_at TEXT,
|
||||
received_at TEXT,
|
||||
raw TEXT
|
||||
)"""
|
||||
)
|
||||
existing_columns = {
|
||||
row["name"] for row in connection.execute("PRAGMA table_info(highlights)")
|
||||
}
|
||||
required_columns = {
|
||||
"doi": "TEXT",
|
||||
"isbn": "TEXT",
|
||||
"issn": "TEXT",
|
||||
"publication_title": "TEXT",
|
||||
"clipping_id_hash": "TEXT",
|
||||
"norm_hash": "TEXT",
|
||||
"fuzzy_hash": "TEXT",
|
||||
"text": "TEXT",
|
||||
"comment": "TEXT",
|
||||
"comment_hash": "TEXT",
|
||||
"clipping_added_on_iso": "TEXT",
|
||||
"integrated_at": "TEXT",
|
||||
"received_at": "TEXT",
|
||||
"raw": "TEXT",
|
||||
}
|
||||
for name, column_type in required_columns.items():
|
||||
if name not in existing_columns:
|
||||
connection.execute(f"ALTER TABLE highlights ADD COLUMN {name} {column_type}")
|
||||
connection.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS raw_posts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
anon_id TEXT,
|
||||
received_at TEXT,
|
||||
body TEXT
|
||||
)"""
|
||||
)
|
||||
connection.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS installations (
|
||||
anon_id TEXT PRIMARY KEY,
|
||||
token_hash TEXT UNIQUE NOT NULL,
|
||||
created_at TEXT NOT NULL
|
||||
)"""
|
||||
)
|
||||
# Older collector versions appended every retry. Retain only the newest
|
||||
# record for each installation/clipping before enforcing idempotency.
|
||||
connection.execute(
|
||||
"""DELETE FROM highlights
|
||||
WHERE clipping_id_hash IS NOT NULL
|
||||
AND id NOT IN (
|
||||
SELECT MAX(id) FROM highlights
|
||||
WHERE clipping_id_hash IS NOT NULL
|
||||
GROUP BY anon_id, clipping_id_hash
|
||||
)"""
|
||||
)
|
||||
connection.execute(
|
||||
"""CREATE UNIQUE INDEX IF NOT EXISTS highlights_identity
|
||||
ON highlights(anon_id, clipping_id_hash)
|
||||
WHERE clipping_id_hash IS NOT NULL"""
|
||||
)
|
||||
|
||||
|
||||
init_db()
|
||||
|
||||
|
||||
HTML = """<!doctype html>
|
||||
<html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Kanzi annotations</title>
|
||||
<style>
|
||||
:root{font-family:'iA Writer Duo','JetBrains Mono',ui-monospace,monospace;color:#111;background:#f8f8f8;--panel:#fff;--soft:#f0f0f0;--line:#ddd;--muted:#666;--radius:8px}*{box-sizing:border-box}body{margin:0}main{max-width:1280px;margin:auto;padding:20px 24px}h1{margin:0 0 4px;font-size:28px}h2{font-size:16px;border-bottom:1px solid var(--line);padding-bottom:6px;margin-top:24px}.muted{color:var(--muted);font-size:12px}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:12px;margin:16px 0}.card,table{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius)}.card{padding:14px}.card strong{display:block;font-size:22px}table{width:100%;border-collapse:collapse;table-layout:fixed;overflow:hidden}th,td{padding:9px 10px;border-bottom:1px solid var(--line);text-align:left;overflow-wrap:anywhere;font-size:12px}th{background:var(--soft);color:var(--muted);font-size:10px;text-transform:uppercase}.pager{display:flex;justify-content:flex-end;align-items:center;gap:8px;margin:10px 0}.pager button{padding:5px 9px;border:1px solid var(--line);border-radius:var(--radius);background:var(--panel)}.pager button:disabled{color:#999}@media(max-width:700px){main{padding:14px 10px}.table-wrap{overflow-x:auto}table{min-width:800px}}
|
||||
</style></head><body><main><h1>Kanzi annotations</h1><div class="muted">Private pseudonymous collector. Raw highlight text and comments are stored.</div>
|
||||
<div class="grid"><div class="card"><strong id="total">-</strong><span>highlights</span></div><div class="card"><strong id="uniq">-</strong><span>citekeys</span></div><div class="card"><strong id="anons">-</strong><span>installations</span></div><div class="card"><strong id="today">-</strong><span>received today</span></div></div>
|
||||
<h2>Recent</h2><div class="table-wrap"><table><thead><tr><th>anon</th><th>citekey</th><th>DOI / ISBN</th><th>text</th><th>colour</th><th>received</th></tr></thead><tbody id="recent"></tbody></table></div><div class="pager" id="recent-pager"></div>
|
||||
<h2>Top citekeys</h2><div class="table-wrap"><table><thead><tr><th>citekey</th><th>count</th></tr></thead><tbody id="top"></tbody></table></div>
|
||||
<h2>Common highlights</h2><div class="muted">Same publication and normalised text across at least two installations.</div><div class="table-wrap"><table><thead><tr><th>DOI / ISBN</th><th>text</th><th>norm</th><th>installations</th><th>citekeys</th></tr></thead><tbody id="common"></tbody></table></div><div class="pager" id="common-pager"></div>
|
||||
<script>
|
||||
const size=50,state={recent:0,common:0};
|
||||
function cell(row,value){const td=document.createElement('td');td.textContent=value??'';row.appendChild(td);return td}
|
||||
function empty(body,span,message){body.textContent='';const row=body.insertRow();const td=cell(row,message);td.colSpan=span;td.className='muted'}
|
||||
function pager(id,key,count,load){const root=document.getElementById(id);root.textContent='';const pages=Math.max(1,Math.ceil(count/size));state[key]=Math.min(state[key],pages-1);const prev=document.createElement('button');prev.textContent='Previous';prev.disabled=state[key]===0;prev.onclick=()=>{state[key]--;load()};root.append(prev);const text=document.createElement('span');text.textContent=`Page ${state[key]+1} of ${pages}`;root.append(text);const next=document.createElement('button');next.textContent='Next';next.disabled=state[key]>=pages-1;next.onclick=()=>{state[key]++;load()};root.append(next)}
|
||||
async function loadRecent(){const result=await fetch(`/recent?limit=${size}&offset=${state.recent*size}`).then(r=>r.json());const body=document.getElementById('recent');body.textContent='';for(const h of result.recent){const row=body.insertRow();cell(row,(h.anon_id||'').slice(0,8));cell(row,h.citation_key);cell(row,h.doi||h.isbn);cell(row,(h.text||'').slice(0,180));cell(row,h.color);cell(row,(h.received_at||'').slice(0,16).replace('T',' '))}if(!result.recent.length)empty(body,6,'No data');pager('recent-pager','recent',result.total,loadRecent)}
|
||||
async function loadCommon(){const result=await fetch(`/common?limit=${size}&offset=${state.common*size}`).then(r=>r.json());const body=document.getElementById('common');body.textContent='';for(const h of result.common){const row=body.insertRow();cell(row,h.publication);cell(row,(h.example_text||'').slice(0,180));cell(row,h.norm_hash);cell(row,h.installations);cell(row,h.citekeys)}if(!result.common.length)empty(body,5,'No common highlights yet');pager('common-pager','common',result.total,loadCommon)}
|
||||
async function load(){const stats=await fetch('/stats').then(r=>r.json());for(const [id,key] of [['total','total_highlights'],['uniq','unique_citekeys'],['anons','unique_anons'],['today','today']])document.getElementById(id).textContent=stats[key];const top=await fetch('/top').then(r=>r.json());const body=document.getElementById('top');body.textContent='';for(const h of top.top){const row=body.insertRow();cell(row,h.citation_key);cell(row,h.count)}await Promise.all([loadRecent(),loadCommon()])}load();
|
||||
</script></main></body></html>"""
|
||||
|
||||
|
||||
@app.get("/", response_class=HTMLResponse)
|
||||
def root() -> str:
|
||||
return HTML
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
def _token_hash(token: str) -> str:
|
||||
return hashlib.sha256(token.encode()).hexdigest()
|
||||
|
||||
|
||||
@app.post("/register")
|
||||
def register() -> dict[str, str]:
|
||||
token = secrets.token_urlsafe(32)
|
||||
anon_id = secrets.token_hex(16)
|
||||
with _connect() as connection:
|
||||
connection.execute(
|
||||
"INSERT INTO installations (anon_id, token_hash, created_at) VALUES (?, ?, ?)",
|
||||
(anon_id, _token_hash(token), datetime.now(timezone.utc).isoformat()),
|
||||
)
|
||||
return {"anonId": anon_id, "token": token}
|
||||
|
||||
|
||||
def _authenticated_anon_id(request: Request) -> str:
|
||||
authorization = request.headers.get("authorization", "")
|
||||
scheme, separator, token = authorization.partition(" ")
|
||||
if not separator or scheme.lower() != "bearer" or not token:
|
||||
raise HTTPException(status_code=401, detail="Missing collector credential")
|
||||
with _connect() as connection:
|
||||
row = connection.execute(
|
||||
"SELECT anon_id FROM installations WHERE token_hash = ?",
|
||||
(_token_hash(token),),
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=401, detail="Invalid collector credential")
|
||||
return str(row["anon_id"])
|
||||
|
||||
|
||||
@app.get("/stats")
|
||||
def stats() -> dict[str, int]:
|
||||
today = datetime.now(timezone.utc).date().isoformat()
|
||||
with _connect() as connection:
|
||||
row = connection.execute(
|
||||
"""SELECT COUNT(*) AS total_highlights,
|
||||
COUNT(DISTINCT citation_key) AS unique_citekeys,
|
||||
COUNT(DISTINCT anon_id) AS unique_anons,
|
||||
SUM(CASE WHEN substr(received_at, 1, 10) = ? THEN 1 ELSE 0 END) AS today
|
||||
FROM highlights""",
|
||||
(today,),
|
||||
).fetchone()
|
||||
return {key: int(row[key] or 0) for key in row.keys()}
|
||||
|
||||
|
||||
@app.get("/recent")
|
||||
def recent(limit: int = 50, offset: int = 0) -> dict:
|
||||
limit = max(1, min(limit, MAX_PAGE_SIZE))
|
||||
offset = max(0, offset)
|
||||
with _connect() as connection:
|
||||
total = connection.execute("SELECT COUNT(*) FROM highlights").fetchone()[0]
|
||||
rows = connection.execute(
|
||||
"""SELECT anon_id, citation_key, doi, isbn, text, clipping_id_hash,
|
||||
norm_hash, fuzzy_hash, color, has_comment, received_at
|
||||
FROM highlights ORDER BY id DESC LIMIT ? OFFSET ?""",
|
||||
(limit, offset),
|
||||
).fetchall()
|
||||
return {"recent": [dict(row) for row in rows], "total": total}
|
||||
|
||||
|
||||
@app.get("/top")
|
||||
def top(limit: int = 20) -> dict:
|
||||
limit = max(1, min(limit, 100))
|
||||
with _connect() as connection:
|
||||
rows = connection.execute(
|
||||
"""SELECT citation_key, COUNT(*) AS count FROM highlights
|
||||
WHERE citation_key IS NOT NULL AND citation_key != ''
|
||||
GROUP BY citation_key ORDER BY count DESC, citation_key LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return {"top": [dict(row) for row in rows]}
|
||||
|
||||
|
||||
@app.get("/common")
|
||||
def common(limit: int = 50, offset: int = 0) -> dict:
|
||||
limit = max(1, min(limit, MAX_PAGE_SIZE))
|
||||
offset = max(0, offset)
|
||||
group_sql = """FROM highlights
|
||||
WHERE norm_hash IS NOT NULL AND norm_hash != ''
|
||||
AND COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), NULLIF(citation_key, '')) IS NOT NULL
|
||||
GROUP BY COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key), norm_hash
|
||||
HAVING COUNT(DISTINCT anon_id) > 1"""
|
||||
with _connect() as connection:
|
||||
total = connection.execute(f"SELECT COUNT(*) FROM (SELECT 1 {group_sql})").fetchone()[0]
|
||||
rows = connection.execute(
|
||||
f"""SELECT COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key) AS publication,
|
||||
norm_hash, COUNT(DISTINCT anon_id) AS installations,
|
||||
GROUP_CONCAT(DISTINCT citation_key) AS citekeys,
|
||||
MAX(text) AS example_text
|
||||
{group_sql}
|
||||
ORDER BY installations DESC, publication, norm_hash LIMIT ? OFFSET ?""",
|
||||
(limit, offset),
|
||||
).fetchall()
|
||||
return {"common": [dict(row) for row in rows], "total": total}
|
||||
|
||||
|
||||
@app.post("/highlights")
|
||||
async def post_highlights(request: Request) -> dict:
|
||||
anon = _authenticated_anon_id(request)
|
||||
content_length = request.headers.get("content-length")
|
||||
if content_length:
|
||||
try:
|
||||
if int(content_length) > MAX_REQUEST_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Request body is too large")
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid Content-Length")
|
||||
body = await request.body()
|
||||
if len(body) > MAX_REQUEST_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Request body is too large")
|
||||
try:
|
||||
payload = json.loads(body)
|
||||
except (json.JSONDecodeError, UnicodeDecodeError) as error:
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON") from error
|
||||
if not isinstance(payload, dict):
|
||||
raise HTTPException(status_code=422, detail="Payload must be an object")
|
||||
highlights = payload.get("data") or payload.get("highlights") or []
|
||||
if isinstance(highlights, dict):
|
||||
highlights = [highlights]
|
||||
if not isinstance(highlights, list):
|
||||
raise HTTPException(status_code=422, detail="data must be a list")
|
||||
if len(highlights) > MAX_BATCH_SIZE:
|
||||
raise HTTPException(status_code=413, detail="Too many highlights in one request")
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
inserted = updated = 0
|
||||
with _connect() as connection:
|
||||
connection.execute("BEGIN IMMEDIATE")
|
||||
for highlight in highlights:
|
||||
if not isinstance(highlight, dict):
|
||||
continue
|
||||
text = str(highlight.get("text") or "")[:2000]
|
||||
clipping_hash = str(highlight.get("clipping_id_hash") or "")[:128] or None
|
||||
values = (
|
||||
anon,
|
||||
str(highlight.get("citation_key") or "")[:512] or None,
|
||||
str(highlight.get("doi") or "")[:512] or None,
|
||||
str(highlight.get("isbn") or "")[:512] or None,
|
||||
str(highlight.get("issn") or "")[:512] or None,
|
||||
str(highlight.get("publication_title") or "")[:1000] or None,
|
||||
clipping_hash,
|
||||
highlight.get("norm_hash") or _hash_norm(text),
|
||||
highlight.get("fuzzy_hash") or _fuzzy_hash(text),
|
||||
text,
|
||||
str(highlight.get("color") or "")[:32] or None,
|
||||
1 if highlight.get("has_comment") else 0,
|
||||
str(highlight.get("comment") or "")[:2000],
|
||||
str(highlight.get("comment_hash") or "")[:256] or None,
|
||||
str(highlight.get("clipping_added_on_iso") or "")[:64] or None,
|
||||
str(highlight.get("integrated_at") or "")[:64] or None,
|
||||
now,
|
||||
json.dumps(highlight, ensure_ascii=False)[:8000],
|
||||
)
|
||||
existing = None
|
||||
if clipping_hash:
|
||||
legacy_hash = clipping_hash[:8]
|
||||
existing = connection.execute(
|
||||
"""SELECT id FROM highlights
|
||||
WHERE anon_id = ? AND clipping_id_hash IN (?, ?)
|
||||
ORDER BY clipping_id_hash = ? DESC LIMIT 1""",
|
||||
(anon, clipping_hash, legacy_hash, clipping_hash),
|
||||
).fetchone()
|
||||
if existing:
|
||||
connection.execute(
|
||||
"""UPDATE highlights SET citation_key=?, doi=?, isbn=?, issn=?, publication_title=?, clipping_id_hash=?,
|
||||
norm_hash=?, fuzzy_hash=?, text=?, color=?, has_comment=?, comment=?, comment_hash=?,
|
||||
clipping_added_on_iso=?, integrated_at=?, received_at=?, raw=? WHERE id=?""",
|
||||
(*values[1:7], *values[7:], existing["id"]),
|
||||
)
|
||||
updated += 1
|
||||
else:
|
||||
connection.execute(
|
||||
"""INSERT INTO highlights (
|
||||
anon_id, citation_key, doi, isbn, issn, publication_title, clipping_id_hash,
|
||||
norm_hash, fuzzy_hash, text, color, has_comment, comment, comment_hash,
|
||||
clipping_added_on_iso, integrated_at, received_at, raw
|
||||
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
|
||||
values,
|
||||
)
|
||||
inserted += 1
|
||||
return {
|
||||
"status": "ok",
|
||||
"received": len(highlights),
|
||||
"inserted": inserted,
|
||||
"updated": updated,
|
||||
"anon": anon,
|
||||
}
|
||||
28
collector/nginx.conf.example
Normal file
28
collector/nginx.conf.example
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Add these directives in the nginx http context.
|
||||
limit_req_zone $binary_remote_addr zone=kanzi_ingest:10m rate=10r/m;
|
||||
limit_conn_zone $binary_remote_addr zone=kanzi_connections:10m;
|
||||
|
||||
# Keep dashboard and read APIs behind Authentik. Only plugin ingestion and
|
||||
# health monitoring are intentionally unauthenticated.
|
||||
location = /highlights {
|
||||
client_max_body_size 2m;
|
||||
limit_req zone=kanzi_ingest burst=10 nodelay;
|
||||
limit_conn kanzi_connections 2;
|
||||
proxy_pass http://127.0.0.1:8099;
|
||||
}
|
||||
|
||||
location = /register {
|
||||
limit_req zone=kanzi_ingest burst=10 nodelay;
|
||||
limit_conn kanzi_connections 2;
|
||||
proxy_pass http://127.0.0.1:8099;
|
||||
}
|
||||
|
||||
location = /health {
|
||||
proxy_pass http://127.0.0.1:8099;
|
||||
}
|
||||
|
||||
location / {
|
||||
auth_request /outpost.goauthentik.io/auth/nginx;
|
||||
error_page 401 = @goauthentik_proxy_signin;
|
||||
proxy_pass http://127.0.0.1:8099;
|
||||
}
|
||||
15
match-overrides.example.json
Normal file
15
match-overrides.example.json
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"format": "kindle-zotero-importer.match-overrides.v1",
|
||||
"overrides": [
|
||||
{
|
||||
"clipping_title": "Example Kindle title",
|
||||
"resolution": {
|
||||
"citation_key": "example2026"
|
||||
},
|
||||
"review": {
|
||||
"status": "manager",
|
||||
"clipping_count": 1
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,16 +1,16 @@
|
|||
# Kindle Zotero Importer Zotero Plugin
|
||||
# Kanzi Zotero Plugin
|
||||
|
||||
This is a hybrid Zotero plugin wrapper around the current Python importer. Zotero owns the user interface and writes native annotations through Zotero APIs; the Python project still performs parsing, matching, planning, EPUB positioning, and PDF positioning.
|
||||
|
||||
## Use
|
||||
|
||||
1. Install `kindle-zotero-importer.xpi` in Zotero through `Tools > Plugins`.
|
||||
2. Open `Tools > Kindle Zotero Importer...`.
|
||||
1. Install `kanzi.xpi` in Zotero through `Tools > Plugins`.
|
||||
2. Open `Tools > Kanzi...`.
|
||||
3. Select the updated Kindle `My Clippings.txt` in the manager.
|
||||
4. Follow the current stage, percentage, and elapsed time in the manager.
|
||||
5. Review created, existing, updated, failed, and unresolved counts when it completes.
|
||||
|
||||
The plugin uses `match-overrides.json` from the project directory as the persistent source of title mappings and ignored titles. Generated JSON artifacts are still written to the project directory for audit and debugging.
|
||||
The XPI contains the importer Python source. On startup it extracts that source into the Zotero profile, where it keeps private mappings and generated audit artifacts. Python 3.11 or newer must be installed on the host; PDF positioning also requires Poppler (`pdftotext`, `pdftohtml`, and `pdfinfo`) and optionally uses `qpdf` for encrypted files. The plugin uses `match-overrides.json` there as the persistent source of title mappings and ignored titles.
|
||||
|
||||
## Build XPI
|
||||
|
||||
|
|
@ -20,6 +20,6 @@ From the repository root:
|
|||
python scripts/build_plugin.py
|
||||
```
|
||||
|
||||
The output is `dist/kindle-zotero-importer.xpi`.
|
||||
The output is `dist/kanzi.xpi`.
|
||||
|
||||
The plugin manifest targets Zotero `6.999` through `10.0.*`.
|
||||
|
|
|
|||
856
plugin/bootstrap.js
vendored
856
plugin/bootstrap.js
vendored
File diff suppressed because it is too large
Load diff
1310
plugin/manager.html
1310
plugin/manager.html
File diff suppressed because it is too large
Load diff
|
|
@ -1,14 +1,14 @@
|
|||
{
|
||||
"manifest_version": 2,
|
||||
"name": "Kanzi",
|
||||
"version": "0.6.14",
|
||||
"version": "0.7.1",
|
||||
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
|
||||
"homepage_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer",
|
||||
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
|
||||
"author": "Utku Bilen Demir",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"id": "kindlezoteroimporter@utkubilen.de",
|
||||
"update_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates.json",
|
||||
"update_url": "https://github.com/UtkuBilenDemir/kanzi/releases/latest/download/updates.json",
|
||||
"strict_min_version": "6.999",
|
||||
"strict_max_version": "10.0.*"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
pref("extensions.kindleZoteroImporter.projectDir", "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer");
|
||||
pref("extensions.kindleZoteroImporter.pythonPath", "/opt/homebrew/bin/python3");
|
||||
pref("extensions.kindleZoteroImporter.zoteroDbPath", "/home/user/Zotero/zotero.sqlite");
|
||||
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "/home/user/Zotero/storage");
|
||||
pref("extensions.kindleZoteroImporter.shareAnonymized", true);
|
||||
pref("extensions.kindleZoteroImporter.projectDir", "");
|
||||
pref("extensions.kindleZoteroImporter.pythonPath", "python3");
|
||||
pref("extensions.kindleZoteroImporter.zoteroDbPath", "");
|
||||
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "");
|
||||
pref("extensions.kindleZoteroImporter.shareAnonymized", false);
|
||||
pref("extensions.kindleZoteroImporter.askedAnonymizedShare", false);
|
||||
pref("extensions.kindleZoteroImporter.shareConsentVersion", 0);
|
||||
|
|
|
|||
|
|
@ -1,11 +1,14 @@
|
|||
[project]
|
||||
name = "kindle-zotero-importer"
|
||||
version = "0.1.0"
|
||||
version = "0.7.1"
|
||||
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.optional-dependencies]
|
||||
collector = ["fastapi>=0.110", "uvicorn>=0.29"]
|
||||
|
||||
[project.scripts]
|
||||
kindle-zotero-importer = "kindle_zotero_importer.cli:main"
|
||||
|
||||
|
|
|
|||
|
|
@ -1,62 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DIST_DIR = ROOT / "dist"
|
||||
OUTPUT = DIST_DIR / "kindle-zotero-importer-canary.xpi"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
DIST_DIR.mkdir(exist_ok=True)
|
||||
if OUTPUT.exists():
|
||||
OUTPUT.unlink()
|
||||
|
||||
manifest = {
|
||||
"manifest_version": 2,
|
||||
"name": "Kindle Zotero Importer Canary",
|
||||
"version": "0.1.0",
|
||||
"description": "Minimal install test for Kindle Zotero Importer.",
|
||||
"homepage_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer",
|
||||
"author": "Utku Bilen Demir",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"id": "kindlezoteroimportercanary@utkubilen.de",
|
||||
"update_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates-canary.json",
|
||||
"strict_min_version": "6.999",
|
||||
"strict_max_version": "9.*",
|
||||
}
|
||||
},
|
||||
}
|
||||
bootstrap = """
|
||||
function install(data, reason) {}
|
||||
function uninstall(data, reason) {}
|
||||
async function startup(data, reason) {
|
||||
await Zotero.initializationPromise;
|
||||
Zotero.debug('Kindle Zotero Importer Canary started');
|
||||
}
|
||||
function shutdown(data, reason) {}
|
||||
""".lstrip()
|
||||
|
||||
prefs = 'pref("extensions.kindleZoteroImporterCanary.enabled", true);\n'
|
||||
|
||||
with zipfile.ZipFile(OUTPUT, "w", zipfile.ZIP_DEFLATED) as archive:
|
||||
for name, content in [
|
||||
("bootstrap.js", bootstrap),
|
||||
("manifest.json", json.dumps(manifest, indent=2) + "\n"),
|
||||
("prefs.js", prefs),
|
||||
]:
|
||||
info = zipfile.ZipInfo(name)
|
||||
info.create_system = 3
|
||||
info.external_attr = 0o644 << 16
|
||||
archive.writestr(info, content, zipfile.ZIP_DEFLATED)
|
||||
|
||||
print(f"Built {OUTPUT}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -8,13 +8,15 @@ from pathlib import Path
|
|||
ROOT = Path(__file__).resolve().parents[1]
|
||||
PLUGIN_DIR = ROOT / "plugin"
|
||||
DIST_DIR = ROOT / "dist"
|
||||
OUTPUT = DIST_DIR / "kindle-zotero-importer.xpi"
|
||||
OUTPUT = DIST_DIR / "kanzi.xpi"
|
||||
LEGACY_OUTPUT = DIST_DIR / "kindle-zotero-importer.xpi"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
DIST_DIR.mkdir(exist_ok=True)
|
||||
if OUTPUT.exists():
|
||||
OUTPUT.unlink()
|
||||
for p in [OUTPUT, LEGACY_OUTPUT]:
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
|
||||
manifest_path = PLUGIN_DIR / "manifest.json"
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
|
|
@ -58,12 +60,19 @@ def main() -> int:
|
|||
info.create_system = 3
|
||||
info.external_attr = 0o644 << 16
|
||||
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
|
||||
runtime_files = [ROOT / "plugin_runner.py", *(ROOT / "src" / "kindle_zotero_importer").glob("*.py")]
|
||||
for path in runtime_files:
|
||||
archive_name = "runtime/" + str(path.relative_to(ROOT))
|
||||
info = zipfile.ZipInfo(archive_name)
|
||||
info.create_system = 3
|
||||
info.external_attr = 0o644 << 16
|
||||
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
|
||||
|
||||
with zipfile.ZipFile(OUTPUT) as archive:
|
||||
names = set(archive.namelist())
|
||||
if "manifest.json" not in names or "bootstrap.js" not in names:
|
||||
if not {"manifest.json", "bootstrap.js", "runtime/plugin_runner.py"}.issubset(names):
|
||||
raise SystemExit(
|
||||
"Invalid XPI: manifest.json and bootstrap.js must be at archive root"
|
||||
"Invalid XPI: plugin files and Python runtime are required"
|
||||
)
|
||||
|
||||
print(f"Built {OUTPUT}")
|
||||
|
|
|
|||
|
|
@ -1,14 +1,20 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .clippings import clippings_to_jsonable, load_clippings
|
||||
from .epub_position import add_epub_positions
|
||||
from .final_plan import build_final_writer_plan
|
||||
from .import_plan import DEFAULT_COLOUR_MAP, build_import_plan
|
||||
from .final_plan import FINAL_FORMAT, build_final_writer_plan
|
||||
from .import_plan import (
|
||||
DEFAULT_COLOUR_MAP,
|
||||
_attach_notes_to_highlights,
|
||||
_related_items_for_citekeys,
|
||||
build_import_plan,
|
||||
)
|
||||
from .matcher import build_match_report, load_json
|
||||
from .mismatch_review import build_mismatch_review
|
||||
from .overrides import generate_override_skeleton, load_overrides
|
||||
|
|
@ -39,6 +45,256 @@ def _load_colour_map(workdir: Path) -> dict[str, str] | None:
|
|||
return None
|
||||
|
||||
|
||||
def _merge_annotation_history(
|
||||
previous_plan: dict | None, current_annotations: list[dict], deletions: list[str]
|
||||
) -> list[dict]:
|
||||
previous = []
|
||||
if previous_plan:
|
||||
previous = previous_plan.get("all_annotations") or previous_plan.get("annotations") or []
|
||||
deleted = set(deletions)
|
||||
by_id = {
|
||||
annotation["clipping_id"]: annotation
|
||||
for annotation in previous
|
||||
if annotation.get("clipping_id") and annotation["clipping_id"] not in deleted
|
||||
}
|
||||
for annotation in current_annotations:
|
||||
clipping_id = annotation.get("clipping_id")
|
||||
if clipping_id:
|
||||
by_id[clipping_id] = annotation
|
||||
return sorted(
|
||||
by_id.values(),
|
||||
key=lambda annotation: (
|
||||
annotation.get("attachment_item_id") or 0,
|
||||
annotation.get("annotation", {}).get("sortIndex") or "",
|
||||
annotation.get("clipping_id") or "",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _refresh_history_relations(
|
||||
annotations: list[dict], zotero_index: dict
|
||||
) -> None:
|
||||
"""Resolve citekeys for old annotations without re-positioning them."""
|
||||
items_by_id = {item["item_id"]: item for item in zotero_index.get("items", [])}
|
||||
for entry in annotations:
|
||||
annotation = entry.get("annotation") or {}
|
||||
entry["related_items"] = _related_items_for_citekeys(
|
||||
(annotation.get("text") or "", annotation.get("comment") or ""),
|
||||
entry.get("parent_item_id"),
|
||||
items_by_id,
|
||||
)
|
||||
|
||||
|
||||
def _changed_note_clipping_ids(
|
||||
current_clippings: list[dict], previous_annotations: list[dict]
|
||||
) -> set[str]:
|
||||
previous_by_id = {
|
||||
annotation.get("clipping_id"): annotation
|
||||
for annotation in previous_annotations
|
||||
if annotation.get("clipping_id")
|
||||
}
|
||||
return {
|
||||
clipping["id"]
|
||||
for clipping in current_clippings
|
||||
if clipping["id"] in previous_by_id
|
||||
and previous_by_id[clipping["id"]].get("note_ids", [])
|
||||
!= clipping.get("note_ids", [])
|
||||
}
|
||||
|
||||
|
||||
def _merge_positioned_history(
|
||||
previous_plan: dict | None, current_plan: dict, source_ids: set[str]
|
||||
) -> dict:
|
||||
by_id = {
|
||||
item.get("clipping", {}).get("id"): item
|
||||
for item in (previous_plan or {}).get("items", [])
|
||||
if item.get("clipping", {}).get("id") in source_ids
|
||||
}
|
||||
for item in current_plan.get("items", []):
|
||||
clipping_id = item.get("clipping", {}).get("id")
|
||||
if clipping_id:
|
||||
by_id[clipping_id] = item
|
||||
merged = dict(current_plan)
|
||||
merged["items"] = list(by_id.values())
|
||||
status_counts: dict[str, int] = {}
|
||||
for item in merged["items"]:
|
||||
status = item.get("status", "unknown")
|
||||
status_counts[status] = status_counts.get(status, 0) + 1
|
||||
merged["status_counts"] = status_counts
|
||||
return merged
|
||||
|
||||
|
||||
def _deletion_ids(
|
||||
previous_integrated_ids: set[str], source_ids: set[str], previous_plan: dict | None
|
||||
) -> set[str]:
|
||||
deleted = set(previous_integrated_ids - source_ids)
|
||||
deleted.update((previous_plan or {}).get("delete_failed_ids", []))
|
||||
return deleted
|
||||
|
||||
|
||||
def _deferred_clipping_ids(
|
||||
previous_plan: dict | None, current_ids: set[str]
|
||||
) -> set[str]:
|
||||
return {
|
||||
clipping_id
|
||||
for clipping_id in (previous_plan or {}).get("deferred_clipping_ids", [])
|
||||
if clipping_id in current_ids
|
||||
}
|
||||
|
||||
|
||||
def _discarded_clipping_ids(
|
||||
previous_plan: dict | None, current_ids: set[str]
|
||||
) -> set[str]:
|
||||
"""Clips the user permanently discarded in Conflicts (e.g. standalone notes)."""
|
||||
return {
|
||||
clipping_id
|
||||
for clipping_id in (previous_plan or {}).get("discarded_clipping_ids", [])
|
||||
if clipping_id in current_ids
|
||||
}
|
||||
|
||||
|
||||
def _changed_override_titles(
|
||||
previous_snapshot: dict | None, current_overrides: dict
|
||||
) -> set[str]:
|
||||
if previous_snapshot is None:
|
||||
return set(current_overrides)
|
||||
return {
|
||||
title
|
||||
for title in set(previous_snapshot) | set(current_overrides)
|
||||
if previous_snapshot.get(title) != current_overrides.get(title)
|
||||
}
|
||||
|
||||
|
||||
def _clipping_ids_for_titles(
|
||||
clippings: list[dict], titles: set[str]
|
||||
) -> set[str]:
|
||||
return {clipping["id"] for clipping in clippings if clipping["title"] in titles}
|
||||
|
||||
|
||||
def _reattempt_failed_with_alternate_attachments(positioned_plan: dict) -> dict:
|
||||
"""Retry text/rect failures against another attachment of the same item.
|
||||
|
||||
`_choose_attachment` commits to one file (e.g. the sole EPUB) without
|
||||
comparing text. When that file lacks the quote but a sibling PDF/EPUB
|
||||
has it (e.g. Dostoyevsky EPUB vs PDF), re-anchor the clone there and
|
||||
keep the winner, tagged `positioned-via-alternate-attachment`.
|
||||
Partial shingle anchors are also upgraded when a sibling holds the
|
||||
full quote.
|
||||
"""
|
||||
retryable = {"epub-text-not-found", "pdf-text-not-found", "pdf-rects-not-found"}
|
||||
retries: list[dict] = []
|
||||
upgrade_ids: set[str] = set()
|
||||
for item in positioned_plan.get("items", []):
|
||||
if item.get("status") in retryable:
|
||||
pass
|
||||
elif item.get("status") == "positioned" and "partial-text-match" in (
|
||||
item.get("problems") or []
|
||||
):
|
||||
clipping_id = (item.get("clipping") or {}).get("id")
|
||||
if clipping_id:
|
||||
upgrade_ids.add(clipping_id)
|
||||
else:
|
||||
continue
|
||||
zotero = item.get("zotero") or {}
|
||||
current = zotero.get("attachment") or {}
|
||||
choices = zotero.get("attachment_choices") or []
|
||||
alternates = [
|
||||
choice
|
||||
for choice in choices
|
||||
if choice.get("item_id") != current.get("item_id")
|
||||
]
|
||||
if not alternates:
|
||||
continue
|
||||
alternates.sort(
|
||||
key=lambda choice: 0
|
||||
if choice.get("content_type") != current.get("content_type")
|
||||
else 1
|
||||
)
|
||||
clone = copy.deepcopy(item)
|
||||
clone_zotero = dict(clone.get("zotero") or {})
|
||||
clone_zotero["attachment"] = alternates[0]
|
||||
clone["zotero"] = clone_zotero
|
||||
clone["status"] = "ready-for-positioning"
|
||||
clone["problems"] = []
|
||||
retries.append(clone)
|
||||
if not retries:
|
||||
return positioned_plan
|
||||
mini = {"format": positioned_plan.get("format"), "items": retries}
|
||||
mini = add_epub_positions(mini)
|
||||
mini = add_pdf_positions(mini)
|
||||
winners = {}
|
||||
for attempt in mini.get("items", []):
|
||||
clipping_id = (attempt.get("clipping") or {}).get("id")
|
||||
if clipping_id and attempt.get("status") == "positioned":
|
||||
winners[clipping_id] = attempt
|
||||
if not winners:
|
||||
return positioned_plan
|
||||
merged = []
|
||||
for item in positioned_plan.get("items", []):
|
||||
clipping_id = (item.get("clipping") or {}).get("id")
|
||||
if clipping_id in winners:
|
||||
won = winners[clipping_id]
|
||||
if clipping_id in upgrade_ids and "partial-text-match" in (
|
||||
won.get("problems") or []
|
||||
):
|
||||
merged.append(item)
|
||||
continue
|
||||
won = dict(won)
|
||||
carried = [
|
||||
problem
|
||||
for problem in item.get("problems", [])
|
||||
if problem != "partial-text-match"
|
||||
or "partial-text-match" in (won.get("problems") or [])
|
||||
]
|
||||
won["problems"] = [
|
||||
*carried,
|
||||
"positioned-via-alternate-attachment",
|
||||
]
|
||||
merged.append(won)
|
||||
else:
|
||||
merged.append(item)
|
||||
updated = dict(positioned_plan)
|
||||
updated["items"] = merged
|
||||
status_counts: dict[str, int] = {}
|
||||
for entry in merged:
|
||||
status = entry.get("status", "unknown")
|
||||
status_counts[status] = status_counts.get(status, 0) + 1
|
||||
updated["status_counts"] = status_counts
|
||||
return updated
|
||||
|
||||
|
||||
def _add_previous_imported_comments(
|
||||
annotations: list[dict], previous_plan: dict | None
|
||||
) -> None:
|
||||
previous_by_id = {
|
||||
annotation.get("clipping_id"): annotation
|
||||
for annotation in ((previous_plan or {}).get("all_annotations") or (previous_plan or {}).get("annotations", []))
|
||||
if annotation.get("clipping_id")
|
||||
}
|
||||
for annotation in annotations:
|
||||
previous = previous_by_id.get(annotation.get("clipping_id"))
|
||||
if previous:
|
||||
annotation["previous_imported_comment"] = previous.get("annotation", {}).get(
|
||||
"comment", ""
|
||||
)
|
||||
|
||||
|
||||
def _unresolved_clipping_ids(
|
||||
previous_plan: dict | None, source_ids: set[str]
|
||||
) -> set[str]:
|
||||
return {
|
||||
item.get("clipping", {}).get("id")
|
||||
for item in (previous_plan or {}).get("items", [])
|
||||
if item.get("status") not in {
|
||||
"positioned",
|
||||
"ignored-title",
|
||||
"pdf-position-skipped-non-highlight",
|
||||
"epub-position-skipped-non-highlight",
|
||||
}
|
||||
and item.get("clipping", {}).get("id") in source_ids
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="kindle-zotero-importer",
|
||||
|
|
@ -61,11 +317,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
"index-zotero", help="Index Zotero items and attachments read-only"
|
||||
)
|
||||
index_parser.add_argument(
|
||||
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite"
|
||||
"--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
|
||||
)
|
||||
index_parser.add_argument(
|
||||
"--storage-root",
|
||||
default="/home/user/Zotero/storage",
|
||||
default=str(Path.home() / "Zotero" / "storage"),
|
||||
help="Path to Zotero storage directory for storage: attachments",
|
||||
)
|
||||
index_parser.add_argument(
|
||||
|
|
@ -175,11 +431,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
help="Directory for generated importer artifacts",
|
||||
)
|
||||
run_parser.add_argument(
|
||||
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite"
|
||||
"--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
|
||||
)
|
||||
run_parser.add_argument(
|
||||
"--storage-root",
|
||||
default="/home/user/Zotero/storage",
|
||||
default=str(Path.home() / "Zotero" / "storage"),
|
||||
help="Path to Zotero storage directory for storage: attachments",
|
||||
)
|
||||
run_parser.add_argument(
|
||||
|
|
@ -203,6 +459,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
action="store_true",
|
||||
help="Force full re-import from scratch, ignoring incremental state (processes all clippings)",
|
||||
)
|
||||
run_parser.add_argument(
|
||||
"--retry-unresolved",
|
||||
action="store_true",
|
||||
help="Retry previously unresolved positioning and attachment cases",
|
||||
)
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
|
|
@ -373,37 +634,55 @@ def main(argv: list[str] | None = None) -> int:
|
|||
report_progress(2, "Reading clippings", "Parsing My Clippings.txt")
|
||||
new_clippings_list = load_clippings(args.clippings_file)
|
||||
new_clippings = clippings_to_jsonable(new_clippings_list)
|
||||
new_ids = {c["id"] for c in new_clippings.get("clippings", [])}
|
||||
source_clippings = new_clippings.get("clippings", [])
|
||||
source_ids = {c["id"] for c in source_clippings}
|
||||
effective_clippings = _attach_notes_to_highlights(source_clippings)
|
||||
new_ids = {c["id"] for c in effective_clippings}
|
||||
|
||||
# Incremental state: previous clippings and previous integrated ids
|
||||
is_incremental = not args.full
|
||||
prev_clippings_ids: set[str] = set()
|
||||
prev_effective_ids: set[str] = set()
|
||||
prev_integrated_ids: set[str] = set()
|
||||
prev_final_plan = None
|
||||
prev_positioned_plan = None
|
||||
prev_final_path = artifact("import-plan.final.json")
|
||||
if prev_final_path.exists():
|
||||
try:
|
||||
prev_final_plan = load_json(str(prev_final_path))
|
||||
previous_annotations = prev_final_plan.get("all_annotations") or prev_final_plan.get("annotations", [])
|
||||
prev_integrated_ids = {a.get("clipping_id") for a in previous_annotations if a.get("clipping_id")}
|
||||
except Exception:
|
||||
prev_final_plan = None
|
||||
prev_integrated_ids = set()
|
||||
if is_incremental:
|
||||
prev_clippings_path = artifact("clippings.json")
|
||||
prev_final_path = artifact("import-plan.final.json")
|
||||
prev_positioned_path = artifact("import-plan.positioned.json")
|
||||
if prev_clippings_path.exists():
|
||||
try:
|
||||
prev_clippings_ids = {c["id"] for c in load_json(str(prev_clippings_path)).get("clippings", [])}
|
||||
previous_clippings = load_json(str(prev_clippings_path)).get("clippings", [])
|
||||
prev_clippings_ids = {c["id"] for c in previous_clippings}
|
||||
prev_effective_ids = {
|
||||
c["id"] for c in _attach_notes_to_highlights(previous_clippings)
|
||||
}
|
||||
except Exception:
|
||||
prev_clippings_ids = set()
|
||||
if prev_final_path.exists():
|
||||
prev_effective_ids = set()
|
||||
if prev_positioned_path.exists():
|
||||
try:
|
||||
prev_final_plan = load_json(str(prev_final_path))
|
||||
prev_integrated_ids = {a.get("clipping_id") for a in prev_final_plan.get("annotations", []) if a.get("clipping_id")}
|
||||
prev_positioned_plan = load_json(str(prev_positioned_path))
|
||||
except Exception:
|
||||
prev_final_plan = None
|
||||
prev_integrated_ids = set()
|
||||
prev_positioned_plan = None
|
||||
# If no previous state, fall back to full
|
||||
if not prev_clippings_ids and not prev_integrated_ids:
|
||||
is_incremental = False
|
||||
|
||||
if is_incremental:
|
||||
# New or previously not integrated (skipped) -> need to (re)process
|
||||
to_process_ids = set(new_ids - prev_integrated_ids)
|
||||
# Only source entries not seen on the preceding run need positioning.
|
||||
# Previously unresolved entries are re-queued when their override changes.
|
||||
to_process_ids = set(new_ids - prev_effective_ids)
|
||||
# Deletions: previously integrated but now absent (removed or changed text)
|
||||
deletions_ids = sorted(prev_integrated_ids - new_ids)
|
||||
deletions_ids = _deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
|
||||
# Also re-queue any previously integrated clippings whose title now has an override
|
||||
# (mapping may have changed, need to move annotation). We need overrides dict for this.
|
||||
# Load overrides early for this check (already loaded later, but do here for incremental decision)
|
||||
|
|
@ -412,39 +691,37 @@ def main(argv: list[str] | None = None) -> int:
|
|||
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
|
||||
except Exception:
|
||||
_ov_for_inc = {}
|
||||
for c in new_clippings_list:
|
||||
if c.id in prev_integrated_ids and c.title in _ov_for_inc:
|
||||
to_process_ids.add(c.id)
|
||||
# Also re-queue highlights whose bracket colour would change the annotation
|
||||
# (so you can recolour a past highlight by adding [r]/[o] etc. to its note or highlight text)
|
||||
if colour_map:
|
||||
import re as _re
|
||||
# Build map of new notes by title for quick lookup
|
||||
new_notes_by_title: dict[str, list] = {}
|
||||
for n in new_clippings_list:
|
||||
if n.kind == "note" and n.id not in prev_integrated_ids:
|
||||
# new note (not yet integrated) — check if it has a bracket colour
|
||||
m = _re.match(r"^\s*\[([^\]]+)\]\s*", n.text or "")
|
||||
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
|
||||
new_notes_by_title.setdefault(n.title, []).append(n)
|
||||
for c in new_clippings_list:
|
||||
if c.id not in prev_integrated_ids:
|
||||
continue
|
||||
if c.kind != "highlight":
|
||||
continue
|
||||
has_bracket = False
|
||||
# check highlight text itself
|
||||
for t in [c.text or "", getattr(c, "comment", "") or ""]:
|
||||
m = _re.match(r"^\s*\[([^\]]+)\]\s*", t)
|
||||
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
|
||||
has_bracket = True
|
||||
break
|
||||
# also check if a *new* note for same title would attach to this highlight (by location/page proximity)
|
||||
# Approximate: any new note with same title is considered for re-queue (conservative, ensures recolours)
|
||||
if not has_bracket and c.title in new_notes_by_title:
|
||||
has_bracket = True
|
||||
if has_bracket:
|
||||
to_process_ids.add(c.id)
|
||||
changed_override_titles = _changed_override_titles(
|
||||
(prev_final_plan or {}).get("overrides_snapshot"), _ov_for_inc
|
||||
)
|
||||
to_process_ids.update(
|
||||
_clipping_ids_for_titles(effective_clippings, changed_override_titles)
|
||||
)
|
||||
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
|
||||
for annotation in previous_annotations:
|
||||
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
|
||||
deletions_ids.add(annotation["clipping_id"])
|
||||
newly_added_source_ids = source_ids - prev_clippings_ids
|
||||
for c in effective_clippings:
|
||||
if set(c.get("note_ids", [])) & newly_added_source_ids:
|
||||
to_process_ids.add(c["id"])
|
||||
to_process_ids.update(
|
||||
_changed_note_clipping_ids(effective_clippings, previous_annotations)
|
||||
)
|
||||
to_process_ids.update(
|
||||
clipping_id
|
||||
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
|
||||
if clipping_id in new_ids
|
||||
)
|
||||
to_process_ids.update(_deferred_clipping_ids(prev_final_plan, new_ids))
|
||||
if args.retry_unresolved and prev_positioned_plan:
|
||||
to_process_ids.update(
|
||||
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
|
||||
)
|
||||
# Permanently discarded clips (user action in Conflicts) never re-queue.
|
||||
to_process_ids.difference_update(
|
||||
_discarded_clipping_ids(prev_final_plan, new_ids)
|
||||
)
|
||||
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
|
||||
try:
|
||||
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
|
||||
|
|
@ -455,6 +732,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
to_process_ids.update(prev_integrated_ids)
|
||||
except Exception:
|
||||
pass
|
||||
deletions_ids = sorted(deletions_ids)
|
||||
# Also handle deleted overrides: if a previously integrated title's override was deleted,
|
||||
# its annotations should be deleted. Detect via previous final plan's titles vs current overrides.
|
||||
# For incremental, we don't have previous overrides, so we handle deletions via the main deletions list
|
||||
|
|
@ -466,7 +744,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
# handled as not in to_process_ids? Actually it is still in new_ids, but we re-queued it above only if title in overrides.
|
||||
# For now, rely on full re-import for deleted-override cleanup.
|
||||
# Build filtered clippings for this delta run
|
||||
filtered_list = [c for c in new_clippings_list if c.id in to_process_ids]
|
||||
filtered_list = [c for c in effective_clippings if c["id"] in to_process_ids]
|
||||
# If nothing to do and no deletions, still need to report but skip expensive steps
|
||||
if not filtered_list and not deletions_ids:
|
||||
report_progress(15, "Matching titles", "No new highlights — incremental skip")
|
||||
|
|
@ -479,6 +757,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
matches = build_match_report(clippings, zotero_index, overrides)
|
||||
override_skeleton = generate_override_skeleton(matches)
|
||||
plan = {"format": "kindle-zotero-importer.import-plan.v1", "items": []}
|
||||
epub_plan = {"format": plan["format"], "items": []}
|
||||
positioned_plan = {"format": plan["format"], "items": []}
|
||||
final_plan = {"format": FINAL_FORMAT, "source_format": positioned_plan.get("format"), "annotation_count": 0, "skipped_counts": {}, "annotations": [], "deletions": [], "is_incremental": True, "incremental_stats": {"new_clippings": len(new_ids), "prev_integrated": len(prev_integrated_ids), "to_process": 0, "deletions": 0}}
|
||||
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
|
||||
|
|
@ -492,7 +771,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if overrides_path.exists():
|
||||
overrides = load_overrides(load_json(str(overrides_path)))
|
||||
# Only the delta clippings go through matching/positioning
|
||||
delta_clippings = clippings_to_jsonable(filtered_list)
|
||||
delta_clippings = {
|
||||
"format": new_clippings.get("format"),
|
||||
"count": len(filtered_list),
|
||||
"clippings": filtered_list,
|
||||
}
|
||||
report_progress(15, "Matching titles", f"Matching {len(filtered_list)} new/pending highlights (incremental)")
|
||||
matches = build_match_report(delta_clippings, zotero_index, overrides)
|
||||
override_skeleton = generate_override_skeleton(matches)
|
||||
|
|
@ -504,6 +787,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
epub_plan = add_epub_positions(plan)
|
||||
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
|
||||
positioned_plan = add_pdf_positions(epub_plan)
|
||||
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
|
||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||
final_plan = build_final_writer_plan(positioned_plan)
|
||||
# Attach deletions for writer
|
||||
|
|
@ -537,14 +821,51 @@ def main(argv: list[str] | None = None) -> int:
|
|||
epub_plan = add_epub_positions(plan)
|
||||
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
|
||||
positioned_plan = add_pdf_positions(epub_plan)
|
||||
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
|
||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||
final_plan = build_final_writer_plan(positioned_plan)
|
||||
final_plan["deletions"] = []
|
||||
final_plan["deletions"] = sorted(
|
||||
_deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
|
||||
)
|
||||
final_plan["is_incremental"] = False
|
||||
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
|
||||
final_plan["colorMap"] = final_plan["colourMap"]
|
||||
mismatch_review = build_mismatch_review(positioned_plan, matches)
|
||||
|
||||
if is_incremental:
|
||||
positioned_plan = _merge_positioned_history(
|
||||
prev_positioned_plan, positioned_plan, new_ids
|
||||
)
|
||||
matches = build_match_report(new_clippings, zotero_index, overrides)
|
||||
override_skeleton = generate_override_skeleton(matches)
|
||||
mismatch_review = build_mismatch_review(positioned_plan, matches)
|
||||
|
||||
_add_previous_imported_comments(
|
||||
final_plan.get("annotations", []), prev_final_plan
|
||||
)
|
||||
|
||||
delta_annotation_count = len(final_plan.get("annotations", []))
|
||||
if is_incremental:
|
||||
final_plan["all_annotations"] = _merge_annotation_history(
|
||||
prev_final_plan,
|
||||
final_plan.get("annotations", []),
|
||||
final_plan.get("deletions", []),
|
||||
)
|
||||
else:
|
||||
final_plan["all_annotations"] = _merge_annotation_history(
|
||||
prev_final_plan,
|
||||
final_plan.get("annotations", []),
|
||||
final_plan.get("deletions", []),
|
||||
)
|
||||
_refresh_history_relations(final_plan["all_annotations"], zotero_index)
|
||||
final_plan["annotation_count"] = len(final_plan["all_annotations"])
|
||||
final_plan["delta_annotation_count"] = delta_annotation_count
|
||||
final_plan["processed_clipping_ids"] = sorted(new_ids)
|
||||
final_plan["overrides_snapshot"] = overrides or {}
|
||||
final_plan["discarded_clipping_ids"] = sorted(
|
||||
_discarded_clipping_ids(prev_final_plan, new_ids)
|
||||
)
|
||||
|
||||
indent = 2 if args.pretty else None
|
||||
outputs = {
|
||||
"clippings": artifact("clippings.json"),
|
||||
|
|
@ -595,7 +916,8 @@ def main(argv: list[str] | None = None) -> int:
|
|||
"counts": {
|
||||
"clippings": len(clippings.get("clippings", [])),
|
||||
"unique_titles": len(matches.get("matches", [])),
|
||||
"final_annotations": final_plan.get("annotation_count", 0),
|
||||
"final_annotations": final_plan.get("delta_annotation_count", 0),
|
||||
"integrated_annotations": final_plan.get("annotation_count", 0),
|
||||
"match_statuses": match_counts,
|
||||
"plan_statuses": status_counts,
|
||||
"skipped_final": final_plan.get("skipped_counts", {}),
|
||||
|
|
|
|||
|
|
@ -80,7 +80,74 @@ def load_clippings(path: str) -> list[Clipping]:
|
|||
return parse_clippings_text(file.read())
|
||||
|
||||
|
||||
def _norm_text(text: str) -> str:
|
||||
import unicodedata
|
||||
|
||||
t = unicodedata.normalize("NFKC", text or "").lower()
|
||||
t = re.sub(r"-\s*\n\s*", "", t)
|
||||
t = re.sub(r"[^\w\s]+", " ", t)
|
||||
t = re.sub(r"\s+", " ", t).strip()
|
||||
return t
|
||||
|
||||
|
||||
def dedup_clippings(clippings: list[Clipping]) -> list[Clipping]:
|
||||
result: list[Clipping] = []
|
||||
exact_highlights: dict[tuple[str, str, str | None, str | None], int] = {}
|
||||
revised_highlights: dict[tuple[str, int], int] = {}
|
||||
seen_other_ids: set[str] = set()
|
||||
|
||||
for c in clippings:
|
||||
if c.kind != "highlight":
|
||||
# Notes carry meaning independently and must never collide with highlights.
|
||||
if c.id not in seen_other_ids:
|
||||
result.append(c)
|
||||
seen_other_ids.add(c.id)
|
||||
continue
|
||||
|
||||
norm = _norm_text(c.text)
|
||||
exact_key = (c.title, norm, c.page, c.location)
|
||||
if exact_key in exact_highlights:
|
||||
continue
|
||||
|
||||
location_start = _range_start(c.location)
|
||||
revision_key = (c.title, location_start) if location_start is not None else None
|
||||
if revision_key is not None and revision_key in revised_highlights:
|
||||
previous_index = revised_highlights[revision_key]
|
||||
previous = result[previous_index]
|
||||
previous_norm = _norm_text(previous.text)
|
||||
if _is_text_revision(previous_norm, norm):
|
||||
# Kindle keeps earlier versions when a highlight is extended. The
|
||||
# longest version represents the final selection.
|
||||
if len(norm) > len(previous_norm):
|
||||
result[previous_index] = c
|
||||
exact_highlights[exact_key] = previous_index
|
||||
continue
|
||||
|
||||
index = len(result)
|
||||
result.append(c)
|
||||
exact_highlights[exact_key] = index
|
||||
if revision_key is not None:
|
||||
revised_highlights[revision_key] = index
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _range_start(value: str | None) -> int | None:
|
||||
if not value:
|
||||
return None
|
||||
match = re.search(r"\d+", value)
|
||||
return int(match.group()) if match else None
|
||||
|
||||
|
||||
def _is_text_revision(left: str, right: str) -> bool:
|
||||
if not left or not right:
|
||||
return False
|
||||
shorter, longer = sorted((left, right), key=len)
|
||||
return len(shorter) >= 20 and longer.startswith(shorter)
|
||||
|
||||
|
||||
def clippings_to_jsonable(clippings: list[Clipping]) -> dict[str, Any]:
|
||||
clippings = dedup_clippings(clippings)
|
||||
return {
|
||||
"format": "kindle-zotero-importer.clippings.v1",
|
||||
"count": len(clippings),
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ from dataclasses import dataclass
|
|||
from html import unescape
|
||||
from html.parser import HTMLParser
|
||||
import posixpath
|
||||
import unicodedata
|
||||
import zipfile
|
||||
import xml.etree.ElementTree as ET
|
||||
from typing import Any
|
||||
|
|
@ -68,6 +69,8 @@ def _position_item(
|
|||
annotation["sortIndex"] = position["sortIndex"]
|
||||
updated["annotation"] = annotation
|
||||
updated["status"] = "positioned"
|
||||
if position.get("partial"):
|
||||
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
|
||||
return updated
|
||||
|
||||
|
||||
|
|
@ -104,11 +107,42 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
|
|||
return None
|
||||
|
||||
match_start = normalized_haystack.find(normalized_quote)
|
||||
if match_start < 0:
|
||||
raw_start: int | None = None
|
||||
raw_end: int | None = None
|
||||
partial = False
|
||||
if match_start >= 0:
|
||||
match_end = match_start + len(normalized_quote)
|
||||
raw_start = haystack_map[match_start]
|
||||
raw_end = haystack_map[match_end - 1] + 1
|
||||
else:
|
||||
stripped_quote, _ = _strip_with_map(normalized_quote)
|
||||
if stripped_quote and len(stripped_quote) >= 20:
|
||||
stripped_haystack, stripped_map = _strip_with_map(normalized_haystack)
|
||||
strip_start = stripped_haystack.find(stripped_quote)
|
||||
strip_len = len(stripped_quote)
|
||||
if strip_start < 0:
|
||||
hit = _partial_stripped_range(stripped_haystack, stripped_quote)
|
||||
if hit is not None:
|
||||
strip_start, strip_end = hit
|
||||
strip_len = strip_end - strip_start
|
||||
partial = True
|
||||
if strip_start >= 0:
|
||||
# stripped_map points into normalized_haystack;
|
||||
# convert via haystack_map back to raw haystack offsets.
|
||||
norm_start = stripped_map[strip_start]
|
||||
norm_end = stripped_map[strip_start + strip_len - 1]
|
||||
raw_start = haystack_map[norm_start]
|
||||
raw_end = haystack_map[norm_end] + 1
|
||||
else:
|
||||
span = _shingle_span(normalized_haystack, _quote_words(normalized_quote))
|
||||
if span is not None:
|
||||
raw_start = haystack_map[span[0]]
|
||||
raw_end = haystack_map[span[1] - 1] + 1
|
||||
partial = True
|
||||
else:
|
||||
partial = False
|
||||
if raw_start is None or raw_end is None:
|
||||
return None
|
||||
match_end = match_start + len(normalized_quote)
|
||||
raw_start = haystack_map[match_start]
|
||||
raw_end = haystack_map[match_end - 1] + 1
|
||||
|
||||
# Account for the spaces inserted between text nodes in haystack construction.
|
||||
adjusted_nodes = _nodes_for_joined_text(text_nodes)
|
||||
|
|
@ -139,6 +173,7 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
|
|||
"value": value,
|
||||
},
|
||||
"sortIndex": f"{spine_step:05d}|{raw_start:08d}",
|
||||
"partial": partial,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -213,6 +248,11 @@ def _clean_text(text: str) -> str:
|
|||
|
||||
|
||||
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||
text = unicodedata.normalize("NFKC", text)
|
||||
text = (
|
||||
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
|
||||
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
|
||||
)
|
||||
normalized = []
|
||||
mapping = []
|
||||
last_was_space = False
|
||||
|
|
@ -232,6 +272,58 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
|||
return "".join(normalized), mapping
|
||||
|
||||
|
||||
def _strip_with_map(text: str) -> tuple[str, list[int]]:
|
||||
text = unicodedata.normalize("NFKC", text).casefold()
|
||||
stripped: list[str] = []
|
||||
mapping: list[int] = []
|
||||
for index, char in enumerate(text):
|
||||
if char.isalnum():
|
||||
stripped.append(char)
|
||||
mapping.append(index)
|
||||
return "".join(stripped), mapping
|
||||
|
||||
|
||||
def _partial_stripped_range(
|
||||
haystack: str, quote: str, min_len: int = 30
|
||||
) -> tuple[int, int] | None:
|
||||
if len(quote) < min_len * 2:
|
||||
return None
|
||||
mid = len(quote) // 2
|
||||
for part in (quote[:mid], quote[mid:]):
|
||||
if len(part) < min_len:
|
||||
continue
|
||||
start = haystack.find(part)
|
||||
if start >= 0:
|
||||
return start, start + len(part)
|
||||
return None
|
||||
|
||||
|
||||
def _quote_words(normalized: str) -> list[str]:
|
||||
words: list[str] = []
|
||||
for token in normalized.split():
|
||||
cleaned = "".join(char for char in token if char.isalnum())
|
||||
if cleaned:
|
||||
words.append(cleaned)
|
||||
return words
|
||||
|
||||
|
||||
def _shingle_span(
|
||||
normalized_text: str, quote_words: list[str], width: int = 6
|
||||
) -> tuple[int, int] | None:
|
||||
first: int | None = None
|
||||
last: int | None = None
|
||||
for start in range(len(quote_words) - width + 1):
|
||||
needle = " ".join(quote_words[start : start + width])
|
||||
at = normalized_text.find(needle)
|
||||
if at >= 0:
|
||||
end = at + len(needle)
|
||||
first = at if first is None else min(first, at)
|
||||
last = end if last is None else max(last, end)
|
||||
if first is None or last is None:
|
||||
return None
|
||||
return first, last
|
||||
|
||||
|
||||
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
|
||||
nodes = []
|
||||
cursor = 0
|
||||
|
|
|
|||
|
|
@ -1,9 +1,42 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import hashlib
|
||||
import re
|
||||
import unicodedata
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _normalise(text: str) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
# NFKC, lower, strip, de-hyphenate, collapse whitespace/punct
|
||||
t = unicodedata.normalize("NFKC", text).lower()
|
||||
t = re.sub(r"-\s*\n\s*", "", t) # de-hyphenate line breaks
|
||||
t = re.sub(r"[\u2010-\u2015\u2018\u2019\u201c\u201d]", "'", t)
|
||||
t = re.sub(r"[^a-z0-9]+", " ", t)
|
||||
t = re.sub(r"\s+", " ", t).strip()
|
||||
return t
|
||||
|
||||
|
||||
def _hash_norm(text: str) -> str:
|
||||
return hashlib.sha256(_normalise(text).encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _fuzzy_hash(text: str) -> str:
|
||||
# Trigram Jaccard-ish: sort unique trigrams of normalised text, hash them.
|
||||
# More tolerant to OCR swaps (rn/m, 1/l, 0/O) than strict SHA.
|
||||
norm = _normalise(text)
|
||||
if len(norm) < 3:
|
||||
return _hash_norm(text)
|
||||
trigrams = {norm[i : i + 3] for i in range(len(norm) - 2)}
|
||||
# also add word-shingled version (sorted words) for reordered line breaks
|
||||
words = sorted(norm.split())
|
||||
tri_words = {" ".join(words[i : i + 3]) for i in range(max(1, len(words) - 2))}
|
||||
combined = sorted(trigrams | tri_words)
|
||||
return hashlib.sha256(" ".join(combined).encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
FINAL_FORMAT = "kindle-zotero-importer.zotero-writer-plan.v1"
|
||||
|
||||
|
||||
|
|
@ -22,18 +55,30 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
|
|||
)
|
||||
continue
|
||||
clipping = item["clipping"]
|
||||
# robust identifiers for cross-library unification
|
||||
raw = annotation.get("text") or clipping.get("text") or ""
|
||||
norm_hash = _hash_norm(raw)
|
||||
fuzzy_hash = _fuzzy_hash(raw)
|
||||
annotations.append(
|
||||
{
|
||||
"clipping_id": clipping["id"],
|
||||
"clipping_title": clipping["title"],
|
||||
"clipping_added_on": clipping.get("added_on"),
|
||||
"clipping_added_on_iso": clipping.get("added_on_iso"),
|
||||
"note_ids": clipping.get("note_ids", []),
|
||||
"integrated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
||||
"attachment_item_id": attachment["item_id"],
|
||||
"attachment_key": attachment["key"],
|
||||
"parent_item_id": item["zotero"]["parent_item_id"],
|
||||
"parent_key": item["zotero"]["parent_key"],
|
||||
"related_items": item["zotero"].get("related_items", []),
|
||||
"citation_key": item["zotero"].get("citation_key"),
|
||||
"doi": item["zotero"].get("doi"),
|
||||
"isbn": item["zotero"].get("isbn"),
|
||||
"issn": item["zotero"].get("issn"),
|
||||
"publication_title": item["zotero"].get("publication_title"),
|
||||
"norm_hash": norm_hash,
|
||||
"fuzzy_hash": fuzzy_hash,
|
||||
"annotation": {
|
||||
"type": annotation["type"],
|
||||
"text": annotation.get("text") or "",
|
||||
|
|
|
|||
|
|
@ -124,6 +124,12 @@ def _plan_clipping(
|
|||
zotero_item, overrides_by_title.get(clipping["title"]) or {}
|
||||
)
|
||||
status = "ready-for-positioning" if attachment else attachment_status
|
||||
stub = _annotation_stub(clipping, colour_map) if attachment else None
|
||||
related_items = (
|
||||
_append_citekey_links(stub, clipping, zotero_item, items_by_id)
|
||||
if stub
|
||||
else []
|
||||
)
|
||||
return {
|
||||
"clipping": clipping,
|
||||
"status": status,
|
||||
|
|
@ -133,17 +139,101 @@ def _plan_clipping(
|
|||
"parent_key": zotero_item["key"],
|
||||
"parent_title": zotero_item.get("title"),
|
||||
"citation_key": zotero_item["fields"].get("citationKey"),
|
||||
"doi": zotero_item["fields"].get("DOI"),
|
||||
"isbn": zotero_item["fields"].get("ISBN"),
|
||||
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
|
||||
"url": zotero_item["fields"].get("url"),
|
||||
"publication_title": zotero_item["fields"].get("publicationTitle"),
|
||||
"related_items": related_items,
|
||||
"attachment": attachment,
|
||||
"attachment_choices": zotero_item.get("attachments", []),
|
||||
"expected_attachment_type": _expected_attachment_type(
|
||||
clipping, zotero_item.get("attachments", [])
|
||||
),
|
||||
},
|
||||
"annotation": _annotation_stub(clipping, colour_map) if attachment else None,
|
||||
"annotation": stub,
|
||||
"problems": problems,
|
||||
}
|
||||
|
||||
|
||||
def _append_citekey_links(
|
||||
stub: dict[str, Any],
|
||||
clipping: dict[str, Any],
|
||||
parent_item: dict[str, Any],
|
||||
items_by_id: dict[int, dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Turn `@citekey` mentions into clickable Zotero item links.
|
||||
|
||||
Kindle-side `@key` tokens (Obsidian-style) resolve against the library
|
||||
index; matches to *other* items are appended to the annotation comment
|
||||
as `zotero://select` links, which Zotero renders clickable. Self-links
|
||||
to the parent item are skipped.
|
||||
"""
|
||||
related_items = _related_items_for_citekeys(
|
||||
(clipping.get("text") or "", clipping.get("comment") or ""),
|
||||
parent_item.get("item_id"),
|
||||
items_by_id,
|
||||
)
|
||||
links = [
|
||||
f"@{target['citation_key']} {target['link']}" for target in related_items
|
||||
]
|
||||
if not links:
|
||||
return []
|
||||
existing = stub.get("comment") or ""
|
||||
block = "Related: " + " · ".join(links)
|
||||
if block not in existing:
|
||||
stub["comment"] = f"{existing}\n\n{block}" if existing else block
|
||||
return related_items
|
||||
|
||||
|
||||
def _select_link(item: dict[str, Any]) -> str:
|
||||
"""Library-aware `zotero://select` link for an indexed item.
|
||||
|
||||
Group-library targets need the `/groups/{groupID}/items/{key}` form;
|
||||
personal-library (or unknown) targets use `/library/items/{key}`.
|
||||
"""
|
||||
group_id = item.get("group_id")
|
||||
if group_id:
|
||||
return f"zotero://select/groups/{group_id}/items/{item['key']}"
|
||||
return f"zotero://select/library/items/{item['key']}"
|
||||
|
||||
|
||||
def _related_items_for_citekeys(
|
||||
sources: tuple[str, ...],
|
||||
parent_item_id: int | None,
|
||||
items_by_id: dict[int, dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
tokens = {
|
||||
token.casefold()
|
||||
for source in sources
|
||||
for token in re.findall(r"@([A-Za-z0-9_][\w:.-]*)", source or "")
|
||||
}
|
||||
if not tokens:
|
||||
return []
|
||||
by_citekey = {
|
||||
str(item.get("fields", {}).get("citationKey", "")).casefold(): item
|
||||
for item in items_by_id.values()
|
||||
if item.get("fields", {}).get("citationKey")
|
||||
}
|
||||
related = []
|
||||
seen_item_ids = set()
|
||||
for token in sorted(tokens):
|
||||
target = by_citekey.get(token)
|
||||
target_id = target.get("item_id") if target else None
|
||||
if not target or target_id == parent_item_id or target_id in seen_item_ids:
|
||||
continue
|
||||
seen_item_ids.add(target_id)
|
||||
related.append(
|
||||
{
|
||||
"item_id": target_id,
|
||||
"key": target["key"],
|
||||
"citation_key": target["fields"]["citationKey"],
|
||||
"link": _select_link(target),
|
||||
}
|
||||
)
|
||||
return related
|
||||
|
||||
|
||||
def _choose_attachment(
|
||||
zotero_item: dict[str, Any],
|
||||
override: dict[str, Any] | None = None,
|
||||
|
|
|
|||
|
|
@ -7,15 +7,17 @@ import re
|
|||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unicodedata
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
_TOOL_PATHS = {
|
||||
"pdftotext": ("/home/user/bin/pdftotext", "/opt/homebrew/bin/pdftotext"),
|
||||
"pdftohtml": ("/home/user/bin/pdftohtml", "/opt/homebrew/bin/pdftohtml"),
|
||||
"pdfinfo": ("/home/user/bin/pdfinfo", "/opt/homebrew/bin/pdfinfo"),
|
||||
"qpdf": ("/home/user/bin/qpdf", "/opt/homebrew/bin/qpdf"),
|
||||
"pdftotext": (str(Path.home() / "bin" / "pdftotext"), "/opt/homebrew/bin/pdftotext"),
|
||||
"pdftohtml": (str(Path.home() / "bin" / "pdftohtml"), "/opt/homebrew/bin/pdftohtml"),
|
||||
"pdfinfo": (str(Path.home() / "bin" / "pdfinfo"), "/opt/homebrew/bin/pdfinfo"),
|
||||
"qpdf": (str(Path.home() / "bin" / "qpdf"), "/opt/homebrew/bin/qpdf"),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -29,6 +31,15 @@ def _tool(name: str) -> str:
|
|||
return name
|
||||
|
||||
|
||||
def _missing_required_tools() -> list[str]:
|
||||
missing = []
|
||||
for name in ("pdftotext", "pdftohtml", "pdfinfo"):
|
||||
path = _tool(name)
|
||||
if path == name and shutil.which(name) is None:
|
||||
missing.append(name)
|
||||
return missing
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PDFWord:
|
||||
text: str
|
||||
|
|
@ -54,18 +65,29 @@ def add_pdf_positions(plan: dict[str, Any]) -> dict[str, Any]:
|
|||
size_cache: dict[tuple[str, int], tuple[float, float]] = {}
|
||||
decrypted_cache: dict[str, str] = {}
|
||||
positioned = []
|
||||
missing_tools = _missing_required_tools()
|
||||
with tempfile.TemporaryDirectory(prefix="kindle-zotero-pdf-") as temp_dir:
|
||||
for item in plan["items"]:
|
||||
positioned.append(
|
||||
_position_item(
|
||||
item,
|
||||
text_cache,
|
||||
xml_cache,
|
||||
size_cache,
|
||||
decrypted_cache,
|
||||
temp_dir,
|
||||
attachment = (item.get("zotero") or {}).get("attachment") or {}
|
||||
if (
|
||||
missing_tools
|
||||
and item.get("status") == "ready-for-positioning"
|
||||
and attachment.get("content_type") == "application/pdf"
|
||||
):
|
||||
positioned.append(
|
||||
_with_problem(item, f"pdf-tools-missing:{','.join(missing_tools)}")
|
||||
)
|
||||
else:
|
||||
positioned.append(
|
||||
_position_item(
|
||||
item,
|
||||
text_cache,
|
||||
xml_cache,
|
||||
size_cache,
|
||||
decrypted_cache,
|
||||
temp_dir,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
status_counts: dict[str, int] = {}
|
||||
for item in positioned:
|
||||
|
|
@ -105,7 +127,12 @@ def _position_item(
|
|||
text_pages, item["clipping"]["text"], item["clipping"].get("page")
|
||||
)
|
||||
if page_index is None:
|
||||
return _with_problem(item, "pdf-text-not-found")
|
||||
failed = _with_problem(item, "pdf-text-not-found")
|
||||
hint = _page_hint_text(text_pages, item["clipping"].get("page"))
|
||||
if hint:
|
||||
failed = dict(failed)
|
||||
failed["page_hint"] = hint
|
||||
return failed
|
||||
page_xml = xml_cache.setdefault(
|
||||
(extraction_path, page_index),
|
||||
extract_pdf_page_xml(extraction_path, page_index),
|
||||
|
|
@ -131,6 +158,8 @@ def _position_item(
|
|||
annotation["sortIndex"] = position.get("sortIndex")
|
||||
updated["annotation"] = annotation
|
||||
updated["status"] = "positioned"
|
||||
if position.get("partial"):
|
||||
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
|
||||
return updated
|
||||
|
||||
|
||||
|
|
@ -206,6 +235,37 @@ def find_pdf_text_page(
|
|||
normalized_page, _ = _normalize_with_map(pages[index])
|
||||
if normalized_quote in normalized_page:
|
||||
return index
|
||||
# Fallback: alphanumeric-only search tolerates PDF extraction artifacts
|
||||
# such as "o f" for "of", hyphenation, and punctuation differences.
|
||||
stripped_quote, _ = _strip_with_map(quote)
|
||||
if stripped_quote and len(stripped_quote) >= 20:
|
||||
for index in page_order:
|
||||
stripped_page, _ = _strip_with_map(pages[index])
|
||||
if stripped_quote in stripped_page:
|
||||
return index
|
||||
# Partial fallback: long highlights may span pages or contain a
|
||||
# divergent middle (edition differences). Anchor to either half so
|
||||
# Zotero-style word search behaviour is approximated.
|
||||
for index in page_order:
|
||||
stripped_page, _ = _strip_with_map(pages[index])
|
||||
if _partial_stripped_hit(stripped_page, stripped_quote):
|
||||
return index
|
||||
# Shingle fallback: scattered OCR substitutions (e.g. Kindle
|
||||
# "yadsınaraz" vs PDF "yadsınamaz") break every long substring but
|
||||
# leave most 6-word runs intact. Rank pages by shingle hits.
|
||||
quote_words = _quote_words(normalized_quote)
|
||||
if len(quote_words) >= 6:
|
||||
required = 2 if len(quote_words) >= 12 else 1
|
||||
best_index: int | None = None
|
||||
best_hits = 0
|
||||
for index in page_order:
|
||||
normalized_page, _ = _normalize_with_map(pages[index])
|
||||
hits = _shingle_hits(normalized_page, quote_words)
|
||||
if hits > best_hits:
|
||||
best_hits = hits
|
||||
best_index = index
|
||||
if best_index is not None and best_hits >= required:
|
||||
return best_index
|
||||
return None
|
||||
|
||||
|
||||
|
|
@ -269,25 +329,52 @@ def find_pdf_rects(
|
|||
text = " ".join(word.text for word in page.words)
|
||||
normalized_text, text_map = _normalize_with_map(text)
|
||||
start = normalized_text.find(normalized_quote)
|
||||
if start < 0:
|
||||
raw_start: int | None = None
|
||||
raw_end: int | None = None
|
||||
partial = False
|
||||
if start >= 0:
|
||||
end = start + len(normalized_quote)
|
||||
raw_start = text_map[start]
|
||||
raw_end = text_map[end - 1] + 1
|
||||
else:
|
||||
stripped_quote, _ = _strip_with_map(quote)
|
||||
if stripped_quote and len(stripped_quote) >= 20:
|
||||
stripped_text, stripped_map = _strip_with_map(text)
|
||||
strip_start = stripped_text.find(stripped_quote)
|
||||
if strip_start >= 0:
|
||||
strip_end = strip_start + len(stripped_quote)
|
||||
raw_start = stripped_map[strip_start]
|
||||
raw_end = stripped_map[strip_end - 1] + 1
|
||||
else:
|
||||
hit = _partial_stripped_range(stripped_text, stripped_quote)
|
||||
if hit is not None:
|
||||
strip_start, strip_end = hit
|
||||
raw_start = stripped_map[strip_start]
|
||||
raw_end = stripped_map[strip_end - 1] + 1
|
||||
partial = True
|
||||
else:
|
||||
span = _shingle_span(normalized_text, _quote_words(normalized_quote))
|
||||
if span is not None:
|
||||
raw_start = text_map[span[0]]
|
||||
raw_end = text_map[span[1] - 1] + 1
|
||||
partial = True
|
||||
if raw_start is None or raw_end is None:
|
||||
return None
|
||||
end = start + len(normalized_quote)
|
||||
raw_start = text_map[start]
|
||||
raw_end = text_map[end - 1] + 1
|
||||
matched_words = [
|
||||
word for word in page.words if word.end >= raw_start and word.start <= raw_end
|
||||
]
|
||||
if not matched_words:
|
||||
return None
|
||||
pdf_width, pdf_height = size_cache.setdefault(
|
||||
(path, page.index),
|
||||
_pdf_page_size(path, page.index + 1, page.width, page.height),
|
||||
)
|
||||
size_key = (path, page.index)
|
||||
if size_key not in size_cache:
|
||||
size_cache[size_key] = _pdf_page_size(path, page.index + 1, page.width, page.height)
|
||||
pdf_width, pdf_height = size_cache[size_key]
|
||||
rects = _word_rects(matched_words, page, pdf_width, pdf_height)
|
||||
return {
|
||||
"pageIndex": page.index,
|
||||
"rects": rects,
|
||||
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
|
||||
"partial": partial,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -340,6 +427,11 @@ def _pdf_page_size(
|
|||
|
||||
|
||||
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||
text = unicodedata.normalize("NFKC", text)
|
||||
text = (
|
||||
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
|
||||
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
|
||||
)
|
||||
normalized = []
|
||||
mapping = []
|
||||
last_was_space = False
|
||||
|
|
@ -359,6 +451,112 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
|||
return "".join(normalized), mapping
|
||||
|
||||
|
||||
def _strip_with_map(text: str) -> tuple[str, list[int]]:
|
||||
"""Alphanumeric-only folding with index mapping.
|
||||
|
||||
Removes spaces and punctuation so PDF artifacts like "o f" for "of",
|
||||
hyphenated line breaks, and curly-quote/dash variants still match.
|
||||
Mapping points back into the input string for rect recovery.
|
||||
"""
|
||||
text = unicodedata.normalize("NFKC", text).casefold()
|
||||
stripped: list[str] = []
|
||||
mapping: list[int] = []
|
||||
for index, char in enumerate(text):
|
||||
if char.isalnum():
|
||||
stripped.append(char)
|
||||
mapping.append(index)
|
||||
return "".join(stripped), mapping
|
||||
|
||||
|
||||
def _partial_stripped_range(
|
||||
haystack: str, quote: str, min_len: int = 30
|
||||
) -> tuple[int, int] | None:
|
||||
"""Best half-quote hit for long highlights spanning pages/editions."""
|
||||
if len(quote) < min_len * 2:
|
||||
return None
|
||||
mid = len(quote) // 2
|
||||
for part in (quote[:mid], quote[mid:]):
|
||||
if len(part) < min_len:
|
||||
continue
|
||||
start = haystack.find(part)
|
||||
if start >= 0:
|
||||
return start, start + len(part)
|
||||
return None
|
||||
|
||||
|
||||
def _partial_stripped_hit(haystack: str, quote: str, min_len: int = 30) -> bool:
|
||||
return _partial_stripped_range(haystack, quote, min_len) is not None
|
||||
|
||||
|
||||
def _quote_words(normalized: str, width: int = 6) -> list[str]:
|
||||
del width
|
||||
words: list[str] = []
|
||||
for token in normalized.split():
|
||||
cleaned = "".join(char for char in token if char.isalnum())
|
||||
if cleaned:
|
||||
words.append(cleaned)
|
||||
return words
|
||||
|
||||
|
||||
def _shingle_hits(normalized_page: str, quote_words: list[str], width: int = 6) -> int:
|
||||
hits = 0
|
||||
seen: set[tuple[str, ...]] = set()
|
||||
for start in range(len(quote_words) - width + 1):
|
||||
shingle = tuple(quote_words[start : start + width])
|
||||
if shingle in seen:
|
||||
continue
|
||||
seen.add(shingle)
|
||||
if " ".join(shingle) in normalized_page:
|
||||
hits += 1
|
||||
return hits
|
||||
|
||||
|
||||
def _shingle_span(
|
||||
normalized_text: str, quote_words: list[str], width: int = 6
|
||||
) -> tuple[int, int] | None:
|
||||
first: int | None = None
|
||||
last: int | None = None
|
||||
for start in range(len(quote_words) - width + 1):
|
||||
needle = " ".join(quote_words[start : start + width])
|
||||
at = normalized_text.find(needle)
|
||||
if at >= 0:
|
||||
end = at + len(needle)
|
||||
first = at if first is None else min(first, at)
|
||||
last = end if last is None else max(last, end)
|
||||
if first is None or last is None:
|
||||
return None
|
||||
return first, last
|
||||
|
||||
|
||||
def _page_hint_text(
|
||||
pages: list[str], kindle_page: str | None, radius: int = 1, limit: int = 1200
|
||||
) -> dict[str, Any] | None:
|
||||
"""Snippet of the PDF pages around the Kindle page hint.
|
||||
|
||||
Lets the user eyeball which part a `pdf-text-not-found` highlight
|
||||
belongs to and place it manually. Truncated to keep artifacts small.
|
||||
"""
|
||||
if not kindle_page or not kindle_page.isdigit():
|
||||
return None
|
||||
center = int(kindle_page) - 1
|
||||
if center < 0 or center >= len(pages):
|
||||
return None
|
||||
parts = []
|
||||
for index in range(max(0, center - radius), min(len(pages), center + radius + 1)):
|
||||
snippet = " ".join(pages[index].split())
|
||||
if snippet:
|
||||
parts.append(f"[PDF p.{index + 1}] {snippet[:limit]}")
|
||||
if not parts:
|
||||
return None
|
||||
return {
|
||||
"kindle_page": kindle_page,
|
||||
"pdf_pages": list(
|
||||
range(max(1, center - radius + 1), min(len(pages), center + radius + 1) + 1)
|
||||
),
|
||||
"text": "\n\n".join(parts)[: limit * 2 + 64],
|
||||
}
|
||||
|
||||
|
||||
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
|
||||
updated = dict(item)
|
||||
updated["status"] = problem
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ class ZoteroItem:
|
|||
key: str
|
||||
item_type: str
|
||||
title: str | None
|
||||
library_id: int | None = None
|
||||
group_id: int | None = None
|
||||
creators: list[str] = field(default_factory=list)
|
||||
fields: dict[str, str] = field(default_factory=dict)
|
||||
attachments: list[ZoteroAttachment] = field(default_factory=list)
|
||||
|
|
@ -47,15 +49,19 @@ def build_zotero_index(db_path: str, storage_root: str | None = None) -> dict[st
|
|||
creators = _load_creators(connection)
|
||||
attachments = _load_attachments(connection, storage_root)
|
||||
|
||||
groups = _load_groups(connection)
|
||||
indexed_items: list[ZoteroItem] = []
|
||||
for row in items:
|
||||
fields = _metadata_for_item(row)
|
||||
library_id = row["libraryID"]
|
||||
indexed_items.append(
|
||||
ZoteroItem(
|
||||
item_id=row["itemID"],
|
||||
key=row["key"],
|
||||
item_type=row["typeName"],
|
||||
title=fields.get("title"),
|
||||
library_id=library_id,
|
||||
group_id=groups.get(library_id),
|
||||
creators=creators.get(row["itemID"], []),
|
||||
fields=fields,
|
||||
attachments=attachments.get(row["itemID"], []),
|
||||
|
|
@ -80,6 +86,7 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
|
|||
SELECT
|
||||
i.itemID,
|
||||
i.key,
|
||||
i.libraryID,
|
||||
it.typeName,
|
||||
{field_columns}
|
||||
FROM items i
|
||||
|
|
@ -95,6 +102,14 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
|
|||
return list(connection.execute(query))
|
||||
|
||||
|
||||
def _load_groups(connection: sqlite3.Connection) -> dict[int, int]:
|
||||
try:
|
||||
rows = connection.execute("SELECT groupID, libraryID FROM groups")
|
||||
except Exception:
|
||||
return {}
|
||||
return {row["libraryID"]: row["groupID"] for row in rows}
|
||||
|
||||
|
||||
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
|
||||
rows = connection.execute(
|
||||
"""
|
||||
|
|
|
|||
15
tests/fixtures/windows-smoke/My Clippings.txt
vendored
Normal file
15
tests/fixtures/windows-smoke/My Clippings.txt
vendored
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
Kanzi Dummy EPUB
|
||||
- Your Highlight on page 1 | location 100-120 | Added on Tuesday, 18 May 2021 14:18:13
|
||||
|
||||
This is a test highlight long enough to import for EPUB.
|
||||
==========
|
||||
Kanzi Dummy PDF
|
||||
- Your Highlight on page 2 | location 200-220 | Added on Tuesday, 18 May 2021 14:20:00
|
||||
|
||||
This is a test highlight long enough to import for PDF.
|
||||
==========
|
||||
Unknown Book Title XYZ
|
||||
- Your Highlight on page 1 | location 10-11 | Added on Tuesday, 18 May 2021 14:18:13
|
||||
|
||||
This should land in Conflicts for manual mapping.
|
||||
==========
|
||||
36
tests/fixtures/windows-smoke/checklist.txt
vendored
Normal file
36
tests/fixtures/windows-smoke/checklist.txt
vendored
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
Kanzi Windows smoke test (Parallels VM)
|
||||
|
||||
COPY THESE 3 FILES INTO THE VM:
|
||||
- My Clippings.txt
|
||||
- kanzi-dummy.epub
|
||||
- kanzi-dummy.pdf
|
||||
- dist/kanzi.xpi (from repo root, same cross-platform build)
|
||||
|
||||
1) PowerShell:
|
||||
winget install --id Python.Python.3.13 --exact
|
||||
winget install --id oschwartz10612.Poppler --exact
|
||||
winget install --id QPDF.QPDF --exact
|
||||
python --version
|
||||
pdftotext -v
|
||||
pdfinfo -v
|
||||
qpdf --version
|
||||
|
||||
2) Fully quit + reopen Zotero (picks up PATH).
|
||||
|
||||
3) Fresh Zotero, sync off. Create:
|
||||
- Book "Kanzi Dummy EPUB" + attach kanzi-dummy.epub
|
||||
- Book "Kanzi Dummy PDF" + attach kanzi-dummy.pdf
|
||||
|
||||
4) Tools -> Kanzi -> Choose My Clippings.txt (the copied one).
|
||||
Leave Python path empty to test auto-detect.
|
||||
|
||||
EXPECT:
|
||||
- 2 highlights position and await review, 1 title in Conflicts
|
||||
("Unknown Book Title XYZ").
|
||||
- Summary shows new Status by Kindle Title table with source counts.
|
||||
- Save a mapping for the unknown title -> Re-import processes it.
|
||||
- Table headers sort and persist across pages.
|
||||
|
||||
IF PYTHON NOT FOUND:
|
||||
Settings shows detected path hint; paste full python.exe path,
|
||||
Save Settings, retry.
|
||||
BIN
tests/fixtures/windows-smoke/kanzi-dummy.epub
vendored
Normal file
BIN
tests/fixtures/windows-smoke/kanzi-dummy.epub
vendored
Normal file
Binary file not shown.
33
tests/fixtures/windows-smoke/kanzi-dummy.pdf
vendored
Normal file
33
tests/fixtures/windows-smoke/kanzi-dummy.pdf
vendored
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<< /Type /Catalog /Pages 2 0 R >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Length 134 >>
|
||||
stream
|
||||
BT /F1 12 Tf 72 720 Td (Kanzi Dummy PDF) Tj ET
|
||||
BT /F1 12 Tf 72 700 Td (This is a test highlight long enough to import for PDF.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
xref
|
||||
0 6
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000058 00000 n
|
||||
0000000115 00000 n
|
||||
0000000241 00000 n
|
||||
0000000311 00000 n
|
||||
trailer
|
||||
<< /Size 6 /Root 1 0 R >>
|
||||
startxref
|
||||
495
|
||||
%%EOF
|
||||
93
tests/fixtures/windows-smoke/make_fixtures.py
vendored
Normal file
93
tests/fixtures/windows-smoke/make_fixtures.py
vendored
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
"""Generate minimal EPUB + PDF fixtures for the Windows smoke test.
|
||||
|
||||
Run: python3 tests/fixtures/windows-smoke/make_fixtures.py
|
||||
Outputs kanzi-dummy.epub and kanzi-dummy.pdf next to this script.
|
||||
Both contain the highlight sentences from 'My Clippings.txt' so
|
||||
EPUB CFI and PDF text positioning can succeed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import zipfile
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
EPUB_TEXT = "This is a test highlight long enough to import for EPUB."
|
||||
PDF_TEXT = "This is a test highlight long enough to import for PDF."
|
||||
|
||||
|
||||
def make_epub(path: Path) -> None:
|
||||
chapter = f"""<?xml version="1.0" encoding="utf-8"?>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||||
<head><title>Kanzi Dummy</title></head>
|
||||
<body>
|
||||
<h1>Kanzi Dummy EPUB</h1>
|
||||
<p>Some intro text before the highlight.</p>
|
||||
<p>{EPUB_TEXT}</p>
|
||||
<p>Some trailing text after the highlight.</p>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
container = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
|
||||
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
|
||||
</container>
|
||||
"""
|
||||
opf = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<package version="3.0" unique-identifier="uid" xmlns="http://www.idpf.org/2007/opf">
|
||||
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Kanzi Dummy EPUB</dc:title><dc:identifier id="uid">kanzi-dummy-epub</dc:identifier><dc:language>en</dc:language></metadata>
|
||||
<manifest><item id="ch1" href="chapter1.xhtml" media-type="application/xhtml+xml"/></manifest>
|
||||
<spine><itemref idref="ch1"/></spine>
|
||||
</package>
|
||||
"""
|
||||
with zipfile.ZipFile(path, "w") as z:
|
||||
z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
|
||||
z.writestr("META-INF/container.xml", container, compress_type=zipfile.ZIP_DEFLATED)
|
||||
z.writestr("OEBPS/content.opf", opf, compress_type=zipfile.ZIP_DEFLATED)
|
||||
z.writestr("OEBPS/chapter1.xhtml", chapter, compress_type=zipfile.ZIP_DEFLATED)
|
||||
|
||||
|
||||
def _pdf_escape(text: str) -> str:
|
||||
return text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
|
||||
|
||||
|
||||
def make_pdf(path: Path) -> None:
|
||||
# Minimal one-page PDF with Helvetica text. pdftotext extracts it fine.
|
||||
content = (
|
||||
f"BT /F1 12 Tf 72 720 Td ({_pdf_escape('Kanzi Dummy PDF')}) Tj ET\n"
|
||||
f"BT /F1 12 Tf 72 700 Td ({_pdf_escape(PDF_TEXT)}) Tj ET\n"
|
||||
)
|
||||
objects: list[bytes] = []
|
||||
objects.append(b"<< /Type /Catalog /Pages 2 0 R >>")
|
||||
objects.append(b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
|
||||
objects.append(
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
|
||||
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"
|
||||
)
|
||||
objects.append(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
|
||||
objects.append(f"<< /Length {len(content.encode('latin-1'))} >>\nstream\n{content}endstream".encode("latin-1"))
|
||||
|
||||
out = bytearray(b"%PDF-1.4\n")
|
||||
offsets = [0]
|
||||
for i, body in enumerate(objects, start=1):
|
||||
offsets.append(len(out))
|
||||
out += f"{i} 0 obj\n".encode("ascii") + body + b"\nendobj\n"
|
||||
xref_pos = len(out)
|
||||
out += f"xref\n0 {len(objects) + 1}\n".encode("ascii")
|
||||
out += b"0000000000 65535 f \n"
|
||||
for off in offsets[1:]:
|
||||
out += f"{off:010d} 00000 n \n".encode("ascii")
|
||||
out += (
|
||||
f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
|
||||
f"startxref\n{xref_pos}\n%%EOF\n"
|
||||
).encode("ascii")
|
||||
path.write_bytes(bytes(out))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
make_epub(HERE / "kanzi-dummy.epub")
|
||||
make_pdf(HERE / "kanzi-dummy.pdf")
|
||||
print("wrote kanzi-dummy.epub + kanzi-dummy.pdf")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -2,12 +2,14 @@ import sys
|
|||
sys.path.insert(0, "src")
|
||||
sys.path.insert(0, ".")
|
||||
import importlib.util
|
||||
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy"]:
|
||||
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy","test_incremental","test_collector"]:
|
||||
spec = importlib.util.spec_from_file_location(name, f"tests/{name}.py")
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
for attr in dir(mod):
|
||||
if attr.startswith("test_"):
|
||||
if attr == "test_incremental_ids":
|
||||
continue
|
||||
getattr(mod, attr)()
|
||||
print(f"✓ {name}.{attr}")
|
||||
print("All Python tests passed")
|
||||
|
|
|
|||
|
|
@ -1,4 +1,9 @@
|
|||
from kindle_zotero_importer.clippings import parse_clippings_text, _stable_id
|
||||
from kindle_zotero_importer.clippings import (
|
||||
Clipping,
|
||||
_stable_id,
|
||||
dedup_clippings,
|
||||
parse_clippings_text,
|
||||
)
|
||||
|
||||
def test_parse_and_id_stable():
|
||||
txt = "My Book (Author)\n- Your Highlight at location 10-20 | Added on Monday, 6 October 2025 11:32:29\n\nSome text\n==========\n"
|
||||
|
|
@ -15,3 +20,57 @@ def test_added_on_iso():
|
|||
txt = "B\n- Your Highlight at location 1 | Added on Tuesday, 18 May 2021 14:18:13\n\nText\n==========\n"
|
||||
c = parse_clippings_text(txt)[0]
|
||||
assert c.added_on_iso.startswith("2021-05-18T14:18:13")
|
||||
|
||||
|
||||
def _clipping(identifier, kind, text, location):
|
||||
return Clipping(
|
||||
id=identifier,
|
||||
title="Book",
|
||||
kind=kind,
|
||||
text=text,
|
||||
raw_detail="detail",
|
||||
location=location,
|
||||
)
|
||||
|
||||
|
||||
def test_dedup_keeps_longest_progressive_highlight_revision():
|
||||
short = _clipping("short", "highlight", "A selection long enough to compare", "53-54")
|
||||
long = _clipping(
|
||||
"long",
|
||||
"highlight",
|
||||
"A selection long enough to compare and its extended ending",
|
||||
"53-55",
|
||||
)
|
||||
|
||||
assert dedup_clippings([short, long]) == [long]
|
||||
|
||||
|
||||
def test_dedup_preserves_unrelated_text_with_same_location_start():
|
||||
short = _clipping(
|
||||
"short",
|
||||
"highlight",
|
||||
"J. he is actually of obscure extraction and this is long enough",
|
||||
"53-54",
|
||||
)
|
||||
corrected = _clipping(
|
||||
"corrected",
|
||||
"highlight",
|
||||
"A. J. he is actually of obscure extraction and this is long enough",
|
||||
"53-55",
|
||||
)
|
||||
|
||||
assert dedup_clippings([short, corrected]) == [short, corrected]
|
||||
|
||||
|
||||
def test_dedup_preserves_same_text_at_different_locations():
|
||||
first = _clipping("first", "highlight", "A repeated sentence in a book", "10-11")
|
||||
second = _clipping("second", "highlight", "A repeated sentence in a book", "80-81")
|
||||
|
||||
assert dedup_clippings([first, second]) == [first, second]
|
||||
|
||||
|
||||
def test_dedup_never_collides_notes_with_highlights():
|
||||
highlight = _clipping("highlight", "highlight", "Same meaningful text", "10-11")
|
||||
note = _clipping("note", "note", "Same meaningful text", "11")
|
||||
|
||||
assert dedup_clippings([highlight, note]) == [highlight, note]
|
||||
|
|
|
|||
63
tests/test_collector.py
Normal file
63
tests/test_collector.py
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
|
||||
os.environ.setdefault(
|
||||
"KANZI_ANNOTATION_DB", str(Path(tempfile.gettempdir()) / "kanzi-test-import.db")
|
||||
)
|
||||
|
||||
try:
|
||||
import fastapi # noqa: F401
|
||||
except ModuleNotFoundError:
|
||||
class _FastAPI:
|
||||
def __init__(self, **_kwargs):
|
||||
pass
|
||||
|
||||
def get(self, *_args, **_kwargs):
|
||||
return lambda function: function
|
||||
|
||||
post = get
|
||||
|
||||
class _HTTPException(Exception):
|
||||
def __init__(self, status_code: int, detail: str):
|
||||
super().__init__(detail)
|
||||
self.status_code = status_code
|
||||
self.detail = detail
|
||||
|
||||
fake_fastapi = types.ModuleType("fastapi")
|
||||
fake_fastapi.FastAPI = _FastAPI
|
||||
fake_fastapi.HTTPException = _HTTPException
|
||||
fake_fastapi.Request = object
|
||||
fake_responses = types.ModuleType("fastapi.responses")
|
||||
fake_responses.HTMLResponse = str
|
||||
sys.modules["fastapi"] = fake_fastapi
|
||||
sys.modules["fastapi.responses"] = fake_responses
|
||||
|
||||
from collector import app as collector
|
||||
from fastapi import HTTPException
|
||||
|
||||
|
||||
class _Request:
|
||||
def __init__(self, token: str | None = None):
|
||||
self.headers = {"authorization": f"Bearer {token}"} if token else {}
|
||||
|
||||
|
||||
def test_collector_issues_and_verifies_installation_credentials():
|
||||
original_db = collector.DB
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
collector.DB = Path(temporary) / "annotations.db"
|
||||
collector.init_db()
|
||||
credentials = collector.register()
|
||||
|
||||
assert collector._authenticated_anon_id(
|
||||
_Request(credentials["token"])
|
||||
) == credentials["anonId"]
|
||||
try:
|
||||
collector._authenticated_anon_id(_Request("wrong"))
|
||||
except HTTPException as error:
|
||||
assert error.status_code == 401
|
||||
else:
|
||||
raise AssertionError("Invalid collector token was accepted")
|
||||
collector.DB = original_db
|
||||
|
|
@ -4,7 +4,7 @@ def test_final_includes_kindle_id_tag():
|
|||
positioned = {
|
||||
"format":"x",
|
||||
"items":[
|
||||
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None},
|
||||
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None,"note_ids":["note1"]},
|
||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
|
||||
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
||||
]
|
||||
|
|
@ -15,6 +15,7 @@ def test_final_includes_kindle_id_tag():
|
|||
assert {"name":"kindle-import"} in tags
|
||||
assert {"name":"kindle-id:abc123"} in tags
|
||||
assert plan["annotations"][0]["clipping_id"]=="abc123"
|
||||
assert plan["annotations"][0]["note_ids"] == ["note1"]
|
||||
|
||||
def test_final_skips_non_positioned():
|
||||
positioned = {"format":"x","items":[{"status":"epub-text-not-found","clipping":{"id":"a"},"zotero":{},"annotation":{}}]}
|
||||
|
|
|
|||
|
|
@ -28,10 +28,206 @@ def test_pdf_fallback_tag():
|
|||
"format":"x",
|
||||
"items":[
|
||||
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
|
||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
|
||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1","related_items":[{"item_id":20,"key":"P2","citation_key":"c2"}]},
|
||||
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
||||
]
|
||||
}
|
||||
plan = build_final_writer_plan(positioned)
|
||||
tags = plan["annotations"][0]["annotation"]["tags"]
|
||||
assert any(t["name"].startswith("kindle-id:") for t in tags)
|
||||
assert plan["annotations"][0]["related_items"][0]["item_id"] == 20
|
||||
|
||||
|
||||
def test_pdf_tolerates_of_split_artifact():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page, find_pdf_rects, PDFPageXML, PDFWord
|
||||
quote = "nothing other than idolatry of the machine and which"
|
||||
page_text = "nothing other than idolatry o f the machine and which"
|
||||
assert find_pdf_text_page([page_text], quote) == 0
|
||||
words = []
|
||||
cursor = 0
|
||||
for w in page_text.split():
|
||||
words.append(PDFWord(text=w, left=0.0, top=0.0, width=10.0, height=10.0, start=cursor, end=cursor + len(w)))
|
||||
cursor += len(w) + 1
|
||||
page = PDFPageXML(index=0, width=100.0, height=100.0, words=words)
|
||||
rects = find_pdf_rects("dummy.pdf", page, quote, {("dummy.pdf", 0): (100.0, 100.0)})
|
||||
assert rects is not None
|
||||
assert rects["pageIndex"] == 0
|
||||
assert rects["rects"]
|
||||
|
||||
|
||||
def test_pdf_tolerates_punctuation_and_quotes():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||
quote = "This, then, gives rise to “intemperate technicism”."
|
||||
page = "this then gives rise to intemperate technicism"
|
||||
assert find_pdf_text_page([page], quote) == 0
|
||||
|
||||
|
||||
def test_pdf_partial_half_anchors_spanning_text():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||
first = "A" * 70
|
||||
second = "Z" * 70
|
||||
quote = first + second
|
||||
page = "xx " + first.lower() + " yy"
|
||||
assert find_pdf_text_page([page], quote) == 0
|
||||
|
||||
|
||||
def test_pdf_shingle_fallback_scattered_ocr_errors():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||
quote = (
|
||||
"Gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
|
||||
"buna inanmamasım sağlayacak güç ve yeteneği bulur mucize yadsınaraz "
|
||||
"dururda bile olsa boyun eğmez"
|
||||
)
|
||||
page = (
|
||||
"gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
|
||||
"buna inanmaması sağlayacak güç ve yeteneği bulur mucize yadsınamaz "
|
||||
"dururda bile olsa boyun eğmez"
|
||||
)
|
||||
assert find_pdf_text_page([page], quote) == 0
|
||||
|
||||
|
||||
def test_page_hint_snippet_around_kindle_page():
|
||||
from kindle_zotero_importer.pdf_position import _page_hint_text
|
||||
pages = ["", "intro words here", "target page words here", "trailing words"]
|
||||
hint = _page_hint_text(pages, "3")
|
||||
assert hint is not None
|
||||
assert hint["kindle_page"] == "3"
|
||||
assert "target page words" in hint["text"]
|
||||
assert _page_hint_text(pages, None) is None
|
||||
assert _page_hint_text(pages, "99") is None
|
||||
|
||||
|
||||
def test_citekey_mentions_become_zotero_links():
|
||||
from kindle_zotero_importer.import_plan import build_import_plan
|
||||
clippings_payload = {
|
||||
"format": "x",
|
||||
"clippings": [
|
||||
{
|
||||
"id": "c1",
|
||||
"title": "Book A",
|
||||
"kind": "highlight",
|
||||
"text": "As argued by @bookb2020, this follows.",
|
||||
"page": None,
|
||||
"location": "10",
|
||||
}
|
||||
],
|
||||
}
|
||||
zotero_index = {
|
||||
"format": "x",
|
||||
"items": [
|
||||
{
|
||||
"item_id": 1,
|
||||
"key": "PARENT01",
|
||||
"title": "Book A",
|
||||
"fields": {"citationKey": "booka2020"},
|
||||
"attachments": [
|
||||
{
|
||||
"item_id": 11,
|
||||
"key": "ATTACH01",
|
||||
"title": "pdf",
|
||||
"content_type": "application/pdf",
|
||||
"path": "x.pdf",
|
||||
"resolved_path": "x.pdf",
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"title": "Book B",
|
||||
"fields": {"citationKey": "bookb2020"},
|
||||
"attachments": [],
|
||||
},
|
||||
],
|
||||
}
|
||||
match_report = {
|
||||
"format": "x",
|
||||
"matches": [
|
||||
{
|
||||
"clipping_title": "Book A",
|
||||
"status": "matched",
|
||||
"candidates": [{"item_id": 1}],
|
||||
}
|
||||
],
|
||||
}
|
||||
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||
comment = plan["items"][0]["annotation"]["comment"] or ""
|
||||
assert "@bookb2020" in comment
|
||||
assert "zotero://select/library/items/TARGET02" in comment
|
||||
assert plan["items"][0]["zotero"]["related_items"] == [
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"citation_key": "bookb2020",
|
||||
"link": "zotero://select/library/items/TARGET02",
|
||||
}
|
||||
]
|
||||
# self-links are skipped
|
||||
clippings_payload["clippings"][0]["text"] = "See @booka2020 here."
|
||||
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||
assert "zotero://select" not in (plan["items"][0]["annotation"]["comment"] or "")
|
||||
assert plan["items"][0]["zotero"]["related_items"] == []
|
||||
|
||||
|
||||
def test_historical_annotations_gain_citekey_relations():
|
||||
from kindle_zotero_importer.cli import _refresh_history_relations
|
||||
history = [
|
||||
{
|
||||
"clipping_id": "old",
|
||||
"parent_item_id": 1,
|
||||
"annotation": {
|
||||
"text": "An old highlight",
|
||||
"comment": "body without organs @deleuze1983",
|
||||
},
|
||||
}
|
||||
]
|
||||
zotero_index = {
|
||||
"items": [
|
||||
{
|
||||
"item_id": 1,
|
||||
"key": "SOURCE01",
|
||||
"fields": {"citationKey": "simondon2017a"},
|
||||
},
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"fields": {"citationKey": "deleuze1983"},
|
||||
},
|
||||
]
|
||||
}
|
||||
_refresh_history_relations(history, zotero_index)
|
||||
assert history[0]["related_items"] == [
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"citation_key": "deleuze1983",
|
||||
"link": "zotero://select/library/items/TARGET02",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_group_library_targets_use_group_select_links():
|
||||
from kindle_zotero_importer.import_plan import _select_link
|
||||
assert (
|
||||
_select_link({"key": "ABC123", "group_id": 6644344})
|
||||
== "zotero://select/groups/6644344/items/ABC123"
|
||||
)
|
||||
assert (
|
||||
_select_link({"key": "ABC123", "group_id": None})
|
||||
== "zotero://select/library/items/ABC123"
|
||||
)
|
||||
assert (
|
||||
_select_link({"key": "ABC123"})
|
||||
== "zotero://select/library/items/ABC123"
|
||||
)
|
||||
|
||||
|
||||
def test_epub_tolerates_extraction_artifacts():
|
||||
from kindle_zotero_importer.epub_position import find_epub_cfi, TextNode
|
||||
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="idolatry o f the machine", start=0, end=24)]
|
||||
quote = "idolatry of the machine and which"
|
||||
# quote is longer than node, so exact fails; use full haystack variant
|
||||
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="nothing other than idolatry o f the machine and which", start=0, end=53)]
|
||||
result = find_epub_cfi(nodes, quote)
|
||||
assert result is not None
|
||||
assert result["selector"]["value"].startswith("epubcfi(")
|
||||
|
|
|
|||
|
|
@ -1,5 +1,22 @@
|
|||
import json, tempfile, pathlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
|
||||
from kindle_zotero_importer import cli
|
||||
from kindle_zotero_importer.cli import (
|
||||
_add_previous_imported_comments,
|
||||
_changed_note_clipping_ids,
|
||||
_changed_override_titles,
|
||||
_clipping_ids_for_titles,
|
||||
_deferred_clipping_ids,
|
||||
_deletion_ids,
|
||||
_discarded_clipping_ids,
|
||||
_merge_annotation_history,
|
||||
_merge_positioned_history,
|
||||
_unresolved_clipping_ids,
|
||||
)
|
||||
from kindle_zotero_importer.clippings import clippings_to_jsonable, load_clippings
|
||||
from kindle_zotero_importer import pdf_position
|
||||
|
||||
def test_incremental_ids(tmp_path: Path):
|
||||
# simulate previous final with one integrated id
|
||||
|
|
@ -17,3 +34,324 @@ def test_incremental_ids(tmp_path: Path):
|
|||
# changed: aaa text changed -> new id ddd, old aaa no longer in new -> deletion
|
||||
new_ids2 = {"ddd","bbb","ccc"}
|
||||
assert (prev_integrated - new_ids2) == {"aaa"}
|
||||
|
||||
|
||||
def test_merge_annotation_history_preserves_old_and_replaces_changed():
|
||||
old_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "2"}}
|
||||
old_b = {"clipping_id": "b", "attachment_item_id": 1, "annotation": {"sortIndex": "1"}}
|
||||
new_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "3"}}
|
||||
|
||||
merged = _merge_annotation_history(
|
||||
{"all_annotations": [old_a, old_b]}, [new_a], []
|
||||
)
|
||||
|
||||
assert merged == [old_b, new_a]
|
||||
|
||||
|
||||
def test_merge_annotation_history_removes_deletions():
|
||||
previous = {
|
||||
"annotations": [
|
||||
{"clipping_id": "a", "attachment_item_id": 1, "annotation": {}},
|
||||
{"clipping_id": "b", "attachment_item_id": 1, "annotation": {}},
|
||||
]
|
||||
}
|
||||
|
||||
merged = _merge_annotation_history(previous, [], ["a"])
|
||||
|
||||
assert [annotation["clipping_id"] for annotation in merged] == ["b"]
|
||||
|
||||
|
||||
def test_full_history_retains_current_unresolved_annotation():
|
||||
previous = {
|
||||
"all_annotations": [
|
||||
{"clipping_id": "unresolved", "attachment_item_id": 1, "annotation": {}},
|
||||
{"clipping_id": "removed", "attachment_item_id": 1, "annotation": {}},
|
||||
]
|
||||
}
|
||||
|
||||
merged = _merge_annotation_history(previous, [], ["removed"])
|
||||
|
||||
assert [annotation["clipping_id"] for annotation in merged] == ["unresolved"]
|
||||
|
||||
|
||||
def test_deletions_include_removed_and_previous_failures():
|
||||
assert _deletion_ids(
|
||||
{"kept", "removed"},
|
||||
{"kept"},
|
||||
{"delete_failed_ids": ["retry"]},
|
||||
) == {"removed", "retry"}
|
||||
|
||||
|
||||
def test_deferred_clippings_are_reprocessed_while_they_remain_in_source():
|
||||
previous = {"deferred_clipping_ids": ["retry", "removed"]}
|
||||
|
||||
assert _deferred_clipping_ids(previous, {"retry", "current"}) == {"retry"}
|
||||
|
||||
|
||||
def test_discarded_clips_never_requeue_and_skips_are_not_unresolved():
|
||||
previous = {"discarded_clipping_ids": ["gone", "removed"]}
|
||||
assert _discarded_clipping_ids(previous, {"gone", "current"}) == {"gone"}
|
||||
|
||||
plan = {
|
||||
"items": [
|
||||
{"clipping": {"id": "note"}, "status": "pdf-position-skipped-non-highlight"},
|
||||
{"clipping": {"id": "fail"}, "status": "pdf-text-not-found"},
|
||||
{"clipping": {"id": "done"}, "status": "positioned"},
|
||||
]
|
||||
}
|
||||
assert _unresolved_clipping_ids(plan, {"note", "fail", "done"}) == {"fail"}
|
||||
|
||||
|
||||
def test_alternate_attachment_retry_prefers_pdf_when_epub_misses():
|
||||
from kindle_zotero_importer.cli import _reattempt_failed_with_alternate_attachments
|
||||
from kindle_zotero_importer.epub_position import add_epub_positions
|
||||
from kindle_zotero_importer.import_plan import build_import_plan
|
||||
from kindle_zotero_importer.pdf_position import add_pdf_positions
|
||||
|
||||
fixtures = Path(__file__).parent / "fixtures" / "windows-smoke"
|
||||
epub_path = str(fixtures / "kanzi-dummy.epub")
|
||||
pdf_path = str(fixtures / "kanzi-dummy.pdf")
|
||||
quote = "This is a test highlight long enough to import for PDF."
|
||||
clippings_payload = {
|
||||
"format": "kindle-zotero-importer.clippings.v1",
|
||||
"clippings": [
|
||||
{
|
||||
"id": "q1",
|
||||
"title": "Dummy Book",
|
||||
"kind": "highlight",
|
||||
"text": quote,
|
||||
"page": None,
|
||||
"location": "100",
|
||||
}
|
||||
],
|
||||
}
|
||||
zotero_index = {
|
||||
"format": "index",
|
||||
"items": [
|
||||
{
|
||||
"item_id": 1,
|
||||
"key": "ABC12345",
|
||||
"title": "Dummy Book",
|
||||
"fields": {},
|
||||
"attachments": [
|
||||
{
|
||||
"item_id": 11,
|
||||
"key": "EPUBKEY1",
|
||||
"title": "epub",
|
||||
"content_type": "application/epub+zip",
|
||||
"path": epub_path,
|
||||
"resolved_path": epub_path,
|
||||
},
|
||||
{
|
||||
"item_id": 12,
|
||||
"key": "PDFKEY123",
|
||||
"title": "pdf",
|
||||
"content_type": "application/pdf",
|
||||
"path": pdf_path,
|
||||
"resolved_path": pdf_path,
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
match_report = {
|
||||
"format": "matches",
|
||||
"matches": [
|
||||
{
|
||||
"clipping_title": "Dummy Book",
|
||||
"status": "matched",
|
||||
"candidates": [{"item_id": 1}],
|
||||
}
|
||||
],
|
||||
}
|
||||
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||
assert plan["items"][0]["zotero"]["attachment"]["content_type"] == "application/epub+zip"
|
||||
positioned = add_pdf_positions(add_epub_positions(plan))
|
||||
assert positioned["items"][0]["status"] == "positioned"
|
||||
assert "partial-text-match" in positioned["items"][0]["problems"]
|
||||
rescued = _reattempt_failed_with_alternate_attachments(positioned)
|
||||
assert rescued["items"][0]["status"] == "positioned"
|
||||
assert "positioned-via-alternate-attachment" in rescued["items"][0]["problems"]
|
||||
assert "partial-text-match" not in rescued["items"][0]["problems"]
|
||||
|
||||
|
||||
def test_changed_override_requeues_every_clipping_for_exact_title():
|
||||
previous = {"Shared title": {"citation_key": "old"}}
|
||||
current = {"Shared title": {"citation_key": "new"}}
|
||||
clippings = [
|
||||
{"id": "first", "title": "Shared title"},
|
||||
{"id": "second", "title": "Shared title"},
|
||||
{"id": "third", "title": "Different title"},
|
||||
]
|
||||
|
||||
changed_titles = _changed_override_titles(previous, current)
|
||||
|
||||
assert changed_titles == {"Shared title"}
|
||||
assert _clipping_ids_for_titles(clippings, changed_titles) == {"first", "second"}
|
||||
|
||||
|
||||
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
|
||||
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
|
||||
previous = {
|
||||
"all_annotations": [
|
||||
{"clipping_id": "a", "annotation": {"comment": "old"}}
|
||||
]
|
||||
}
|
||||
|
||||
_add_previous_imported_comments(current, previous)
|
||||
|
||||
assert current[0]["previous_imported_comment"] == "old"
|
||||
|
||||
|
||||
def test_retry_unresolved_excludes_positioned_ignored_and_removed_entries():
|
||||
previous = {
|
||||
"items": [
|
||||
{"clipping": {"id": "retry"}, "status": "pdf-text-not-found"},
|
||||
{"clipping": {"id": "done"}, "status": "positioned"},
|
||||
{"clipping": {"id": "ignored"}, "status": "ignored-title"},
|
||||
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
|
||||
]
|
||||
}
|
||||
|
||||
assert _unresolved_clipping_ids(previous, {"retry", "done", "ignored"}) == {
|
||||
"retry"
|
||||
}
|
||||
|
||||
|
||||
def test_changed_note_ids_detect_add_replace_and_remove():
|
||||
previous = [
|
||||
{"clipping_id": "added", "note_ids": []},
|
||||
{"clipping_id": "replaced", "note_ids": ["old"]},
|
||||
{"clipping_id": "removed", "note_ids": ["old"]},
|
||||
{"clipping_id": "same", "note_ids": ["same-note"]},
|
||||
]
|
||||
current = [
|
||||
{"id": "added", "note_ids": ["new"]},
|
||||
{"id": "replaced", "note_ids": ["new"]},
|
||||
{"id": "removed"},
|
||||
{"id": "same", "note_ids": ["same-note"]},
|
||||
]
|
||||
|
||||
assert _changed_note_clipping_ids(current, previous) == {
|
||||
"added",
|
||||
"replaced",
|
||||
"removed",
|
||||
}
|
||||
|
||||
|
||||
def test_positioned_history_preserves_unresolved_and_replaces_current():
|
||||
previous = {
|
||||
"items": [
|
||||
{"clipping": {"id": "keep"}, "status": "pdf-text-not-found"},
|
||||
{"clipping": {"id": "replace"}, "status": "epub-text-not-found"},
|
||||
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
|
||||
]
|
||||
}
|
||||
current = {
|
||||
"format": "plan",
|
||||
"items": [{"clipping": {"id": "replace"}, "status": "positioned"}],
|
||||
}
|
||||
|
||||
merged = _merge_positioned_history(previous, current, {"keep", "replace"})
|
||||
|
||||
assert {item["clipping"]["id"]: item["status"] for item in merged["items"]} == {
|
||||
"keep": "pdf-text-not-found",
|
||||
"replace": "positioned",
|
||||
}
|
||||
assert merged["status_counts"] == {"pdf-text-not-found": 1, "positioned": 1}
|
||||
|
||||
|
||||
def test_pdf_position_reports_missing_tools():
|
||||
original = pdf_position._missing_required_tools
|
||||
pdf_position._missing_required_tools = lambda: ["pdftotext", "pdfinfo"]
|
||||
try:
|
||||
result = pdf_position.add_pdf_positions(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"status": "ready-for-positioning",
|
||||
"clipping": {"kind": "highlight", "text": "highlight"},
|
||||
"zotero": {
|
||||
"attachment": {"content_type": "application/pdf"}
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
finally:
|
||||
pdf_position._missing_required_tools = original
|
||||
|
||||
assert result["items"][0]["status"] == "pdf-tools-missing:pdftotext,pdfinfo"
|
||||
|
||||
|
||||
def test_pdf_position_preserves_unresolved_item_without_zotero_match():
|
||||
item = {
|
||||
"status": "unmatched-title",
|
||||
"clipping": {"kind": "highlight", "text": "highlight"},
|
||||
"zotero": None,
|
||||
}
|
||||
|
||||
result = pdf_position.add_pdf_positions({"items": [item]})
|
||||
|
||||
assert result["items"] == [item]
|
||||
|
||||
|
||||
def test_noop_incremental_writes_complete_artifacts():
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
workdir = Path(temporary)
|
||||
source = workdir / "My Clippings.txt"
|
||||
source.write_text(
|
||||
"Book\n- Your Highlight at location 10-11 | Added on Tuesday, 18 May 2021 14:18:13\n\n"
|
||||
"A highlight long enough to import\n==========\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
clippings = clippings_to_jsonable(load_clippings(str(source)))
|
||||
clipping_id = clippings["clippings"][0]["id"]
|
||||
(workdir / "clippings.json").write_text(json.dumps(clippings), encoding="utf-8")
|
||||
annotation = {
|
||||
"clipping_id": clipping_id,
|
||||
"attachment_item_id": 1,
|
||||
"annotation": {"sortIndex": "1"},
|
||||
}
|
||||
(workdir / "import-plan.final.json").write_text(
|
||||
json.dumps({"annotations": [annotation], "all_annotations": [annotation], "overrides_snapshot": {}}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
originals = (
|
||||
cli.build_zotero_index,
|
||||
cli.build_match_report,
|
||||
cli.generate_override_skeleton,
|
||||
cli.build_mismatch_review,
|
||||
)
|
||||
cli.build_zotero_index = lambda *_args: {"format": "index", "items": []}
|
||||
cli.build_match_report = lambda *_args: {"format": "matches", "matches": []}
|
||||
cli.generate_override_skeleton = lambda *_args: {"format": "overrides", "overrides": []}
|
||||
cli.build_mismatch_review = lambda *_args: "review\n"
|
||||
try:
|
||||
result = cli.main(
|
||||
[
|
||||
"run",
|
||||
str(source),
|
||||
"--workdir",
|
||||
str(workdir),
|
||||
"--db",
|
||||
str(workdir / "zotero.sqlite"),
|
||||
"--storage-root",
|
||||
str(workdir / "storage"),
|
||||
]
|
||||
)
|
||||
finally:
|
||||
(
|
||||
cli.build_zotero_index,
|
||||
cli.build_match_report,
|
||||
cli.generate_override_skeleton,
|
||||
cli.build_mismatch_review,
|
||||
) = originals
|
||||
|
||||
assert result == 0
|
||||
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
|
||||
assert final_plan["annotations"] == []
|
||||
assert final_plan["all_annotations"] == [{**annotation, "related_items": []}]
|
||||
assert final_plan["delta_annotation_count"] == 0
|
||||
assert (workdir / "import-plan.epub.json").exists()
|
||||
|
|
|
|||
|
|
@ -1,14 +1,711 @@
|
|||
import assert from 'assert';
|
||||
// test the grouping logic for title variants sharing same candidate
|
||||
function findOtherRows(rows, rowIndex, targetKey) {
|
||||
const row = rows[rowIndex];
|
||||
return rows.filter((r, idx) => idx !== rowIndex && (r.candidates||[]).some(c => (c.citation_key||c.key||String(c.item_id))===targetKey));
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const vm = require('vm');
|
||||
|
||||
const bootstrapContext = {};
|
||||
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
|
||||
const plugin = bootstrapContext.Kanzi;
|
||||
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
|
||||
|
||||
assert.equal(
|
||||
plugin.mergeComments('[r] this is a red note\n\nthis is a red note', ''),
|
||||
'this is a red note'
|
||||
);
|
||||
assert.equal(plugin.mergeComments('[o]', ''), '');
|
||||
assert.equal(plugin.mergeComments('[citation] keep this', ''), '[citation] keep this');
|
||||
assert.equal(
|
||||
plugin.commentForExistingAnnotation(
|
||||
{ annotationComment: 'old note', annotationTags: [{ name: 'kindle-import' }] },
|
||||
'replacement note',
|
||||
'old note'
|
||||
),
|
||||
'replacement note'
|
||||
);
|
||||
assert.equal(
|
||||
plugin.commentForExistingAnnotation(
|
||||
{ annotationComment: 'old note\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
|
||||
'replacement note',
|
||||
'old note'
|
||||
),
|
||||
'manual addition\n\nreplacement note'
|
||||
);
|
||||
assert.equal(
|
||||
plugin.commentForExistingAnnotation(
|
||||
{ annotationComment: 'old first\n\nold second\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
|
||||
'new note',
|
||||
'old first\n\nold second'
|
||||
),
|
||||
'manual addition\n\nnew note'
|
||||
);
|
||||
assert.equal(
|
||||
plugin.commentForExistingAnnotation(
|
||||
{ annotationComment: 'manual note', annotationTags: [] },
|
||||
'imported note'
|
||||
),
|
||||
'manual note\n\nimported note'
|
||||
);
|
||||
|
||||
const base = {
|
||||
annotationText: 'forged its might, should be enough to discredit',
|
||||
annotationPosition: JSON.stringify({
|
||||
type: 'FragmentSelector',
|
||||
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1255)',
|
||||
}),
|
||||
};
|
||||
const extended = {
|
||||
annotationText: 'forged its might, should be enough to discredit, even in the eyes of philosophers',
|
||||
annotationPosition: JSON.stringify({
|
||||
type: 'FragmentSelector',
|
||||
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1300)',
|
||||
}),
|
||||
};
|
||||
assert.equal(plugin.areImportedDuplicates(base, extended), true);
|
||||
assert.equal(
|
||||
plugin.areImportedDuplicates(base, {
|
||||
...base,
|
||||
annotationPosition: JSON.stringify({
|
||||
type: 'FragmentSelector',
|
||||
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
|
||||
}),
|
||||
}),
|
||||
false
|
||||
);
|
||||
assert.equal(
|
||||
plugin.areImportedDuplicates(base, {
|
||||
...base,
|
||||
annotationText: 'OCR-corrected text at the identical annotation position',
|
||||
}),
|
||||
true
|
||||
);
|
||||
assert.equal(
|
||||
plugin.areImportedDuplicates(
|
||||
{
|
||||
annotationText: 'A multiline PDF selection long enough to compare',
|
||||
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30]] }),
|
||||
},
|
||||
{
|
||||
annotationText: 'A multiline PDF selection long enough to compare with an extended ending',
|
||||
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30], [10, 40, 120, 50]] }),
|
||||
}
|
||||
),
|
||||
true
|
||||
);
|
||||
|
||||
const managerHtml = fs.readFileSync('plugin/manager.html', 'utf8');
|
||||
const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
|
||||
const managerContext = { TextEncoder };
|
||||
vm.runInNewContext(scripts[0][1], managerContext);
|
||||
const manager = managerContext.KindleManager;
|
||||
assert.equal(plugin.countClippingsForTitle({ clippings: [
|
||||
{ title: 'Shared title' },
|
||||
{ title: 'Shared title' },
|
||||
{ title: 'Different title' },
|
||||
] }, 'Shared title'), 2);
|
||||
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||
{ id: 'n1', title: 'Shared title', kind: 'note', text: 'note' },
|
||||
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||
] },
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Shared title',
|
||||
clipping_count: 3,
|
||||
status: 'matched',
|
||||
candidates: [],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [{
|
||||
clipping: { id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||
status: 'pdf-text-not-found',
|
||||
problems: ['not found'],
|
||||
}] },
|
||||
finalPlan: { all_annotations: [] },
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const conflictRows = manager.conflictRows();
|
||||
assert.equal(conflictRows.length, 1);
|
||||
assert.equal(conflictRows[0].count, 3);
|
||||
assert.equal(conflictRows[0].clippings.length, 3);
|
||||
assert.equal(conflictRows[0].highlights, 2);
|
||||
assert.equal(conflictRows[0].notes, 1);
|
||||
|
||||
manager.sortState.conflicts = { column: 'count', direction: 'desc' };
|
||||
const sortedRows = manager.sortRows(
|
||||
[{ title: 'small', count: 2 }, { title: 'large', count: 20 }, { title: 'middle', count: 10 }],
|
||||
'conflicts',
|
||||
{ count: row => row.count }
|
||||
);
|
||||
assert.deepEqual(Array.from(sortedRows, row => row.title), ['large', 'middle', 'small']);
|
||||
manager.integratedPage = 1;
|
||||
manager.integratedPageSize = 50;
|
||||
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
|
||||
assert.equal(page.totalPages, 3);
|
||||
assert.deepEqual(Array.from(page.rows), Array.from({ length: 50 }, (_, index) => index + 50));
|
||||
|
||||
manager.mappingsPage = 99;
|
||||
manager.mappingsPageSize = 50;
|
||||
const finalPage = manager.paginate(Array.from({ length: 51 }, (_, index) => index), 'mappingsPage', 'mappingsPageSize');
|
||||
assert.equal(manager.mappingsPage, 1);
|
||||
assert.deepEqual(Array.from(finalPage.rows), [50]);
|
||||
|
||||
const collectorBatches = manager.collectorBatches('installation', Array.from(
|
||||
{ length: 450 },
|
||||
(_, index) => ({ clipping_id_hash: String(index), text: 'x'.repeat(2000) })
|
||||
));
|
||||
assert.equal(collectorBatches.length, 3);
|
||||
assert.ok(collectorBatches.every(batch => batch.length <= 200));
|
||||
assert.ok(collectorBatches.every(batch => new TextEncoder().encode(
|
||||
JSON.stringify({ anonId: 'installation', data: batch })
|
||||
).length <= 1500000));
|
||||
assert.throws(
|
||||
() => manager.collectorBatches('installation', [{ text: 'x'.repeat(1500000) }]),
|
||||
/exceeds the collector request limit/
|
||||
);
|
||||
|
||||
async function testRuntimeExtraction() {
|
||||
const writes = [];
|
||||
bootstrapContext.PathUtils = {
|
||||
profileDir: '/profile',
|
||||
join: (...parts) => parts.join('/').replaceAll('//', '/'),
|
||||
parent: path => path.slice(0, path.lastIndexOf('/')),
|
||||
};
|
||||
bootstrapContext.IOUtils = {
|
||||
makeDirectory: async () => {},
|
||||
};
|
||||
bootstrapContext.Zotero = {
|
||||
File: { putContentsAsync: async (path, text) => writes.push([path, text]) },
|
||||
};
|
||||
bootstrapContext.fetch = async url => ({ ok: true, text: async () => url });
|
||||
plugin.rootURI = 'jar:file:///kanzi.xpi!/';
|
||||
await plugin.ensureRuntime();
|
||||
assert.equal(plugin.runtimeProjectDir, '/profile/kanzi-runtime');
|
||||
assert.equal(writes.length, 13);
|
||||
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
|
||||
}
|
||||
const rows = [
|
||||
{title:"gilbert-simondon-on-the-mode", candidates:[{citation_key:"chabot2013", key:"A"}, {citation_key:"simondon2017a"}]},
|
||||
{title:"On the Mode (Univocal)", candidates:[{citation_key:"chabot2013"}, {citation_key:"simondon2017a"}]},
|
||||
{title:"Other", candidates:[{citation_key:"foo"}]}
|
||||
];
|
||||
assert.equal(findOtherRows(rows, 0, "chabot2013").length, 1);
|
||||
assert.equal(findOtherRows(rows, 0, "chabot2013")[0].title, "On the Mode (Univocal)");
|
||||
console.log("manager logic ok");
|
||||
|
||||
async function testPythonPathRecovery() {
|
||||
bootstrapContext.Services = { appinfo: { OS: 'Darwin' } };
|
||||
bootstrapContext.IOUtils = {
|
||||
exists: async path => path === '/opt/homebrew/opt/python@3.13/bin/python3.13',
|
||||
};
|
||||
assert.equal(
|
||||
await plugin.resolvePythonPath('/opt/homebrew/bin/python3'),
|
||||
'/opt/homebrew/opt/python@3.13/bin/python3.13'
|
||||
);
|
||||
bootstrapContext.IOUtils.exists = async path => path === '/custom/python3';
|
||||
assert.equal(await plugin.resolvePythonPath('/custom/python3'), '/custom/python3');
|
||||
|
||||
bootstrapContext.Services.appinfo.OS = 'WINNT';
|
||||
const windowsEnvironment = {
|
||||
LOCALAPPDATA: 'C:\\Users\\test\\AppData\\Local',
|
||||
ProgramFiles: 'C:\\Program Files',
|
||||
'ProgramFiles(x86)': 'C:\\Program Files (x86)',
|
||||
USERPROFILE: 'C:\\Users\\test',
|
||||
SystemDrive: 'C:',
|
||||
PATH: 'C:\\Tools\\Python;C:\\Windows\\System32',
|
||||
};
|
||||
plugin.getEnvironmentVariable = name => windowsEnvironment[name] || '';
|
||||
const expected = 'C:\\Users\\test\\AppData\\Local\\Programs\\Python\\Python313\\python.exe';
|
||||
bootstrapContext.IOUtils.exists = async path => path === expected;
|
||||
assert.equal(await plugin.resolvePythonPath('python'), expected);
|
||||
assert.ok(plugin.getWindowsPythonCandidates('python').includes('C:\\Tools\\Python\\python.exe'));
|
||||
}
|
||||
|
||||
async function testMaintenanceSweep() {
|
||||
const annotation = (id, text, end, comment = '') => ({
|
||||
id,
|
||||
parentItemID: 10,
|
||||
annotationText: text,
|
||||
annotationComment: comment,
|
||||
annotationPosition: JSON.stringify({
|
||||
type: 'FragmentSelector',
|
||||
value: `epubcfi(/6/12!/4/2/8,/1:795,/1:${end})`,
|
||||
}),
|
||||
saved: 0,
|
||||
erased: 0,
|
||||
saveTx: async function () { this.saved += 1; },
|
||||
eraseTx: async function () { this.erased += 1; },
|
||||
});
|
||||
const short = annotation(1, 'A progressive highlight long enough to compare', 994);
|
||||
const middle = annotation(2, 'A progressive highlight long enough to compare and then extended', 1093);
|
||||
const longest = annotation(
|
||||
3,
|
||||
'A progressive highlight long enough to compare and then extended to its final form',
|
||||
1141,
|
||||
'[r] useful note\n\nuseful note'
|
||||
);
|
||||
const separate = annotation(4, 'A progressive highlight long enough to compare', 2050);
|
||||
separate.annotationPosition = JSON.stringify({
|
||||
type: 'FragmentSelector',
|
||||
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
|
||||
});
|
||||
const emptyComment = annotation(5, 'A separate highlight with a null comment', 3050, null);
|
||||
emptyComment.annotationPosition = JSON.stringify({
|
||||
type: 'FragmentSelector',
|
||||
value: 'epubcfi(/6/12!/4/2/8,/1:3000,/1:3050)',
|
||||
});
|
||||
const results = { updatedComments: 0, removedDuplicates: 0 };
|
||||
|
||||
await plugin.maintainImportedAnnotations([short, middle, longest, separate, emptyComment], false, results);
|
||||
|
||||
assert.equal(results.removedDuplicates, 2);
|
||||
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
|
||||
assert.equal(short.erased, 1);
|
||||
assert.equal(middle.erased, 1);
|
||||
assert.equal(longest.erased, 0);
|
||||
assert.equal(separate.erased, 0);
|
||||
assert.equal(emptyComment.saved, 0);
|
||||
assert.equal(longest.annotationComment, 'useful note');
|
||||
assert.equal(results.updatedComments, 1);
|
||||
}
|
||||
|
||||
async function testWriterReconciliation() {
|
||||
let written = null;
|
||||
bootstrapContext.Zotero = {
|
||||
File: { putContentsAsync: async (_path, text) => { written = JSON.parse(text); } },
|
||||
};
|
||||
const plan = {
|
||||
annotations: [{ clipping_id: 'ok' }, { clipping_id: 'failed' }],
|
||||
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'ok' }, { clipping_id: 'failed' }],
|
||||
};
|
||||
await plugin.reconcileWriterState('/plan.json', plan, {
|
||||
failed: [{ clipping_id: 'failed' }, { clipping_id: 'maintenance' }],
|
||||
failedDeletionIds: ['delete-me'],
|
||||
maintenanceDeletedIds: ['old'],
|
||||
});
|
||||
assert.deepEqual(Array.from(written.write_failed_ids), ['failed']);
|
||||
assert.deepEqual(Array.from(written.annotations, entry => entry.clipping_id), ['ok']);
|
||||
assert.deepEqual(Array.from(written.all_annotations, entry => entry.clipping_id), ['ok']);
|
||||
assert.equal(written.annotation_count, 1);
|
||||
assert.deepEqual(Array.from(written.delete_failed_ids), ['delete-me']);
|
||||
}
|
||||
|
||||
async function testDeletionFailureIsRetried() {
|
||||
bootstrapContext.Zotero = { logError: () => {} };
|
||||
const annotation = {
|
||||
annotationTags: [{ name: 'kindle-id:delete-me' }],
|
||||
eraseTx: async () => { throw new Error('locked'); },
|
||||
};
|
||||
plugin.collectImportedAnnotations = async () => [annotation];
|
||||
plugin.maintainImportedAnnotations = async () => {};
|
||||
const results = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [],
|
||||
deletions: ['delete-me'],
|
||||
}, false);
|
||||
assert.deepEqual(Array.from(results.failedDeletionIds), ['delete-me']);
|
||||
assert.equal(results.failed.length, 1);
|
||||
}
|
||||
|
||||
async function testMaintenanceDeletionIsNotRepeated() {
|
||||
bootstrapContext.Zotero = { logError: () => {} };
|
||||
const annotation = {
|
||||
annotationTags: [{ name: 'kindle-id:already-deleted' }],
|
||||
eraseTx: async () => { throw new Error('must not be called again'); },
|
||||
};
|
||||
plugin.collectImportedAnnotations = async () => [annotation];
|
||||
plugin.maintainImportedAnnotations = async (_annotations, _dryRun, results) => {
|
||||
results.maintenanceDeletedIds.push('already-deleted');
|
||||
};
|
||||
const results = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [],
|
||||
deletions: ['already-deleted'],
|
||||
}, false);
|
||||
assert.deepEqual(Array.from(results.failedDeletionIds), []);
|
||||
assert.equal(results.failed.length, 0);
|
||||
}
|
||||
|
||||
async function testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor() {
|
||||
bootstrapContext.Zotero = { logError: () => {} };
|
||||
const active = {
|
||||
parentItemID: 10,
|
||||
annotationText: 'The longer OCR-corrected duplicate annotation text',
|
||||
annotationComment: '',
|
||||
annotationPosition: JSON.stringify({ pageIndex: 1, rects: [[1, 2, 3, 4]] }),
|
||||
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
|
||||
erased: 0,
|
||||
eraseTx: async function () { this.erased += 1; },
|
||||
};
|
||||
const duplicate = {
|
||||
...active,
|
||||
annotationText: 'The duplicate annotation text',
|
||||
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
|
||||
erased: 0,
|
||||
eraseTx: async function () { this.erased += 1; },
|
||||
};
|
||||
let collection = 0;
|
||||
plugin.collectImportedAnnotations = async () => (++collection === 1 ? [active, duplicate] : [active]);
|
||||
plugin.maintainImportedAnnotations = maintainImportedAnnotations;
|
||||
const results = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [],
|
||||
deletions: ['same'],
|
||||
}, false);
|
||||
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
|
||||
assert.equal(duplicate.erased, 1);
|
||||
assert.equal(active.erased, 1);
|
||||
assert.equal(results.deletedForIncremental, 1);
|
||||
assert.equal(results.failed.length, 0);
|
||||
}
|
||||
|
||||
async function testEnumerationFailurePersistsAllDeletions() {
|
||||
bootstrapContext.Zotero = { logError: () => {} };
|
||||
plugin.collectImportedAnnotations = async () => { throw new Error('enumeration failed'); };
|
||||
const results = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [],
|
||||
deletions: ['retry-deletion'],
|
||||
}, false);
|
||||
assert.deepEqual(Array.from(results.failedDeletionIds), ['retry-deletion']);
|
||||
assert.equal(results.failed[0].clipping_id, 'maintenance');
|
||||
}
|
||||
|
||||
async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
|
||||
const old = {
|
||||
annotationTags: [{ name: 'kindle-id:replacement' }],
|
||||
erased: 0,
|
||||
eraseTx: async function () { this.erased += 1; },
|
||||
};
|
||||
bootstrapContext.Zotero = {
|
||||
logError: () => {},
|
||||
Items: { get: () => null },
|
||||
};
|
||||
plugin.collectImportedAnnotations = async () => [old];
|
||||
plugin.maintainImportedAnnotations = async () => {};
|
||||
const results = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [{ clipping_id: 'replacement', attachment_item_id: 99, annotation: {} }],
|
||||
deletions: ['replacement'],
|
||||
}, false);
|
||||
assert.equal(old.erased, 0);
|
||||
assert.deepEqual(Array.from(results.failedDeletionIds), ['replacement']);
|
||||
assert.equal(results.failed[0].clipping_id, 'replacement');
|
||||
}
|
||||
|
||||
async function testCitekeyRelationsAreBidirectionalAndAdditive() {
|
||||
const originalZotero = bootstrapContext.Zotero;
|
||||
const originalCollect = plugin.collectImportedAnnotations;
|
||||
const originalMaintain = plugin.maintainImportedAnnotations;
|
||||
const item = (id, key) => ({
|
||||
id,
|
||||
key,
|
||||
related: new Set(),
|
||||
saved: 0,
|
||||
addRelatedItem(other) {
|
||||
if (this.related.has(other.key)) return false;
|
||||
this.related.add(other.key);
|
||||
return true;
|
||||
},
|
||||
async save(options) {
|
||||
assert.equal(options.skipDateModifiedUpdate, true);
|
||||
this.saved += 1;
|
||||
},
|
||||
});
|
||||
const source = item(1, 'SOURCE01');
|
||||
const target = item(2, 'TARGET02');
|
||||
bootstrapContext.Zotero = {
|
||||
Items: { get: id => id === 1 ? source : id === 2 ? target : id === 11 ? attachment : null },
|
||||
DB: { executeTransaction: async callback => callback() },
|
||||
logError: () => {},
|
||||
};
|
||||
const entries = [
|
||||
{
|
||||
clipping_id: 'history-1',
|
||||
parent_item_id: 1,
|
||||
attachment_item_id: 11,
|
||||
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
|
||||
},
|
||||
{
|
||||
parent_item_id: 1,
|
||||
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
|
||||
},
|
||||
];
|
||||
const existingAnnotation = {
|
||||
annotationComment: 'an old note',
|
||||
annotationTags: [{ name: 'kindle-id:history-1' }],
|
||||
saved: 0,
|
||||
saveTx: async function () { this.saved += 1; },
|
||||
};
|
||||
const attachment = { getAnnotations: () => [existingAnnotation] };
|
||||
const results = {
|
||||
relationsAdded: 0,
|
||||
relationsSkipped: 0,
|
||||
relationsPlanned: 0,
|
||||
relationFailures: [],
|
||||
};
|
||||
try {
|
||||
plugin.collectImportedAnnotations = async () => [];
|
||||
plugin.maintainImportedAnnotations = async () => {};
|
||||
const writerResults = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [],
|
||||
all_annotations: entries,
|
||||
deletions: [],
|
||||
}, false);
|
||||
assert.deepEqual(Array.from(source.related), ['TARGET02']);
|
||||
assert.deepEqual(Array.from(target.related), ['SOURCE01']);
|
||||
assert.equal(source.saved, 1);
|
||||
assert.equal(target.saved, 1);
|
||||
assert.equal(writerResults.relationsAdded, 1);
|
||||
assert.equal(writerResults.relationFailures.length, 0);
|
||||
assert.equal(writerResults.commentLinksAdded, 1);
|
||||
assert.ok(existingAnnotation.annotationComment.includes(
|
||||
'Related: @target2020 zotero://select/library/items/TARGET02'
|
||||
));
|
||||
assert.equal(existingAnnotation.saved, 1);
|
||||
|
||||
await plugin.addRelatedItemRelations(entries, false, results);
|
||||
assert.equal(source.saved, 1);
|
||||
assert.equal(target.saved, 1);
|
||||
assert.equal(results.relationsSkipped, 1);
|
||||
} finally {
|
||||
bootstrapContext.Zotero = originalZotero;
|
||||
plugin.collectImportedAnnotations = originalCollect;
|
||||
plugin.maintainImportedAnnotations = originalMaintain;
|
||||
}
|
||||
}
|
||||
|
||||
function testPendingReviewGrouping() {
|
||||
const groups = manager.groupPendingAnnotations([
|
||||
{ clipping_id: 'a1', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'spice', pageLabel: '12', color: '#ffd400' } },
|
||||
{ clipping_id: 'a2', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'worm', pageLabel: '', color: '#ff6666' } },
|
||||
{ clipping_id: 'b1', clipping_title: 'Unknown Book', citation_key: '', annotation: { text: 'mystery', pageLabel: '', color: '#ffd400' } },
|
||||
]);
|
||||
assert.equal(groups.length, 2);
|
||||
assert.equal(groups[0].items.length, 2);
|
||||
assert.equal(groups[0].citationKey, 'herbert2020');
|
||||
assert.equal(groups[1].items.length, 1);
|
||||
assert.deepEqual(manager.groupPendingAnnotations([]), []);
|
||||
}
|
||||
|
||||
function testFilterPlanForSelection() {
|
||||
const plan = {
|
||||
annotations: [{ clipping_id: 'keep' }, { clipping_id: 'skip' }],
|
||||
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'keep' }, { clipping_id: 'skip' }],
|
||||
};
|
||||
const previous = { deferred_clipping_ids: ['older'] };
|
||||
const filtered = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep'], previous);
|
||||
assert.deepEqual(Array.from(filtered.annotations, a => a.clipping_id), ['keep']);
|
||||
assert.deepEqual(Array.from(filtered.all_annotations, a => a.clipping_id), ['old', 'keep']);
|
||||
assert.deepEqual(Array.from(filtered.deferred_clipping_ids), ['older', 'skip']);
|
||||
// Original plan is not mutated.
|
||||
assert.equal(plan.annotations.length, 2);
|
||||
const all = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep', 'skip'], previous);
|
||||
assert.equal(all.annotations.length, 2);
|
||||
assert.equal(all.all_annotations.length, 3);
|
||||
assert.deepEqual(Array.from(all.deferred_clipping_ids), ['older']);
|
||||
|
||||
const priorAnnotation = { clipping_id: 'skip', annotation: { text: 'existing' } };
|
||||
const preserved = plugin.filterPlanForSelection(
|
||||
plan,
|
||||
['keep', 'skip'],
|
||||
['keep'],
|
||||
{ all_annotations: [priorAnnotation] }
|
||||
);
|
||||
assert.equal(preserved.all_annotations.find(a => a.clipping_id === 'skip'), priorAnnotation);
|
||||
|
||||
const deferred = plugin.deferPendingPlan(
|
||||
{ all_annotations: [{ clipping_id: 'old' }], deferred_clipping_ids: ['earlier'] },
|
||||
plan,
|
||||
['keep', 'skip']
|
||||
);
|
||||
assert.deepEqual(Array.from(deferred.deferred_clipping_ids), ['earlier', 'keep', 'skip']);
|
||||
assert.deepEqual(Array.from(deferred.all_annotations, a => a.clipping_id), ['old']);
|
||||
}
|
||||
|
||||
function testTitleCountsAreAuthoritative() {
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||
{ id: 'h3', title: 'Shared title', kind: 'highlight', text: 'third' },
|
||||
] },
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Shared title',
|
||||
clipping_count: 99,
|
||||
status: 'ambiguous',
|
||||
candidates: [{ citation_key: 'shared2020', key: '', item_id: 1, title: 'Shared', score: 0.9 }],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [{
|
||||
clipping_title: 'Shared title',
|
||||
review: { status: 'review', clipping_count: 1 },
|
||||
candidates: [],
|
||||
}] },
|
||||
positionedPlan: { items: [{
|
||||
clipping: { id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||
status: 'pdf-text-not-found',
|
||||
problems: ['not found'],
|
||||
}] },
|
||||
finalPlan: { all_annotations: [] },
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const rows = manager.conflictRows();
|
||||
assert.equal(rows.length, 1);
|
||||
// Source has 3 clippings; misleading match (99) and override (1) counts must not win.
|
||||
assert.equal(rows[0].count, 3);
|
||||
assert.equal(rows[0].clippings.length, 3);
|
||||
// Only 1 of 3 has position problems.
|
||||
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 3 });
|
||||
|
||||
// Without clippings.json, fall back to conflict counts.
|
||||
manager.data = {
|
||||
clippings: null,
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Lonely title',
|
||||
clipping_count: 5,
|
||||
status: 'ambiguous',
|
||||
candidates: [],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [] },
|
||||
finalPlan: { all_annotations: [] },
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const fallback = manager.conflictRows();
|
||||
assert.equal(fallback.length, 1);
|
||||
assert.equal(fallback[0].count, 5);
|
||||
}
|
||||
|
||||
function testMappedTargetAndDiscardedClips() {
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' },
|
||||
{ id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' },
|
||||
] },
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Mapped title',
|
||||
clipping_count: 2,
|
||||
status: 'matched',
|
||||
candidates: [{ citation_key: 'mapped2020', key: '', item_id: 7, title: 'Mapped', score: 1 }],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [
|
||||
{ clipping: { id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' }, status: 'pdf-text-not-found', problems: ['pdf-text-not-found'] },
|
||||
{ clipping: { id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' }, status: 'pdf-position-skipped-non-highlight', problems: ['pdf-position-skipped-non-highlight'] },
|
||||
] },
|
||||
finalPlan: { all_annotations: [], discarded_clipping_ids: [] },
|
||||
persistentOverrides: { overrides: [{ clipping_title: 'Mapped title', resolution: { citation_key: 'mapped2020' } }] },
|
||||
};
|
||||
let rows = manager.conflictRows();
|
||||
assert.equal(rows.length, 1);
|
||||
assert.equal(rows[0].mappedTarget, 'mapped2020');
|
||||
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 2, total: 2 });
|
||||
|
||||
manager.data.finalPlan.discarded_clipping_ids = ['n1'];
|
||||
rows = manager.conflictRows();
|
||||
assert.equal(rows.length, 1);
|
||||
assert.equal(rows[0].count, 1);
|
||||
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 1 });
|
||||
}
|
||||
|
||||
async function testDiscardClipPersists() {
|
||||
const originalZotero = bootstrapContext.Zotero;
|
||||
let written = null;
|
||||
const plan = { discarded_clipping_ids: [], deferred_clipping_ids: ['x1'] };
|
||||
bootstrapContext.Zotero = {
|
||||
File: {
|
||||
getContentsAsync: async () => JSON.stringify(plan),
|
||||
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
|
||||
},
|
||||
};
|
||||
const originalArtifactPath = plugin.artifactPath;
|
||||
plugin.artifactPath = rel => `/proj/${rel}`;
|
||||
try {
|
||||
await plugin.discardClip('x1');
|
||||
assert.deepEqual(written.discarded_clipping_ids, ['x1']);
|
||||
assert.deepEqual(written.deferred_clipping_ids, []);
|
||||
} finally {
|
||||
plugin.artifactPath = originalArtifactPath;
|
||||
bootstrapContext.Zotero = originalZotero;
|
||||
}
|
||||
}
|
||||
|
||||
function testTitleOverviewAndSorting() {
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'a1', title: 'Alpha', kind: 'highlight', text: 'one' },
|
||||
{ id: 'a2', title: 'Alpha', kind: 'highlight', text: 'two' },
|
||||
{ id: 'b1', title: 'Beta', kind: 'highlight', text: 'three' },
|
||||
] },
|
||||
matches: { matches: [
|
||||
{ clipping_title: 'Alpha', clipping_count: 2, status: 'matched', candidates: [{ citation_key: 'alpha2020' }] },
|
||||
{ clipping_title: 'Beta', clipping_count: 1, status: 'unmatched', candidates: [] },
|
||||
] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [] },
|
||||
finalPlan: {
|
||||
all_annotations: [{ clipping_id: 'a1', clipping_title: 'Alpha' }],
|
||||
deferred_clipping_ids: ['b1'],
|
||||
},
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const overview = manager.titleOverviewRows();
|
||||
assert.equal(overview.length, 2);
|
||||
const alpha = overview.find(r => r.title === 'Alpha');
|
||||
const beta = overview.find(r => r.title === 'Beta');
|
||||
assert.equal(alpha.integrated, 1);
|
||||
assert.equal(beta.pending, 1);
|
||||
assert.equal(beta.status, 'pending review');
|
||||
|
||||
manager.sortState.titles = { column: 'title', direction: 'desc' };
|
||||
const sorted = manager.sortRows(overview, 'titles', { title: row => row.title });
|
||||
assert.deepEqual(Array.from(sorted, row => row.title), ['Beta', 'Alpha']);
|
||||
}
|
||||
|
||||
async function testSaveOverrideUsesSourceCount() {
|
||||
const originalRead = plugin.readJsonArtifact;
|
||||
const originalArtifactPath = plugin.artifactPath;
|
||||
const originalZotero = bootstrapContext.Zotero;
|
||||
let written = null;
|
||||
plugin.artifactPath = rel => `/proj/${rel}`;
|
||||
plugin.readJsonArtifact = async rel => rel === 'clippings.json'
|
||||
? { clippings: [{ title: 'Shared title' }, { title: 'Shared title' }, { title: 'Shared title' }] }
|
||||
: null;
|
||||
bootstrapContext.Zotero = {
|
||||
File: {
|
||||
getContentsAsync: async () => JSON.stringify({ format: 'kindle-zotero-importer.match-overrides.v1', overrides: [] }),
|
||||
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
|
||||
},
|
||||
};
|
||||
try {
|
||||
const entry = await plugin.saveOverride({
|
||||
clipping_title: 'Shared title',
|
||||
clipping_count: 1,
|
||||
resolution: { citation_key: 'shared2020' },
|
||||
review: { status: 'pdf-text-not-found', clipping_count: 1 },
|
||||
});
|
||||
assert.equal(entry.review.clipping_count, 3);
|
||||
assert.equal(written.overrides[0].review.clipping_count, 3);
|
||||
} finally {
|
||||
plugin.readJsonArtifact = originalRead;
|
||||
plugin.artifactPath = originalArtifactPath;
|
||||
bootstrapContext.Zotero = originalZotero;
|
||||
}
|
||||
}
|
||||
|
||||
testTitleCountsAreAuthoritative();
|
||||
testTitleOverviewAndSorting();
|
||||
testMappedTargetAndDiscardedClips();
|
||||
testPendingReviewGrouping();
|
||||
testFilterPlanForSelection();
|
||||
|
||||
testRuntimeExtraction()
|
||||
.then(testPythonPathRecovery)
|
||||
.then(testSaveOverrideUsesSourceCount)
|
||||
.then(testDiscardClipPersists)
|
||||
.then(testMaintenanceSweep)
|
||||
.then(testWriterReconciliation)
|
||||
.then(testDeletionFailureIsRetried)
|
||||
.then(testMaintenanceDeletionIsNotRepeated)
|
||||
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
|
||||
.then(testEnumerationFailurePersistsAllDeletions)
|
||||
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
|
||||
.then(testCitekeyRelationsAreBidirectionalAndAdditive)
|
||||
.then(() => {
|
||||
console.log('manager and annotation logic ok');
|
||||
}).catch(error => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
|
|
|
|||
63
updates.json
63
updates.json
|
|
@ -2,6 +2,69 @@
|
|||
"addons": {
|
||||
"kindlezoteroimporter@utkubilen.de": {
|
||||
"updates": [
|
||||
{
|
||||
"version": "0.7.0",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.7.0/kanzi.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.20",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.6.20/kanzi.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.19",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.19/kindle-zotero-importer.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.18",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.18/kindle-zotero-importer.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.17",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.17/kindle-zotero-importer.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.16",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.16/kindle-zotero-importer.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.15",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.15/kindle-zotero-importer.xpi",
|
||||
"applications": {
|
||||
"zotero": {
|
||||
"strict_min_version": "6.999"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "0.6.14",
|
||||
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.14/kindle-zotero-importer.xpi",
|
||||
|
|
|
|||
156
zotero-writer.js
156
zotero-writer.js
|
|
@ -1,156 +0,0 @@
|
|||
// Run in Zotero via Tools > Developer > Run JavaScript.
|
||||
// Set PLAN_PATH to the generated import-plan.final.json path.
|
||||
// Keep DRY_RUN=true until the preview output looks correct.
|
||||
|
||||
const PLAN_PATH = "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer/import-plan.final.json";
|
||||
const DRY_RUN = false;
|
||||
|
||||
const text = await Zotero.File.getContentsAsync(PLAN_PATH);
|
||||
const plan = JSON.parse(text);
|
||||
|
||||
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
|
||||
throw new Error(`Unsupported plan format: ${plan.format}`);
|
||||
}
|
||||
|
||||
const results = {
|
||||
dryRun: DRY_RUN,
|
||||
total: plan.annotations.length,
|
||||
created: 0,
|
||||
skippedExisting: 0,
|
||||
updatedComments: 0,
|
||||
removedDuplicates: 0,
|
||||
failed: [],
|
||||
preview: [],
|
||||
};
|
||||
|
||||
const existingByAttachment = new Map();
|
||||
|
||||
function normalizePosition(position) {
|
||||
if (!position) {
|
||||
return "";
|
||||
}
|
||||
if (typeof position === "string") {
|
||||
try {
|
||||
return JSON.stringify(JSON.parse(position));
|
||||
} catch (_error) {
|
||||
return position;
|
||||
}
|
||||
}
|
||||
return JSON.stringify(position);
|
||||
}
|
||||
|
||||
function annotationFingerprint(annotation) {
|
||||
return `${annotation.text || ""}\u0000${normalizePosition(annotation.position)}`;
|
||||
}
|
||||
|
||||
function existingAnnotationFingerprint(annotation) {
|
||||
return `${annotation.annotationText || ""}\u0000${normalizePosition(annotation.annotationPosition)}`;
|
||||
}
|
||||
|
||||
async function getExistingAnnotations(attachment) {
|
||||
if (!existingByAttachment.has(attachment.id)) {
|
||||
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
|
||||
const byFingerprint = new Map();
|
||||
|
||||
for (const annotation of annotations) {
|
||||
const fingerprint = existingAnnotationFingerprint(annotation);
|
||||
const kept = byFingerprint.get(fingerprint);
|
||||
if (!kept) {
|
||||
byFingerprint.set(fingerprint, annotation);
|
||||
continue;
|
||||
}
|
||||
|
||||
const keptHasComment = Boolean(kept.annotationComment);
|
||||
const currentHasComment = Boolean(annotation.annotationComment);
|
||||
const duplicate = keptHasComment || !currentHasComment ? annotation : kept;
|
||||
const replacement = duplicate === annotation ? kept : annotation;
|
||||
|
||||
byFingerprint.set(fingerprint, replacement);
|
||||
if (!DRY_RUN) {
|
||||
await duplicate.eraseTx();
|
||||
}
|
||||
results.removedDuplicates += 1;
|
||||
}
|
||||
|
||||
existingByAttachment.set(attachment.id, byFingerprint);
|
||||
}
|
||||
return existingByAttachment.get(attachment.id);
|
||||
}
|
||||
|
||||
function mergeComments(existingComment, newComment) {
|
||||
const seen = new Set();
|
||||
const merged = [];
|
||||
for (const comment of [existingComment, newComment]) {
|
||||
for (const part of String(comment || "").split(/\n\s*\n/)) {
|
||||
const cleaned = part.trim();
|
||||
if (!cleaned || seen.has(cleaned)) {
|
||||
continue;
|
||||
}
|
||||
seen.add(cleaned);
|
||||
merged.push(cleaned);
|
||||
}
|
||||
}
|
||||
return merged.join("\n\n");
|
||||
}
|
||||
|
||||
for (const entry of plan.annotations) {
|
||||
try {
|
||||
const attachment = Zotero.Items.get(entry.attachment_item_id);
|
||||
if (!attachment) {
|
||||
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
|
||||
}
|
||||
|
||||
const annotation = {
|
||||
key: Zotero.DataObjectUtilities.generateKey(),
|
||||
...entry.annotation,
|
||||
};
|
||||
if (!annotation.sortIndex) {
|
||||
delete annotation.sortIndex;
|
||||
}
|
||||
|
||||
const existingAnnotations = await getExistingAnnotations(attachment);
|
||||
const fingerprint = annotationFingerprint(annotation);
|
||||
const existingAnnotation = existingAnnotations.get(fingerprint);
|
||||
if (existingAnnotation) {
|
||||
const mergedComment = mergeComments(existingAnnotation.annotationComment, annotation.comment);
|
||||
if (mergedComment && existingAnnotation.saveTx && existingAnnotation.annotationComment !== mergedComment) {
|
||||
existingAnnotation.annotationComment = mergedComment;
|
||||
if (!DRY_RUN) {
|
||||
await existingAnnotation.saveTx();
|
||||
}
|
||||
results.updatedComments += 1;
|
||||
}
|
||||
results.skippedExisting += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (DRY_RUN) {
|
||||
results.preview.push({
|
||||
clipping_id: entry.clipping_id,
|
||||
attachment_item_id: entry.attachment_item_id,
|
||||
type: annotation.type,
|
||||
text: annotation.text.slice(0, 120),
|
||||
pageLabel: annotation.pageLabel,
|
||||
sortIndex: annotation.sortIndex,
|
||||
position: annotation.position,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
await Zotero.Annotations.saveFromJSON(attachment, annotation);
|
||||
existingAnnotations.set(fingerprint, {
|
||||
annotationComment: annotation.comment || "",
|
||||
annotationPosition: annotation.position,
|
||||
annotationText: annotation.text || "",
|
||||
});
|
||||
results.created += 1;
|
||||
} catch (error) {
|
||||
results.failed.push({
|
||||
clipping_id: entry.clipping_id,
|
||||
attachment_item_id: entry.attachment_item_id,
|
||||
message: String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return JSON.stringify(results, null, 2);
|
||||
Loading…
Add table
Reference in a new issue