Compare commits

...

13 commits

Author SHA1 Message Date
Utku Bilen Demir
aa3fc5ebc5 Kanzi 0.7.1: safer imports and resilient positioning 2026-10-15 14:01:05 +02:00
Utku Bilen Demir
495f50989d docs: use inline code for K-an-Z-I in heading (visible without italic) 2026-09-25 10:13:39 +02:00
Utku Bilen Demir
ff0d0c572c docs: show K-an-Z-I in heading via italic (visible inside bold heading) 2026-09-25 10:11:48 +02:00
Utku Bilen Demir
608098d378 docs: make K-an-Z-I visible outside bold heading 2026-09-25 10:10:39 +02:00
Utku Bilen Demir
afac75153e docs: bold K-an-Z-I in Kanzi title 2026-09-25 10:09:20 +02:00
Utku Bilen Demir
936971b7f8 docs: clarify Kanzi — Kindle annotations to Zotero Importer (K-an-Z-I) 2026-09-25 10:08:15 +02:00
Utku Bilen Demir
d365ca28e1 Kanzi 0.7.0: safe incremental imports and collector hardening 2026-09-24 14:57:00 +02:00
Utku Bilen Demir
de435fec00 gitignore docs/mismatch-review.md — generated, exposes discussion 2026-09-24 01:17:18 +02:00
Utku Bilen Demir
5fb9b8631d gitignore announcement drafts (docs/announcement*.md) 2026-09-24 01:15:17 +02:00
Utku Bilen Demir
deaa105be2 Kanzi 0.6.20: kanzi.xpi only, dedup identical highlights, pagination 50
- scripts/build_plugin.py: OUTPUT kanzi.xpi (was kindle-zotero-importer.xpi), clean legacy
- plugin/manifest.json: homepage/update_url -> kanzi repo, name Kanzi 0.6.20
- README: Install kanzi.xpi from kanzi/releases, build -> dist/kanzi.xpi
- updates.json: 0.6.20 link -> kanzi/kanzi.xpi
- src/kindle_zotero_importer/clippings.py: dedup_clippings() collapse identical highlights (same title + norm text) to fix badiou yellow×3
- plugin/bootstrap.js: norm dedup for existing yellow over yellow, mergeComments bracket stripped
2026-09-24 01:01:49 +02:00
Utku Bilen Demir
d9062d9a33 Kanzi 0.6.19: kanzi.xpi + sharp black unified + pagination 50
- dist/kanzi.xpi (copy of kindle-zotero-importer.xpi) for Release v0.6.19 alongside old name for auto-update compat
- plugin/manager.html: sharp black please give me money+ (0 radius) centred, pagination 50 for Integrated/Conflicts/Mappings (Prev/Next, 25/50/100/200/500/All, Expand 50)
- README centred flat-square 111111 black sharp badge
- 0.6.18 double [r] fix + 0.6.19 incremental recolours already in main
2026-09-24 00:38:30 +02:00
Utku Bilen Demir
8e8bd31fc8 gitignore MEMORY.md — not for public 2026-09-24 00:21:17 +02:00
Utku Bilen Demir
edf777e8f7 Kanzi 0.6.18: fix double bracket comment + missing text/DOI
- plugin/bootstrap.js: mergeComments now strips leading [code] before dedup, so [r] red note + red note collapse to single red note (was double [r] + stripped)
- plugin/manager.html: 0.6.17 text: a.annotation?.text now sent, vis shows text snippet and DOI/ISBN (simondon was isbn, not DOI, correctly)
- Bump to 0.6.18
2026-09-24 00:00:52 +02:00
34 changed files with 4771 additions and 579 deletions

5
.gitignore vendored
View file

@ -2,7 +2,7 @@ clippings.json
zotero-index.json zotero-index.json
matches.json matches.json
match-overrides.generated.json match-overrides.generated.json
!match-overrides.json match-overrides.json
import-plan*.json import-plan*.json
*.import-plan.json *.import-plan.json
__pycache__/ __pycache__/
@ -15,3 +15,6 @@ plugin-summary.json
plugin-progress.json plugin-progress.json
plugin-config.json plugin-config.json
kindle-import-manager.html kindle-import-manager.html
MEMORY.md
docs/announcement*.md
docs/mismatch-review.md

View file

@ -1,9 +1,9 @@
> [!CAUTION] > [!CAUTION]
> **This project has been heavily vibecoded.** > **This project has been heavily vibecoded.**
# Kanzi — Kindle annotations to Zotero # Kanzi — `K`indle `an`notations to `Z`otero `I`mporter
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero, no terminal needed. Formerly *Kindle Zotero Importer*. Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero. Once the required host tools are installed, imports run from the manager without CLI commands. Formerly *Kindle Zotero Importer*.
<p align="center"> <p align="center">
@ -13,22 +13,45 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
# Install # Install
1. Download `kindle-zotero-importer.xpi` from [Releases](../../releases) (latest `0.6.17`, or `0.6.16-beta` for preview). 1. Download `kanzi.xpi` from [Releases](https://github.com/UtkuBilenDemir/kanzi/releases) (latest `0.7.0`).
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero. 2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
3. `Tools` → `Kanzi…` to open the manager. 3. `Tools` → `Kanzi…` to open the manager.
Works with Zotero 7 to 10, macOS, Windows and Linux, PDF and EPUB. Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. Kanzi detects standard Python installations automatically on macOS, Linux and Windows.
## Windows
Open PowerShell and install the host tools:
```powershell
winget install --id Python.Python.3.13 --exact
winget install --id oschwartz10612.Poppler --exact
winget install --id QPDF.QPDF --exact # Optional: handles copy-protected PDFs
```
Fully quit and reopen Zotero after installation so it receives the updated `PATH`, then install `kanzi.xpi` and open `Tools` → `Kanzi…`. Kanzi checks standard per-user and system Python locations as well as `PATH`; you should not need to enter an executable path manually. EPUB-only imports require Python but not Poppler.
To verify the installation in a new PowerShell window:
```powershell
python --version
pdftotext -v
pdftohtml -v
pdfinfo -v
qpdf --version # Optional
```
# Use # Use
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress. 1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything. 2. **Review before writing.** New highlights are grouped by title and selected by default. Untick individual highlights or whole titles, then press `Integrate selected`. Skipped highlights return on the next run. `Cancel` and closing the manager write nothing to Zotero and do not mark pending highlights as integrated.
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate. 3. **Check Integrated.** After integration, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Tick `Full re-import from scratch` only if you want to rebuild everything. 4. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows one expandable row for that title with up to three suggestions `citekey · title (score%)`. Expand it to inspect individual highlights and positioning problems. Click `Use` on the right candidate, enter a citation key, Zotero key or item ID under `Use Custom`, or choose `Ignore Title`. Tables can be sorted and resized.
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching. 5. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new, changed, deferred or removed highlights are processed. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`. 6. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
7. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
Your choices are saved in `match-overrides.json` in the project folder. Back it up, share it, or delete an entry to undo a mapping. Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
# Coloured highlights # Coloured highlights
@ -62,15 +85,15 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
`Tools` → `Kanzi…` opens the manager. `Tools` → `Kanzi…` opens the manager.
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. **Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. After positioning, the pending review lets you select exactly what is written to Zotero.
**Integrated.** The last `import-plan.final.json` annotations. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right, 400 shown, sortable and resizable. **Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate. **Conflicts.** One row per unresolved Kindle title, combining title-match and attachment/position statuses. Expand a row to inspect its highlights and positioning problems. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. Columns are sortable and resizable.
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable. **Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, `Share anonymised annotations` (on by default, private, hashed only), and `Highlight colours` as above. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed. **Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, annotation sharing, and `Highlight colours` as above. Sharing is off until you explicitly opt in and can be disabled here at any time. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
**Artifacts.** `Mismatch review`, `Persistent overrides`, `Generated suggestions`, `Positioned plan`, `Final writer plan` and `Plugin summary`. Each has `Open` and `Reveal`. **Artifacts.** `Mismatch review`, `Persistent overrides`, `Generated suggestions`, `Positioned plan`, `Final writer plan` and `Plugin summary`. Each has `Open` and `Reveal`.
@ -79,19 +102,19 @@ Incremental is default. Only new, changed or colour-changed highlights are repos
# For developers / CLI # For developers / CLI
```sh ```sh
python scripts/build_plugin.py # → dist/kindle-zotero-importer.xpi python scripts/build_plugin.py # → dist/kanzi.xpi, including the Python runtime
python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --pretty python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --pretty
python -m kindle_zotero_importer run ... --full # ignore incremental and reprocess all python -m kindle_zotero_importer run ... --full # ignore incremental and reprocess all
``` ```
PDF positioning uses Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly. PDF positioning uses host-installed Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
# Releases # Releases
Stable releases are on `Releases` with `kindle-zotero-importer.xpi` and `updates.json` attached. Pre-releases (`beta`) are marked `Pre-release` on GitHub. Zotero auto-updates from `releases/latest/download/updates.json`. Stable releases are on `Releases` with `kanzi.xpi` and `updates.json` attached. Pre-releases (`beta`) are marked `Pre-release` on GitHub. Zotero auto-updates from `releases/latest/download/updates.json`.
# Support # Support
Issues and pull requests welcome at [Issues](../../issues). Issues and pull requests welcome at [Issues](../../issues).
Anonymised sharing is on by default to help future development. It sends only citation key, hashed highlight, colour, has comment and note, added on and integrated at. It never sends raw highlight text. Opt out any time in `Manager` → `Settings` → uncheck `Share anonymised annotations`. Annotation sharing is off until you explicitly opt in. If enabled, it sends a random pseudonymous installation ID, citation key and publication metadata (including DOI/ISBN when available), raw highlight text and comments, normalised/fuzzy hashes, colour, and dates to `annotation.utkubilen.de`. The collector does not receive your Zotero account identity, but free text can contain identifying information and therefore is not anonymous. Change this any time in `Manager` → `Settings`.

357
collector/app.py Normal file
View file

@ -0,0 +1,357 @@
from __future__ import annotations
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path
import re
import secrets
import sqlite3
import unicodedata
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import HTMLResponse
DB = Path(os.environ.get("KANZI_ANNOTATION_DB", Path(__file__).parent / "annotations.db"))
MAX_PAGE_SIZE = 200
MAX_REQUEST_BYTES = 2_000_000
MAX_BATCH_SIZE = 5_000
app = FastAPI(title="Kanzi Annotation Collector")
def _normalise(text: str) -> str:
value = unicodedata.normalize("NFKC", text or "").lower()
value = re.sub(r"-\s*\n\s*", "", value)
value = re.sub(r"[\u2010-\u2015\u2018\u2019\u201c\u201d]", "'", value)
value = re.sub(r"[^a-z0-9]+", " ", value)
return re.sub(r"\s+", " ", value).strip()
def _hash_norm(text: str) -> str:
return hashlib.sha256(_normalise(text).encode()).hexdigest()[:16]
def _fuzzy_hash(text: str) -> str:
norm = _normalise(text)
if len(norm) < 3:
return _hash_norm(text)
trigrams = {norm[index : index + 3] for index in range(len(norm) - 2)}
words = sorted(norm.split())
word_trigrams = {
" ".join(words[index : index + 3])
for index in range(max(1, len(words) - 2))
}
return hashlib.sha256(" ".join(sorted(trigrams | word_trigrams)).encode()).hexdigest()[:16]
def _connect() -> sqlite3.Connection:
DB.parent.mkdir(parents=True, exist_ok=True)
connection = sqlite3.connect(DB)
connection.row_factory = sqlite3.Row
connection.execute("PRAGMA busy_timeout = 5000")
connection.execute("PRAGMA journal_mode = WAL")
return connection
def init_db() -> None:
with _connect() as connection:
connection.execute(
"""CREATE TABLE IF NOT EXISTS highlights (
id INTEGER PRIMARY KEY AUTOINCREMENT,
anon_id TEXT,
citation_key TEXT,
doi TEXT,
isbn TEXT,
issn TEXT,
publication_title TEXT,
clipping_id_hash TEXT,
norm_hash TEXT,
fuzzy_hash TEXT,
text TEXT,
color TEXT,
has_comment INTEGER,
comment TEXT,
comment_hash TEXT,
clipping_added_on_iso TEXT,
integrated_at TEXT,
received_at TEXT,
raw TEXT
)"""
)
existing_columns = {
row["name"] for row in connection.execute("PRAGMA table_info(highlights)")
}
required_columns = {
"doi": "TEXT",
"isbn": "TEXT",
"issn": "TEXT",
"publication_title": "TEXT",
"clipping_id_hash": "TEXT",
"norm_hash": "TEXT",
"fuzzy_hash": "TEXT",
"text": "TEXT",
"comment": "TEXT",
"comment_hash": "TEXT",
"clipping_added_on_iso": "TEXT",
"integrated_at": "TEXT",
"received_at": "TEXT",
"raw": "TEXT",
}
for name, column_type in required_columns.items():
if name not in existing_columns:
connection.execute(f"ALTER TABLE highlights ADD COLUMN {name} {column_type}")
connection.execute(
"""CREATE TABLE IF NOT EXISTS raw_posts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
anon_id TEXT,
received_at TEXT,
body TEXT
)"""
)
connection.execute(
"""CREATE TABLE IF NOT EXISTS installations (
anon_id TEXT PRIMARY KEY,
token_hash TEXT UNIQUE NOT NULL,
created_at TEXT NOT NULL
)"""
)
# Older collector versions appended every retry. Retain only the newest
# record for each installation/clipping before enforcing idempotency.
connection.execute(
"""DELETE FROM highlights
WHERE clipping_id_hash IS NOT NULL
AND id NOT IN (
SELECT MAX(id) FROM highlights
WHERE clipping_id_hash IS NOT NULL
GROUP BY anon_id, clipping_id_hash
)"""
)
connection.execute(
"""CREATE UNIQUE INDEX IF NOT EXISTS highlights_identity
ON highlights(anon_id, clipping_id_hash)
WHERE clipping_id_hash IS NOT NULL"""
)
init_db()
HTML = """<!doctype html>
<html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Kanzi annotations</title>
<style>
:root{font-family:'iA Writer Duo','JetBrains Mono',ui-monospace,monospace;color:#111;background:#f8f8f8;--panel:#fff;--soft:#f0f0f0;--line:#ddd;--muted:#666;--radius:8px}*{box-sizing:border-box}body{margin:0}main{max-width:1280px;margin:auto;padding:20px 24px}h1{margin:0 0 4px;font-size:28px}h2{font-size:16px;border-bottom:1px solid var(--line);padding-bottom:6px;margin-top:24px}.muted{color:var(--muted);font-size:12px}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:12px;margin:16px 0}.card,table{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius)}.card{padding:14px}.card strong{display:block;font-size:22px}table{width:100%;border-collapse:collapse;table-layout:fixed;overflow:hidden}th,td{padding:9px 10px;border-bottom:1px solid var(--line);text-align:left;overflow-wrap:anywhere;font-size:12px}th{background:var(--soft);color:var(--muted);font-size:10px;text-transform:uppercase}.pager{display:flex;justify-content:flex-end;align-items:center;gap:8px;margin:10px 0}.pager button{padding:5px 9px;border:1px solid var(--line);border-radius:var(--radius);background:var(--panel)}.pager button:disabled{color:#999}@media(max-width:700px){main{padding:14px 10px}.table-wrap{overflow-x:auto}table{min-width:800px}}
</style></head><body><main><h1>Kanzi annotations</h1><div class="muted">Private pseudonymous collector. Raw highlight text and comments are stored.</div>
<div class="grid"><div class="card"><strong id="total">-</strong><span>highlights</span></div><div class="card"><strong id="uniq">-</strong><span>citekeys</span></div><div class="card"><strong id="anons">-</strong><span>installations</span></div><div class="card"><strong id="today">-</strong><span>received today</span></div></div>
<h2>Recent</h2><div class="table-wrap"><table><thead><tr><th>anon</th><th>citekey</th><th>DOI / ISBN</th><th>text</th><th>colour</th><th>received</th></tr></thead><tbody id="recent"></tbody></table></div><div class="pager" id="recent-pager"></div>
<h2>Top citekeys</h2><div class="table-wrap"><table><thead><tr><th>citekey</th><th>count</th></tr></thead><tbody id="top"></tbody></table></div>
<h2>Common highlights</h2><div class="muted">Same publication and normalised text across at least two installations.</div><div class="table-wrap"><table><thead><tr><th>DOI / ISBN</th><th>text</th><th>norm</th><th>installations</th><th>citekeys</th></tr></thead><tbody id="common"></tbody></table></div><div class="pager" id="common-pager"></div>
<script>
const size=50,state={recent:0,common:0};
function cell(row,value){const td=document.createElement('td');td.textContent=value??'';row.appendChild(td);return td}
function empty(body,span,message){body.textContent='';const row=body.insertRow();const td=cell(row,message);td.colSpan=span;td.className='muted'}
function pager(id,key,count,load){const root=document.getElementById(id);root.textContent='';const pages=Math.max(1,Math.ceil(count/size));state[key]=Math.min(state[key],pages-1);const prev=document.createElement('button');prev.textContent='Previous';prev.disabled=state[key]===0;prev.onclick=()=>{state[key]--;load()};root.append(prev);const text=document.createElement('span');text.textContent=`Page ${state[key]+1} of ${pages}`;root.append(text);const next=document.createElement('button');next.textContent='Next';next.disabled=state[key]>=pages-1;next.onclick=()=>{state[key]++;load()};root.append(next)}
async function loadRecent(){const result=await fetch(`/recent?limit=${size}&offset=${state.recent*size}`).then(r=>r.json());const body=document.getElementById('recent');body.textContent='';for(const h of result.recent){const row=body.insertRow();cell(row,(h.anon_id||'').slice(0,8));cell(row,h.citation_key);cell(row,h.doi||h.isbn);cell(row,(h.text||'').slice(0,180));cell(row,h.color);cell(row,(h.received_at||'').slice(0,16).replace('T',' '))}if(!result.recent.length)empty(body,6,'No data');pager('recent-pager','recent',result.total,loadRecent)}
async function loadCommon(){const result=await fetch(`/common?limit=${size}&offset=${state.common*size}`).then(r=>r.json());const body=document.getElementById('common');body.textContent='';for(const h of result.common){const row=body.insertRow();cell(row,h.publication);cell(row,(h.example_text||'').slice(0,180));cell(row,h.norm_hash);cell(row,h.installations);cell(row,h.citekeys)}if(!result.common.length)empty(body,5,'No common highlights yet');pager('common-pager','common',result.total,loadCommon)}
async function load(){const stats=await fetch('/stats').then(r=>r.json());for(const [id,key] of [['total','total_highlights'],['uniq','unique_citekeys'],['anons','unique_anons'],['today','today']])document.getElementById(id).textContent=stats[key];const top=await fetch('/top').then(r=>r.json());const body=document.getElementById('top');body.textContent='';for(const h of top.top){const row=body.insertRow();cell(row,h.citation_key);cell(row,h.count)}await Promise.all([loadRecent(),loadCommon()])}load();
</script></main></body></html>"""
@app.get("/", response_class=HTMLResponse)
def root() -> str:
return HTML
@app.get("/health")
def health() -> dict[str, str]:
return {"status": "ok"}
def _token_hash(token: str) -> str:
return hashlib.sha256(token.encode()).hexdigest()
@app.post("/register")
def register() -> dict[str, str]:
token = secrets.token_urlsafe(32)
anon_id = secrets.token_hex(16)
with _connect() as connection:
connection.execute(
"INSERT INTO installations (anon_id, token_hash, created_at) VALUES (?, ?, ?)",
(anon_id, _token_hash(token), datetime.now(timezone.utc).isoformat()),
)
return {"anonId": anon_id, "token": token}
def _authenticated_anon_id(request: Request) -> str:
authorization = request.headers.get("authorization", "")
scheme, separator, token = authorization.partition(" ")
if not separator or scheme.lower() != "bearer" or not token:
raise HTTPException(status_code=401, detail="Missing collector credential")
with _connect() as connection:
row = connection.execute(
"SELECT anon_id FROM installations WHERE token_hash = ?",
(_token_hash(token),),
).fetchone()
if not row:
raise HTTPException(status_code=401, detail="Invalid collector credential")
return str(row["anon_id"])
@app.get("/stats")
def stats() -> dict[str, int]:
today = datetime.now(timezone.utc).date().isoformat()
with _connect() as connection:
row = connection.execute(
"""SELECT COUNT(*) AS total_highlights,
COUNT(DISTINCT citation_key) AS unique_citekeys,
COUNT(DISTINCT anon_id) AS unique_anons,
SUM(CASE WHEN substr(received_at, 1, 10) = ? THEN 1 ELSE 0 END) AS today
FROM highlights""",
(today,),
).fetchone()
return {key: int(row[key] or 0) for key in row.keys()}
@app.get("/recent")
def recent(limit: int = 50, offset: int = 0) -> dict:
limit = max(1, min(limit, MAX_PAGE_SIZE))
offset = max(0, offset)
with _connect() as connection:
total = connection.execute("SELECT COUNT(*) FROM highlights").fetchone()[0]
rows = connection.execute(
"""SELECT anon_id, citation_key, doi, isbn, text, clipping_id_hash,
norm_hash, fuzzy_hash, color, has_comment, received_at
FROM highlights ORDER BY id DESC LIMIT ? OFFSET ?""",
(limit, offset),
).fetchall()
return {"recent": [dict(row) for row in rows], "total": total}
@app.get("/top")
def top(limit: int = 20) -> dict:
limit = max(1, min(limit, 100))
with _connect() as connection:
rows = connection.execute(
"""SELECT citation_key, COUNT(*) AS count FROM highlights
WHERE citation_key IS NOT NULL AND citation_key != ''
GROUP BY citation_key ORDER BY count DESC, citation_key LIMIT ?""",
(limit,),
).fetchall()
return {"top": [dict(row) for row in rows]}
@app.get("/common")
def common(limit: int = 50, offset: int = 0) -> dict:
limit = max(1, min(limit, MAX_PAGE_SIZE))
offset = max(0, offset)
group_sql = """FROM highlights
WHERE norm_hash IS NOT NULL AND norm_hash != ''
AND COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), NULLIF(citation_key, '')) IS NOT NULL
GROUP BY COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key), norm_hash
HAVING COUNT(DISTINCT anon_id) > 1"""
with _connect() as connection:
total = connection.execute(f"SELECT COUNT(*) FROM (SELECT 1 {group_sql})").fetchone()[0]
rows = connection.execute(
f"""SELECT COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key) AS publication,
norm_hash, COUNT(DISTINCT anon_id) AS installations,
GROUP_CONCAT(DISTINCT citation_key) AS citekeys,
MAX(text) AS example_text
{group_sql}
ORDER BY installations DESC, publication, norm_hash LIMIT ? OFFSET ?""",
(limit, offset),
).fetchall()
return {"common": [dict(row) for row in rows], "total": total}
@app.post("/highlights")
async def post_highlights(request: Request) -> dict:
anon = _authenticated_anon_id(request)
content_length = request.headers.get("content-length")
if content_length:
try:
if int(content_length) > MAX_REQUEST_BYTES:
raise HTTPException(status_code=413, detail="Request body is too large")
except ValueError:
raise HTTPException(status_code=400, detail="Invalid Content-Length")
body = await request.body()
if len(body) > MAX_REQUEST_BYTES:
raise HTTPException(status_code=413, detail="Request body is too large")
try:
payload = json.loads(body)
except (json.JSONDecodeError, UnicodeDecodeError) as error:
raise HTTPException(status_code=400, detail="Invalid JSON") from error
if not isinstance(payload, dict):
raise HTTPException(status_code=422, detail="Payload must be an object")
highlights = payload.get("data") or payload.get("highlights") or []
if isinstance(highlights, dict):
highlights = [highlights]
if not isinstance(highlights, list):
raise HTTPException(status_code=422, detail="data must be a list")
if len(highlights) > MAX_BATCH_SIZE:
raise HTTPException(status_code=413, detail="Too many highlights in one request")
now = datetime.now(timezone.utc).isoformat()
inserted = updated = 0
with _connect() as connection:
connection.execute("BEGIN IMMEDIATE")
for highlight in highlights:
if not isinstance(highlight, dict):
continue
text = str(highlight.get("text") or "")[:2000]
clipping_hash = str(highlight.get("clipping_id_hash") or "")[:128] or None
values = (
anon,
str(highlight.get("citation_key") or "")[:512] or None,
str(highlight.get("doi") or "")[:512] or None,
str(highlight.get("isbn") or "")[:512] or None,
str(highlight.get("issn") or "")[:512] or None,
str(highlight.get("publication_title") or "")[:1000] or None,
clipping_hash,
highlight.get("norm_hash") or _hash_norm(text),
highlight.get("fuzzy_hash") or _fuzzy_hash(text),
text,
str(highlight.get("color") or "")[:32] or None,
1 if highlight.get("has_comment") else 0,
str(highlight.get("comment") or "")[:2000],
str(highlight.get("comment_hash") or "")[:256] or None,
str(highlight.get("clipping_added_on_iso") or "")[:64] or None,
str(highlight.get("integrated_at") or "")[:64] or None,
now,
json.dumps(highlight, ensure_ascii=False)[:8000],
)
existing = None
if clipping_hash:
legacy_hash = clipping_hash[:8]
existing = connection.execute(
"""SELECT id FROM highlights
WHERE anon_id = ? AND clipping_id_hash IN (?, ?)
ORDER BY clipping_id_hash = ? DESC LIMIT 1""",
(anon, clipping_hash, legacy_hash, clipping_hash),
).fetchone()
if existing:
connection.execute(
"""UPDATE highlights SET citation_key=?, doi=?, isbn=?, issn=?, publication_title=?, clipping_id_hash=?,
norm_hash=?, fuzzy_hash=?, text=?, color=?, has_comment=?, comment=?, comment_hash=?,
clipping_added_on_iso=?, integrated_at=?, received_at=?, raw=? WHERE id=?""",
(*values[1:7], *values[7:], existing["id"]),
)
updated += 1
else:
connection.execute(
"""INSERT INTO highlights (
anon_id, citation_key, doi, isbn, issn, publication_title, clipping_id_hash,
norm_hash, fuzzy_hash, text, color, has_comment, comment, comment_hash,
clipping_added_on_iso, integrated_at, received_at, raw
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
values,
)
inserted += 1
return {
"status": "ok",
"received": len(highlights),
"inserted": inserted,
"updated": updated,
"anon": anon,
}

View file

@ -0,0 +1,28 @@
# Add these directives in the nginx http context.
limit_req_zone $binary_remote_addr zone=kanzi_ingest:10m rate=10r/m;
limit_conn_zone $binary_remote_addr zone=kanzi_connections:10m;
# Keep dashboard and read APIs behind Authentik. Only plugin ingestion and
# health monitoring are intentionally unauthenticated.
location = /highlights {
client_max_body_size 2m;
limit_req zone=kanzi_ingest burst=10 nodelay;
limit_conn kanzi_connections 2;
proxy_pass http://127.0.0.1:8099;
}
location = /register {
limit_req zone=kanzi_ingest burst=10 nodelay;
limit_conn kanzi_connections 2;
proxy_pass http://127.0.0.1:8099;
}
location = /health {
proxy_pass http://127.0.0.1:8099;
}
location / {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
proxy_pass http://127.0.0.1:8099;
}

View file

@ -0,0 +1,15 @@
{
"format": "kindle-zotero-importer.match-overrides.v1",
"overrides": [
{
"clipping_title": "Example Kindle title",
"resolution": {
"citation_key": "example2026"
},
"review": {
"status": "manager",
"clipping_count": 1
}
}
]
}

View file

@ -1,16 +1,16 @@
# Kindle Zotero Importer Zotero Plugin # Kanzi Zotero Plugin
This is a hybrid Zotero plugin wrapper around the current Python importer. Zotero owns the user interface and writes native annotations through Zotero APIs; the Python project still performs parsing, matching, planning, EPUB positioning, and PDF positioning. This is a hybrid Zotero plugin wrapper around the current Python importer. Zotero owns the user interface and writes native annotations through Zotero APIs; the Python project still performs parsing, matching, planning, EPUB positioning, and PDF positioning.
## Use ## Use
1. Install `kindle-zotero-importer.xpi` in Zotero through `Tools > Plugins`. 1. Install `kanzi.xpi` in Zotero through `Tools > Plugins`.
2. Open `Tools > Kindle Zotero Importer...`. 2. Open `Tools > Kanzi...`.
3. Select the updated Kindle `My Clippings.txt` in the manager. 3. Select the updated Kindle `My Clippings.txt` in the manager.
4. Follow the current stage, percentage, and elapsed time in the manager. 4. Follow the current stage, percentage, and elapsed time in the manager.
5. Review created, existing, updated, failed, and unresolved counts when it completes. 5. Review created, existing, updated, failed, and unresolved counts when it completes.
The plugin uses `match-overrides.json` from the project directory as the persistent source of title mappings and ignored titles. Generated JSON artifacts are still written to the project directory for audit and debugging. The XPI contains the importer Python source. On startup it extracts that source into the Zotero profile, where it keeps private mappings and generated audit artifacts. Python 3.11 or newer must be installed on the host; PDF positioning also requires Poppler (`pdftotext`, `pdftohtml`, and `pdfinfo`) and optionally uses `qpdf` for encrypted files. The plugin uses `match-overrides.json` there as the persistent source of title mappings and ignored titles.
## Build XPI ## Build XPI
@ -20,6 +20,6 @@ From the repository root:
python scripts/build_plugin.py python scripts/build_plugin.py
``` ```
The output is `dist/kindle-zotero-importer.xpi`. The output is `dist/kanzi.xpi`.
The plugin manifest targets Zotero `6.999` through `10.0.*`. The plugin manifest targets Zotero `6.999` through `10.0.*`.

846
plugin/bootstrap.js vendored

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -1,14 +1,14 @@
{ {
"manifest_version": 2, "manifest_version": 2,
"name": "Kanzi", "name": "Kanzi",
"version": "0.6.17", "version": "0.7.1",
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.", "description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
"homepage_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer", "homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
"author": "Utku Bilen Demir", "author": "Utku Bilen Demir",
"applications": { "applications": {
"zotero": { "zotero": {
"id": "kindlezoteroimporter@utkubilen.de", "id": "kindlezoteroimporter@utkubilen.de",
"update_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates.json", "update_url": "https://github.com/UtkuBilenDemir/kanzi/releases/latest/download/updates.json",
"strict_min_version": "6.999", "strict_min_version": "6.999",
"strict_max_version": "10.0.*" "strict_max_version": "10.0.*"
} }

View file

@ -1,6 +1,7 @@
pref("extensions.kindleZoteroImporter.projectDir", "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer"); pref("extensions.kindleZoteroImporter.projectDir", "");
pref("extensions.kindleZoteroImporter.pythonPath", "/opt/homebrew/bin/python3"); pref("extensions.kindleZoteroImporter.pythonPath", "python3");
pref("extensions.kindleZoteroImporter.zoteroDbPath", "/home/user/Zotero/zotero.sqlite"); pref("extensions.kindleZoteroImporter.zoteroDbPath", "");
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "/home/user/Zotero/storage"); pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "");
pref("extensions.kindleZoteroImporter.shareAnonymized", true); pref("extensions.kindleZoteroImporter.shareAnonymized", false);
pref("extensions.kindleZoteroImporter.askedAnonymizedShare", false); pref("extensions.kindleZoteroImporter.askedAnonymizedShare", false);
pref("extensions.kindleZoteroImporter.shareConsentVersion", 0);

View file

@ -1,11 +1,14 @@
[project] [project]
name = "kindle-zotero-importer" name = "kindle-zotero-importer"
version = "0.1.0" version = "0.7.1"
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files." description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
dependencies = [] dependencies = []
[project.optional-dependencies]
collector = ["fastapi>=0.110", "uvicorn>=0.29"]
[project.scripts] [project.scripts]
kindle-zotero-importer = "kindle_zotero_importer.cli:main" kindle-zotero-importer = "kindle_zotero_importer.cli:main"

View file

@ -1,62 +0,0 @@
from __future__ import annotations
import json
import zipfile
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
DIST_DIR = ROOT / "dist"
OUTPUT = DIST_DIR / "kindle-zotero-importer-canary.xpi"
def main() -> int:
DIST_DIR.mkdir(exist_ok=True)
if OUTPUT.exists():
OUTPUT.unlink()
manifest = {
"manifest_version": 2,
"name": "Kindle Zotero Importer Canary",
"version": "0.1.0",
"description": "Minimal install test for Kindle Zotero Importer.",
"homepage_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer",
"author": "Utku Bilen Demir",
"applications": {
"zotero": {
"id": "kindlezoteroimportercanary@utkubilen.de",
"update_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates-canary.json",
"strict_min_version": "6.999",
"strict_max_version": "9.*",
}
},
}
bootstrap = """
function install(data, reason) {}
function uninstall(data, reason) {}
async function startup(data, reason) {
await Zotero.initializationPromise;
Zotero.debug('Kindle Zotero Importer Canary started');
}
function shutdown(data, reason) {}
""".lstrip()
prefs = 'pref("extensions.kindleZoteroImporterCanary.enabled", true);\n'
with zipfile.ZipFile(OUTPUT, "w", zipfile.ZIP_DEFLATED) as archive:
for name, content in [
("bootstrap.js", bootstrap),
("manifest.json", json.dumps(manifest, indent=2) + "\n"),
("prefs.js", prefs),
]:
info = zipfile.ZipInfo(name)
info.create_system = 3
info.external_attr = 0o644 << 16
archive.writestr(info, content, zipfile.ZIP_DEFLATED)
print(f"Built {OUTPUT}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -8,13 +8,15 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
PLUGIN_DIR = ROOT / "plugin" PLUGIN_DIR = ROOT / "plugin"
DIST_DIR = ROOT / "dist" DIST_DIR = ROOT / "dist"
OUTPUT = DIST_DIR / "kindle-zotero-importer.xpi" OUTPUT = DIST_DIR / "kanzi.xpi"
LEGACY_OUTPUT = DIST_DIR / "kindle-zotero-importer.xpi"
def main() -> int: def main() -> int:
DIST_DIR.mkdir(exist_ok=True) DIST_DIR.mkdir(exist_ok=True)
if OUTPUT.exists(): for p in [OUTPUT, LEGACY_OUTPUT]:
OUTPUT.unlink() if p.exists():
p.unlink()
manifest_path = PLUGIN_DIR / "manifest.json" manifest_path = PLUGIN_DIR / "manifest.json"
manifest = json.loads(manifest_path.read_text(encoding="utf-8")) manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
@ -58,12 +60,19 @@ def main() -> int:
info.create_system = 3 info.create_system = 3
info.external_attr = 0o644 << 16 info.external_attr = 0o644 << 16
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED) archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
runtime_files = [ROOT / "plugin_runner.py", *(ROOT / "src" / "kindle_zotero_importer").glob("*.py")]
for path in runtime_files:
archive_name = "runtime/" + str(path.relative_to(ROOT))
info = zipfile.ZipInfo(archive_name)
info.create_system = 3
info.external_attr = 0o644 << 16
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
with zipfile.ZipFile(OUTPUT) as archive: with zipfile.ZipFile(OUTPUT) as archive:
names = set(archive.namelist()) names = set(archive.namelist())
if "manifest.json" not in names or "bootstrap.js" not in names: if not {"manifest.json", "bootstrap.js", "runtime/plugin_runner.py"}.issubset(names):
raise SystemExit( raise SystemExit(
"Invalid XPI: manifest.json and bootstrap.js must be at archive root" "Invalid XPI: plugin files and Python runtime are required"
) )
print(f"Built {OUTPUT}") print(f"Built {OUTPUT}")

View file

@ -1,14 +1,20 @@
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import copy
import json import json
import sys import sys
from pathlib import Path from pathlib import Path
from .clippings import clippings_to_jsonable, load_clippings from .clippings import clippings_to_jsonable, load_clippings
from .epub_position import add_epub_positions from .epub_position import add_epub_positions
from .final_plan import build_final_writer_plan from .final_plan import FINAL_FORMAT, build_final_writer_plan
from .import_plan import DEFAULT_COLOUR_MAP, build_import_plan from .import_plan import (
DEFAULT_COLOUR_MAP,
_attach_notes_to_highlights,
_related_items_for_citekeys,
build_import_plan,
)
from .matcher import build_match_report, load_json from .matcher import build_match_report, load_json
from .mismatch_review import build_mismatch_review from .mismatch_review import build_mismatch_review
from .overrides import generate_override_skeleton, load_overrides from .overrides import generate_override_skeleton, load_overrides
@ -39,6 +45,256 @@ def _load_colour_map(workdir: Path) -> dict[str, str] | None:
return None return None
def _merge_annotation_history(
previous_plan: dict | None, current_annotations: list[dict], deletions: list[str]
) -> list[dict]:
previous = []
if previous_plan:
previous = previous_plan.get("all_annotations") or previous_plan.get("annotations") or []
deleted = set(deletions)
by_id = {
annotation["clipping_id"]: annotation
for annotation in previous
if annotation.get("clipping_id") and annotation["clipping_id"] not in deleted
}
for annotation in current_annotations:
clipping_id = annotation.get("clipping_id")
if clipping_id:
by_id[clipping_id] = annotation
return sorted(
by_id.values(),
key=lambda annotation: (
annotation.get("attachment_item_id") or 0,
annotation.get("annotation", {}).get("sortIndex") or "",
annotation.get("clipping_id") or "",
),
)
def _refresh_history_relations(
annotations: list[dict], zotero_index: dict
) -> None:
"""Resolve citekeys for old annotations without re-positioning them."""
items_by_id = {item["item_id"]: item for item in zotero_index.get("items", [])}
for entry in annotations:
annotation = entry.get("annotation") or {}
entry["related_items"] = _related_items_for_citekeys(
(annotation.get("text") or "", annotation.get("comment") or ""),
entry.get("parent_item_id"),
items_by_id,
)
def _changed_note_clipping_ids(
current_clippings: list[dict], previous_annotations: list[dict]
) -> set[str]:
previous_by_id = {
annotation.get("clipping_id"): annotation
for annotation in previous_annotations
if annotation.get("clipping_id")
}
return {
clipping["id"]
for clipping in current_clippings
if clipping["id"] in previous_by_id
and previous_by_id[clipping["id"]].get("note_ids", [])
!= clipping.get("note_ids", [])
}
def _merge_positioned_history(
previous_plan: dict | None, current_plan: dict, source_ids: set[str]
) -> dict:
by_id = {
item.get("clipping", {}).get("id"): item
for item in (previous_plan or {}).get("items", [])
if item.get("clipping", {}).get("id") in source_ids
}
for item in current_plan.get("items", []):
clipping_id = item.get("clipping", {}).get("id")
if clipping_id:
by_id[clipping_id] = item
merged = dict(current_plan)
merged["items"] = list(by_id.values())
status_counts: dict[str, int] = {}
for item in merged["items"]:
status = item.get("status", "unknown")
status_counts[status] = status_counts.get(status, 0) + 1
merged["status_counts"] = status_counts
return merged
def _deletion_ids(
previous_integrated_ids: set[str], source_ids: set[str], previous_plan: dict | None
) -> set[str]:
deleted = set(previous_integrated_ids - source_ids)
deleted.update((previous_plan or {}).get("delete_failed_ids", []))
return deleted
def _deferred_clipping_ids(
previous_plan: dict | None, current_ids: set[str]
) -> set[str]:
return {
clipping_id
for clipping_id in (previous_plan or {}).get("deferred_clipping_ids", [])
if clipping_id in current_ids
}
def _discarded_clipping_ids(
previous_plan: dict | None, current_ids: set[str]
) -> set[str]:
"""Clips the user permanently discarded in Conflicts (e.g. standalone notes)."""
return {
clipping_id
for clipping_id in (previous_plan or {}).get("discarded_clipping_ids", [])
if clipping_id in current_ids
}
def _changed_override_titles(
previous_snapshot: dict | None, current_overrides: dict
) -> set[str]:
if previous_snapshot is None:
return set(current_overrides)
return {
title
for title in set(previous_snapshot) | set(current_overrides)
if previous_snapshot.get(title) != current_overrides.get(title)
}
def _clipping_ids_for_titles(
clippings: list[dict], titles: set[str]
) -> set[str]:
return {clipping["id"] for clipping in clippings if clipping["title"] in titles}
def _reattempt_failed_with_alternate_attachments(positioned_plan: dict) -> dict:
"""Retry text/rect failures against another attachment of the same item.
`_choose_attachment` commits to one file (e.g. the sole EPUB) without
comparing text. When that file lacks the quote but a sibling PDF/EPUB
has it (e.g. Dostoyevsky EPUB vs PDF), re-anchor the clone there and
keep the winner, tagged `positioned-via-alternate-attachment`.
Partial shingle anchors are also upgraded when a sibling holds the
full quote.
"""
retryable = {"epub-text-not-found", "pdf-text-not-found", "pdf-rects-not-found"}
retries: list[dict] = []
upgrade_ids: set[str] = set()
for item in positioned_plan.get("items", []):
if item.get("status") in retryable:
pass
elif item.get("status") == "positioned" and "partial-text-match" in (
item.get("problems") or []
):
clipping_id = (item.get("clipping") or {}).get("id")
if clipping_id:
upgrade_ids.add(clipping_id)
else:
continue
zotero = item.get("zotero") or {}
current = zotero.get("attachment") or {}
choices = zotero.get("attachment_choices") or []
alternates = [
choice
for choice in choices
if choice.get("item_id") != current.get("item_id")
]
if not alternates:
continue
alternates.sort(
key=lambda choice: 0
if choice.get("content_type") != current.get("content_type")
else 1
)
clone = copy.deepcopy(item)
clone_zotero = dict(clone.get("zotero") or {})
clone_zotero["attachment"] = alternates[0]
clone["zotero"] = clone_zotero
clone["status"] = "ready-for-positioning"
clone["problems"] = []
retries.append(clone)
if not retries:
return positioned_plan
mini = {"format": positioned_plan.get("format"), "items": retries}
mini = add_epub_positions(mini)
mini = add_pdf_positions(mini)
winners = {}
for attempt in mini.get("items", []):
clipping_id = (attempt.get("clipping") or {}).get("id")
if clipping_id and attempt.get("status") == "positioned":
winners[clipping_id] = attempt
if not winners:
return positioned_plan
merged = []
for item in positioned_plan.get("items", []):
clipping_id = (item.get("clipping") or {}).get("id")
if clipping_id in winners:
won = winners[clipping_id]
if clipping_id in upgrade_ids and "partial-text-match" in (
won.get("problems") or []
):
merged.append(item)
continue
won = dict(won)
carried = [
problem
for problem in item.get("problems", [])
if problem != "partial-text-match"
or "partial-text-match" in (won.get("problems") or [])
]
won["problems"] = [
*carried,
"positioned-via-alternate-attachment",
]
merged.append(won)
else:
merged.append(item)
updated = dict(positioned_plan)
updated["items"] = merged
status_counts: dict[str, int] = {}
for entry in merged:
status = entry.get("status", "unknown")
status_counts[status] = status_counts.get(status, 0) + 1
updated["status_counts"] = status_counts
return updated
def _add_previous_imported_comments(
annotations: list[dict], previous_plan: dict | None
) -> None:
previous_by_id = {
annotation.get("clipping_id"): annotation
for annotation in ((previous_plan or {}).get("all_annotations") or (previous_plan or {}).get("annotations", []))
if annotation.get("clipping_id")
}
for annotation in annotations:
previous = previous_by_id.get(annotation.get("clipping_id"))
if previous:
annotation["previous_imported_comment"] = previous.get("annotation", {}).get(
"comment", ""
)
def _unresolved_clipping_ids(
previous_plan: dict | None, source_ids: set[str]
) -> set[str]:
return {
item.get("clipping", {}).get("id")
for item in (previous_plan or {}).get("items", [])
if item.get("status") not in {
"positioned",
"ignored-title",
"pdf-position-skipped-non-highlight",
"epub-position-skipped-non-highlight",
}
and item.get("clipping", {}).get("id") in source_ids
}
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(
prog="kindle-zotero-importer", prog="kindle-zotero-importer",
@ -61,11 +317,11 @@ def main(argv: list[str] | None = None) -> int:
"index-zotero", help="Index Zotero items and attachments read-only" "index-zotero", help="Index Zotero items and attachments read-only"
) )
index_parser.add_argument( index_parser.add_argument(
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite" "--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
) )
index_parser.add_argument( index_parser.add_argument(
"--storage-root", "--storage-root",
default="/home/user/Zotero/storage", default=str(Path.home() / "Zotero" / "storage"),
help="Path to Zotero storage directory for storage: attachments", help="Path to Zotero storage directory for storage: attachments",
) )
index_parser.add_argument( index_parser.add_argument(
@ -175,11 +431,11 @@ def main(argv: list[str] | None = None) -> int:
help="Directory for generated importer artifacts", help="Directory for generated importer artifacts",
) )
run_parser.add_argument( run_parser.add_argument(
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite" "--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
) )
run_parser.add_argument( run_parser.add_argument(
"--storage-root", "--storage-root",
default="/home/user/Zotero/storage", default=str(Path.home() / "Zotero" / "storage"),
help="Path to Zotero storage directory for storage: attachments", help="Path to Zotero storage directory for storage: attachments",
) )
run_parser.add_argument( run_parser.add_argument(
@ -203,6 +459,11 @@ def main(argv: list[str] | None = None) -> int:
action="store_true", action="store_true",
help="Force full re-import from scratch, ignoring incremental state (processes all clippings)", help="Force full re-import from scratch, ignoring incremental state (processes all clippings)",
) )
run_parser.add_argument(
"--retry-unresolved",
action="store_true",
help="Retry previously unresolved positioning and attachment cases",
)
args = parser.parse_args(argv) args = parser.parse_args(argv)
@ -373,37 +634,55 @@ def main(argv: list[str] | None = None) -> int:
report_progress(2, "Reading clippings", "Parsing My Clippings.txt") report_progress(2, "Reading clippings", "Parsing My Clippings.txt")
new_clippings_list = load_clippings(args.clippings_file) new_clippings_list = load_clippings(args.clippings_file)
new_clippings = clippings_to_jsonable(new_clippings_list) new_clippings = clippings_to_jsonable(new_clippings_list)
new_ids = {c["id"] for c in new_clippings.get("clippings", [])} source_clippings = new_clippings.get("clippings", [])
source_ids = {c["id"] for c in source_clippings}
effective_clippings = _attach_notes_to_highlights(source_clippings)
new_ids = {c["id"] for c in effective_clippings}
# Incremental state: previous clippings and previous integrated ids # Incremental state: previous clippings and previous integrated ids
is_incremental = not args.full is_incremental = not args.full
prev_clippings_ids: set[str] = set() prev_clippings_ids: set[str] = set()
prev_effective_ids: set[str] = set()
prev_integrated_ids: set[str] = set() prev_integrated_ids: set[str] = set()
prev_final_plan = None prev_final_plan = None
if is_incremental: prev_positioned_plan = None
prev_clippings_path = artifact("clippings.json")
prev_final_path = artifact("import-plan.final.json") prev_final_path = artifact("import-plan.final.json")
if prev_clippings_path.exists():
try:
prev_clippings_ids = {c["id"] for c in load_json(str(prev_clippings_path)).get("clippings", [])}
except Exception:
prev_clippings_ids = set()
if prev_final_path.exists(): if prev_final_path.exists():
try: try:
prev_final_plan = load_json(str(prev_final_path)) prev_final_plan = load_json(str(prev_final_path))
prev_integrated_ids = {a.get("clipping_id") for a in prev_final_plan.get("annotations", []) if a.get("clipping_id")} previous_annotations = prev_final_plan.get("all_annotations") or prev_final_plan.get("annotations", [])
prev_integrated_ids = {a.get("clipping_id") for a in previous_annotations if a.get("clipping_id")}
except Exception: except Exception:
prev_final_plan = None prev_final_plan = None
prev_integrated_ids = set() prev_integrated_ids = set()
if is_incremental:
prev_clippings_path = artifact("clippings.json")
prev_positioned_path = artifact("import-plan.positioned.json")
if prev_clippings_path.exists():
try:
previous_clippings = load_json(str(prev_clippings_path)).get("clippings", [])
prev_clippings_ids = {c["id"] for c in previous_clippings}
prev_effective_ids = {
c["id"] for c in _attach_notes_to_highlights(previous_clippings)
}
except Exception:
prev_clippings_ids = set()
prev_effective_ids = set()
if prev_positioned_path.exists():
try:
prev_positioned_plan = load_json(str(prev_positioned_path))
except Exception:
prev_positioned_plan = None
# If no previous state, fall back to full # If no previous state, fall back to full
if not prev_clippings_ids and not prev_integrated_ids: if not prev_clippings_ids and not prev_integrated_ids:
is_incremental = False is_incremental = False
if is_incremental: if is_incremental:
# New or previously not integrated (skipped) -> need to (re)process # Only source entries not seen on the preceding run need positioning.
to_process_ids = set(new_ids - prev_integrated_ids) # Previously unresolved entries are re-queued when their override changes.
to_process_ids = set(new_ids - prev_effective_ids)
# Deletions: previously integrated but now absent (removed or changed text) # Deletions: previously integrated but now absent (removed or changed text)
deletions_ids = sorted(prev_integrated_ids - new_ids) deletions_ids = _deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
# Also re-queue any previously integrated clippings whose title now has an override # Also re-queue any previously integrated clippings whose title now has an override
# (mapping may have changed, need to move annotation). We need overrides dict for this. # (mapping may have changed, need to move annotation). We need overrides dict for this.
# Load overrides early for this check (already loaded later, but do here for incremental decision) # Load overrides early for this check (already loaded later, but do here for incremental decision)
@ -412,39 +691,37 @@ def main(argv: list[str] | None = None) -> int:
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {} _ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
except Exception: except Exception:
_ov_for_inc = {} _ov_for_inc = {}
for c in new_clippings_list: changed_override_titles = _changed_override_titles(
if c.id in prev_integrated_ids and c.title in _ov_for_inc: (prev_final_plan or {}).get("overrides_snapshot"), _ov_for_inc
to_process_ids.add(c.id) )
# Also re-queue highlights whose bracket colour would change the annotation to_process_ids.update(
# (so you can recolour a past highlight by adding [r]/[o] etc. to its note or highlight text) _clipping_ids_for_titles(effective_clippings, changed_override_titles)
if colour_map: )
import re as _re previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
# Build map of new notes by title for quick lookup for annotation in previous_annotations:
new_notes_by_title: dict[str, list] = {} if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
for n in new_clippings_list: deletions_ids.add(annotation["clipping_id"])
if n.kind == "note" and n.id not in prev_integrated_ids: newly_added_source_ids = source_ids - prev_clippings_ids
# new note (not yet integrated) — check if it has a bracket colour for c in effective_clippings:
m = _re.match(r"^\s*\[([^\]]+)\]\s*", n.text or "") if set(c.get("note_ids", [])) & newly_added_source_ids:
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}: to_process_ids.add(c["id"])
new_notes_by_title.setdefault(n.title, []).append(n) to_process_ids.update(
for c in new_clippings_list: _changed_note_clipping_ids(effective_clippings, previous_annotations)
if c.id not in prev_integrated_ids: )
continue to_process_ids.update(
if c.kind != "highlight": clipping_id
continue for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
has_bracket = False if clipping_id in new_ids
# check highlight text itself )
for t in [c.text or "", getattr(c, "comment", "") or ""]: to_process_ids.update(_deferred_clipping_ids(prev_final_plan, new_ids))
m = _re.match(r"^\s*\[([^\]]+)\]\s*", t) if args.retry_unresolved and prev_positioned_plan:
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}: to_process_ids.update(
has_bracket = True _unresolved_clipping_ids(prev_positioned_plan, new_ids)
break )
# also check if a *new* note for same title would attach to this highlight (by location/page proximity) # Permanently discarded clips (user action in Conflicts) never re-queue.
# Approximate: any new note with same title is considered for re-queue (conservative, ensures recolours) to_process_ids.difference_update(
if not has_bracket and c.title in new_notes_by_title: _discarded_clipping_ids(prev_final_plan, new_ids)
has_bracket = True )
if has_bracket:
to_process_ids.add(c.id)
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours) # If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
try: try:
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap") prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
@ -455,6 +732,7 @@ def main(argv: list[str] | None = None) -> int:
to_process_ids.update(prev_integrated_ids) to_process_ids.update(prev_integrated_ids)
except Exception: except Exception:
pass pass
deletions_ids = sorted(deletions_ids)
# Also handle deleted overrides: if a previously integrated title's override was deleted, # Also handle deleted overrides: if a previously integrated title's override was deleted,
# its annotations should be deleted. Detect via previous final plan's titles vs current overrides. # its annotations should be deleted. Detect via previous final plan's titles vs current overrides.
# For incremental, we don't have previous overrides, so we handle deletions via the main deletions list # For incremental, we don't have previous overrides, so we handle deletions via the main deletions list
@ -466,7 +744,7 @@ def main(argv: list[str] | None = None) -> int:
# handled as not in to_process_ids? Actually it is still in new_ids, but we re-queued it above only if title in overrides. # handled as not in to_process_ids? Actually it is still in new_ids, but we re-queued it above only if title in overrides.
# For now, rely on full re-import for deleted-override cleanup. # For now, rely on full re-import for deleted-override cleanup.
# Build filtered clippings for this delta run # Build filtered clippings for this delta run
filtered_list = [c for c in new_clippings_list if c.id in to_process_ids] filtered_list = [c for c in effective_clippings if c["id"] in to_process_ids]
# If nothing to do and no deletions, still need to report but skip expensive steps # If nothing to do and no deletions, still need to report but skip expensive steps
if not filtered_list and not deletions_ids: if not filtered_list and not deletions_ids:
report_progress(15, "Matching titles", "No new highlights — incremental skip") report_progress(15, "Matching titles", "No new highlights — incremental skip")
@ -479,6 +757,7 @@ def main(argv: list[str] | None = None) -> int:
matches = build_match_report(clippings, zotero_index, overrides) matches = build_match_report(clippings, zotero_index, overrides)
override_skeleton = generate_override_skeleton(matches) override_skeleton = generate_override_skeleton(matches)
plan = {"format": "kindle-zotero-importer.import-plan.v1", "items": []} plan = {"format": "kindle-zotero-importer.import-plan.v1", "items": []}
epub_plan = {"format": plan["format"], "items": []}
positioned_plan = {"format": plan["format"], "items": []} positioned_plan = {"format": plan["format"], "items": []}
final_plan = {"format": FINAL_FORMAT, "source_format": positioned_plan.get("format"), "annotation_count": 0, "skipped_counts": {}, "annotations": [], "deletions": [], "is_incremental": True, "incremental_stats": {"new_clippings": len(new_ids), "prev_integrated": len(prev_integrated_ids), "to_process": 0, "deletions": 0}} final_plan = {"format": FINAL_FORMAT, "source_format": positioned_plan.get("format"), "annotation_count": 0, "skipped_counts": {}, "annotations": [], "deletions": [], "is_incremental": True, "incremental_stats": {"new_clippings": len(new_ids), "prev_integrated": len(prev_integrated_ids), "to_process": 0, "deletions": 0}}
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
@ -492,7 +771,11 @@ def main(argv: list[str] | None = None) -> int:
if overrides_path.exists(): if overrides_path.exists():
overrides = load_overrides(load_json(str(overrides_path))) overrides = load_overrides(load_json(str(overrides_path)))
# Only the delta clippings go through matching/positioning # Only the delta clippings go through matching/positioning
delta_clippings = clippings_to_jsonable(filtered_list) delta_clippings = {
"format": new_clippings.get("format"),
"count": len(filtered_list),
"clippings": filtered_list,
}
report_progress(15, "Matching titles", f"Matching {len(filtered_list)} new/pending highlights (incremental)") report_progress(15, "Matching titles", f"Matching {len(filtered_list)} new/pending highlights (incremental)")
matches = build_match_report(delta_clippings, zotero_index, overrides) matches = build_match_report(delta_clippings, zotero_index, overrides)
override_skeleton = generate_override_skeleton(matches) override_skeleton = generate_override_skeleton(matches)
@ -504,6 +787,7 @@ def main(argv: list[str] | None = None) -> int:
epub_plan = add_epub_positions(plan) epub_plan = add_epub_positions(plan)
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)") report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
positioned_plan = add_pdf_positions(epub_plan) positioned_plan = add_pdf_positions(epub_plan)
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report") report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan) final_plan = build_final_writer_plan(positioned_plan)
# Attach deletions for writer # Attach deletions for writer
@ -537,14 +821,51 @@ def main(argv: list[str] | None = None) -> int:
epub_plan = add_epub_positions(plan) epub_plan = add_epub_positions(plan)
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage") report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
positioned_plan = add_pdf_positions(epub_plan) positioned_plan = add_pdf_positions(epub_plan)
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report") report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan) final_plan = build_final_writer_plan(positioned_plan)
final_plan["deletions"] = [] final_plan["deletions"] = sorted(
_deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
)
final_plan["is_incremental"] = False final_plan["is_incremental"] = False
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
final_plan["colorMap"] = final_plan["colourMap"] final_plan["colorMap"] = final_plan["colourMap"]
mismatch_review = build_mismatch_review(positioned_plan, matches) mismatch_review = build_mismatch_review(positioned_plan, matches)
if is_incremental:
positioned_plan = _merge_positioned_history(
prev_positioned_plan, positioned_plan, new_ids
)
matches = build_match_report(new_clippings, zotero_index, overrides)
override_skeleton = generate_override_skeleton(matches)
mismatch_review = build_mismatch_review(positioned_plan, matches)
_add_previous_imported_comments(
final_plan.get("annotations", []), prev_final_plan
)
delta_annotation_count = len(final_plan.get("annotations", []))
if is_incremental:
final_plan["all_annotations"] = _merge_annotation_history(
prev_final_plan,
final_plan.get("annotations", []),
final_plan.get("deletions", []),
)
else:
final_plan["all_annotations"] = _merge_annotation_history(
prev_final_plan,
final_plan.get("annotations", []),
final_plan.get("deletions", []),
)
_refresh_history_relations(final_plan["all_annotations"], zotero_index)
final_plan["annotation_count"] = len(final_plan["all_annotations"])
final_plan["delta_annotation_count"] = delta_annotation_count
final_plan["processed_clipping_ids"] = sorted(new_ids)
final_plan["overrides_snapshot"] = overrides or {}
final_plan["discarded_clipping_ids"] = sorted(
_discarded_clipping_ids(prev_final_plan, new_ids)
)
indent = 2 if args.pretty else None indent = 2 if args.pretty else None
outputs = { outputs = {
"clippings": artifact("clippings.json"), "clippings": artifact("clippings.json"),
@ -595,7 +916,8 @@ def main(argv: list[str] | None = None) -> int:
"counts": { "counts": {
"clippings": len(clippings.get("clippings", [])), "clippings": len(clippings.get("clippings", [])),
"unique_titles": len(matches.get("matches", [])), "unique_titles": len(matches.get("matches", [])),
"final_annotations": final_plan.get("annotation_count", 0), "final_annotations": final_plan.get("delta_annotation_count", 0),
"integrated_annotations": final_plan.get("annotation_count", 0),
"match_statuses": match_counts, "match_statuses": match_counts,
"plan_statuses": status_counts, "plan_statuses": status_counts,
"skipped_final": final_plan.get("skipped_counts", {}), "skipped_final": final_plan.get("skipped_counts", {}),

View file

@ -80,7 +80,74 @@ def load_clippings(path: str) -> list[Clipping]:
return parse_clippings_text(file.read()) return parse_clippings_text(file.read())
def _norm_text(text: str) -> str:
import unicodedata
t = unicodedata.normalize("NFKC", text or "").lower()
t = re.sub(r"-\s*\n\s*", "", t)
t = re.sub(r"[^\w\s]+", " ", t)
t = re.sub(r"\s+", " ", t).strip()
return t
def dedup_clippings(clippings: list[Clipping]) -> list[Clipping]:
result: list[Clipping] = []
exact_highlights: dict[tuple[str, str, str | None, str | None], int] = {}
revised_highlights: dict[tuple[str, int], int] = {}
seen_other_ids: set[str] = set()
for c in clippings:
if c.kind != "highlight":
# Notes carry meaning independently and must never collide with highlights.
if c.id not in seen_other_ids:
result.append(c)
seen_other_ids.add(c.id)
continue
norm = _norm_text(c.text)
exact_key = (c.title, norm, c.page, c.location)
if exact_key in exact_highlights:
continue
location_start = _range_start(c.location)
revision_key = (c.title, location_start) if location_start is not None else None
if revision_key is not None and revision_key in revised_highlights:
previous_index = revised_highlights[revision_key]
previous = result[previous_index]
previous_norm = _norm_text(previous.text)
if _is_text_revision(previous_norm, norm):
# Kindle keeps earlier versions when a highlight is extended. The
# longest version represents the final selection.
if len(norm) > len(previous_norm):
result[previous_index] = c
exact_highlights[exact_key] = previous_index
continue
index = len(result)
result.append(c)
exact_highlights[exact_key] = index
if revision_key is not None:
revised_highlights[revision_key] = index
return result
def _range_start(value: str | None) -> int | None:
if not value:
return None
match = re.search(r"\d+", value)
return int(match.group()) if match else None
def _is_text_revision(left: str, right: str) -> bool:
if not left or not right:
return False
shorter, longer = sorted((left, right), key=len)
return len(shorter) >= 20 and longer.startswith(shorter)
def clippings_to_jsonable(clippings: list[Clipping]) -> dict[str, Any]: def clippings_to_jsonable(clippings: list[Clipping]) -> dict[str, Any]:
clippings = dedup_clippings(clippings)
return { return {
"format": "kindle-zotero-importer.clippings.v1", "format": "kindle-zotero-importer.clippings.v1",
"count": len(clippings), "count": len(clippings),

View file

@ -4,6 +4,7 @@ from dataclasses import dataclass
from html import unescape from html import unescape
from html.parser import HTMLParser from html.parser import HTMLParser
import posixpath import posixpath
import unicodedata
import zipfile import zipfile
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from typing import Any from typing import Any
@ -68,6 +69,8 @@ def _position_item(
annotation["sortIndex"] = position["sortIndex"] annotation["sortIndex"] = position["sortIndex"]
updated["annotation"] = annotation updated["annotation"] = annotation
updated["status"] = "positioned" updated["status"] = "positioned"
if position.get("partial"):
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
return updated return updated
@ -104,11 +107,42 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
return None return None
match_start = normalized_haystack.find(normalized_quote) match_start = normalized_haystack.find(normalized_quote)
if match_start < 0: raw_start: int | None = None
return None raw_end: int | None = None
partial = False
if match_start >= 0:
match_end = match_start + len(normalized_quote) match_end = match_start + len(normalized_quote)
raw_start = haystack_map[match_start] raw_start = haystack_map[match_start]
raw_end = haystack_map[match_end - 1] + 1 raw_end = haystack_map[match_end - 1] + 1
else:
stripped_quote, _ = _strip_with_map(normalized_quote)
if stripped_quote and len(stripped_quote) >= 20:
stripped_haystack, stripped_map = _strip_with_map(normalized_haystack)
strip_start = stripped_haystack.find(stripped_quote)
strip_len = len(stripped_quote)
if strip_start < 0:
hit = _partial_stripped_range(stripped_haystack, stripped_quote)
if hit is not None:
strip_start, strip_end = hit
strip_len = strip_end - strip_start
partial = True
if strip_start >= 0:
# stripped_map points into normalized_haystack;
# convert via haystack_map back to raw haystack offsets.
norm_start = stripped_map[strip_start]
norm_end = stripped_map[strip_start + strip_len - 1]
raw_start = haystack_map[norm_start]
raw_end = haystack_map[norm_end] + 1
else:
span = _shingle_span(normalized_haystack, _quote_words(normalized_quote))
if span is not None:
raw_start = haystack_map[span[0]]
raw_end = haystack_map[span[1] - 1] + 1
partial = True
else:
partial = False
if raw_start is None or raw_end is None:
return None
# Account for the spaces inserted between text nodes in haystack construction. # Account for the spaces inserted between text nodes in haystack construction.
adjusted_nodes = _nodes_for_joined_text(text_nodes) adjusted_nodes = _nodes_for_joined_text(text_nodes)
@ -139,6 +173,7 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
"value": value, "value": value,
}, },
"sortIndex": f"{spine_step:05d}|{raw_start:08d}", "sortIndex": f"{spine_step:05d}|{raw_start:08d}",
"partial": partial,
} }
@ -213,6 +248,11 @@ def _clean_text(text: str) -> str:
def _normalize_with_map(text: str) -> tuple[str, list[int]]: def _normalize_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text)
text = (
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
)
normalized = [] normalized = []
mapping = [] mapping = []
last_was_space = False last_was_space = False
@ -232,6 +272,58 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
return "".join(normalized), mapping return "".join(normalized), mapping
def _strip_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text).casefold()
stripped: list[str] = []
mapping: list[int] = []
for index, char in enumerate(text):
if char.isalnum():
stripped.append(char)
mapping.append(index)
return "".join(stripped), mapping
def _partial_stripped_range(
haystack: str, quote: str, min_len: int = 30
) -> tuple[int, int] | None:
if len(quote) < min_len * 2:
return None
mid = len(quote) // 2
for part in (quote[:mid], quote[mid:]):
if len(part) < min_len:
continue
start = haystack.find(part)
if start >= 0:
return start, start + len(part)
return None
def _quote_words(normalized: str) -> list[str]:
words: list[str] = []
for token in normalized.split():
cleaned = "".join(char for char in token if char.isalnum())
if cleaned:
words.append(cleaned)
return words
def _shingle_span(
normalized_text: str, quote_words: list[str], width: int = 6
) -> tuple[int, int] | None:
first: int | None = None
last: int | None = None
for start in range(len(quote_words) - width + 1):
needle = " ".join(quote_words[start : start + width])
at = normalized_text.find(needle)
if at >= 0:
end = at + len(needle)
first = at if first is None else min(first, at)
last = end if last is None else max(last, end)
if first is None or last is None:
return None
return first, last
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]: def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
nodes = [] nodes = []
cursor = 0 cursor = 0

View file

@ -65,11 +65,13 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
"clipping_title": clipping["title"], "clipping_title": clipping["title"],
"clipping_added_on": clipping.get("added_on"), "clipping_added_on": clipping.get("added_on"),
"clipping_added_on_iso": clipping.get("added_on_iso"), "clipping_added_on_iso": clipping.get("added_on_iso"),
"note_ids": clipping.get("note_ids", []),
"integrated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(), "integrated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"attachment_item_id": attachment["item_id"], "attachment_item_id": attachment["item_id"],
"attachment_key": attachment["key"], "attachment_key": attachment["key"],
"parent_item_id": item["zotero"]["parent_item_id"], "parent_item_id": item["zotero"]["parent_item_id"],
"parent_key": item["zotero"]["parent_key"], "parent_key": item["zotero"]["parent_key"],
"related_items": item["zotero"].get("related_items", []),
"citation_key": item["zotero"].get("citation_key"), "citation_key": item["zotero"].get("citation_key"),
"doi": item["zotero"].get("doi"), "doi": item["zotero"].get("doi"),
"isbn": item["zotero"].get("isbn"), "isbn": item["zotero"].get("isbn"),

View file

@ -124,6 +124,12 @@ def _plan_clipping(
zotero_item, overrides_by_title.get(clipping["title"]) or {} zotero_item, overrides_by_title.get(clipping["title"]) or {}
) )
status = "ready-for-positioning" if attachment else attachment_status status = "ready-for-positioning" if attachment else attachment_status
stub = _annotation_stub(clipping, colour_map) if attachment else None
related_items = (
_append_citekey_links(stub, clipping, zotero_item, items_by_id)
if stub
else []
)
return { return {
"clipping": clipping, "clipping": clipping,
"status": status, "status": status,
@ -138,17 +144,96 @@ def _plan_clipping(
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None, "issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
"url": zotero_item["fields"].get("url"), "url": zotero_item["fields"].get("url"),
"publication_title": zotero_item["fields"].get("publicationTitle"), "publication_title": zotero_item["fields"].get("publicationTitle"),
"related_items": related_items,
"attachment": attachment, "attachment": attachment,
"attachment_choices": zotero_item.get("attachments", []), "attachment_choices": zotero_item.get("attachments", []),
"expected_attachment_type": _expected_attachment_type( "expected_attachment_type": _expected_attachment_type(
clipping, zotero_item.get("attachments", []) clipping, zotero_item.get("attachments", [])
), ),
}, },
"annotation": _annotation_stub(clipping, colour_map) if attachment else None, "annotation": stub,
"problems": problems, "problems": problems,
} }
def _append_citekey_links(
stub: dict[str, Any],
clipping: dict[str, Any],
parent_item: dict[str, Any],
items_by_id: dict[int, dict[str, Any]],
) -> list[dict[str, Any]]:
"""Turn `@citekey` mentions into clickable Zotero item links.
Kindle-side `@key` tokens (Obsidian-style) resolve against the library
index; matches to *other* items are appended to the annotation comment
as `zotero://select` links, which Zotero renders clickable. Self-links
to the parent item are skipped.
"""
related_items = _related_items_for_citekeys(
(clipping.get("text") or "", clipping.get("comment") or ""),
parent_item.get("item_id"),
items_by_id,
)
links = [
f"@{target['citation_key']} {target['link']}" for target in related_items
]
if not links:
return []
existing = stub.get("comment") or ""
block = "Related: " + " · ".join(links)
if block not in existing:
stub["comment"] = f"{existing}\n\n{block}" if existing else block
return related_items
def _select_link(item: dict[str, Any]) -> str:
"""Library-aware `zotero://select` link for an indexed item.
Group-library targets need the `/groups/{groupID}/items/{key}` form;
personal-library (or unknown) targets use `/library/items/{key}`.
"""
group_id = item.get("group_id")
if group_id:
return f"zotero://select/groups/{group_id}/items/{item['key']}"
return f"zotero://select/library/items/{item['key']}"
def _related_items_for_citekeys(
sources: tuple[str, ...],
parent_item_id: int | None,
items_by_id: dict[int, dict[str, Any]],
) -> list[dict[str, Any]]:
tokens = {
token.casefold()
for source in sources
for token in re.findall(r"@([A-Za-z0-9_][\w:.-]*)", source or "")
}
if not tokens:
return []
by_citekey = {
str(item.get("fields", {}).get("citationKey", "")).casefold(): item
for item in items_by_id.values()
if item.get("fields", {}).get("citationKey")
}
related = []
seen_item_ids = set()
for token in sorted(tokens):
target = by_citekey.get(token)
target_id = target.get("item_id") if target else None
if not target or target_id == parent_item_id or target_id in seen_item_ids:
continue
seen_item_ids.add(target_id)
related.append(
{
"item_id": target_id,
"key": target["key"],
"citation_key": target["fields"]["citationKey"],
"link": _select_link(target),
}
)
return related
def _choose_attachment( def _choose_attachment(
zotero_item: dict[str, Any], zotero_item: dict[str, Any],
override: dict[str, Any] | None = None, override: dict[str, Any] | None = None,

View file

@ -7,15 +7,17 @@ import re
import shutil import shutil
import subprocess import subprocess
import tempfile import tempfile
import unicodedata
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Any from typing import Any
_TOOL_PATHS = { _TOOL_PATHS = {
"pdftotext": ("/home/user/bin/pdftotext", "/opt/homebrew/bin/pdftotext"), "pdftotext": (str(Path.home() / "bin" / "pdftotext"), "/opt/homebrew/bin/pdftotext"),
"pdftohtml": ("/home/user/bin/pdftohtml", "/opt/homebrew/bin/pdftohtml"), "pdftohtml": (str(Path.home() / "bin" / "pdftohtml"), "/opt/homebrew/bin/pdftohtml"),
"pdfinfo": ("/home/user/bin/pdfinfo", "/opt/homebrew/bin/pdfinfo"), "pdfinfo": (str(Path.home() / "bin" / "pdfinfo"), "/opt/homebrew/bin/pdfinfo"),
"qpdf": ("/home/user/bin/qpdf", "/opt/homebrew/bin/qpdf"), "qpdf": (str(Path.home() / "bin" / "qpdf"), "/opt/homebrew/bin/qpdf"),
} }
@ -29,6 +31,15 @@ def _tool(name: str) -> str:
return name return name
def _missing_required_tools() -> list[str]:
missing = []
for name in ("pdftotext", "pdftohtml", "pdfinfo"):
path = _tool(name)
if path == name and shutil.which(name) is None:
missing.append(name)
return missing
@dataclass(frozen=True) @dataclass(frozen=True)
class PDFWord: class PDFWord:
text: str text: str
@ -54,8 +65,19 @@ def add_pdf_positions(plan: dict[str, Any]) -> dict[str, Any]:
size_cache: dict[tuple[str, int], tuple[float, float]] = {} size_cache: dict[tuple[str, int], tuple[float, float]] = {}
decrypted_cache: dict[str, str] = {} decrypted_cache: dict[str, str] = {}
positioned = [] positioned = []
missing_tools = _missing_required_tools()
with tempfile.TemporaryDirectory(prefix="kindle-zotero-pdf-") as temp_dir: with tempfile.TemporaryDirectory(prefix="kindle-zotero-pdf-") as temp_dir:
for item in plan["items"]: for item in plan["items"]:
attachment = (item.get("zotero") or {}).get("attachment") or {}
if (
missing_tools
and item.get("status") == "ready-for-positioning"
and attachment.get("content_type") == "application/pdf"
):
positioned.append(
_with_problem(item, f"pdf-tools-missing:{','.join(missing_tools)}")
)
else:
positioned.append( positioned.append(
_position_item( _position_item(
item, item,
@ -105,7 +127,12 @@ def _position_item(
text_pages, item["clipping"]["text"], item["clipping"].get("page") text_pages, item["clipping"]["text"], item["clipping"].get("page")
) )
if page_index is None: if page_index is None:
return _with_problem(item, "pdf-text-not-found") failed = _with_problem(item, "pdf-text-not-found")
hint = _page_hint_text(text_pages, item["clipping"].get("page"))
if hint:
failed = dict(failed)
failed["page_hint"] = hint
return failed
page_xml = xml_cache.setdefault( page_xml = xml_cache.setdefault(
(extraction_path, page_index), (extraction_path, page_index),
extract_pdf_page_xml(extraction_path, page_index), extract_pdf_page_xml(extraction_path, page_index),
@ -131,6 +158,8 @@ def _position_item(
annotation["sortIndex"] = position.get("sortIndex") annotation["sortIndex"] = position.get("sortIndex")
updated["annotation"] = annotation updated["annotation"] = annotation
updated["status"] = "positioned" updated["status"] = "positioned"
if position.get("partial"):
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
return updated return updated
@ -206,6 +235,37 @@ def find_pdf_text_page(
normalized_page, _ = _normalize_with_map(pages[index]) normalized_page, _ = _normalize_with_map(pages[index])
if normalized_quote in normalized_page: if normalized_quote in normalized_page:
return index return index
# Fallback: alphanumeric-only search tolerates PDF extraction artifacts
# such as "o f" for "of", hyphenation, and punctuation differences.
stripped_quote, _ = _strip_with_map(quote)
if stripped_quote and len(stripped_quote) >= 20:
for index in page_order:
stripped_page, _ = _strip_with_map(pages[index])
if stripped_quote in stripped_page:
return index
# Partial fallback: long highlights may span pages or contain a
# divergent middle (edition differences). Anchor to either half so
# Zotero-style word search behaviour is approximated.
for index in page_order:
stripped_page, _ = _strip_with_map(pages[index])
if _partial_stripped_hit(stripped_page, stripped_quote):
return index
# Shingle fallback: scattered OCR substitutions (e.g. Kindle
# "yadsınaraz" vs PDF "yadsınamaz") break every long substring but
# leave most 6-word runs intact. Rank pages by shingle hits.
quote_words = _quote_words(normalized_quote)
if len(quote_words) >= 6:
required = 2 if len(quote_words) >= 12 else 1
best_index: int | None = None
best_hits = 0
for index in page_order:
normalized_page, _ = _normalize_with_map(pages[index])
hits = _shingle_hits(normalized_page, quote_words)
if hits > best_hits:
best_hits = hits
best_index = index
if best_index is not None and best_hits >= required:
return best_index
return None return None
@ -269,25 +329,52 @@ def find_pdf_rects(
text = " ".join(word.text for word in page.words) text = " ".join(word.text for word in page.words)
normalized_text, text_map = _normalize_with_map(text) normalized_text, text_map = _normalize_with_map(text)
start = normalized_text.find(normalized_quote) start = normalized_text.find(normalized_quote)
if start < 0: raw_start: int | None = None
return None raw_end: int | None = None
partial = False
if start >= 0:
end = start + len(normalized_quote) end = start + len(normalized_quote)
raw_start = text_map[start] raw_start = text_map[start]
raw_end = text_map[end - 1] + 1 raw_end = text_map[end - 1] + 1
else:
stripped_quote, _ = _strip_with_map(quote)
if stripped_quote and len(stripped_quote) >= 20:
stripped_text, stripped_map = _strip_with_map(text)
strip_start = stripped_text.find(stripped_quote)
if strip_start >= 0:
strip_end = strip_start + len(stripped_quote)
raw_start = stripped_map[strip_start]
raw_end = stripped_map[strip_end - 1] + 1
else:
hit = _partial_stripped_range(stripped_text, stripped_quote)
if hit is not None:
strip_start, strip_end = hit
raw_start = stripped_map[strip_start]
raw_end = stripped_map[strip_end - 1] + 1
partial = True
else:
span = _shingle_span(normalized_text, _quote_words(normalized_quote))
if span is not None:
raw_start = text_map[span[0]]
raw_end = text_map[span[1] - 1] + 1
partial = True
if raw_start is None or raw_end is None:
return None
matched_words = [ matched_words = [
word for word in page.words if word.end >= raw_start and word.start <= raw_end word for word in page.words if word.end >= raw_start and word.start <= raw_end
] ]
if not matched_words: if not matched_words:
return None return None
pdf_width, pdf_height = size_cache.setdefault( size_key = (path, page.index)
(path, page.index), if size_key not in size_cache:
_pdf_page_size(path, page.index + 1, page.width, page.height), size_cache[size_key] = _pdf_page_size(path, page.index + 1, page.width, page.height)
) pdf_width, pdf_height = size_cache[size_key]
rects = _word_rects(matched_words, page, pdf_width, pdf_height) rects = _word_rects(matched_words, page, pdf_width, pdf_height)
return { return {
"pageIndex": page.index, "pageIndex": page.index,
"rects": rects, "rects": rects,
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}", "sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
"partial": partial,
} }
@ -340,6 +427,11 @@ def _pdf_page_size(
def _normalize_with_map(text: str) -> tuple[str, list[int]]: def _normalize_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text)
text = (
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
)
normalized = [] normalized = []
mapping = [] mapping = []
last_was_space = False last_was_space = False
@ -359,6 +451,112 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
return "".join(normalized), mapping return "".join(normalized), mapping
def _strip_with_map(text: str) -> tuple[str, list[int]]:
"""Alphanumeric-only folding with index mapping.
Removes spaces and punctuation so PDF artifacts like "o f" for "of",
hyphenated line breaks, and curly-quote/dash variants still match.
Mapping points back into the input string for rect recovery.
"""
text = unicodedata.normalize("NFKC", text).casefold()
stripped: list[str] = []
mapping: list[int] = []
for index, char in enumerate(text):
if char.isalnum():
stripped.append(char)
mapping.append(index)
return "".join(stripped), mapping
def _partial_stripped_range(
haystack: str, quote: str, min_len: int = 30
) -> tuple[int, int] | None:
"""Best half-quote hit for long highlights spanning pages/editions."""
if len(quote) < min_len * 2:
return None
mid = len(quote) // 2
for part in (quote[:mid], quote[mid:]):
if len(part) < min_len:
continue
start = haystack.find(part)
if start >= 0:
return start, start + len(part)
return None
def _partial_stripped_hit(haystack: str, quote: str, min_len: int = 30) -> bool:
return _partial_stripped_range(haystack, quote, min_len) is not None
def _quote_words(normalized: str, width: int = 6) -> list[str]:
del width
words: list[str] = []
for token in normalized.split():
cleaned = "".join(char for char in token if char.isalnum())
if cleaned:
words.append(cleaned)
return words
def _shingle_hits(normalized_page: str, quote_words: list[str], width: int = 6) -> int:
hits = 0
seen: set[tuple[str, ...]] = set()
for start in range(len(quote_words) - width + 1):
shingle = tuple(quote_words[start : start + width])
if shingle in seen:
continue
seen.add(shingle)
if " ".join(shingle) in normalized_page:
hits += 1
return hits
def _shingle_span(
normalized_text: str, quote_words: list[str], width: int = 6
) -> tuple[int, int] | None:
first: int | None = None
last: int | None = None
for start in range(len(quote_words) - width + 1):
needle = " ".join(quote_words[start : start + width])
at = normalized_text.find(needle)
if at >= 0:
end = at + len(needle)
first = at if first is None else min(first, at)
last = end if last is None else max(last, end)
if first is None or last is None:
return None
return first, last
def _page_hint_text(
pages: list[str], kindle_page: str | None, radius: int = 1, limit: int = 1200
) -> dict[str, Any] | None:
"""Snippet of the PDF pages around the Kindle page hint.
Lets the user eyeball which part a `pdf-text-not-found` highlight
belongs to and place it manually. Truncated to keep artifacts small.
"""
if not kindle_page or not kindle_page.isdigit():
return None
center = int(kindle_page) - 1
if center < 0 or center >= len(pages):
return None
parts = []
for index in range(max(0, center - radius), min(len(pages), center + radius + 1)):
snippet = " ".join(pages[index].split())
if snippet:
parts.append(f"[PDF p.{index + 1}] {snippet[:limit]}")
if not parts:
return None
return {
"kindle_page": kindle_page,
"pdf_pages": list(
range(max(1, center - radius + 1), min(len(pages), center + radius + 1) + 1)
),
"text": "\n\n".join(parts)[: limit * 2 + 64],
}
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]: def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
updated = dict(item) updated = dict(item)
updated["status"] = problem updated["status"] = problem

View file

@ -34,6 +34,8 @@ class ZoteroItem:
key: str key: str
item_type: str item_type: str
title: str | None title: str | None
library_id: int | None = None
group_id: int | None = None
creators: list[str] = field(default_factory=list) creators: list[str] = field(default_factory=list)
fields: dict[str, str] = field(default_factory=dict) fields: dict[str, str] = field(default_factory=dict)
attachments: list[ZoteroAttachment] = field(default_factory=list) attachments: list[ZoteroAttachment] = field(default_factory=list)
@ -47,15 +49,19 @@ def build_zotero_index(db_path: str, storage_root: str | None = None) -> dict[st
creators = _load_creators(connection) creators = _load_creators(connection)
attachments = _load_attachments(connection, storage_root) attachments = _load_attachments(connection, storage_root)
groups = _load_groups(connection)
indexed_items: list[ZoteroItem] = [] indexed_items: list[ZoteroItem] = []
for row in items: for row in items:
fields = _metadata_for_item(row) fields = _metadata_for_item(row)
library_id = row["libraryID"]
indexed_items.append( indexed_items.append(
ZoteroItem( ZoteroItem(
item_id=row["itemID"], item_id=row["itemID"],
key=row["key"], key=row["key"],
item_type=row["typeName"], item_type=row["typeName"],
title=fields.get("title"), title=fields.get("title"),
library_id=library_id,
group_id=groups.get(library_id),
creators=creators.get(row["itemID"], []), creators=creators.get(row["itemID"], []),
fields=fields, fields=fields,
attachments=attachments.get(row["itemID"], []), attachments=attachments.get(row["itemID"], []),
@ -80,6 +86,7 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
SELECT SELECT
i.itemID, i.itemID,
i.key, i.key,
i.libraryID,
it.typeName, it.typeName,
{field_columns} {field_columns}
FROM items i FROM items i
@ -95,6 +102,14 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
return list(connection.execute(query)) return list(connection.execute(query))
def _load_groups(connection: sqlite3.Connection) -> dict[int, int]:
try:
rows = connection.execute("SELECT groupID, libraryID FROM groups")
except Exception:
return {}
return {row["libraryID"]: row["groupID"] for row in rows}
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]: def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
rows = connection.execute( rows = connection.execute(
""" """

View file

@ -0,0 +1,15 @@
Kanzi Dummy EPUB
- Your Highlight on page 1 | location 100-120 | Added on Tuesday, 18 May 2021 14:18:13
This is a test highlight long enough to import for EPUB.
==========
Kanzi Dummy PDF
- Your Highlight on page 2 | location 200-220 | Added on Tuesday, 18 May 2021 14:20:00
This is a test highlight long enough to import for PDF.
==========
Unknown Book Title XYZ
- Your Highlight on page 1 | location 10-11 | Added on Tuesday, 18 May 2021 14:18:13
This should land in Conflicts for manual mapping.
==========

View file

@ -0,0 +1,36 @@
Kanzi Windows smoke test (Parallels VM)
COPY THESE 3 FILES INTO THE VM:
- My Clippings.txt
- kanzi-dummy.epub
- kanzi-dummy.pdf
- dist/kanzi.xpi (from repo root, same cross-platform build)
1) PowerShell:
winget install --id Python.Python.3.13 --exact
winget install --id oschwartz10612.Poppler --exact
winget install --id QPDF.QPDF --exact
python --version
pdftotext -v
pdfinfo -v
qpdf --version
2) Fully quit + reopen Zotero (picks up PATH).
3) Fresh Zotero, sync off. Create:
- Book "Kanzi Dummy EPUB" + attach kanzi-dummy.epub
- Book "Kanzi Dummy PDF" + attach kanzi-dummy.pdf
4) Tools -> Kanzi -> Choose My Clippings.txt (the copied one).
Leave Python path empty to test auto-detect.
EXPECT:
- 2 highlights position and await review, 1 title in Conflicts
("Unknown Book Title XYZ").
- Summary shows new Status by Kindle Title table with source counts.
- Save a mapping for the unknown title -> Re-import processes it.
- Table headers sort and persist across pages.
IF PYTHON NOT FOUND:
Settings shows detected path hint; paste full python.exe path,
Save Settings, retry.

Binary file not shown.

View file

@ -0,0 +1,33 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
endobj
4 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
endobj
5 0 obj
<< /Length 134 >>
stream
BT /F1 12 Tf 72 720 Td (Kanzi Dummy PDF) Tj ET
BT /F1 12 Tf 72 700 Td (This is a test highlight long enough to import for PDF.) Tj ET
endstream
endobj
xref
0 6
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000241 00000 n
0000000311 00000 n
trailer
<< /Size 6 /Root 1 0 R >>
startxref
495
%%EOF

View file

@ -0,0 +1,93 @@
"""Generate minimal EPUB + PDF fixtures for the Windows smoke test.
Run: python3 tests/fixtures/windows-smoke/make_fixtures.py
Outputs kanzi-dummy.epub and kanzi-dummy.pdf next to this script.
Both contain the highlight sentences from 'My Clippings.txt' so
EPUB CFI and PDF text positioning can succeed.
"""
from __future__ import annotations
from pathlib import Path
import zipfile
HERE = Path(__file__).resolve().parent
EPUB_TEXT = "This is a test highlight long enough to import for EPUB."
PDF_TEXT = "This is a test highlight long enough to import for PDF."
def make_epub(path: Path) -> None:
chapter = f"""<?xml version="1.0" encoding="utf-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head><title>Kanzi Dummy</title></head>
<body>
<h1>Kanzi Dummy EPUB</h1>
<p>Some intro text before the highlight.</p>
<p>{EPUB_TEXT}</p>
<p>Some trailing text after the highlight.</p>
</body>
</html>
"""
container = """<?xml version="1.0" encoding="utf-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
</container>
"""
opf = """<?xml version="1.0" encoding="utf-8"?>
<package version="3.0" unique-identifier="uid" xmlns="http://www.idpf.org/2007/opf">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Kanzi Dummy EPUB</dc:title><dc:identifier id="uid">kanzi-dummy-epub</dc:identifier><dc:language>en</dc:language></metadata>
<manifest><item id="ch1" href="chapter1.xhtml" media-type="application/xhtml+xml"/></manifest>
<spine><itemref idref="ch1"/></spine>
</package>
"""
with zipfile.ZipFile(path, "w") as z:
z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
z.writestr("META-INF/container.xml", container, compress_type=zipfile.ZIP_DEFLATED)
z.writestr("OEBPS/content.opf", opf, compress_type=zipfile.ZIP_DEFLATED)
z.writestr("OEBPS/chapter1.xhtml", chapter, compress_type=zipfile.ZIP_DEFLATED)
def _pdf_escape(text: str) -> str:
return text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
def make_pdf(path: Path) -> None:
# Minimal one-page PDF with Helvetica text. pdftotext extracts it fine.
content = (
f"BT /F1 12 Tf 72 720 Td ({_pdf_escape('Kanzi Dummy PDF')}) Tj ET\n"
f"BT /F1 12 Tf 72 700 Td ({_pdf_escape(PDF_TEXT)}) Tj ET\n"
)
objects: list[bytes] = []
objects.append(b"<< /Type /Catalog /Pages 2 0 R >>")
objects.append(b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
objects.append(
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"
)
objects.append(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
objects.append(f"<< /Length {len(content.encode('latin-1'))} >>\nstream\n{content}endstream".encode("latin-1"))
out = bytearray(b"%PDF-1.4\n")
offsets = [0]
for i, body in enumerate(objects, start=1):
offsets.append(len(out))
out += f"{i} 0 obj\n".encode("ascii") + body + b"\nendobj\n"
xref_pos = len(out)
out += f"xref\n0 {len(objects) + 1}\n".encode("ascii")
out += b"0000000000 65535 f \n"
for off in offsets[1:]:
out += f"{off:010d} 00000 n \n".encode("ascii")
out += (
f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
f"startxref\n{xref_pos}\n%%EOF\n"
).encode("ascii")
path.write_bytes(bytes(out))
def main() -> None:
make_epub(HERE / "kanzi-dummy.epub")
make_pdf(HERE / "kanzi-dummy.pdf")
print("wrote kanzi-dummy.epub + kanzi-dummy.pdf")
if __name__ == "__main__":
main()

View file

@ -2,12 +2,14 @@ import sys
sys.path.insert(0, "src") sys.path.insert(0, "src")
sys.path.insert(0, ".") sys.path.insert(0, ".")
import importlib.util import importlib.util
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy"]: for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy","test_incremental","test_collector"]:
spec = importlib.util.spec_from_file_location(name, f"tests/{name}.py") spec = importlib.util.spec_from_file_location(name, f"tests/{name}.py")
mod = importlib.util.module_from_spec(spec) mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod) spec.loader.exec_module(mod)
for attr in dir(mod): for attr in dir(mod):
if attr.startswith("test_"): if attr.startswith("test_"):
if attr == "test_incremental_ids":
continue
getattr(mod, attr)() getattr(mod, attr)()
print(f"✓ {name}.{attr}") print(f"✓ {name}.{attr}")
print("All Python tests passed") print("All Python tests passed")

View file

@ -1,4 +1,9 @@
from kindle_zotero_importer.clippings import parse_clippings_text, _stable_id from kindle_zotero_importer.clippings import (
Clipping,
_stable_id,
dedup_clippings,
parse_clippings_text,
)
def test_parse_and_id_stable(): def test_parse_and_id_stable():
txt = "My Book (Author)\n- Your Highlight at location 10-20 | Added on Monday, 6 October 2025 11:32:29\n\nSome text\n==========\n" txt = "My Book (Author)\n- Your Highlight at location 10-20 | Added on Monday, 6 October 2025 11:32:29\n\nSome text\n==========\n"
@ -15,3 +20,57 @@ def test_added_on_iso():
txt = "B\n- Your Highlight at location 1 | Added on Tuesday, 18 May 2021 14:18:13\n\nText\n==========\n" txt = "B\n- Your Highlight at location 1 | Added on Tuesday, 18 May 2021 14:18:13\n\nText\n==========\n"
c = parse_clippings_text(txt)[0] c = parse_clippings_text(txt)[0]
assert c.added_on_iso.startswith("2021-05-18T14:18:13") assert c.added_on_iso.startswith("2021-05-18T14:18:13")
def _clipping(identifier, kind, text, location):
return Clipping(
id=identifier,
title="Book",
kind=kind,
text=text,
raw_detail="detail",
location=location,
)
def test_dedup_keeps_longest_progressive_highlight_revision():
short = _clipping("short", "highlight", "A selection long enough to compare", "53-54")
long = _clipping(
"long",
"highlight",
"A selection long enough to compare and its extended ending",
"53-55",
)
assert dedup_clippings([short, long]) == [long]
def test_dedup_preserves_unrelated_text_with_same_location_start():
short = _clipping(
"short",
"highlight",
"J. he is actually of obscure extraction and this is long enough",
"53-54",
)
corrected = _clipping(
"corrected",
"highlight",
"A. J. he is actually of obscure extraction and this is long enough",
"53-55",
)
assert dedup_clippings([short, corrected]) == [short, corrected]
def test_dedup_preserves_same_text_at_different_locations():
first = _clipping("first", "highlight", "A repeated sentence in a book", "10-11")
second = _clipping("second", "highlight", "A repeated sentence in a book", "80-81")
assert dedup_clippings([first, second]) == [first, second]
def test_dedup_never_collides_notes_with_highlights():
highlight = _clipping("highlight", "highlight", "Same meaningful text", "10-11")
note = _clipping("note", "note", "Same meaningful text", "11")
assert dedup_clippings([highlight, note]) == [highlight, note]

63
tests/test_collector.py Normal file
View file

@ -0,0 +1,63 @@
import os
from pathlib import Path
import sys
import tempfile
import types
os.environ.setdefault(
"KANZI_ANNOTATION_DB", str(Path(tempfile.gettempdir()) / "kanzi-test-import.db")
)
try:
import fastapi # noqa: F401
except ModuleNotFoundError:
class _FastAPI:
def __init__(self, **_kwargs):
pass
def get(self, *_args, **_kwargs):
return lambda function: function
post = get
class _HTTPException(Exception):
def __init__(self, status_code: int, detail: str):
super().__init__(detail)
self.status_code = status_code
self.detail = detail
fake_fastapi = types.ModuleType("fastapi")
fake_fastapi.FastAPI = _FastAPI
fake_fastapi.HTTPException = _HTTPException
fake_fastapi.Request = object
fake_responses = types.ModuleType("fastapi.responses")
fake_responses.HTMLResponse = str
sys.modules["fastapi"] = fake_fastapi
sys.modules["fastapi.responses"] = fake_responses
from collector import app as collector
from fastapi import HTTPException
class _Request:
def __init__(self, token: str | None = None):
self.headers = {"authorization": f"Bearer {token}"} if token else {}
def test_collector_issues_and_verifies_installation_credentials():
original_db = collector.DB
with tempfile.TemporaryDirectory() as temporary:
collector.DB = Path(temporary) / "annotations.db"
collector.init_db()
credentials = collector.register()
assert collector._authenticated_anon_id(
_Request(credentials["token"])
) == credentials["anonId"]
try:
collector._authenticated_anon_id(_Request("wrong"))
except HTTPException as error:
assert error.status_code == 401
else:
raise AssertionError("Invalid collector token was accepted")
collector.DB = original_db

View file

@ -4,7 +4,7 @@ def test_final_includes_kindle_id_tag():
positioned = { positioned = {
"format":"x", "format":"x",
"items":[ "items":[
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None}, {"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None,"note_ids":["note1"]},
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"}, "zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}} "annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
] ]
@ -15,6 +15,7 @@ def test_final_includes_kindle_id_tag():
assert {"name":"kindle-import"} in tags assert {"name":"kindle-import"} in tags
assert {"name":"kindle-id:abc123"} in tags assert {"name":"kindle-id:abc123"} in tags
assert plan["annotations"][0]["clipping_id"]=="abc123" assert plan["annotations"][0]["clipping_id"]=="abc123"
assert plan["annotations"][0]["note_ids"] == ["note1"]
def test_final_skips_non_positioned(): def test_final_skips_non_positioned():
positioned = {"format":"x","items":[{"status":"epub-text-not-found","clipping":{"id":"a"},"zotero":{},"annotation":{}}]} positioned = {"format":"x","items":[{"status":"epub-text-not-found","clipping":{"id":"a"},"zotero":{},"annotation":{}}]}

View file

@ -28,10 +28,206 @@ def test_pdf_fallback_tag():
"format":"x", "format":"x",
"items":[ "items":[
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None}, {"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"}, "zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1","related_items":[{"item_id":20,"key":"P2","citation_key":"c2"}]},
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}} "annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
] ]
} }
plan = build_final_writer_plan(positioned) plan = build_final_writer_plan(positioned)
tags = plan["annotations"][0]["annotation"]["tags"] tags = plan["annotations"][0]["annotation"]["tags"]
assert any(t["name"].startswith("kindle-id:") for t in tags) assert any(t["name"].startswith("kindle-id:") for t in tags)
assert plan["annotations"][0]["related_items"][0]["item_id"] == 20
def test_pdf_tolerates_of_split_artifact():
from kindle_zotero_importer.pdf_position import find_pdf_text_page, find_pdf_rects, PDFPageXML, PDFWord
quote = "nothing other than idolatry of the machine and which"
page_text = "nothing other than idolatry o f the machine and which"
assert find_pdf_text_page([page_text], quote) == 0
words = []
cursor = 0
for w in page_text.split():
words.append(PDFWord(text=w, left=0.0, top=0.0, width=10.0, height=10.0, start=cursor, end=cursor + len(w)))
cursor += len(w) + 1
page = PDFPageXML(index=0, width=100.0, height=100.0, words=words)
rects = find_pdf_rects("dummy.pdf", page, quote, {("dummy.pdf", 0): (100.0, 100.0)})
assert rects is not None
assert rects["pageIndex"] == 0
assert rects["rects"]
def test_pdf_tolerates_punctuation_and_quotes():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
quote = "This, then, gives rise to “intemperate technicism”."
page = "this then gives rise to intemperate technicism"
assert find_pdf_text_page([page], quote) == 0
def test_pdf_partial_half_anchors_spanning_text():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
first = "A" * 70
second = "Z" * 70
quote = first + second
page = "xx " + first.lower() + " yy"
assert find_pdf_text_page([page], quote) == 0
def test_pdf_shingle_fallback_scattered_ocr_errors():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
quote = (
"Gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
"buna inanmamasım sağlayacak güç ve yeteneği bulur mucize yadsınaraz "
"dururda bile olsa boyun eğmez"
)
page = (
"gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
"buna inanmaması sağlayacak güç ve yeteneği bulur mucize yadsınamaz "
"dururda bile olsa boyun eğmez"
)
assert find_pdf_text_page([page], quote) == 0
def test_page_hint_snippet_around_kindle_page():
from kindle_zotero_importer.pdf_position import _page_hint_text
pages = ["", "intro words here", "target page words here", "trailing words"]
hint = _page_hint_text(pages, "3")
assert hint is not None
assert hint["kindle_page"] == "3"
assert "target page words" in hint["text"]
assert _page_hint_text(pages, None) is None
assert _page_hint_text(pages, "99") is None
def test_citekey_mentions_become_zotero_links():
from kindle_zotero_importer.import_plan import build_import_plan
clippings_payload = {
"format": "x",
"clippings": [
{
"id": "c1",
"title": "Book A",
"kind": "highlight",
"text": "As argued by @bookb2020, this follows.",
"page": None,
"location": "10",
}
],
}
zotero_index = {
"format": "x",
"items": [
{
"item_id": 1,
"key": "PARENT01",
"title": "Book A",
"fields": {"citationKey": "booka2020"},
"attachments": [
{
"item_id": 11,
"key": "ATTACH01",
"title": "pdf",
"content_type": "application/pdf",
"path": "x.pdf",
"resolved_path": "x.pdf",
}
],
},
{
"item_id": 2,
"key": "TARGET02",
"title": "Book B",
"fields": {"citationKey": "bookb2020"},
"attachments": [],
},
],
}
match_report = {
"format": "x",
"matches": [
{
"clipping_title": "Book A",
"status": "matched",
"candidates": [{"item_id": 1}],
}
],
}
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
comment = plan["items"][0]["annotation"]["comment"] or ""
assert "@bookb2020" in comment
assert "zotero://select/library/items/TARGET02" in comment
assert plan["items"][0]["zotero"]["related_items"] == [
{
"item_id": 2,
"key": "TARGET02",
"citation_key": "bookb2020",
"link": "zotero://select/library/items/TARGET02",
}
]
# self-links are skipped
clippings_payload["clippings"][0]["text"] = "See @booka2020 here."
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
assert "zotero://select" not in (plan["items"][0]["annotation"]["comment"] or "")
assert plan["items"][0]["zotero"]["related_items"] == []
def test_historical_annotations_gain_citekey_relations():
from kindle_zotero_importer.cli import _refresh_history_relations
history = [
{
"clipping_id": "old",
"parent_item_id": 1,
"annotation": {
"text": "An old highlight",
"comment": "body without organs @deleuze1983",
},
}
]
zotero_index = {
"items": [
{
"item_id": 1,
"key": "SOURCE01",
"fields": {"citationKey": "simondon2017a"},
},
{
"item_id": 2,
"key": "TARGET02",
"fields": {"citationKey": "deleuze1983"},
},
]
}
_refresh_history_relations(history, zotero_index)
assert history[0]["related_items"] == [
{
"item_id": 2,
"key": "TARGET02",
"citation_key": "deleuze1983",
"link": "zotero://select/library/items/TARGET02",
}
]
def test_group_library_targets_use_group_select_links():
from kindle_zotero_importer.import_plan import _select_link
assert (
_select_link({"key": "ABC123", "group_id": 6644344})
== "zotero://select/groups/6644344/items/ABC123"
)
assert (
_select_link({"key": "ABC123", "group_id": None})
== "zotero://select/library/items/ABC123"
)
assert (
_select_link({"key": "ABC123"})
== "zotero://select/library/items/ABC123"
)
def test_epub_tolerates_extraction_artifacts():
from kindle_zotero_importer.epub_position import find_epub_cfi, TextNode
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="idolatry o f the machine", start=0, end=24)]
quote = "idolatry of the machine and which"
# quote is longer than node, so exact fails; use full haystack variant
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="nothing other than idolatry o f the machine and which", start=0, end=53)]
result = find_epub_cfi(nodes, quote)
assert result is not None
assert result["selector"]["value"].startswith("epubcfi(")

View file

@ -1,5 +1,22 @@
import json, tempfile, pathlib import json
from pathlib import Path from pathlib import Path
import tempfile
from kindle_zotero_importer import cli
from kindle_zotero_importer.cli import (
_add_previous_imported_comments,
_changed_note_clipping_ids,
_changed_override_titles,
_clipping_ids_for_titles,
_deferred_clipping_ids,
_deletion_ids,
_discarded_clipping_ids,
_merge_annotation_history,
_merge_positioned_history,
_unresolved_clipping_ids,
)
from kindle_zotero_importer.clippings import clippings_to_jsonable, load_clippings
from kindle_zotero_importer import pdf_position
def test_incremental_ids(tmp_path: Path): def test_incremental_ids(tmp_path: Path):
# simulate previous final with one integrated id # simulate previous final with one integrated id
@ -17,3 +34,324 @@ def test_incremental_ids(tmp_path: Path):
# changed: aaa text changed -> new id ddd, old aaa no longer in new -> deletion # changed: aaa text changed -> new id ddd, old aaa no longer in new -> deletion
new_ids2 = {"ddd","bbb","ccc"} new_ids2 = {"ddd","bbb","ccc"}
assert (prev_integrated - new_ids2) == {"aaa"} assert (prev_integrated - new_ids2) == {"aaa"}
def test_merge_annotation_history_preserves_old_and_replaces_changed():
old_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "2"}}
old_b = {"clipping_id": "b", "attachment_item_id": 1, "annotation": {"sortIndex": "1"}}
new_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "3"}}
merged = _merge_annotation_history(
{"all_annotations": [old_a, old_b]}, [new_a], []
)
assert merged == [old_b, new_a]
def test_merge_annotation_history_removes_deletions():
previous = {
"annotations": [
{"clipping_id": "a", "attachment_item_id": 1, "annotation": {}},
{"clipping_id": "b", "attachment_item_id": 1, "annotation": {}},
]
}
merged = _merge_annotation_history(previous, [], ["a"])
assert [annotation["clipping_id"] for annotation in merged] == ["b"]
def test_full_history_retains_current_unresolved_annotation():
previous = {
"all_annotations": [
{"clipping_id": "unresolved", "attachment_item_id": 1, "annotation": {}},
{"clipping_id": "removed", "attachment_item_id": 1, "annotation": {}},
]
}
merged = _merge_annotation_history(previous, [], ["removed"])
assert [annotation["clipping_id"] for annotation in merged] == ["unresolved"]
def test_deletions_include_removed_and_previous_failures():
assert _deletion_ids(
{"kept", "removed"},
{"kept"},
{"delete_failed_ids": ["retry"]},
) == {"removed", "retry"}
def test_deferred_clippings_are_reprocessed_while_they_remain_in_source():
previous = {"deferred_clipping_ids": ["retry", "removed"]}
assert _deferred_clipping_ids(previous, {"retry", "current"}) == {"retry"}
def test_discarded_clips_never_requeue_and_skips_are_not_unresolved():
previous = {"discarded_clipping_ids": ["gone", "removed"]}
assert _discarded_clipping_ids(previous, {"gone", "current"}) == {"gone"}
plan = {
"items": [
{"clipping": {"id": "note"}, "status": "pdf-position-skipped-non-highlight"},
{"clipping": {"id": "fail"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "done"}, "status": "positioned"},
]
}
assert _unresolved_clipping_ids(plan, {"note", "fail", "done"}) == {"fail"}
def test_alternate_attachment_retry_prefers_pdf_when_epub_misses():
from kindle_zotero_importer.cli import _reattempt_failed_with_alternate_attachments
from kindle_zotero_importer.epub_position import add_epub_positions
from kindle_zotero_importer.import_plan import build_import_plan
from kindle_zotero_importer.pdf_position import add_pdf_positions
fixtures = Path(__file__).parent / "fixtures" / "windows-smoke"
epub_path = str(fixtures / "kanzi-dummy.epub")
pdf_path = str(fixtures / "kanzi-dummy.pdf")
quote = "This is a test highlight long enough to import for PDF."
clippings_payload = {
"format": "kindle-zotero-importer.clippings.v1",
"clippings": [
{
"id": "q1",
"title": "Dummy Book",
"kind": "highlight",
"text": quote,
"page": None,
"location": "100",
}
],
}
zotero_index = {
"format": "index",
"items": [
{
"item_id": 1,
"key": "ABC12345",
"title": "Dummy Book",
"fields": {},
"attachments": [
{
"item_id": 11,
"key": "EPUBKEY1",
"title": "epub",
"content_type": "application/epub+zip",
"path": epub_path,
"resolved_path": epub_path,
},
{
"item_id": 12,
"key": "PDFKEY123",
"title": "pdf",
"content_type": "application/pdf",
"path": pdf_path,
"resolved_path": pdf_path,
},
],
}
],
}
match_report = {
"format": "matches",
"matches": [
{
"clipping_title": "Dummy Book",
"status": "matched",
"candidates": [{"item_id": 1}],
}
],
}
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
assert plan["items"][0]["zotero"]["attachment"]["content_type"] == "application/epub+zip"
positioned = add_pdf_positions(add_epub_positions(plan))
assert positioned["items"][0]["status"] == "positioned"
assert "partial-text-match" in positioned["items"][0]["problems"]
rescued = _reattempt_failed_with_alternate_attachments(positioned)
assert rescued["items"][0]["status"] == "positioned"
assert "positioned-via-alternate-attachment" in rescued["items"][0]["problems"]
assert "partial-text-match" not in rescued["items"][0]["problems"]
def test_changed_override_requeues_every_clipping_for_exact_title():
previous = {"Shared title": {"citation_key": "old"}}
current = {"Shared title": {"citation_key": "new"}}
clippings = [
{"id": "first", "title": "Shared title"},
{"id": "second", "title": "Shared title"},
{"id": "third", "title": "Different title"},
]
changed_titles = _changed_override_titles(previous, current)
assert changed_titles == {"Shared title"}
assert _clipping_ids_for_titles(clippings, changed_titles) == {"first", "second"}
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
previous = {
"all_annotations": [
{"clipping_id": "a", "annotation": {"comment": "old"}}
]
}
_add_previous_imported_comments(current, previous)
assert current[0]["previous_imported_comment"] == "old"
def test_retry_unresolved_excludes_positioned_ignored_and_removed_entries():
previous = {
"items": [
{"clipping": {"id": "retry"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "done"}, "status": "positioned"},
{"clipping": {"id": "ignored"}, "status": "ignored-title"},
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
]
}
assert _unresolved_clipping_ids(previous, {"retry", "done", "ignored"}) == {
"retry"
}
def test_changed_note_ids_detect_add_replace_and_remove():
previous = [
{"clipping_id": "added", "note_ids": []},
{"clipping_id": "replaced", "note_ids": ["old"]},
{"clipping_id": "removed", "note_ids": ["old"]},
{"clipping_id": "same", "note_ids": ["same-note"]},
]
current = [
{"id": "added", "note_ids": ["new"]},
{"id": "replaced", "note_ids": ["new"]},
{"id": "removed"},
{"id": "same", "note_ids": ["same-note"]},
]
assert _changed_note_clipping_ids(current, previous) == {
"added",
"replaced",
"removed",
}
def test_positioned_history_preserves_unresolved_and_replaces_current():
previous = {
"items": [
{"clipping": {"id": "keep"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "replace"}, "status": "epub-text-not-found"},
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
]
}
current = {
"format": "plan",
"items": [{"clipping": {"id": "replace"}, "status": "positioned"}],
}
merged = _merge_positioned_history(previous, current, {"keep", "replace"})
assert {item["clipping"]["id"]: item["status"] for item in merged["items"]} == {
"keep": "pdf-text-not-found",
"replace": "positioned",
}
assert merged["status_counts"] == {"pdf-text-not-found": 1, "positioned": 1}
def test_pdf_position_reports_missing_tools():
original = pdf_position._missing_required_tools
pdf_position._missing_required_tools = lambda: ["pdftotext", "pdfinfo"]
try:
result = pdf_position.add_pdf_positions(
{
"items": [
{
"status": "ready-for-positioning",
"clipping": {"kind": "highlight", "text": "highlight"},
"zotero": {
"attachment": {"content_type": "application/pdf"}
},
}
]
}
)
finally:
pdf_position._missing_required_tools = original
assert result["items"][0]["status"] == "pdf-tools-missing:pdftotext,pdfinfo"
def test_pdf_position_preserves_unresolved_item_without_zotero_match():
item = {
"status": "unmatched-title",
"clipping": {"kind": "highlight", "text": "highlight"},
"zotero": None,
}
result = pdf_position.add_pdf_positions({"items": [item]})
assert result["items"] == [item]
def test_noop_incremental_writes_complete_artifacts():
with tempfile.TemporaryDirectory() as temporary:
workdir = Path(temporary)
source = workdir / "My Clippings.txt"
source.write_text(
"Book\n- Your Highlight at location 10-11 | Added on Tuesday, 18 May 2021 14:18:13\n\n"
"A highlight long enough to import\n==========\n",
encoding="utf-8",
)
clippings = clippings_to_jsonable(load_clippings(str(source)))
clipping_id = clippings["clippings"][0]["id"]
(workdir / "clippings.json").write_text(json.dumps(clippings), encoding="utf-8")
annotation = {
"clipping_id": clipping_id,
"attachment_item_id": 1,
"annotation": {"sortIndex": "1"},
}
(workdir / "import-plan.final.json").write_text(
json.dumps({"annotations": [annotation], "all_annotations": [annotation], "overrides_snapshot": {}}),
encoding="utf-8",
)
originals = (
cli.build_zotero_index,
cli.build_match_report,
cli.generate_override_skeleton,
cli.build_mismatch_review,
)
cli.build_zotero_index = lambda *_args: {"format": "index", "items": []}
cli.build_match_report = lambda *_args: {"format": "matches", "matches": []}
cli.generate_override_skeleton = lambda *_args: {"format": "overrides", "overrides": []}
cli.build_mismatch_review = lambda *_args: "review\n"
try:
result = cli.main(
[
"run",
str(source),
"--workdir",
str(workdir),
"--db",
str(workdir / "zotero.sqlite"),
"--storage-root",
str(workdir / "storage"),
]
)
finally:
(
cli.build_zotero_index,
cli.build_match_report,
cli.generate_override_skeleton,
cli.build_mismatch_review,
) = originals
assert result == 0
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
assert final_plan["annotations"] == []
assert final_plan["all_annotations"] == [{**annotation, "related_items": []}]
assert final_plan["delta_annotation_count"] == 0
assert (workdir / "import-plan.epub.json").exists()

View file

@ -1,14 +1,711 @@
import assert from 'assert'; const assert = require('assert');
// test the grouping logic for title variants sharing same candidate const fs = require('fs');
function findOtherRows(rows, rowIndex, targetKey) { const vm = require('vm');
const row = rows[rowIndex];
return rows.filter((r, idx) => idx !== rowIndex && (r.candidates||[]).some(c => (c.citation_key||c.key||String(c.item_id))===targetKey)); const bootstrapContext = {};
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
const plugin = bootstrapContext.Kanzi;
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
assert.equal(
plugin.mergeComments('[r] this is a red note\n\nthis is a red note', ''),
'this is a red note'
);
assert.equal(plugin.mergeComments('[o]', ''), '');
assert.equal(plugin.mergeComments('[citation] keep this', ''), '[citation] keep this');
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'old note', annotationTags: [{ name: 'kindle-import' }] },
'replacement note',
'old note'
),
'replacement note'
);
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'old note\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
'replacement note',
'old note'
),
'manual addition\n\nreplacement note'
);
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'old first\n\nold second\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
'new note',
'old first\n\nold second'
),
'manual addition\n\nnew note'
);
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'manual note', annotationTags: [] },
'imported note'
),
'manual note\n\nimported note'
);
const base = {
annotationText: 'forged its might, should be enough to discredit',
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1255)',
}),
};
const extended = {
annotationText: 'forged its might, should be enough to discredit, even in the eyes of philosophers',
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1300)',
}),
};
assert.equal(plugin.areImportedDuplicates(base, extended), true);
assert.equal(
plugin.areImportedDuplicates(base, {
...base,
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
}),
}),
false
);
assert.equal(
plugin.areImportedDuplicates(base, {
...base,
annotationText: 'OCR-corrected text at the identical annotation position',
}),
true
);
assert.equal(
plugin.areImportedDuplicates(
{
annotationText: 'A multiline PDF selection long enough to compare',
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30]] }),
},
{
annotationText: 'A multiline PDF selection long enough to compare with an extended ending',
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30], [10, 40, 120, 50]] }),
} }
const rows = [ ),
{title:"gilbert-simondon-on-the-mode", candidates:[{citation_key:"chabot2013", key:"A"}, {citation_key:"simondon2017a"}]}, true
{title:"On the Mode (Univocal)", candidates:[{citation_key:"chabot2013"}, {citation_key:"simondon2017a"}]}, );
{title:"Other", candidates:[{citation_key:"foo"}]}
const managerHtml = fs.readFileSync('plugin/manager.html', 'utf8');
const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
const managerContext = { TextEncoder };
vm.runInNewContext(scripts[0][1], managerContext);
const manager = managerContext.KindleManager;
assert.equal(plugin.countClippingsForTitle({ clippings: [
{ title: 'Shared title' },
{ title: 'Shared title' },
{ title: 'Different title' },
] }, 'Shared title'), 2);
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
{ id: 'n1', title: 'Shared title', kind: 'note', text: 'note' },
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
] },
matches: { matches: [{
clipping_title: 'Shared title',
clipping_count: 3,
status: 'matched',
candidates: [],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [{
clipping: { id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
status: 'pdf-text-not-found',
problems: ['not found'],
}] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const conflictRows = manager.conflictRows();
assert.equal(conflictRows.length, 1);
assert.equal(conflictRows[0].count, 3);
assert.equal(conflictRows[0].clippings.length, 3);
assert.equal(conflictRows[0].highlights, 2);
assert.equal(conflictRows[0].notes, 1);
manager.sortState.conflicts = { column: 'count', direction: 'desc' };
const sortedRows = manager.sortRows(
[{ title: 'small', count: 2 }, { title: 'large', count: 20 }, { title: 'middle', count: 10 }],
'conflicts',
{ count: row => row.count }
);
assert.deepEqual(Array.from(sortedRows, row => row.title), ['large', 'middle', 'small']);
manager.integratedPage = 1;
manager.integratedPageSize = 50;
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
assert.equal(page.totalPages, 3);
assert.deepEqual(Array.from(page.rows), Array.from({ length: 50 }, (_, index) => index + 50));
manager.mappingsPage = 99;
manager.mappingsPageSize = 50;
const finalPage = manager.paginate(Array.from({ length: 51 }, (_, index) => index), 'mappingsPage', 'mappingsPageSize');
assert.equal(manager.mappingsPage, 1);
assert.deepEqual(Array.from(finalPage.rows), [50]);
const collectorBatches = manager.collectorBatches('installation', Array.from(
{ length: 450 },
(_, index) => ({ clipping_id_hash: String(index), text: 'x'.repeat(2000) })
));
assert.equal(collectorBatches.length, 3);
assert.ok(collectorBatches.every(batch => batch.length <= 200));
assert.ok(collectorBatches.every(batch => new TextEncoder().encode(
JSON.stringify({ anonId: 'installation', data: batch })
).length <= 1500000));
assert.throws(
() => manager.collectorBatches('installation', [{ text: 'x'.repeat(1500000) }]),
/exceeds the collector request limit/
);
async function testRuntimeExtraction() {
const writes = [];
bootstrapContext.PathUtils = {
profileDir: '/profile',
join: (...parts) => parts.join('/').replaceAll('//', '/'),
parent: path => path.slice(0, path.lastIndexOf('/')),
};
bootstrapContext.IOUtils = {
makeDirectory: async () => {},
};
bootstrapContext.Zotero = {
File: { putContentsAsync: async (path, text) => writes.push([path, text]) },
};
bootstrapContext.fetch = async url => ({ ok: true, text: async () => url });
plugin.rootURI = 'jar:file:///kanzi.xpi!/';
await plugin.ensureRuntime();
assert.equal(plugin.runtimeProjectDir, '/profile/kanzi-runtime');
assert.equal(writes.length, 13);
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
}
async function testPythonPathRecovery() {
bootstrapContext.Services = { appinfo: { OS: 'Darwin' } };
bootstrapContext.IOUtils = {
exists: async path => path === '/opt/homebrew/opt/python@3.13/bin/python3.13',
};
assert.equal(
await plugin.resolvePythonPath('/opt/homebrew/bin/python3'),
'/opt/homebrew/opt/python@3.13/bin/python3.13'
);
bootstrapContext.IOUtils.exists = async path => path === '/custom/python3';
assert.equal(await plugin.resolvePythonPath('/custom/python3'), '/custom/python3');
bootstrapContext.Services.appinfo.OS = 'WINNT';
const windowsEnvironment = {
LOCALAPPDATA: 'C:\\Users\\test\\AppData\\Local',
ProgramFiles: 'C:\\Program Files',
'ProgramFiles(x86)': 'C:\\Program Files (x86)',
USERPROFILE: 'C:\\Users\\test',
SystemDrive: 'C:',
PATH: 'C:\\Tools\\Python;C:\\Windows\\System32',
};
plugin.getEnvironmentVariable = name => windowsEnvironment[name] || '';
const expected = 'C:\\Users\\test\\AppData\\Local\\Programs\\Python\\Python313\\python.exe';
bootstrapContext.IOUtils.exists = async path => path === expected;
assert.equal(await plugin.resolvePythonPath('python'), expected);
assert.ok(plugin.getWindowsPythonCandidates('python').includes('C:\\Tools\\Python\\python.exe'));
}
async function testMaintenanceSweep() {
const annotation = (id, text, end, comment = '') => ({
id,
parentItemID: 10,
annotationText: text,
annotationComment: comment,
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: `epubcfi(/6/12!/4/2/8,/1:795,/1:${end})`,
}),
saved: 0,
erased: 0,
saveTx: async function () { this.saved += 1; },
eraseTx: async function () { this.erased += 1; },
});
const short = annotation(1, 'A progressive highlight long enough to compare', 994);
const middle = annotation(2, 'A progressive highlight long enough to compare and then extended', 1093);
const longest = annotation(
3,
'A progressive highlight long enough to compare and then extended to its final form',
1141,
'[r] useful note\n\nuseful note'
);
const separate = annotation(4, 'A progressive highlight long enough to compare', 2050);
separate.annotationPosition = JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
});
const emptyComment = annotation(5, 'A separate highlight with a null comment', 3050, null);
emptyComment.annotationPosition = JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:3000,/1:3050)',
});
const results = { updatedComments: 0, removedDuplicates: 0 };
await plugin.maintainImportedAnnotations([short, middle, longest, separate, emptyComment], false, results);
assert.equal(results.removedDuplicates, 2);
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
assert.equal(short.erased, 1);
assert.equal(middle.erased, 1);
assert.equal(longest.erased, 0);
assert.equal(separate.erased, 0);
assert.equal(emptyComment.saved, 0);
assert.equal(longest.annotationComment, 'useful note');
assert.equal(results.updatedComments, 1);
}
async function testWriterReconciliation() {
let written = null;
bootstrapContext.Zotero = {
File: { putContentsAsync: async (_path, text) => { written = JSON.parse(text); } },
};
const plan = {
annotations: [{ clipping_id: 'ok' }, { clipping_id: 'failed' }],
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'ok' }, { clipping_id: 'failed' }],
};
await plugin.reconcileWriterState('/plan.json', plan, {
failed: [{ clipping_id: 'failed' }, { clipping_id: 'maintenance' }],
failedDeletionIds: ['delete-me'],
maintenanceDeletedIds: ['old'],
});
assert.deepEqual(Array.from(written.write_failed_ids), ['failed']);
assert.deepEqual(Array.from(written.annotations, entry => entry.clipping_id), ['ok']);
assert.deepEqual(Array.from(written.all_annotations, entry => entry.clipping_id), ['ok']);
assert.equal(written.annotation_count, 1);
assert.deepEqual(Array.from(written.delete_failed_ids), ['delete-me']);
}
async function testDeletionFailureIsRetried() {
bootstrapContext.Zotero = { logError: () => {} };
const annotation = {
annotationTags: [{ name: 'kindle-id:delete-me' }],
eraseTx: async () => { throw new Error('locked'); },
};
plugin.collectImportedAnnotations = async () => [annotation];
plugin.maintainImportedAnnotations = async () => {};
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['delete-me'],
}, false);
assert.deepEqual(Array.from(results.failedDeletionIds), ['delete-me']);
assert.equal(results.failed.length, 1);
}
async function testMaintenanceDeletionIsNotRepeated() {
bootstrapContext.Zotero = { logError: () => {} };
const annotation = {
annotationTags: [{ name: 'kindle-id:already-deleted' }],
eraseTx: async () => { throw new Error('must not be called again'); },
};
plugin.collectImportedAnnotations = async () => [annotation];
plugin.maintainImportedAnnotations = async (_annotations, _dryRun, results) => {
results.maintenanceDeletedIds.push('already-deleted');
};
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['already-deleted'],
}, false);
assert.deepEqual(Array.from(results.failedDeletionIds), []);
assert.equal(results.failed.length, 0);
}
async function testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor() {
bootstrapContext.Zotero = { logError: () => {} };
const active = {
parentItemID: 10,
annotationText: 'The longer OCR-corrected duplicate annotation text',
annotationComment: '',
annotationPosition: JSON.stringify({ pageIndex: 1, rects: [[1, 2, 3, 4]] }),
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
erased: 0,
eraseTx: async function () { this.erased += 1; },
};
const duplicate = {
...active,
annotationText: 'The duplicate annotation text',
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
erased: 0,
eraseTx: async function () { this.erased += 1; },
};
let collection = 0;
plugin.collectImportedAnnotations = async () => (++collection === 1 ? [active, duplicate] : [active]);
plugin.maintainImportedAnnotations = maintainImportedAnnotations;
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['same'],
}, false);
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
assert.equal(duplicate.erased, 1);
assert.equal(active.erased, 1);
assert.equal(results.deletedForIncremental, 1);
assert.equal(results.failed.length, 0);
}
async function testEnumerationFailurePersistsAllDeletions() {
bootstrapContext.Zotero = { logError: () => {} };
plugin.collectImportedAnnotations = async () => { throw new Error('enumeration failed'); };
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['retry-deletion'],
}, false);
assert.deepEqual(Array.from(results.failedDeletionIds), ['retry-deletion']);
assert.equal(results.failed[0].clipping_id, 'maintenance');
}
async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
const old = {
annotationTags: [{ name: 'kindle-id:replacement' }],
erased: 0,
eraseTx: async function () { this.erased += 1; },
};
bootstrapContext.Zotero = {
logError: () => {},
Items: { get: () => null },
};
plugin.collectImportedAnnotations = async () => [old];
plugin.maintainImportedAnnotations = async () => {};
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [{ clipping_id: 'replacement', attachment_item_id: 99, annotation: {} }],
deletions: ['replacement'],
}, false);
assert.equal(old.erased, 0);
assert.deepEqual(Array.from(results.failedDeletionIds), ['replacement']);
assert.equal(results.failed[0].clipping_id, 'replacement');
}
async function testCitekeyRelationsAreBidirectionalAndAdditive() {
const originalZotero = bootstrapContext.Zotero;
const originalCollect = plugin.collectImportedAnnotations;
const originalMaintain = plugin.maintainImportedAnnotations;
const item = (id, key) => ({
id,
key,
related: new Set(),
saved: 0,
addRelatedItem(other) {
if (this.related.has(other.key)) return false;
this.related.add(other.key);
return true;
},
async save(options) {
assert.equal(options.skipDateModifiedUpdate, true);
this.saved += 1;
},
});
const source = item(1, 'SOURCE01');
const target = item(2, 'TARGET02');
bootstrapContext.Zotero = {
Items: { get: id => id === 1 ? source : id === 2 ? target : id === 11 ? attachment : null },
DB: { executeTransaction: async callback => callback() },
logError: () => {},
};
const entries = [
{
clipping_id: 'history-1',
parent_item_id: 1,
attachment_item_id: 11,
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
},
{
parent_item_id: 1,
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
},
]; ];
assert.equal(findOtherRows(rows, 0, "chabot2013").length, 1); const existingAnnotation = {
assert.equal(findOtherRows(rows, 0, "chabot2013")[0].title, "On the Mode (Univocal)"); annotationComment: 'an old note',
console.log("manager logic ok"); annotationTags: [{ name: 'kindle-id:history-1' }],
saved: 0,
saveTx: async function () { this.saved += 1; },
};
const attachment = { getAnnotations: () => [existingAnnotation] };
const results = {
relationsAdded: 0,
relationsSkipped: 0,
relationsPlanned: 0,
relationFailures: [],
};
try {
plugin.collectImportedAnnotations = async () => [];
plugin.maintainImportedAnnotations = async () => {};
const writerResults = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
all_annotations: entries,
deletions: [],
}, false);
assert.deepEqual(Array.from(source.related), ['TARGET02']);
assert.deepEqual(Array.from(target.related), ['SOURCE01']);
assert.equal(source.saved, 1);
assert.equal(target.saved, 1);
assert.equal(writerResults.relationsAdded, 1);
assert.equal(writerResults.relationFailures.length, 0);
assert.equal(writerResults.commentLinksAdded, 1);
assert.ok(existingAnnotation.annotationComment.includes(
'Related: @target2020 zotero://select/library/items/TARGET02'
));
assert.equal(existingAnnotation.saved, 1);
await plugin.addRelatedItemRelations(entries, false, results);
assert.equal(source.saved, 1);
assert.equal(target.saved, 1);
assert.equal(results.relationsSkipped, 1);
} finally {
bootstrapContext.Zotero = originalZotero;
plugin.collectImportedAnnotations = originalCollect;
plugin.maintainImportedAnnotations = originalMaintain;
}
}
function testPendingReviewGrouping() {
const groups = manager.groupPendingAnnotations([
{ clipping_id: 'a1', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'spice', pageLabel: '12', color: '#ffd400' } },
{ clipping_id: 'a2', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'worm', pageLabel: '', color: '#ff6666' } },
{ clipping_id: 'b1', clipping_title: 'Unknown Book', citation_key: '', annotation: { text: 'mystery', pageLabel: '', color: '#ffd400' } },
]);
assert.equal(groups.length, 2);
assert.equal(groups[0].items.length, 2);
assert.equal(groups[0].citationKey, 'herbert2020');
assert.equal(groups[1].items.length, 1);
assert.deepEqual(manager.groupPendingAnnotations([]), []);
}
function testFilterPlanForSelection() {
const plan = {
annotations: [{ clipping_id: 'keep' }, { clipping_id: 'skip' }],
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'keep' }, { clipping_id: 'skip' }],
};
const previous = { deferred_clipping_ids: ['older'] };
const filtered = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep'], previous);
assert.deepEqual(Array.from(filtered.annotations, a => a.clipping_id), ['keep']);
assert.deepEqual(Array.from(filtered.all_annotations, a => a.clipping_id), ['old', 'keep']);
assert.deepEqual(Array.from(filtered.deferred_clipping_ids), ['older', 'skip']);
// Original plan is not mutated.
assert.equal(plan.annotations.length, 2);
const all = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep', 'skip'], previous);
assert.equal(all.annotations.length, 2);
assert.equal(all.all_annotations.length, 3);
assert.deepEqual(Array.from(all.deferred_clipping_ids), ['older']);
const priorAnnotation = { clipping_id: 'skip', annotation: { text: 'existing' } };
const preserved = plugin.filterPlanForSelection(
plan,
['keep', 'skip'],
['keep'],
{ all_annotations: [priorAnnotation] }
);
assert.equal(preserved.all_annotations.find(a => a.clipping_id === 'skip'), priorAnnotation);
const deferred = plugin.deferPendingPlan(
{ all_annotations: [{ clipping_id: 'old' }], deferred_clipping_ids: ['earlier'] },
plan,
['keep', 'skip']
);
assert.deepEqual(Array.from(deferred.deferred_clipping_ids), ['earlier', 'keep', 'skip']);
assert.deepEqual(Array.from(deferred.all_annotations, a => a.clipping_id), ['old']);
}
function testTitleCountsAreAuthoritative() {
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
{ id: 'h3', title: 'Shared title', kind: 'highlight', text: 'third' },
] },
matches: { matches: [{
clipping_title: 'Shared title',
clipping_count: 99,
status: 'ambiguous',
candidates: [{ citation_key: 'shared2020', key: '', item_id: 1, title: 'Shared', score: 0.9 }],
}] },
generatedOverrides: { overrides: [{
clipping_title: 'Shared title',
review: { status: 'review', clipping_count: 1 },
candidates: [],
}] },
positionedPlan: { items: [{
clipping: { id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
status: 'pdf-text-not-found',
problems: ['not found'],
}] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const rows = manager.conflictRows();
assert.equal(rows.length, 1);
// Source has 3 clippings; misleading match (99) and override (1) counts must not win.
assert.equal(rows[0].count, 3);
assert.equal(rows[0].clippings.length, 3);
// Only 1 of 3 has position problems.
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 3 });
// Without clippings.json, fall back to conflict counts.
manager.data = {
clippings: null,
matches: { matches: [{
clipping_title: 'Lonely title',
clipping_count: 5,
status: 'ambiguous',
candidates: [],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const fallback = manager.conflictRows();
assert.equal(fallback.length, 1);
assert.equal(fallback[0].count, 5);
}
function testMappedTargetAndDiscardedClips() {
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' },
{ id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' },
] },
matches: { matches: [{
clipping_title: 'Mapped title',
clipping_count: 2,
status: 'matched',
candidates: [{ citation_key: 'mapped2020', key: '', item_id: 7, title: 'Mapped', score: 1 }],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [
{ clipping: { id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' }, status: 'pdf-text-not-found', problems: ['pdf-text-not-found'] },
{ clipping: { id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' }, status: 'pdf-position-skipped-non-highlight', problems: ['pdf-position-skipped-non-highlight'] },
] },
finalPlan: { all_annotations: [], discarded_clipping_ids: [] },
persistentOverrides: { overrides: [{ clipping_title: 'Mapped title', resolution: { citation_key: 'mapped2020' } }] },
};
let rows = manager.conflictRows();
assert.equal(rows.length, 1);
assert.equal(rows[0].mappedTarget, 'mapped2020');
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 2, total: 2 });
manager.data.finalPlan.discarded_clipping_ids = ['n1'];
rows = manager.conflictRows();
assert.equal(rows.length, 1);
assert.equal(rows[0].count, 1);
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 1 });
}
async function testDiscardClipPersists() {
const originalZotero = bootstrapContext.Zotero;
let written = null;
const plan = { discarded_clipping_ids: [], deferred_clipping_ids: ['x1'] };
bootstrapContext.Zotero = {
File: {
getContentsAsync: async () => JSON.stringify(plan),
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
},
};
const originalArtifactPath = plugin.artifactPath;
plugin.artifactPath = rel => `/proj/${rel}`;
try {
await plugin.discardClip('x1');
assert.deepEqual(written.discarded_clipping_ids, ['x1']);
assert.deepEqual(written.deferred_clipping_ids, []);
} finally {
plugin.artifactPath = originalArtifactPath;
bootstrapContext.Zotero = originalZotero;
}
}
function testTitleOverviewAndSorting() {
manager.data = {
clippings: { clippings: [
{ id: 'a1', title: 'Alpha', kind: 'highlight', text: 'one' },
{ id: 'a2', title: 'Alpha', kind: 'highlight', text: 'two' },
{ id: 'b1', title: 'Beta', kind: 'highlight', text: 'three' },
] },
matches: { matches: [
{ clipping_title: 'Alpha', clipping_count: 2, status: 'matched', candidates: [{ citation_key: 'alpha2020' }] },
{ clipping_title: 'Beta', clipping_count: 1, status: 'unmatched', candidates: [] },
] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [] },
finalPlan: {
all_annotations: [{ clipping_id: 'a1', clipping_title: 'Alpha' }],
deferred_clipping_ids: ['b1'],
},
persistentOverrides: { overrides: [] },
};
const overview = manager.titleOverviewRows();
assert.equal(overview.length, 2);
const alpha = overview.find(r => r.title === 'Alpha');
const beta = overview.find(r => r.title === 'Beta');
assert.equal(alpha.integrated, 1);
assert.equal(beta.pending, 1);
assert.equal(beta.status, 'pending review');
manager.sortState.titles = { column: 'title', direction: 'desc' };
const sorted = manager.sortRows(overview, 'titles', { title: row => row.title });
assert.deepEqual(Array.from(sorted, row => row.title), ['Beta', 'Alpha']);
}
async function testSaveOverrideUsesSourceCount() {
const originalRead = plugin.readJsonArtifact;
const originalArtifactPath = plugin.artifactPath;
const originalZotero = bootstrapContext.Zotero;
let written = null;
plugin.artifactPath = rel => `/proj/${rel}`;
plugin.readJsonArtifact = async rel => rel === 'clippings.json'
? { clippings: [{ title: 'Shared title' }, { title: 'Shared title' }, { title: 'Shared title' }] }
: null;
bootstrapContext.Zotero = {
File: {
getContentsAsync: async () => JSON.stringify({ format: 'kindle-zotero-importer.match-overrides.v1', overrides: [] }),
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
},
};
try {
const entry = await plugin.saveOverride({
clipping_title: 'Shared title',
clipping_count: 1,
resolution: { citation_key: 'shared2020' },
review: { status: 'pdf-text-not-found', clipping_count: 1 },
});
assert.equal(entry.review.clipping_count, 3);
assert.equal(written.overrides[0].review.clipping_count, 3);
} finally {
plugin.readJsonArtifact = originalRead;
plugin.artifactPath = originalArtifactPath;
bootstrapContext.Zotero = originalZotero;
}
}
testTitleCountsAreAuthoritative();
testTitleOverviewAndSorting();
testMappedTargetAndDiscardedClips();
testPendingReviewGrouping();
testFilterPlanForSelection();
testRuntimeExtraction()
.then(testPythonPathRecovery)
.then(testSaveOverrideUsesSourceCount)
.then(testDiscardClipPersists)
.then(testMaintenanceSweep)
.then(testWriterReconciliation)
.then(testDeletionFailureIsRetried)
.then(testMaintenanceDeletionIsNotRepeated)
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
.then(testEnumerationFailurePersistsAllDeletions)
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
.then(testCitekeyRelationsAreBidirectionalAndAdditive)
.then(() => {
console.log('manager and annotation logic ok');
}).catch(error => {
console.error(error);
process.exitCode = 1;
});

View file

@ -2,6 +2,42 @@
"addons": { "addons": {
"kindlezoteroimporter@utkubilen.de": { "kindlezoteroimporter@utkubilen.de": {
"updates": [ "updates": [
{
"version": "0.7.0",
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.7.0/kanzi.xpi",
"applications": {
"zotero": {
"strict_min_version": "6.999"
}
}
},
{
"version": "0.6.20",
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.6.20/kanzi.xpi",
"applications": {
"zotero": {
"strict_min_version": "6.999"
}
}
},
{
"version": "0.6.19",
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.19/kindle-zotero-importer.xpi",
"applications": {
"zotero": {
"strict_min_version": "6.999"
}
}
},
{
"version": "0.6.18",
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.18/kindle-zotero-importer.xpi",
"applications": {
"zotero": {
"strict_min_version": "6.999"
}
}
},
{ {
"version": "0.6.17", "version": "0.6.17",
"update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.17/kindle-zotero-importer.xpi", "update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.17/kindle-zotero-importer.xpi",

View file

@ -1,156 +0,0 @@
// Run in Zotero via Tools > Developer > Run JavaScript.
// Set PLAN_PATH to the generated import-plan.final.json path.
// Keep DRY_RUN=true until the preview output looks correct.
const PLAN_PATH = "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer/import-plan.final.json";
const DRY_RUN = false;
const text = await Zotero.File.getContentsAsync(PLAN_PATH);
const plan = JSON.parse(text);
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
throw new Error(`Unsupported plan format: ${plan.format}`);
}
const results = {
dryRun: DRY_RUN,
total: plan.annotations.length,
created: 0,
skippedExisting: 0,
updatedComments: 0,
removedDuplicates: 0,
failed: [],
preview: [],
};
const existingByAttachment = new Map();
function normalizePosition(position) {
if (!position) {
return "";
}
if (typeof position === "string") {
try {
return JSON.stringify(JSON.parse(position));
} catch (_error) {
return position;
}
}
return JSON.stringify(position);
}
function annotationFingerprint(annotation) {
return `${annotation.text || ""}\u0000${normalizePosition(annotation.position)}`;
}
function existingAnnotationFingerprint(annotation) {
return `${annotation.annotationText || ""}\u0000${normalizePosition(annotation.annotationPosition)}`;
}
async function getExistingAnnotations(attachment) {
if (!existingByAttachment.has(attachment.id)) {
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
const byFingerprint = new Map();
for (const annotation of annotations) {
const fingerprint = existingAnnotationFingerprint(annotation);
const kept = byFingerprint.get(fingerprint);
if (!kept) {
byFingerprint.set(fingerprint, annotation);
continue;
}
const keptHasComment = Boolean(kept.annotationComment);
const currentHasComment = Boolean(annotation.annotationComment);
const duplicate = keptHasComment || !currentHasComment ? annotation : kept;
const replacement = duplicate === annotation ? kept : annotation;
byFingerprint.set(fingerprint, replacement);
if (!DRY_RUN) {
await duplicate.eraseTx();
}
results.removedDuplicates += 1;
}
existingByAttachment.set(attachment.id, byFingerprint);
}
return existingByAttachment.get(attachment.id);
}
function mergeComments(existingComment, newComment) {
const seen = new Set();
const merged = [];
for (const comment of [existingComment, newComment]) {
for (const part of String(comment || "").split(/\n\s*\n/)) {
const cleaned = part.trim();
if (!cleaned || seen.has(cleaned)) {
continue;
}
seen.add(cleaned);
merged.push(cleaned);
}
}
return merged.join("\n\n");
}
for (const entry of plan.annotations) {
try {
const attachment = Zotero.Items.get(entry.attachment_item_id);
if (!attachment) {
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
}
const annotation = {
key: Zotero.DataObjectUtilities.generateKey(),
...entry.annotation,
};
if (!annotation.sortIndex) {
delete annotation.sortIndex;
}
const existingAnnotations = await getExistingAnnotations(attachment);
const fingerprint = annotationFingerprint(annotation);
const existingAnnotation = existingAnnotations.get(fingerprint);
if (existingAnnotation) {
const mergedComment = mergeComments(existingAnnotation.annotationComment, annotation.comment);
if (mergedComment && existingAnnotation.saveTx && existingAnnotation.annotationComment !== mergedComment) {
existingAnnotation.annotationComment = mergedComment;
if (!DRY_RUN) {
await existingAnnotation.saveTx();
}
results.updatedComments += 1;
}
results.skippedExisting += 1;
continue;
}
if (DRY_RUN) {
results.preview.push({
clipping_id: entry.clipping_id,
attachment_item_id: entry.attachment_item_id,
type: annotation.type,
text: annotation.text.slice(0, 120),
pageLabel: annotation.pageLabel,
sortIndex: annotation.sortIndex,
position: annotation.position,
});
continue;
}
await Zotero.Annotations.saveFromJSON(attachment, annotation);
existingAnnotations.set(fingerprint, {
annotationComment: annotation.comment || "",
annotationPosition: annotation.position,
annotationText: annotation.text || "",
});
results.created += 1;
} catch (error) {
results.failed.push({
clipping_id: entry.clipping_id,
attachment_item_id: entry.attachment_item_id,
message: String(error),
});
}
}
return JSON.stringify(results, null, 2);