Kanzi 0.7.0: safe incremental imports and collector hardening

This commit is contained in:
Utku Bilen Demir 2026-09-24 14:57:00 +02:00
parent de435fec00
commit d365ca28e1
25 changed files with 1920 additions and 596 deletions

2
.gitignore vendored
View file

@ -2,7 +2,7 @@ clippings.json
zotero-index.json
matches.json
match-overrides.generated.json
!match-overrides.json
match-overrides.json
import-plan*.json
*.import-plan.json
__pycache__/

View file

@ -3,7 +3,7 @@
# Kanzi — Kindle annotations to Zotero
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero, no terminal needed. Formerly *Kindle Zotero Importer*.
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero. Once the required host tools are installed, imports run from the manager without CLI commands. Formerly *Kindle Zotero Importer*.
<p align="center">
@ -13,22 +13,22 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
# Install
1. Download `kanzi.xpi` from [Releases](https://github.com/UtkuBilenDemir/kanzi/releases) (latest `0.6.20`, or `0.6.19-beta` for preview).
1. Download `kanzi.xpi` from [Releases](https://github.com/UtkuBilenDemir/kanzi/releases) (latest `0.7.0`).
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
3. `Tools` → `Kanzi…` to open the manager.
Works with Zotero 7 to 10, macOS, Windows and Linux, PDF and EPUB.
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. macOS and Linux are tested. Windows requires setting the full Python executable path in `Settings` and installing the PDF tools separately.
# Use
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Tick `Full re-import from scratch` only if you want to rebuild everything.
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
Your choices are saved in `match-overrides.json` in the project folder. Back it up, share it, or delete an entry to undo a mapping.
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
# Coloured highlights
@ -64,13 +64,13 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
**Integrated.** The last `import-plan.final.json` annotations. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right, 400 shown, sortable and resizable.
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, `Share anonymised annotations` (on by default, private, hashed only), and `Highlight colours` as above. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, annotation sharing, and `Highlight colours` as above. Sharing is off until you explicitly opt in and can be disabled here at any time. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
**Artifacts.** `Mismatch review`, `Persistent overrides`, `Generated suggestions`, `Positioned plan`, `Final writer plan` and `Plugin summary`. Each has `Open` and `Reveal`.
@ -79,12 +79,12 @@ Incremental is default. Only new, changed or colour-changed highlights are repos
# For developers / CLI
```sh
python scripts/build_plugin.py # → dist/kanzi.xpi
python scripts/build_plugin.py # → dist/kanzi.xpi, including the Python runtime
python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --pretty
python -m kindle_zotero_importer run ... --full # ignore incremental and reprocess all
```
PDF positioning uses Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
PDF positioning uses host-installed Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
# Releases
@ -94,4 +94,4 @@ Stable releases are on `Releases` with `kanzi.xpi` and `updates.json` attached.
Issues and pull requests welcome at [Issues](../../issues).
Anonymised sharing is on by default to help future development. It sends only citation key, hashed highlight, colour, has comment and note, added on and integrated at. It never sends raw highlight text. Opt out any time in `Manager` → `Settings` → uncheck `Share anonymised annotations`.
Annotation sharing is off until you explicitly opt in. If enabled, it sends a random pseudonymous installation ID, citation key and publication metadata (including DOI/ISBN when available), raw highlight text and comments, normalised/fuzzy hashes, colour, and dates to `annotation.utkubilen.de`. The collector does not receive your Zotero account identity, but free text can contain identifying information and therefore is not anonymous. Change this any time in `Manager` → `Settings`.

357
collector/app.py Normal file
View file

@ -0,0 +1,357 @@
from __future__ import annotations
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path
import re
import secrets
import sqlite3
import unicodedata
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import HTMLResponse
DB = Path(os.environ.get("KANZI_ANNOTATION_DB", Path(__file__).parent / "annotations.db"))
MAX_PAGE_SIZE = 200
MAX_REQUEST_BYTES = 2_000_000
MAX_BATCH_SIZE = 5_000
app = FastAPI(title="Kanzi Annotation Collector")
def _normalise(text: str) -> str:
value = unicodedata.normalize("NFKC", text or "").lower()
value = re.sub(r"-\s*\n\s*", "", value)
value = re.sub(r"[\u2010-\u2015\u2018\u2019\u201c\u201d]", "'", value)
value = re.sub(r"[^a-z0-9]+", " ", value)
return re.sub(r"\s+", " ", value).strip()
def _hash_norm(text: str) -> str:
return hashlib.sha256(_normalise(text).encode()).hexdigest()[:16]
def _fuzzy_hash(text: str) -> str:
norm = _normalise(text)
if len(norm) < 3:
return _hash_norm(text)
trigrams = {norm[index : index + 3] for index in range(len(norm) - 2)}
words = sorted(norm.split())
word_trigrams = {
" ".join(words[index : index + 3])
for index in range(max(1, len(words) - 2))
}
return hashlib.sha256(" ".join(sorted(trigrams | word_trigrams)).encode()).hexdigest()[:16]
def _connect() -> sqlite3.Connection:
DB.parent.mkdir(parents=True, exist_ok=True)
connection = sqlite3.connect(DB)
connection.row_factory = sqlite3.Row
connection.execute("PRAGMA busy_timeout = 5000")
connection.execute("PRAGMA journal_mode = WAL")
return connection
def init_db() -> None:
with _connect() as connection:
connection.execute(
"""CREATE TABLE IF NOT EXISTS highlights (
id INTEGER PRIMARY KEY AUTOINCREMENT,
anon_id TEXT,
citation_key TEXT,
doi TEXT,
isbn TEXT,
issn TEXT,
publication_title TEXT,
clipping_id_hash TEXT,
norm_hash TEXT,
fuzzy_hash TEXT,
text TEXT,
color TEXT,
has_comment INTEGER,
comment TEXT,
comment_hash TEXT,
clipping_added_on_iso TEXT,
integrated_at TEXT,
received_at TEXT,
raw TEXT
)"""
)
existing_columns = {
row["name"] for row in connection.execute("PRAGMA table_info(highlights)")
}
required_columns = {
"doi": "TEXT",
"isbn": "TEXT",
"issn": "TEXT",
"publication_title": "TEXT",
"clipping_id_hash": "TEXT",
"norm_hash": "TEXT",
"fuzzy_hash": "TEXT",
"text": "TEXT",
"comment": "TEXT",
"comment_hash": "TEXT",
"clipping_added_on_iso": "TEXT",
"integrated_at": "TEXT",
"received_at": "TEXT",
"raw": "TEXT",
}
for name, column_type in required_columns.items():
if name not in existing_columns:
connection.execute(f"ALTER TABLE highlights ADD COLUMN {name} {column_type}")
connection.execute(
"""CREATE TABLE IF NOT EXISTS raw_posts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
anon_id TEXT,
received_at TEXT,
body TEXT
)"""
)
connection.execute(
"""CREATE TABLE IF NOT EXISTS installations (
anon_id TEXT PRIMARY KEY,
token_hash TEXT UNIQUE NOT NULL,
created_at TEXT NOT NULL
)"""
)
# Older collector versions appended every retry. Retain only the newest
# record for each installation/clipping before enforcing idempotency.
connection.execute(
"""DELETE FROM highlights
WHERE clipping_id_hash IS NOT NULL
AND id NOT IN (
SELECT MAX(id) FROM highlights
WHERE clipping_id_hash IS NOT NULL
GROUP BY anon_id, clipping_id_hash
)"""
)
connection.execute(
"""CREATE UNIQUE INDEX IF NOT EXISTS highlights_identity
ON highlights(anon_id, clipping_id_hash)
WHERE clipping_id_hash IS NOT NULL"""
)
init_db()
HTML = """<!doctype html>
<html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Kanzi annotations</title>
<style>
:root{font-family:'iA Writer Duo','JetBrains Mono',ui-monospace,monospace;color:#111;background:#f8f8f8;--panel:#fff;--soft:#f0f0f0;--line:#ddd;--muted:#666;--radius:8px}*{box-sizing:border-box}body{margin:0}main{max-width:1280px;margin:auto;padding:20px 24px}h1{margin:0 0 4px;font-size:28px}h2{font-size:16px;border-bottom:1px solid var(--line);padding-bottom:6px;margin-top:24px}.muted{color:var(--muted);font-size:12px}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:12px;margin:16px 0}.card,table{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius)}.card{padding:14px}.card strong{display:block;font-size:22px}table{width:100%;border-collapse:collapse;table-layout:fixed;overflow:hidden}th,td{padding:9px 10px;border-bottom:1px solid var(--line);text-align:left;overflow-wrap:anywhere;font-size:12px}th{background:var(--soft);color:var(--muted);font-size:10px;text-transform:uppercase}.pager{display:flex;justify-content:flex-end;align-items:center;gap:8px;margin:10px 0}.pager button{padding:5px 9px;border:1px solid var(--line);border-radius:var(--radius);background:var(--panel)}.pager button:disabled{color:#999}@media(max-width:700px){main{padding:14px 10px}.table-wrap{overflow-x:auto}table{min-width:800px}}
</style></head><body><main><h1>Kanzi annotations</h1><div class="muted">Private pseudonymous collector. Raw highlight text and comments are stored.</div>
<div class="grid"><div class="card"><strong id="total">-</strong><span>highlights</span></div><div class="card"><strong id="uniq">-</strong><span>citekeys</span></div><div class="card"><strong id="anons">-</strong><span>installations</span></div><div class="card"><strong id="today">-</strong><span>received today</span></div></div>
<h2>Recent</h2><div class="table-wrap"><table><thead><tr><th>anon</th><th>citekey</th><th>DOI / ISBN</th><th>text</th><th>colour</th><th>received</th></tr></thead><tbody id="recent"></tbody></table></div><div class="pager" id="recent-pager"></div>
<h2>Top citekeys</h2><div class="table-wrap"><table><thead><tr><th>citekey</th><th>count</th></tr></thead><tbody id="top"></tbody></table></div>
<h2>Common highlights</h2><div class="muted">Same publication and normalised text across at least two installations.</div><div class="table-wrap"><table><thead><tr><th>DOI / ISBN</th><th>text</th><th>norm</th><th>installations</th><th>citekeys</th></tr></thead><tbody id="common"></tbody></table></div><div class="pager" id="common-pager"></div>
<script>
const size=50,state={recent:0,common:0};
function cell(row,value){const td=document.createElement('td');td.textContent=value??'';row.appendChild(td);return td}
function empty(body,span,message){body.textContent='';const row=body.insertRow();const td=cell(row,message);td.colSpan=span;td.className='muted'}
function pager(id,key,count,load){const root=document.getElementById(id);root.textContent='';const pages=Math.max(1,Math.ceil(count/size));state[key]=Math.min(state[key],pages-1);const prev=document.createElement('button');prev.textContent='Previous';prev.disabled=state[key]===0;prev.onclick=()=>{state[key]--;load()};root.append(prev);const text=document.createElement('span');text.textContent=`Page ${state[key]+1} of ${pages}`;root.append(text);const next=document.createElement('button');next.textContent='Next';next.disabled=state[key]>=pages-1;next.onclick=()=>{state[key]++;load()};root.append(next)}
async function loadRecent(){const result=await fetch(`/recent?limit=${size}&offset=${state.recent*size}`).then(r=>r.json());const body=document.getElementById('recent');body.textContent='';for(const h of result.recent){const row=body.insertRow();cell(row,(h.anon_id||'').slice(0,8));cell(row,h.citation_key);cell(row,h.doi||h.isbn);cell(row,(h.text||'').slice(0,180));cell(row,h.color);cell(row,(h.received_at||'').slice(0,16).replace('T',' '))}if(!result.recent.length)empty(body,6,'No data');pager('recent-pager','recent',result.total,loadRecent)}
async function loadCommon(){const result=await fetch(`/common?limit=${size}&offset=${state.common*size}`).then(r=>r.json());const body=document.getElementById('common');body.textContent='';for(const h of result.common){const row=body.insertRow();cell(row,h.publication);cell(row,(h.example_text||'').slice(0,180));cell(row,h.norm_hash);cell(row,h.installations);cell(row,h.citekeys)}if(!result.common.length)empty(body,5,'No common highlights yet');pager('common-pager','common',result.total,loadCommon)}
async function load(){const stats=await fetch('/stats').then(r=>r.json());for(const [id,key] of [['total','total_highlights'],['uniq','unique_citekeys'],['anons','unique_anons'],['today','today']])document.getElementById(id).textContent=stats[key];const top=await fetch('/top').then(r=>r.json());const body=document.getElementById('top');body.textContent='';for(const h of top.top){const row=body.insertRow();cell(row,h.citation_key);cell(row,h.count)}await Promise.all([loadRecent(),loadCommon()])}load();
</script></main></body></html>"""
@app.get("/", response_class=HTMLResponse)
def root() -> str:
return HTML
@app.get("/health")
def health() -> dict[str, str]:
return {"status": "ok"}
def _token_hash(token: str) -> str:
return hashlib.sha256(token.encode()).hexdigest()
@app.post("/register")
def register() -> dict[str, str]:
token = secrets.token_urlsafe(32)
anon_id = secrets.token_hex(16)
with _connect() as connection:
connection.execute(
"INSERT INTO installations (anon_id, token_hash, created_at) VALUES (?, ?, ?)",
(anon_id, _token_hash(token), datetime.now(timezone.utc).isoformat()),
)
return {"anonId": anon_id, "token": token}
def _authenticated_anon_id(request: Request) -> str:
authorization = request.headers.get("authorization", "")
scheme, separator, token = authorization.partition(" ")
if not separator or scheme.lower() != "bearer" or not token:
raise HTTPException(status_code=401, detail="Missing collector credential")
with _connect() as connection:
row = connection.execute(
"SELECT anon_id FROM installations WHERE token_hash = ?",
(_token_hash(token),),
).fetchone()
if not row:
raise HTTPException(status_code=401, detail="Invalid collector credential")
return str(row["anon_id"])
@app.get("/stats")
def stats() -> dict[str, int]:
today = datetime.now(timezone.utc).date().isoformat()
with _connect() as connection:
row = connection.execute(
"""SELECT COUNT(*) AS total_highlights,
COUNT(DISTINCT citation_key) AS unique_citekeys,
COUNT(DISTINCT anon_id) AS unique_anons,
SUM(CASE WHEN substr(received_at, 1, 10) = ? THEN 1 ELSE 0 END) AS today
FROM highlights""",
(today,),
).fetchone()
return {key: int(row[key] or 0) for key in row.keys()}
@app.get("/recent")
def recent(limit: int = 50, offset: int = 0) -> dict:
limit = max(1, min(limit, MAX_PAGE_SIZE))
offset = max(0, offset)
with _connect() as connection:
total = connection.execute("SELECT COUNT(*) FROM highlights").fetchone()[0]
rows = connection.execute(
"""SELECT anon_id, citation_key, doi, isbn, text, clipping_id_hash,
norm_hash, fuzzy_hash, color, has_comment, received_at
FROM highlights ORDER BY id DESC LIMIT ? OFFSET ?""",
(limit, offset),
).fetchall()
return {"recent": [dict(row) for row in rows], "total": total}
@app.get("/top")
def top(limit: int = 20) -> dict:
limit = max(1, min(limit, 100))
with _connect() as connection:
rows = connection.execute(
"""SELECT citation_key, COUNT(*) AS count FROM highlights
WHERE citation_key IS NOT NULL AND citation_key != ''
GROUP BY citation_key ORDER BY count DESC, citation_key LIMIT ?""",
(limit,),
).fetchall()
return {"top": [dict(row) for row in rows]}
@app.get("/common")
def common(limit: int = 50, offset: int = 0) -> dict:
limit = max(1, min(limit, MAX_PAGE_SIZE))
offset = max(0, offset)
group_sql = """FROM highlights
WHERE norm_hash IS NOT NULL AND norm_hash != ''
AND COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), NULLIF(citation_key, '')) IS NOT NULL
GROUP BY COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key), norm_hash
HAVING COUNT(DISTINCT anon_id) > 1"""
with _connect() as connection:
total = connection.execute(f"SELECT COUNT(*) FROM (SELECT 1 {group_sql})").fetchone()[0]
rows = connection.execute(
f"""SELECT COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key) AS publication,
norm_hash, COUNT(DISTINCT anon_id) AS installations,
GROUP_CONCAT(DISTINCT citation_key) AS citekeys,
MAX(text) AS example_text
{group_sql}
ORDER BY installations DESC, publication, norm_hash LIMIT ? OFFSET ?""",
(limit, offset),
).fetchall()
return {"common": [dict(row) for row in rows], "total": total}
@app.post("/highlights")
async def post_highlights(request: Request) -> dict:
anon = _authenticated_anon_id(request)
content_length = request.headers.get("content-length")
if content_length:
try:
if int(content_length) > MAX_REQUEST_BYTES:
raise HTTPException(status_code=413, detail="Request body is too large")
except ValueError:
raise HTTPException(status_code=400, detail="Invalid Content-Length")
body = await request.body()
if len(body) > MAX_REQUEST_BYTES:
raise HTTPException(status_code=413, detail="Request body is too large")
try:
payload = json.loads(body)
except (json.JSONDecodeError, UnicodeDecodeError) as error:
raise HTTPException(status_code=400, detail="Invalid JSON") from error
if not isinstance(payload, dict):
raise HTTPException(status_code=422, detail="Payload must be an object")
highlights = payload.get("data") or payload.get("highlights") or []
if isinstance(highlights, dict):
highlights = [highlights]
if not isinstance(highlights, list):
raise HTTPException(status_code=422, detail="data must be a list")
if len(highlights) > MAX_BATCH_SIZE:
raise HTTPException(status_code=413, detail="Too many highlights in one request")
now = datetime.now(timezone.utc).isoformat()
inserted = updated = 0
with _connect() as connection:
connection.execute("BEGIN IMMEDIATE")
for highlight in highlights:
if not isinstance(highlight, dict):
continue
text = str(highlight.get("text") or "")[:2000]
clipping_hash = str(highlight.get("clipping_id_hash") or "")[:128] or None
values = (
anon,
str(highlight.get("citation_key") or "")[:512] or None,
str(highlight.get("doi") or "")[:512] or None,
str(highlight.get("isbn") or "")[:512] or None,
str(highlight.get("issn") or "")[:512] or None,
str(highlight.get("publication_title") or "")[:1000] or None,
clipping_hash,
highlight.get("norm_hash") or _hash_norm(text),
highlight.get("fuzzy_hash") or _fuzzy_hash(text),
text,
str(highlight.get("color") or "")[:32] or None,
1 if highlight.get("has_comment") else 0,
str(highlight.get("comment") or "")[:2000],
str(highlight.get("comment_hash") or "")[:256] or None,
str(highlight.get("clipping_added_on_iso") or "")[:64] or None,
str(highlight.get("integrated_at") or "")[:64] or None,
now,
json.dumps(highlight, ensure_ascii=False)[:8000],
)
existing = None
if clipping_hash:
legacy_hash = clipping_hash[:8]
existing = connection.execute(
"""SELECT id FROM highlights
WHERE anon_id = ? AND clipping_id_hash IN (?, ?)
ORDER BY clipping_id_hash = ? DESC LIMIT 1""",
(anon, clipping_hash, legacy_hash, clipping_hash),
).fetchone()
if existing:
connection.execute(
"""UPDATE highlights SET citation_key=?, doi=?, isbn=?, issn=?, publication_title=?, clipping_id_hash=?,
norm_hash=?, fuzzy_hash=?, text=?, color=?, has_comment=?, comment=?, comment_hash=?,
clipping_added_on_iso=?, integrated_at=?, received_at=?, raw=? WHERE id=?""",
(*values[1:7], *values[7:], existing["id"]),
)
updated += 1
else:
connection.execute(
"""INSERT INTO highlights (
anon_id, citation_key, doi, isbn, issn, publication_title, clipping_id_hash,
norm_hash, fuzzy_hash, text, color, has_comment, comment, comment_hash,
clipping_added_on_iso, integrated_at, received_at, raw
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
values,
)
inserted += 1
return {
"status": "ok",
"received": len(highlights),
"inserted": inserted,
"updated": updated,
"anon": anon,
}

View file

@ -0,0 +1,28 @@
# Add these directives in the nginx http context.
limit_req_zone $binary_remote_addr zone=kanzi_ingest:10m rate=10r/m;
limit_conn_zone $binary_remote_addr zone=kanzi_connections:10m;
# Keep dashboard and read APIs behind Authentik. Only plugin ingestion and
# health monitoring are intentionally unauthenticated.
location = /highlights {
client_max_body_size 2m;
limit_req zone=kanzi_ingest burst=10 nodelay;
limit_conn kanzi_connections 2;
proxy_pass http://127.0.0.1:8099;
}
location = /register {
limit_req zone=kanzi_ingest burst=10 nodelay;
limit_conn kanzi_connections 2;
proxy_pass http://127.0.0.1:8099;
}
location = /health {
proxy_pass http://127.0.0.1:8099;
}
location / {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
proxy_pass http://127.0.0.1:8099;
}

View file

@ -0,0 +1,15 @@
{
"format": "kindle-zotero-importer.match-overrides.v1",
"overrides": [
{
"clipping_title": "Example Kindle title",
"resolution": {
"citation_key": "example2026"
},
"review": {
"status": "manager",
"clipping_count": 1
}
}
]
}

View file

@ -1,16 +1,16 @@
# Kindle Zotero Importer Zotero Plugin
# Kanzi Zotero Plugin
This is a hybrid Zotero plugin wrapper around the current Python importer. Zotero owns the user interface and writes native annotations through Zotero APIs; the Python project still performs parsing, matching, planning, EPUB positioning, and PDF positioning.
## Use
1. Install `kindle-zotero-importer.xpi` in Zotero through `Tools > Plugins`.
2. Open `Tools > Kindle Zotero Importer...`.
1. Install `kanzi.xpi` in Zotero through `Tools > Plugins`.
2. Open `Tools > Kanzi...`.
3. Select the updated Kindle `My Clippings.txt` in the manager.
4. Follow the current stage, percentage, and elapsed time in the manager.
5. Review created, existing, updated, failed, and unresolved counts when it completes.
The plugin uses `match-overrides.json` from the project directory as the persistent source of title mappings and ignored titles. Generated JSON artifacts are still written to the project directory for audit and debugging.
The XPI contains the importer Python source. On startup it extracts that source into the Zotero profile, where it keeps private mappings and generated audit artifacts. Python 3.11 or newer must be installed on the host; PDF positioning also requires Poppler (`pdftotext`, `pdftohtml`, and `pdfinfo`) and optionally uses `qpdf` for encrypted files. The plugin uses `match-overrides.json` there as the persistent source of title mappings and ignored titles.
## Build XPI
@ -20,6 +20,6 @@ From the repository root:
python scripts/build_plugin.py
```
The output is `dist/kindle-zotero-importer.xpi`.
The output is `dist/kanzi.xpi`.
The plugin manifest targets Zotero `6.999` through `10.0.*`.

407
plugin/bootstrap.js vendored
View file

@ -1,10 +1,9 @@
var KindleZoteroImporter = {
id: "kindle-zotero-importer@utkubilen.de",
menuItems: [],
defaultProjectDir:
"/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer",
defaultClippingsPath:
"/home/user/Library/Mobile Documents/com~apple~CloudDocs/Projects/test/My Clippings.txt",
runtimeProjectDir: "",
defaultProjectDir: "",
defaultClippingsPath: "",
prefBranch: "extensions.kindleZoteroImporter.",
rootURI: null,
managerWindow: null,
@ -20,9 +19,41 @@ var KindleZoteroImporter = {
[["content", "kindle-zotero-importer", ""]]
);
await Zotero.initializationPromise;
await this.ensureRuntime();
this.addToAllWindows();
},
async ensureRuntime() {
const runtimeDir = PathUtils.join(PathUtils.profileDir, "kanzi-runtime");
const files = [
"plugin_runner.py",
"src/kindle_zotero_importer/__init__.py",
"src/kindle_zotero_importer/__main__.py",
"src/kindle_zotero_importer/cli.py",
"src/kindle_zotero_importer/clippings.py",
"src/kindle_zotero_importer/epub_position.py",
"src/kindle_zotero_importer/final_plan.py",
"src/kindle_zotero_importer/import_plan.py",
"src/kindle_zotero_importer/matcher.py",
"src/kindle_zotero_importer/mismatch_review.py",
"src/kindle_zotero_importer/overrides.py",
"src/kindle_zotero_importer/pdf_position.py",
"src/kindle_zotero_importer/zotero_index.py",
];
await IOUtils.makeDirectory(runtimeDir, { createAncestors: true, ignoreExisting: true });
for (const relativePath of files) {
const destination = PathUtils.join(runtimeDir, ...relativePath.split("/"));
await IOUtils.makeDirectory(PathUtils.parent(destination), {
createAncestors: true,
ignoreExisting: true,
});
const response = await fetch(this.rootURI + "runtime/" + relativePath);
if (!response.ok) throw new Error(`Missing Kanzi runtime file: ${relativePath}`);
await Zotero.File.putContentsAsync(destination, await response.text());
}
this.runtimeProjectDir = runtimeDir;
},
shutdown() {
this.removeMenuItems();
if (this.managerWindow && !this.managerWindow.closed) {
@ -89,7 +120,8 @@ var KindleZoteroImporter = {
getPref(name, fallback) {
try {
return Services.prefs.getCharPref(this.prefBranch + name);
const value = Services.prefs.getCharPref(this.prefBranch + name);
return value || fallback;
} catch (_error) {
return fallback;
}
@ -100,19 +132,21 @@ var KindleZoteroImporter = {
},
getProjectDir() {
return this.getPref("projectDir", this.defaultProjectDir);
return this.getPref("projectDir", this.runtimeProjectDir || this.defaultProjectDir);
},
getPythonPath() {
return this.getPref("pythonPath", "/opt/homebrew/bin/python3");
return this.getPref("pythonPath", "python3");
},
getZoteroDbPath() {
return this.getPref("zoteroDbPath", "/home/user/Zotero/zotero.sqlite");
const dataDir = Zotero.DataDirectory && Zotero.DataDirectory.dir;
return this.getPref("zoteroDbPath", dataDir ? PathUtils.join(dataDir, "zotero.sqlite") : "");
},
getZoteroStorageRoot() {
return this.getPref("zoteroStorageRoot", "/home/user/Zotero/storage");
const dataDir = Zotero.DataDirectory && Zotero.DataDirectory.dir;
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
},
getConfigPath() {
@ -143,6 +177,17 @@ var KindleZoteroImporter = {
zoteroDbPath: this.getZoteroDbPath(),
zoteroStorageRoot: this.getZoteroStorageRoot(),
};
if (settings.pythonPath === "python3") {
const candidates = Services.appinfo.OS === "Darwin"
? ["/opt/homebrew/bin/python3", "/usr/local/bin/python3", "/usr/bin/python3"]
: ["/usr/bin/python3", "/usr/local/bin/python3"];
for (const candidate of candidates) {
if (await IOUtils.exists(candidate)) {
settings.pythonPath = candidate;
break;
}
}
}
try {
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
const parsed = JSON.parse(text);
@ -382,7 +427,7 @@ var KindleZoteroImporter = {
}
},
async runPipeline(clippingsPath, settings, isFull = false) {
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
const python = settings.pythonPath;
const projectDir = settings.projectDir;
const runner = projectDir + "/plugin_runner.py";
@ -409,6 +454,9 @@ var KindleZoteroImporter = {
if (isFull) {
args.push("--full");
}
if (retryUnresolved) {
args.push("--retry-unresolved");
}
await this.exec(python, args);
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
@ -420,10 +468,11 @@ var KindleZoteroImporter = {
return;
}
const isFull = manager.getFullReimport ? manager.getFullReimport() : false;
await this.runManagedImportWithPath(manager, clippingsPath, isFull);
const retryUnresolved = manager.getRetryUnresolved ? manager.getRetryUnresolved() : false;
await this.runManagedImportWithPath(manager, clippingsPath, isFull, retryUnresolved);
},
async runManagedImportWithPath(manager, clippingsPath, isFullOverride) {
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
try {
if (!clippingsPath) {
throw new Error("No clippings file selected");
@ -432,6 +481,9 @@ var KindleZoteroImporter = {
const progressPath = settings.projectDir + "/plugin-progress.json";
// Determine full vs incremental from explicit arg or manager checkbox
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
? retryUnresolvedOverride
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
manager.beginRun(clippingsPath);
try {
await Zotero.File.removeIfExists(progressPath);
@ -450,7 +502,7 @@ var KindleZoteroImporter = {
const pollPromise = poll();
let summary;
try {
summary = await this.runPipeline(clippingsPath, settings, isFull);
summary = await this.runPipeline(clippingsPath, settings, isFull, retryUnresolved);
} finally {
polling = false;
await pollPromise;
@ -464,8 +516,9 @@ var KindleZoteroImporter = {
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
});
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, results);
const data = await this.loadManagerData();
manager.completeRun(summary, results, data);
await manager.completeRun(summary, results, data);
} catch (error) {
Zotero.logError(error);
manager.failRun(String(error && error.stack ? error.stack : error));
@ -548,42 +601,143 @@ var KindleZoteroImporter = {
},
mergeComments(existingComment, newComment) {
const stripBracket = (s) => String(s || "").replace(/^\s*\[[^\]]+\]\s*/, "").trim();
const seen = new Set();
const merged = [];
for (const comment of [existingComment, newComment]) {
for (const part of String(comment || "").split(/\n\s*\n/)) {
const cleaned = part.trim();
const cleaned = this.stripColourCode(part);
if (!cleaned) continue;
// dedup on stripped version too, so "[r] note" and "note" are same
const key = stripBracket(cleaned).toLowerCase() || cleaned.toLowerCase();
if (seen.has(cleaned) || seen.has(key)) {
// if we have stripped version already, prefer stripped
if (seen.has(key) && cleaned !== stripBracket(cleaned)) {
// replace existing bracket version with stripped if we now have stripped
const idx = merged.findIndex(p => stripBracket(p).toLowerCase() === key);
if (idx >= 0) merged[idx] = stripBracket(cleaned);
seen.add(cleaned);
const key = cleaned.toLowerCase();
if (seen.has(key)) continue;
seen.add(key);
continue;
}
continue;
}
seen.add(cleaned);
seen.add(key);
// store stripped version if it had bracket
const stripped = stripBracket(cleaned);
if (stripped !== cleaned) {
merged.push(stripped);
seen.add(stripped);
} else {
merged.push(cleaned);
}
}
}
return merged.join("\n\n");
},
stripColourCode(value) {
return String(value || "")
.replace(/^\s*\[(?:y|yellow|o|orange|r|red|e|grey|gray|g|green|b|blue|bl|p|purple|m|magenta|pink)\]\s*/i, "")
.trim();
},
normalizeAnnotationText(value) {
return String(value || "")
.toLowerCase()
.normalize("NFKC")
.replace(/-\s*\n\s*/g, "")
.replace(/[^\p{L}\p{N}\s]+/gu, " ")
.replace(/\s+/g, " ")
.trim();
},
annotationPositionAnchor(annotation) {
const raw = annotation.annotationPosition || annotation.position;
let position = raw;
if (typeof raw === "string") {
try { position = JSON.parse(raw); } catch (_error) { return raw; }
}
if (!position || typeof position !== "object") return "";
if (position.type === "FragmentSelector" && position.value) {
const value = String(position.value);
const finalComma = value.lastIndexOf(",");
return `epub:${finalComma > 0 ? value.slice(0, finalComma) : value}`;
}
if (Number.isInteger(position.pageIndex) && Array.isArray(position.rects) && position.rects.length) {
const rect = position.rects[0] || [];
const x = Number(rect[0]);
const y = Number(rect[1]);
return `pdf:${position.pageIndex}:${x.toFixed(1)}:${y.toFixed(1)}`;
}
return this.normalizePosition(position);
},
areImportedDuplicates(left, right) {
const leftAnchor = this.annotationPositionAnchor(left);
const rightAnchor = this.annotationPositionAnchor(right);
if (!leftAnchor || leftAnchor !== rightAnchor) return false;
const leftPosition = this.normalizePosition(left.annotationPosition || left.position);
const rightPosition = this.normalizePosition(right.annotationPosition || right.position);
if (leftPosition && leftPosition === rightPosition) return true;
const leftText = this.normalizeAnnotationText(left.annotationText || left.text);
const rightText = this.normalizeAnnotationText(right.annotationText || right.text);
if (!leftText || !rightText) return false;
if (leftText === rightText) return true;
const shorter = leftText.length < rightText.length ? leftText : rightText;
const longer = shorter === leftText ? rightText : leftText;
return shorter.length >= 20 && longer.startsWith(shorter);
},
async collectImportedAnnotations() {
const annotations = [];
let allItems = [];
if (Zotero.DB && Zotero.DB.columnQueryAsync) {
const ids = await Zotero.DB.columnQueryAsync("SELECT itemID FROM itemAnnotations");
allItems = Zotero.Items.getAsync ? await Zotero.Items.getAsync(ids) : Zotero.Items.get(ids);
} else if (Zotero.Items.getAll) {
allItems = await Zotero.Items.getAll();
}
for (const item of allItems || []) {
if (!item || !item.isAnnotation || !item.isAnnotation()) continue;
if (this.hasKindleImportTag(item)) annotations.push(item);
}
return annotations;
},
async maintainImportedAnnotations(annotations, dryRun, results) {
if (!results.maintenanceDeletedIds) results.maintenanceDeletedIds = [];
const byAttachment = new Map();
for (const annotation of annotations) {
const parentID = annotation.parentItemID || annotation.parentID;
if (!byAttachment.has(parentID)) byAttachment.set(parentID, []);
byAttachment.get(parentID).push(annotation);
}
for (const group of byAttachment.values()) {
group.sort((a, b) =>
this.normalizeAnnotationText(b.annotationText).length -
this.normalizeAnnotationText(a.annotationText).length
);
const kept = [];
for (const annotation of group) {
const canonicalComment = this.mergeComments(annotation.annotationComment, "");
if (String(annotation.annotationComment || "") !== canonicalComment) {
annotation.annotationComment = canonicalComment;
results.updatedComments += 1;
if (!dryRun) await this.saveExistingAnnotation(annotation);
}
const duplicateOf = kept.find((candidate) => this.areImportedDuplicates(candidate, annotation));
if (!duplicateOf) {
kept.push(annotation);
continue;
}
const mergedComment = this.mergeComments(duplicateOf.annotationComment, annotation.annotationComment);
if (duplicateOf.annotationComment !== mergedComment) {
duplicateOf.annotationComment = mergedComment;
results.updatedComments += 1;
if (!dryRun) await this.saveExistingAnnotation(duplicateOf);
}
if (!dryRun) await annotation.eraseTx();
const keptClippingIds = new Set(
this.existingTags(duplicateOf)
.filter((tag) => tag.startsWith("kindle-id:"))
.map((tag) => tag.slice("kindle-id:".length))
);
for (const tag of this.existingTags(annotation)) {
const clippingID = tag.startsWith("kindle-id:")
? tag.slice("kindle-id:".length)
: "";
if (clippingID && !keptClippingIds.has(clippingID)) {
results.maintenanceDeletedIds.push(clippingID);
}
}
results.removedDuplicates += 1;
}
}
},
existingSortIndex(annotation) {
return annotation.annotationSortIndex || annotation._annotationSortIndex || annotation.sortIndex || "";
},
@ -607,6 +761,23 @@ var KindleZoteroImporter = {
return this.existingTags(annotation).includes("kindle-import");
},
commentForExistingAnnotation(annotation, newComment, previousImportedComment) {
if (!this.hasKindleImportTag(annotation) || previousImportedComment === undefined) {
return this.mergeComments(annotation.annotationComment, newComment);
}
const previousParts = new Set(
String(previousImportedComment || "")
.split(/\n\s*\n/)
.map((part) => this.stripColourCode(part).toLowerCase())
.filter(Boolean)
);
const manualParts = String(annotation.annotationComment || "")
.split(/\n\s*\n/)
.filter((part) => !previousParts.has(this.stripColourCode(part).toLowerCase()))
.join("\n\n");
return this.mergeComments(manualParts, newComment);
},
confirmRecreateForSortIndex(entry, existingAnnotation, oldSortIndex, newSortIndex) {
const tags = this.existingTags(existingAnnotation).join(", ") || "(none)";
const message = [
@ -643,6 +814,37 @@ var KindleZoteroImporter = {
throw new Error("Existing annotation cannot be saved by this Zotero build");
},
async reconcileWriterState(path, plan, results) {
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
const failedIds = new Set(
(results.failed || [])
.map((failure) => failure.clipping_id)
.filter((clippingID) => plannedIds.has(clippingID))
);
plan.write_failed_ids = Array.from(failedIds).sort();
plan.delete_failed_ids = Array.from(new Set(results.failedDeletionIds || [])).sort();
const maintenanceDeletedIds = new Set(results.maintenanceDeletedIds || []);
if (failedIds.size) {
plan.annotations = (plan.annotations || []).filter(
(entry) => !failedIds.has(entry.clipping_id)
);
plan.all_annotations = (plan.all_annotations || []).filter(
(entry) => !failedIds.has(entry.clipping_id)
);
}
if (maintenanceDeletedIds.size) {
plan.annotations = (plan.annotations || []).filter(
(entry) => !maintenanceDeletedIds.has(entry.clipping_id)
);
plan.all_annotations = (plan.all_annotations || []).filter(
(entry) => !maintenanceDeletedIds.has(entry.clipping_id)
);
}
plan.delta_annotation_count = (plan.annotations || []).length;
plan.annotation_count = (plan.all_annotations || plan.annotations || []).length;
await Zotero.File.putContentsAsync(path, JSON.stringify(plan, null, 2) + "\n");
},
async writeAnnotations(plan, dryRun, onProgress) {
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
throw new Error(`Unsupported plan format: ${plan.format}`);
@ -660,75 +862,31 @@ var KindleZoteroImporter = {
recreateDeclined: 0,
removedDuplicates: 0,
deletedForIncremental: 0,
failedDeletionIds: [],
maintenanceDeletedIds: [],
failed: [],
};
// Incremental deletions: remove annotations whose clipping_id is no longer present
if (!dryRun && plan.deletions && plan.deletions.length) {
const deletionIds = new Set(plan.deletions);
let importedAnnotations = [];
let maintenanceReady = false;
try {
// Find all annotations with kindle-id tag
const allAnnotations = [];
// Collect via attachments to be safe
const allItems = Zotero.Items.getAll ? await Zotero.Items.getAll() : [];
for (const item of allItems) {
if (item.isAnnotation && item.isAnnotation()) {
allAnnotations.push(item);
} else if (item.getAnnotations) {
const anns = item.getAnnotations();
for (const a of anns) allAnnotations.push(a);
}
}
// Deduplicate
const seen = new Set();
for (const ann of allAnnotations) {
if (!ann || seen.has(ann.id)) continue;
seen.add(ann.id);
const tags = this.existingTags(ann);
for (const tag of tags) {
if (tag.startsWith("kindle-id:")) {
const id = tag.substring("kindle-id:".length);
if (deletionIds.has(id)) {
await ann.eraseTx();
results.deletedForIncremental += 1;
break;
}
}
}
}
} catch (e) {
Zotero.logError(e);
importedAnnotations = await this.collectImportedAnnotations();
await this.maintainImportedAnnotations(importedAnnotations, dryRun, results);
if (!dryRun && results.removedDuplicates) {
importedAnnotations = await this.collectImportedAnnotations();
}
maintenanceReady = true;
} catch (error) {
Zotero.logError(error);
results.failed.push({ clipping_id: "maintenance", message: String(error) });
}
const existingByAttachment = new Map();
const normForDedup = (s) => String(s || "").toLowerCase().normalize("NFKC").replace(/-\s*\n\s*/g, "").replace(/[^\w\s]+/g, " ").replace(/\s+/g, " ").trim();
const getExistingAnnotations = async (attachment) => {
if (!existingByAttachment.has(attachment.id)) {
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
const byFingerprint = new Map();
const byNorm = new Map();
for (const annotation of annotations) {
const norm = normForDedup(annotation.annotationText || "");
const dupNorm = byNorm.get(norm);
if (dupNorm && norm) {
// collapse identical highlights (e.g. My Clippings duplicates "forged its might…" x5)
const kept = dupNorm;
const keptHasComment = Boolean(kept.annotationComment);
const curHasComment = Boolean(annotation.annotationComment);
const duplicate = keptHasComment || !curHasComment ? annotation : kept;
const replacement = duplicate === annotation ? kept : annotation;
byNorm.set(norm, replacement);
// also update fingerprint map
const fpDup = this.existingAnnotationFingerprint(duplicate);
const fpKeep = this.existingAnnotationFingerprint(replacement);
byFingerprint.delete(fpDup);
byFingerprint.set(fpKeep, replacement);
if (!dryRun) await duplicate.eraseTx();
results.removedDuplicates += 1;
continue;
}
if (norm) byNorm.set(norm, annotation);
const fingerprint = this.existingAnnotationFingerprint(annotation);
const kept = byFingerprint.get(fingerprint);
if (!kept) {
@ -754,6 +912,7 @@ var KindleZoteroImporter = {
};
let completed = 0;
const successfulEntries = new Map();
for (const entry of plan.annotations) {
try {
const attachment = Zotero.Items.get(entry.attachment_item_id);
@ -771,19 +930,32 @@ var KindleZoteroImporter = {
const existingAnnotations = await getExistingAnnotations(attachment);
const fingerprint = this.annotationFingerprint(annotation);
const existingAnnotation = existingAnnotations.get(fingerprint);
const existingAnnotation = existingAnnotations.get(fingerprint) ||
Array.from(existingAnnotations.values()).find(
(candidate) => this.areImportedDuplicates(candidate, annotation)
);
if (existingAnnotation) {
const mergedComment = this.mergeComments(
existingAnnotation.annotationComment,
annotation.comment
const mergedComment = this.commentForExistingAnnotation(
existingAnnotation,
annotation.comment,
entry.previous_imported_comment
);
let changed = false;
if (mergedComment && existingAnnotation.annotationComment !== mergedComment) {
if (existingAnnotation.annotationComment !== mergedComment) {
existingAnnotation.annotationComment = mergedComment;
changed = true;
results.updatedComments += 1;
}
const existingTags = new Set(this.existingTags(existingAnnotation));
for (const tag of annotation.tags || []) {
const name = tag.name || tag.tag;
if (!name || existingTags.has(name) || !existingAnnotation.addTag) continue;
existingAnnotation.addTag(name);
existingTags.add(name);
changed = true;
}
// colour update (British spelling in code handles both)
const existingColour = existingAnnotation.annotationColor || existingAnnotation.color || "";
const newColour = annotation.color || "";
@ -836,12 +1008,14 @@ var KindleZoteroImporter = {
});
results.recreatedForSortIndex += 1;
results.skippedExisting += 1;
successfulEntries.set(entry.clipping_id, entry);
continue;
}
} else if (sortIndexChanged && dryRun) {
results.updatedSortIndex += 1;
}
results.skippedExisting += 1;
successfulEntries.set(entry.clipping_id, entry);
continue;
}
@ -856,6 +1030,7 @@ var KindleZoteroImporter = {
annotationText: annotation.text || "",
});
results.created += 1;
successfulEntries.set(entry.clipping_id, entry);
} catch (error) {
results.failed.push({
clipping_id: entry.clipping_id,
@ -870,6 +1045,40 @@ var KindleZoteroImporter = {
}
}
if (!dryRun && plan.deletions && plan.deletions.length) {
const deletionIds = new Set(plan.deletions);
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
if (!maintenanceReady) {
results.failedDeletionIds.push(...deletionIds);
} else {
for (const ann of importedAnnotations) {
for (const tag of this.existingTags(ann)) {
if (!tag.startsWith("kindle-id:")) continue;
const id = tag.slice("kindle-id:".length);
if (!deletionIds.has(id) || alreadyDeletedIds.has(id)) break;
const replacement = successfulEntries.get(id);
if (replacement) {
const sameAttachment = (ann.parentItemID || ann.parentID) === replacement.attachment_item_id;
const sameFingerprint = this.existingAnnotationFingerprint(ann) ===
this.annotationFingerprint(replacement.annotation);
if (sameAttachment && sameFingerprint) break;
} else if ((plan.annotations || []).some((entry) => entry.clipping_id === id)) {
results.failedDeletionIds.push(id);
break;
}
try {
await ann.eraseTx();
results.deletedForIncremental += 1;
} catch (error) {
results.failedDeletionIds.push(id);
results.failed.push({ clipping_id: id, message: `Deletion failed: ${error}` });
}
break;
}
}
}
}
return results;
},

View file

@ -90,6 +90,9 @@
.muted { color: var(--muted); font-size: 12px; }
.highlight-text { max-width: 520px; white-space: pre-wrap; word-break: break-word; font-size: 12px; line-height: 1.45; font-family: inherit; }
.integrated-meta { font-size: 10.5px; color: var(--muted); font-family: 'JetBrains Mono', monospace; }
.pagination { display: flex; align-items: center; justify-content: flex-end; gap: 8px; margin: 10px 0; color: var(--muted); font-size: 11px; }
.pagination button, .pagination select { width: auto; padding: 5px 8px; border: 1px solid var(--line); border-radius: var(--radius); background: var(--panel); color: var(--text); font: inherit; }
.pagination button:disabled { color: var(--faint); background: var(--panel-soft); cursor: not-allowed; }
::selection { background: var(--accent); color: white; }
</style>
<script>
@ -175,6 +178,41 @@
if (el) el.textContent = `${minutes}:${String(seconds % 60).padStart(2, '0')} elapsed`;
},
async collectorCredentials() {
let anonId = '';
let token = '';
try { anonId = Services.prefs.getCharPref("extensions.kindleZoteroImporter.collectorAnonId"); } catch(e) {}
try { token = Services.prefs.getCharPref("extensions.kindleZoteroImporter.collectorToken"); } catch(e) {}
if (anonId && token) return {anonId, token};
const response = await fetch("https://annotation.utkubilen.de/register", {method: "POST"});
if (!response.ok) throw new Error(`Collector registration returned HTTP ${response.status}`);
const credentials = await response.json();
if (!credentials.anonId || !credentials.token) throw new Error('Collector registration was incomplete');
Services.prefs.setCharPref("extensions.kindleZoteroImporter.collectorAnonId", credentials.anonId);
Services.prefs.setCharPref("extensions.kindleZoteroImporter.collectorToken", credentials.token);
return credentials;
},
collectorBatches(anonId, data) {
const batches = [];
let current = [];
for (const item of data) {
const candidate = current.concat(item);
const bytes = new TextEncoder().encode(JSON.stringify({anonId, data: candidate})).length;
if (!current.length && bytes > 1500000) {
throw new Error('One annotation exceeds the collector request limit');
}
if (current.length && (current.length >= 200 || bytes > 1500000)) {
batches.push(current);
current = [item];
} else {
current = candidate;
}
}
if (current.length) batches.push(current);
return batches;
},
updateProgress(percent, stage, detail) {
const prog = document.getElementById('progress');
if (prog) prog.value = Math.max(0, Math.min(100, Number(percent) || 0));
@ -186,7 +224,7 @@
if (pctEl) pctEl.textContent = `${Math.round(Number(percent) || 0)}%`;
},
completeRun(summary, results, data) {
async completeRun(summary, results, data) {
this.running = false;
clearInterval(this.timer);
this.updateProgress(100, 'Import complete', 'Annotations have been checked and saved in Zotero');
@ -219,20 +257,14 @@
grid.appendChild(makeCard(results.updatedComments + results.updatedSortIndex, 'updated'));
grid.appendChild(makeCard(results.failed.length, 'failed'));
resultsEl.appendChild(grid);
// Opt-in prompt for anonymised sharing (once, after first successful import)
try {
const hasAsked = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare");
} catch(e) {
// First run: default ON, allow opt-out
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", true);
try {
const anonId = Math.random().toString(36).slice(2,10) + Date.now().toString(36);
Services.prefs.setCharPref("extensions.kindleZoteroImporter.anonId", anonId);
} catch(e2) {}
const consentVersion = 2;
let acceptedVersion = 0;
try { acceptedVersion = Services.prefs.getIntPref("extensions.kindleZoteroImporter.shareConsentVersion"); } catch(e) {}
if (acceptedVersion < consentVersion && (data.finalPlan?.annotations || []).length) {
const accepted = confirm("Share annotation data with the Kanzi developer?\n\nThis sends a random pseudonymous installation ID, publication metadata, raw highlight text and comments, hashes, colour, and dates to annotation.utkubilen.de. Free text may contain identifying information. The dashboard is private, but this data is not anonymous.\n\nChoose OK to opt in, or Cancel to keep sharing disabled. You can change this later in Settings.");
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", accepted);
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare", true);
if (results.created > 0) {
alert("Anonymised sharing is enabled by default to help future development (only citation key, hashed highlight, colour, has comment + note, and dates — never your highlight text alone). You can opt out anytime in Settings → uncheck 'Share anonymised annotations'.");
}
Services.prefs.setIntPref("extensions.kindleZoteroImporter.shareConsentVersion", consentVersion);
}
}
this.data = data;
@ -243,45 +275,41 @@
const bar = document.getElementById('reimport-bar');
if (bar) bar.style.display = 'none';
this.status(`${summary.counts.final_annotations || 0} positioned annotations processed; ${this.rows.length} review rows remain.`, results.failed.length > 0);
// Anonymised sharing if opted in — robust to missing prefs (British spelling)
// Pseudonymous sharing after explicit consent.
try {
let share = true;
try { share = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) { share = true; }
let share = false;
try { share = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) {}
if (share) {
let anonId = "anon";
try { anonId = Services.prefs.getCharPref("extensions.kindleZoteroImporter.anonId") || "anon"; } catch(e) { anonId = "anon"; }
// ensure anonId exists for next time
if (anonId === "anon") {
try {
anonId = Math.random().toString(36).slice(2,10) + Date.now().toString(36);
Services.prefs.setCharPref("extensions.kindleZoteroImporter.anonId", anonId);
} catch(e) {}
}
const {anonId, token} = await this.collectorCredentials();
const anonData = (data.finalPlan?.annotations || []).map(a => ({
citation_key: a.citation_key,
doi: a.doi || null,
isbn: a.isbn || null,
issn: a.issn || null,
publication_title: a.publication_title || null,
clipping_id_hash: a.clipping_id ? a.clipping_id.slice(0,8) : null,
norm_hash: a.norm_hash || null,
fuzzy_hash: a.fuzzy_hash || null,
text: a.annotation?.text || "",
citation_key: (a.citation_key || '').slice(0, 512) || null,
doi: (a.doi || '').slice(0, 512) || null,
isbn: (a.isbn || '').slice(0, 512) || null,
issn: (a.issn || '').slice(0, 512) || null,
publication_title: (a.publication_title || '').slice(0, 1000) || null,
clipping_id_hash: (a.clipping_id || '').slice(0, 128) || null,
norm_hash: (a.norm_hash || '').slice(0, 256) || null,
fuzzy_hash: (a.fuzzy_hash || '').slice(0, 256) || null,
text: (a.annotation?.text || '').slice(0, 2000),
color: a.annotation?.color,
has_comment: !!(a.annotation?.comment),
comment: a.annotation?.comment || "",
comment_hash: a.annotation?.comment ? btoa(a.annotation.comment).slice(0,24) : null,
clipping_added_on_iso: a.clipping_added_on_iso,
integrated_at: a.integrated_at
comment: (a.annotation?.comment || '').slice(0, 2000),
comment_hash: null,
clipping_added_on_iso: (a.clipping_added_on_iso || '').slice(0, 64) || null,
integrated_at: (a.integrated_at || '').slice(0, 64) || null
}));
// Fire and forget, no text sent — private, not public (annotation.utkubilen.de)
fetch("https://annotation.utkubilen.de/highlights", {
for (const batch of this.collectorBatches(anonId, anonData)) {
const response = await fetch("https://annotation.utkubilen.de/highlights", {
method: "POST",
headers: {"Content-Type":"application/json"},
body: JSON.stringify({anonId, data: anonData})
}).catch(()=>{});
headers: {"Content-Type":"application/json", "Authorization": `Bearer ${token}`},
body: JSON.stringify({anonId, data: batch})
});
if (!response.ok) throw new Error(`Collector returned HTTP ${response.status}`);
}
}
} catch(e) {
this.status(`Import succeeded, but annotation sharing failed: ${e}`, true);
}
} catch(e) {}
},
failRun(message) {
@ -298,6 +326,11 @@
return !!(cb && cb.checked);
},
getRetryUnresolved() {
const cb = document.getElementById('retry-unresolved-check');
return !!(cb && cb.checked);
},
updateInfobox() {
const box = document.getElementById('infobox');
if (!box) return;
@ -614,8 +647,54 @@
} catch { return String(iso).slice(0,10); }
},
paginate(items, pageKey, pageSizeKey) {
const totalPages = Math.max(1, Math.ceil(items.length / this[pageSizeKey]));
if (this[pageKey] >= totalPages) this[pageKey] = totalPages - 1;
if (this[pageKey] < 0) this[pageKey] = 0;
const start = this[pageKey] * this[pageSizeKey];
return { rows: items.slice(start, start + this[pageSizeKey]), totalPages };
},
refreshPagination(containerId, pageKey, pageSizeKey, totalPages, refresh) {
const container = document.getElementById(containerId);
if (!container) return;
container.textContent = '';
const button = (label, disabled, action) => {
const el = document.createElement('button');
el.textContent = label;
el.disabled = disabled;
if (!disabled) el.addEventListener('click', action);
return el;
};
container.appendChild(button('Previous', this[pageKey] === 0, () => {
this[pageKey] -= 1;
refresh();
}));
const status = document.createElement('span');
status.textContent = `Page ${this[pageKey] + 1} of ${totalPages}`;
container.appendChild(status);
container.appendChild(button('Next', this[pageKey] >= totalPages - 1, () => {
this[pageKey] += 1;
refresh();
}));
const select = document.createElement('select');
for (const size of [25, 50, 100, 200]) {
const option = document.createElement('option');
option.value = size;
option.textContent = `${size} per page`;
option.selected = size === this[pageSizeKey];
select.appendChild(option);
}
select.addEventListener('change', () => {
this[pageSizeKey] = Number(select.value);
this[pageKey] = 0;
refresh();
});
container.appendChild(select);
},
getIntegratedHighlights() {
const anns = this.data.finalPlan?.annotations || [];
const anns = this.data.finalPlan?.all_annotations || this.data.finalPlan?.annotations || [];
return anns.slice().sort((a,b) => (a.clipping_title || '').localeCompare(b.clipping_title || '') || (a.annotation?.text || '').localeCompare(b.annotation?.text || ''));
},
@ -639,82 +718,11 @@
const dateMap = this.buildClippingDateMap();
const filtered = needle ? all.filter(a => `${a.clipping_title} ${a.annotation?.text || ''} ${a.citation_key || ''} ${a.annotation?.pageLabel || ''} ${a.clipping_added_on_iso || ''} ${a.integrated_at || ''}`.toLowerCase().includes(needle)) : all;
if (resetPage) this.integratedPage = 0;
const totalPages = Math.max(1, Math.ceil(filtered.length / this.integratedPageSize));
if (this.integratedPage >= totalPages) this.integratedPage = totalPages - 1;
const start = this.integratedPage * this.integratedPageSize;
const toShow = filtered.slice(start, start + this.integratedPageSize);
const page = this.paginate(filtered, 'integratedPage', 'integratedPageSize');
const totalPages = page.totalPages;
const toShow = page.rows;
if (countEl) countEl.textContent = `${filtered.length} of ${all.length} — page ${this.integratedPage + 1}/${totalPages} (${this.integratedPageSize} per page)`;
// pagination controls
let pag = document.getElementById('integrated-pagination');
if (!pag) {
pag = document.createElement('div');
pag.id = 'integrated-pagination';
pag.style.display = 'flex';
pag.style.justifyContent = 'space-between';
pag.style.alignItems = 'center';
pag.style.margin = '8px 0';
pag.style.gap = '8px';
pag.style.flexWrap = 'wrap';
tbody.parentElement.parentElement.appendChild(pag);
}
pag.textContent = '';
const mkBtn = (label, disabled, onClick) => {
const b = document.createElement('button');
b.textContent = label;
b.disabled = disabled;
b.style.padding = '4px 8px';
b.style.border = '1px solid var(--line)';
b.style.background = disabled ? 'var(--panel-soft)' : 'var(--panel)';
b.style.borderRadius = 'var(--radius)';
b.style.cursor = disabled ? 'not-allowed' : 'pointer';
b.style.fontSize = '11px';
if (!disabled) b.addEventListener('click', onClick);
return b;
};
const left = document.createElement('div');
left.style.display = 'flex';
left.style.gap = '4px';
left.appendChild(mkBtn('‹ Prev', this.integratedPage === 0, () => { this.integratedPage--; this.refreshIntegrated(); }));
left.appendChild(mkBtn('Next ›', this.integratedPage >= totalPages - 1, () => { this.integratedPage++; this.refreshIntegrated(); }));
const mid = document.createElement('span');
mid.className = 'muted';
mid.textContent = `Page ${this.integratedPage + 1} of ${totalPages}`;
left.appendChild(mid);
pag.appendChild(left);
const right = document.createElement('div');
right.style.display = 'flex';
right.style.gap = '6px';
right.style.alignItems = 'center';
const sel = document.createElement('select');
sel.style.padding = '4px';
sel.style.border = '1px solid var(--line)';
sel.style.borderRadius = 'var(--radius)';
sel.style.fontSize = '11px';
for (const n of [25, 50, 100, 200, 500]) {
const opt = document.createElement('option');
opt.value = n;
opt.textContent = n + ' per page';
if (n === this.integratedPageSize) opt.selected = true;
sel.appendChild(opt);
}
const optAll = document.createElement('option');
optAll.value = filtered.length || 500;
optAll.textContent = 'Show all';
sel.appendChild(optAll);
sel.addEventListener('change', () => { this.integratedPageSize = parseInt(sel.value, 10); this.integratedPage = 0; this.refreshIntegrated(); });
right.appendChild(sel);
const expand = document.createElement('button');
expand.textContent = 'Expand 50 →';
expand.title = 'Show next 50';
expand.style.padding = '4px 8px';
expand.style.border = '1px solid var(--line)';
expand.style.background = 'var(--panel)';
expand.style.borderRadius = 'var(--radius)';
expand.style.cursor = 'pointer';
expand.style.fontSize = '11px';
expand.addEventListener('click', () => { this.integratedPageSize += 50; this.refreshIntegrated(); });
right.appendChild(expand);
pag.appendChild(right);
this.refreshPagination('integrated-pagination', 'integratedPage', 'integratedPageSize', totalPages, () => this.refreshIntegrated());
if (!toShow.length) {
const tr = document.createElement('tr');
const td = document.createElement('td');
@ -768,7 +776,7 @@
}
},
refreshMappings() {
refreshMappings(resetPage = false) {
const tbody = document.getElementById('mappings-rows');
const countEl = document.getElementById('mappings-count');
if (!tbody) return;
@ -776,7 +784,10 @@
const needle = (document.getElementById('mappings-search')?.value || '').toLowerCase();
const overrides = this.data.persistentOverrides?.overrides || [];
const filtered = needle ? overrides.filter(o => `${o.clipping_title} ${o.resolution?.citation_key || ''} ${o.resolution?.zotero_key || ''} ${o.resolution?.ignore ? 'ignored' : ''}`.toLowerCase().includes(needle)) : overrides;
if (countEl) countEl.textContent = `${filtered.length} of ${overrides.length} mappings`;
if (resetPage) this.mappingsPage = 0;
const page = this.paginate(filtered, 'mappingsPage', 'mappingsPageSize');
if (countEl) countEl.textContent = `${filtered.length} of ${overrides.length} mappings — page ${this.mappingsPage + 1}/${page.totalPages}`;
this.refreshPagination('mappings-pagination', 'mappingsPage', 'mappingsPageSize', page.totalPages, () => this.refreshMappings());
if (!filtered.length) {
const tr = document.createElement('tr');
const td = document.createElement('td');
@ -786,7 +797,7 @@
tbody.appendChild(tr);
return;
}
for (const entry of filtered) {
for (const entry of page.rows) {
const tr = document.createElement('tr');
const tdTitle = document.createElement('td');
tdTitle.textContent = entry.clipping_title;
@ -886,6 +897,7 @@
this.settings = { ...this.settings, ...newSettings };
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", shareAnonymized);
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare", true);
Services.prefs.setIntPref("extensions.kindleZoteroImporter.shareConsentVersion", 2);
if (shareAnonymized) {
try {
Services.prefs.getCharPref("extensions.kindleZoteroImporter.anonId");
@ -1005,6 +1017,25 @@
fullLabel.title = 'When checked, re-processes all 2379 clippings from scratch (slow). Unchecked = incremental (only new/changed highlights, fast).';
fullRow.appendChild(fullLabel);
panel.appendChild(fullRow);
const retryRow = document.createElement('div');
retryRow.style.marginTop = '6px';
retryRow.style.display = 'flex';
retryRow.style.alignItems = 'center';
retryRow.style.gap = '6px';
const retryCheck = document.createElement('input');
retryCheck.type = 'checkbox';
retryCheck.id = 'retry-unresolved-check';
retryCheck.style.width = 'auto';
retryRow.appendChild(retryCheck);
const retryLabel = document.createElement('label');
retryLabel.htmlFor = 'retry-unresolved-check';
retryLabel.textContent = 'Retry unresolved attachments and positions';
retryLabel.style.margin = '0';
retryLabel.style.fontSize = '12px';
retryLabel.style.color = 'var(--muted)';
retryLabel.title = 'Retries prior matching and positioning failures without rebuilding annotations that already succeeded.';
retryRow.appendChild(retryLabel);
panel.appendChild(retryRow);
root.appendChild(panel);
const resultsDiv = document.createElement('div');
@ -1081,6 +1112,10 @@
iTbody.id='integrated-rows';
iTable.appendChild(iTbody);
integratedTab.appendChild(iTable);
const iPagination = document.createElement('div');
iPagination.id = 'integrated-pagination';
iPagination.className = 'pagination';
integratedTab.appendChild(iPagination);
root.appendChild(integratedTab);
const conflictsTab = document.createElement('div');
@ -1129,7 +1164,7 @@
search.placeholder = 'Filter conflicts by title, status, candidate, citekey...';
search.style.width = '320px';
search.style.maxWidth = '50%';
search.addEventListener('input', () => this.refreshRows());
search.addEventListener('input', () => this.refreshRows(true));
searchWrap.appendChild(search);
conflictsTab.appendChild(searchWrap);
const table = document.createElement('table');
@ -1140,6 +1175,10 @@
tbody.id = 'rows';
table.appendChild(tbody);
conflictsTab.appendChild(table);
const cPagination = document.createElement('div');
cPagination.id = 'conflicts-pagination';
cPagination.className = 'pagination';
conflictsTab.appendChild(cPagination);
root.appendChild(conflictsTab);
const mappingsTab = document.createElement('div');
@ -1161,7 +1200,7 @@
mSearch.placeholder = 'Filter mappings by title, citekey...';
mSearch.style.width = '320px';
mSearch.style.maxWidth = '50%';
mSearch.addEventListener('input', () => this.refreshMappings());
mSearch.addEventListener('input', () => this.refreshMappings(true));
mSearchWrap.appendChild(mSearch);
mappingsTab.appendChild(mSearchWrap);
const mCount = document.createElement('div');
@ -1177,6 +1216,10 @@
mTbody.id = 'mappings-rows';
mTable.appendChild(mTbody);
mappingsTab.appendChild(mTable);
const mPagination = document.createElement('div');
mPagination.id = 'mappings-pagination';
mPagination.className = 'pagination';
mappingsTab.appendChild(mPagination);
root.appendChild(mappingsTab);
const settingsTab = document.createElement('div');
@ -1207,11 +1250,11 @@
const shareCheck = document.createElement('input');
shareCheck.type = 'checkbox';
shareCheck.id = 'setting-shareAnonymized';
try { shareCheck.checked = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) { shareCheck.checked = true; }
try { shareCheck.checked = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) { shareCheck.checked = false; }
shareRow.appendChild(shareCheck);
const shareLabel = document.createElement('label');
shareLabel.htmlFor = 'setting-shareAnonymized';
shareLabel.textContent = 'Share anonymised annotations to help future development (no highlight text, only citation key + hashed highlight + notes if any)';
shareLabel.textContent = 'Share annotation data to help future development (opt in: pseudonymous ID, publication metadata, raw highlight text and comments, hashes, colour, and dates)';
shareLabel.style.margin = '0';
shareLabel.style.fontSize = '12px';
shareRow.appendChild(shareLabel);
@ -1550,17 +1593,17 @@
this.makeTableResizable(document.querySelector('#tab-conflicts table'));
this.makeTableResizable(document.querySelector('#tab-integrated table'));
this.makeTableResizable(document.querySelector('#tab-mappings table'));
this.makeTableSortable(document.querySelector('#tab-conflicts table'));
this.makeTableSortable(document.querySelector('#tab-integrated table'));
this.makeTableSortable(document.querySelector('#tab-mappings table'));
},
refreshRows() {
refreshRows(resetPage = false) {
const tbody = document.getElementById('rows');
if (!tbody) return;
tbody.textContent = '';
const needle = (document.getElementById('search')?.value || '').toLowerCase();
const filtered = this.rows.filter(row => `${row.source} ${row.status} ${row.title} ${row.detail || ''} ${(row.candidates || []).map(c => `${c.citation_key} ${c.title}`).join(' ')}`.toLowerCase().includes(needle)).slice(0,300);
const filtered = this.rows.filter(row => `${row.source} ${row.status} ${row.title} ${row.detail || ''} ${(row.candidates || []).map(c => `${c.citation_key} ${c.title}`).join(' ')}`.toLowerCase().includes(needle));
if (resetPage) this.conflictsPage = 0;
const page = this.paginate(filtered, 'conflictsPage', 'conflictsPageSize');
this.refreshPagination('conflicts-pagination', 'conflictsPage', 'conflictsPageSize', page.totalPages, () => this.refreshRows());
if (!filtered.length) {
const tr = document.createElement('tr');
const td = document.createElement('td');
@ -1570,8 +1613,8 @@
tbody.appendChild(tr);
return;
}
for (let idx = 0; idx < filtered.length; idx++) {
const row = filtered[idx];
for (let idx = 0; idx < page.rows.length; idx++) {
const row = page.rows[idx];
const globalIndex = this.rows.indexOf(row);
const tr = document.createElement('tr');
const tdSource = document.createElement('td');

View file

@ -1,7 +1,7 @@
{
"manifest_version": 2,
"name": "Kanzi",
"version": "0.6.20",
"version": "0.7.0",
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
"author": "Utku Bilen Demir",

View file

@ -1,6 +1,7 @@
pref("extensions.kindleZoteroImporter.projectDir", "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer");
pref("extensions.kindleZoteroImporter.pythonPath", "/opt/homebrew/bin/python3");
pref("extensions.kindleZoteroImporter.zoteroDbPath", "/home/user/Zotero/zotero.sqlite");
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "/home/user/Zotero/storage");
pref("extensions.kindleZoteroImporter.shareAnonymized", true);
pref("extensions.kindleZoteroImporter.projectDir", "");
pref("extensions.kindleZoteroImporter.pythonPath", "python3");
pref("extensions.kindleZoteroImporter.zoteroDbPath", "");
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "");
pref("extensions.kindleZoteroImporter.shareAnonymized", false);
pref("extensions.kindleZoteroImporter.askedAnonymizedShare", false);
pref("extensions.kindleZoteroImporter.shareConsentVersion", 0);

View file

@ -1,11 +1,14 @@
[project]
name = "kindle-zotero-importer"
version = "0.1.0"
version = "0.7.0"
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
readme = "README.md"
requires-python = ">=3.11"
dependencies = []
[project.optional-dependencies]
collector = ["fastapi>=0.110", "uvicorn>=0.29"]
[project.scripts]
kindle-zotero-importer = "kindle_zotero_importer.cli:main"

View file

@ -1,62 +0,0 @@
from __future__ import annotations
import json
import zipfile
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
DIST_DIR = ROOT / "dist"
OUTPUT = DIST_DIR / "kindle-zotero-importer-canary.xpi"
def main() -> int:
DIST_DIR.mkdir(exist_ok=True)
if OUTPUT.exists():
OUTPUT.unlink()
manifest = {
"manifest_version": 2,
"name": "Kindle Zotero Importer Canary",
"version": "0.1.0",
"description": "Minimal install test for Kindle Zotero Importer.",
"homepage_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer",
"author": "Utku Bilen Demir",
"applications": {
"zotero": {
"id": "kindlezoteroimportercanary@utkubilen.de",
"update_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates-canary.json",
"strict_min_version": "6.999",
"strict_max_version": "9.*",
}
},
}
bootstrap = """
function install(data, reason) {}
function uninstall(data, reason) {}
async function startup(data, reason) {
await Zotero.initializationPromise;
Zotero.debug('Kindle Zotero Importer Canary started');
}
function shutdown(data, reason) {}
""".lstrip()
prefs = 'pref("extensions.kindleZoteroImporterCanary.enabled", true);\n'
with zipfile.ZipFile(OUTPUT, "w", zipfile.ZIP_DEFLATED) as archive:
for name, content in [
("bootstrap.js", bootstrap),
("manifest.json", json.dumps(manifest, indent=2) + "\n"),
("prefs.js", prefs),
]:
info = zipfile.ZipInfo(name)
info.create_system = 3
info.external_attr = 0o644 << 16
archive.writestr(info, content, zipfile.ZIP_DEFLATED)
print(f"Built {OUTPUT}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -60,12 +60,19 @@ def main() -> int:
info.create_system = 3
info.external_attr = 0o644 << 16
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
runtime_files = [ROOT / "plugin_runner.py", *(ROOT / "src" / "kindle_zotero_importer").glob("*.py")]
for path in runtime_files:
archive_name = "runtime/" + str(path.relative_to(ROOT))
info = zipfile.ZipInfo(archive_name)
info.create_system = 3
info.external_attr = 0o644 << 16
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
with zipfile.ZipFile(OUTPUT) as archive:
names = set(archive.namelist())
if "manifest.json" not in names or "bootstrap.js" not in names:
if not {"manifest.json", "bootstrap.js", "runtime/plugin_runner.py"}.issubset(names):
raise SystemExit(
"Invalid XPI: manifest.json and bootstrap.js must be at archive root"
"Invalid XPI: plugin files and Python runtime are required"
)
print(f"Built {OUTPUT}")

View file

@ -7,8 +7,8 @@ from pathlib import Path
from .clippings import clippings_to_jsonable, load_clippings
from .epub_position import add_epub_positions
from .final_plan import build_final_writer_plan
from .import_plan import DEFAULT_COLOUR_MAP, build_import_plan
from .final_plan import FINAL_FORMAT, build_final_writer_plan
from .import_plan import DEFAULT_COLOUR_MAP, _attach_notes_to_highlights, build_import_plan
from .matcher import build_match_report, load_json
from .mismatch_review import build_mismatch_review
from .overrides import generate_override_skeleton, load_overrides
@ -39,6 +39,106 @@ def _load_colour_map(workdir: Path) -> dict[str, str] | None:
return None
def _merge_annotation_history(
previous_plan: dict | None, current_annotations: list[dict], deletions: list[str]
) -> list[dict]:
previous = []
if previous_plan:
previous = previous_plan.get("all_annotations") or previous_plan.get("annotations") or []
deleted = set(deletions)
by_id = {
annotation["clipping_id"]: annotation
for annotation in previous
if annotation.get("clipping_id") and annotation["clipping_id"] not in deleted
}
for annotation in current_annotations:
clipping_id = annotation.get("clipping_id")
if clipping_id:
by_id[clipping_id] = annotation
return sorted(
by_id.values(),
key=lambda annotation: (
annotation.get("attachment_item_id") or 0,
annotation.get("annotation", {}).get("sortIndex") or "",
annotation.get("clipping_id") or "",
),
)
def _changed_note_clipping_ids(
current_clippings: list[dict], previous_annotations: list[dict]
) -> set[str]:
previous_by_id = {
annotation.get("clipping_id"): annotation
for annotation in previous_annotations
if annotation.get("clipping_id")
}
return {
clipping["id"]
for clipping in current_clippings
if clipping["id"] in previous_by_id
and previous_by_id[clipping["id"]].get("note_ids", [])
!= clipping.get("note_ids", [])
}
def _merge_positioned_history(
previous_plan: dict | None, current_plan: dict, source_ids: set[str]
) -> dict:
by_id = {
item.get("clipping", {}).get("id"): item
for item in (previous_plan or {}).get("items", [])
if item.get("clipping", {}).get("id") in source_ids
}
for item in current_plan.get("items", []):
clipping_id = item.get("clipping", {}).get("id")
if clipping_id:
by_id[clipping_id] = item
merged = dict(current_plan)
merged["items"] = list(by_id.values())
status_counts: dict[str, int] = {}
for item in merged["items"]:
status = item.get("status", "unknown")
status_counts[status] = status_counts.get(status, 0) + 1
merged["status_counts"] = status_counts
return merged
def _deletion_ids(
previous_integrated_ids: set[str], source_ids: set[str], previous_plan: dict | None
) -> set[str]:
deleted = set(previous_integrated_ids - source_ids)
deleted.update((previous_plan or {}).get("delete_failed_ids", []))
return deleted
def _add_previous_imported_comments(
annotations: list[dict], previous_plan: dict | None
) -> None:
previous_by_id = {
annotation.get("clipping_id"): annotation
for annotation in ((previous_plan or {}).get("all_annotations") or (previous_plan or {}).get("annotations", []))
if annotation.get("clipping_id")
}
for annotation in annotations:
previous = previous_by_id.get(annotation.get("clipping_id"))
if previous:
annotation["previous_imported_comment"] = previous.get("annotation", {}).get(
"comment", ""
)
def _unresolved_clipping_ids(
previous_plan: dict | None, source_ids: set[str]
) -> set[str]:
return {
item.get("clipping", {}).get("id")
for item in (previous_plan or {}).get("items", [])
if item.get("status") not in {"positioned", "ignored-title"}
and item.get("clipping", {}).get("id") in source_ids
}
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="kindle-zotero-importer",
@ -61,11 +161,11 @@ def main(argv: list[str] | None = None) -> int:
"index-zotero", help="Index Zotero items and attachments read-only"
)
index_parser.add_argument(
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite"
"--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
)
index_parser.add_argument(
"--storage-root",
default="/home/user/Zotero/storage",
default=str(Path.home() / "Zotero" / "storage"),
help="Path to Zotero storage directory for storage: attachments",
)
index_parser.add_argument(
@ -175,11 +275,11 @@ def main(argv: list[str] | None = None) -> int:
help="Directory for generated importer artifacts",
)
run_parser.add_argument(
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite"
"--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
)
run_parser.add_argument(
"--storage-root",
default="/home/user/Zotero/storage",
default=str(Path.home() / "Zotero" / "storage"),
help="Path to Zotero storage directory for storage: attachments",
)
run_parser.add_argument(
@ -203,6 +303,11 @@ def main(argv: list[str] | None = None) -> int:
action="store_true",
help="Force full re-import from scratch, ignoring incremental state (processes all clippings)",
)
run_parser.add_argument(
"--retry-unresolved",
action="store_true",
help="Retry previously unresolved positioning and attachment cases",
)
args = parser.parse_args(argv)
@ -373,37 +478,55 @@ def main(argv: list[str] | None = None) -> int:
report_progress(2, "Reading clippings", "Parsing My Clippings.txt")
new_clippings_list = load_clippings(args.clippings_file)
new_clippings = clippings_to_jsonable(new_clippings_list)
new_ids = {c["id"] for c in new_clippings.get("clippings", [])}
source_clippings = new_clippings.get("clippings", [])
source_ids = {c["id"] for c in source_clippings}
effective_clippings = _attach_notes_to_highlights(source_clippings)
new_ids = {c["id"] for c in effective_clippings}
# Incremental state: previous clippings and previous integrated ids
is_incremental = not args.full
prev_clippings_ids: set[str] = set()
prev_effective_ids: set[str] = set()
prev_integrated_ids: set[str] = set()
prev_final_plan = None
if is_incremental:
prev_clippings_path = artifact("clippings.json")
prev_positioned_plan = None
prev_final_path = artifact("import-plan.final.json")
if prev_clippings_path.exists():
try:
prev_clippings_ids = {c["id"] for c in load_json(str(prev_clippings_path)).get("clippings", [])}
except Exception:
prev_clippings_ids = set()
if prev_final_path.exists():
try:
prev_final_plan = load_json(str(prev_final_path))
prev_integrated_ids = {a.get("clipping_id") for a in prev_final_plan.get("annotations", []) if a.get("clipping_id")}
previous_annotations = prev_final_plan.get("all_annotations") or prev_final_plan.get("annotations", [])
prev_integrated_ids = {a.get("clipping_id") for a in previous_annotations if a.get("clipping_id")}
except Exception:
prev_final_plan = None
prev_integrated_ids = set()
if is_incremental:
prev_clippings_path = artifact("clippings.json")
prev_positioned_path = artifact("import-plan.positioned.json")
if prev_clippings_path.exists():
try:
previous_clippings = load_json(str(prev_clippings_path)).get("clippings", [])
prev_clippings_ids = {c["id"] for c in previous_clippings}
prev_effective_ids = {
c["id"] for c in _attach_notes_to_highlights(previous_clippings)
}
except Exception:
prev_clippings_ids = set()
prev_effective_ids = set()
if prev_positioned_path.exists():
try:
prev_positioned_plan = load_json(str(prev_positioned_path))
except Exception:
prev_positioned_plan = None
# If no previous state, fall back to full
if not prev_clippings_ids and not prev_integrated_ids:
is_incremental = False
if is_incremental:
# New or previously not integrated (skipped) -> need to (re)process
to_process_ids = set(new_ids - prev_integrated_ids)
# Only source entries not seen on the preceding run need positioning.
# Previously unresolved entries are re-queued when their override changes.
to_process_ids = set(new_ids - prev_effective_ids)
# Deletions: previously integrated but now absent (removed or changed text)
deletions_ids = sorted(prev_integrated_ids - new_ids)
deletions_ids = _deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
# Also re-queue any previously integrated clippings whose title now has an override
# (mapping may have changed, need to move annotation). We need overrides dict for this.
# Load overrides early for this check (already loaded later, but do here for incremental decision)
@ -412,54 +535,38 @@ def main(argv: list[str] | None = None) -> int:
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
except Exception:
_ov_for_inc = {}
for c in new_clippings_list:
if c.id in prev_integrated_ids and c.title in _ov_for_inc:
to_process_ids.add(c.id)
# Also re-queue highlights whose bracket colour would change the annotation
# (so you can recolour a past highlight by adding [r]/[o] etc. to its note or highlight text,
# and also fix old double-bracket comments like "[r] note" + "note")
if colour_map:
import re as _re
# Build map of notes by title for quick lookup (for bracket on note)
notes_by_title: dict[str, list] = {}
for n in new_clippings_list:
if n.kind == "note":
m = _re.match(r"^\s*\[([^\]]+)\]\s*", n.text or "")
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
notes_by_title.setdefault(n.title, []).append(n)
# also check for double-bracket case: note text without bracket but highlight already has [r] in Zotero
# For that, we need to re-queue if existing Zotero comment contains "[r]"
for c in new_clippings_list:
if c.id not in prev_integrated_ids:
continue
if c.kind != "highlight":
continue
has_bracket = False
# check highlight text itself
for t in [c.text or "", getattr(c, "comment", "") or ""]:
m = _re.match(r"^\s*\[([^\]]+)\]\s*", t)
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
has_bracket = True
break
# also check if any note for same title has bracket (new or old) — for already integrated highlights,
# the note that would be attached is in new_clippings_list; check all notes for title
if not has_bracket and c.title in notes_by_title:
has_bracket = True
# also check if previous final plan's annotation for this clipping had a bracket comment that is now stripped
# (handles the double "[r] note" + "note" case)
if not has_bracket and prev_final_plan:
for ann in prev_final_plan.get("annotations", []):
if ann.get("clipping_id") == c.id:
prev_comment = ann.get("annotation", {}).get("comment") or ""
# if prev had "[r] note" and new would be "note", they differ
for part in prev_comment.split("\n\n"):
m = _re.match(r"^\s*\[([^\]]+)\]\s*", part.strip())
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
has_bracket = True
break
break
if has_bracket:
to_process_ids.add(c.id)
previous_override_snapshot = (prev_final_plan or {}).get("overrides_snapshot")
if previous_override_snapshot is None:
changed_override_titles = set(_ov_for_inc)
else:
changed_override_titles = {
title
for title in set(previous_override_snapshot) | set(_ov_for_inc)
if previous_override_snapshot.get(title) != _ov_for_inc.get(title)
}
for c in effective_clippings:
if c["title"] in changed_override_titles:
to_process_ids.add(c["id"])
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
for annotation in previous_annotations:
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
deletions_ids.add(annotation["clipping_id"])
newly_added_source_ids = source_ids - prev_clippings_ids
for c in effective_clippings:
if set(c.get("note_ids", [])) & newly_added_source_ids:
to_process_ids.add(c["id"])
to_process_ids.update(
_changed_note_clipping_ids(effective_clippings, previous_annotations)
)
to_process_ids.update(
clipping_id
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
if clipping_id in new_ids
)
if args.retry_unresolved and prev_positioned_plan:
to_process_ids.update(
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
)
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
try:
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
@ -470,6 +577,7 @@ def main(argv: list[str] | None = None) -> int:
to_process_ids.update(prev_integrated_ids)
except Exception:
pass
deletions_ids = sorted(deletions_ids)
# Also handle deleted overrides: if a previously integrated title's override was deleted,
# its annotations should be deleted. Detect via previous final plan's titles vs current overrides.
# For incremental, we don't have previous overrides, so we handle deletions via the main deletions list
@ -481,7 +589,7 @@ def main(argv: list[str] | None = None) -> int:
# handled as not in to_process_ids? Actually it is still in new_ids, but we re-queued it above only if title in overrides.
# For now, rely on full re-import for deleted-override cleanup.
# Build filtered clippings for this delta run
filtered_list = [c for c in new_clippings_list if c.id in to_process_ids]
filtered_list = [c for c in effective_clippings if c["id"] in to_process_ids]
# If nothing to do and no deletions, still need to report but skip expensive steps
if not filtered_list and not deletions_ids:
report_progress(15, "Matching titles", "No new highlights — incremental skip")
@ -494,6 +602,7 @@ def main(argv: list[str] | None = None) -> int:
matches = build_match_report(clippings, zotero_index, overrides)
override_skeleton = generate_override_skeleton(matches)
plan = {"format": "kindle-zotero-importer.import-plan.v1", "items": []}
epub_plan = {"format": plan["format"], "items": []}
positioned_plan = {"format": plan["format"], "items": []}
final_plan = {"format": FINAL_FORMAT, "source_format": positioned_plan.get("format"), "annotation_count": 0, "skipped_counts": {}, "annotations": [], "deletions": [], "is_incremental": True, "incremental_stats": {"new_clippings": len(new_ids), "prev_integrated": len(prev_integrated_ids), "to_process": 0, "deletions": 0}}
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
@ -507,7 +616,11 @@ def main(argv: list[str] | None = None) -> int:
if overrides_path.exists():
overrides = load_overrides(load_json(str(overrides_path)))
# Only the delta clippings go through matching/positioning
delta_clippings = clippings_to_jsonable(filtered_list)
delta_clippings = {
"format": new_clippings.get("format"),
"count": len(filtered_list),
"clippings": filtered_list,
}
report_progress(15, "Matching titles", f"Matching {len(filtered_list)} new/pending highlights (incremental)")
matches = build_match_report(delta_clippings, zotero_index, overrides)
override_skeleton = generate_override_skeleton(matches)
@ -554,12 +667,44 @@ def main(argv: list[str] | None = None) -> int:
positioned_plan = add_pdf_positions(epub_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan)
final_plan["deletions"] = []
final_plan["deletions"] = sorted(
_deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
)
final_plan["is_incremental"] = False
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
final_plan["colorMap"] = final_plan["colourMap"]
mismatch_review = build_mismatch_review(positioned_plan, matches)
if is_incremental:
positioned_plan = _merge_positioned_history(
prev_positioned_plan, positioned_plan, new_ids
)
matches = build_match_report(new_clippings, zotero_index, overrides)
override_skeleton = generate_override_skeleton(matches)
mismatch_review = build_mismatch_review(positioned_plan, matches)
_add_previous_imported_comments(
final_plan.get("annotations", []), prev_final_plan
)
delta_annotation_count = len(final_plan.get("annotations", []))
if is_incremental:
final_plan["all_annotations"] = _merge_annotation_history(
prev_final_plan,
final_plan.get("annotations", []),
final_plan.get("deletions", []),
)
else:
final_plan["all_annotations"] = _merge_annotation_history(
prev_final_plan,
final_plan.get("annotations", []),
final_plan.get("deletions", []),
)
final_plan["annotation_count"] = len(final_plan["all_annotations"])
final_plan["delta_annotation_count"] = delta_annotation_count
final_plan["processed_clipping_ids"] = sorted(new_ids)
final_plan["overrides_snapshot"] = overrides or {}
indent = 2 if args.pretty else None
outputs = {
"clippings": artifact("clippings.json"),
@ -610,7 +755,8 @@ def main(argv: list[str] | None = None) -> int:
"counts": {
"clippings": len(clippings.get("clippings", [])),
"unique_titles": len(matches.get("matches", [])),
"final_annotations": final_plan.get("annotation_count", 0),
"final_annotations": final_plan.get("delta_annotation_count", 0),
"integrated_annotations": final_plan.get("annotation_count", 0),
"match_statuses": match_counts,
"plan_statuses": status_counts,
"skipped_final": final_plan.get("skipped_counts", {}),

View file

@ -91,25 +91,59 @@ def _norm_text(text: str) -> str:
def dedup_clippings(clippings: list[Clipping]) -> list[Clipping]:
seen: dict[tuple[str, str], Clipping] = {}
result: list[Clipping] = []
exact_highlights: dict[tuple[str, str, str | None, str | None], int] = {}
revised_highlights: dict[tuple[str, int], int] = {}
seen_other_ids: set[str] = set()
for c in clippings:
if c.kind not in {"highlight", "note"}:
# keep bookmarks etc. as is
key = (c.title, c.id)
if key not in seen:
seen[key] = c
if c.kind != "highlight":
# Notes carry meaning independently and must never collide with highlights.
if c.id not in seen_other_ids:
result.append(c)
seen_other_ids.add(c.id)
continue
key = (c.title, _norm_text(c.text))
# keep earliest added_on, or first seen
if key not in seen:
seen[key] = c
else:
# if same norm text, keep the one with earlier added_on or shorter location range
prev = seen[key]
# prefer the one with more complete text (longer) or earlier date
if len(c.text) > len(prev.text):
seen[key] = c
return list(seen.values())
norm = _norm_text(c.text)
exact_key = (c.title, norm, c.page, c.location)
if exact_key in exact_highlights:
continue
location_start = _range_start(c.location)
revision_key = (c.title, location_start) if location_start is not None else None
if revision_key is not None and revision_key in revised_highlights:
previous_index = revised_highlights[revision_key]
previous = result[previous_index]
previous_norm = _norm_text(previous.text)
if _is_text_revision(previous_norm, norm):
# Kindle keeps earlier versions when a highlight is extended. The
# longest version represents the final selection.
if len(norm) > len(previous_norm):
result[previous_index] = c
exact_highlights[exact_key] = previous_index
continue
index = len(result)
result.append(c)
exact_highlights[exact_key] = index
if revision_key is not None:
revised_highlights[revision_key] = index
return result
def _range_start(value: str | None) -> int | None:
if not value:
return None
match = re.search(r"\d+", value)
return int(match.group()) if match else None
def _is_text_revision(left: str, right: str) -> bool:
if not left or not right:
return False
shorter, longer = sorted((left, right), key=len)
return len(shorter) >= 20 and longer.startswith(shorter)
def clippings_to_jsonable(clippings: list[Clipping]) -> dict[str, Any]:

View file

@ -65,6 +65,7 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
"clipping_title": clipping["title"],
"clipping_added_on": clipping.get("added_on"),
"clipping_added_on_iso": clipping.get("added_on_iso"),
"note_ids": clipping.get("note_ids", []),
"integrated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"attachment_item_id": attachment["item_id"],
"attachment_key": attachment["key"],

View file

@ -8,14 +8,15 @@ import shutil
import subprocess
import tempfile
import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Any
_TOOL_PATHS = {
"pdftotext": ("/home/user/bin/pdftotext", "/opt/homebrew/bin/pdftotext"),
"pdftohtml": ("/home/user/bin/pdftohtml", "/opt/homebrew/bin/pdftohtml"),
"pdfinfo": ("/home/user/bin/pdfinfo", "/opt/homebrew/bin/pdfinfo"),
"qpdf": ("/home/user/bin/qpdf", "/opt/homebrew/bin/qpdf"),
"pdftotext": (str(Path.home() / "bin" / "pdftotext"), "/opt/homebrew/bin/pdftotext"),
"pdftohtml": (str(Path.home() / "bin" / "pdftohtml"), "/opt/homebrew/bin/pdftohtml"),
"pdfinfo": (str(Path.home() / "bin" / "pdfinfo"), "/opt/homebrew/bin/pdfinfo"),
"qpdf": (str(Path.home() / "bin" / "qpdf"), "/opt/homebrew/bin/qpdf"),
}
@ -29,6 +30,15 @@ def _tool(name: str) -> str:
return name
def _missing_required_tools() -> list[str]:
missing = []
for name in ("pdftotext", "pdftohtml", "pdfinfo"):
path = _tool(name)
if path == name and shutil.which(name) is None:
missing.append(name)
return missing
@dataclass(frozen=True)
class PDFWord:
text: str
@ -54,8 +64,19 @@ def add_pdf_positions(plan: dict[str, Any]) -> dict[str, Any]:
size_cache: dict[tuple[str, int], tuple[float, float]] = {}
decrypted_cache: dict[str, str] = {}
positioned = []
missing_tools = _missing_required_tools()
with tempfile.TemporaryDirectory(prefix="kindle-zotero-pdf-") as temp_dir:
for item in plan["items"]:
attachment = (item.get("zotero") or {}).get("attachment") or {}
if (
missing_tools
and item.get("status") == "ready-for-positioning"
and attachment.get("content_type") == "application/pdf"
):
positioned.append(
_with_problem(item, f"pdf-tools-missing:{','.join(missing_tools)}")
)
else:
positioned.append(
_position_item(
item,

View file

@ -2,12 +2,14 @@ import sys
sys.path.insert(0, "src")
sys.path.insert(0, ".")
import importlib.util
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy"]:
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy","test_incremental","test_collector"]:
spec = importlib.util.spec_from_file_location(name, f"tests/{name}.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
for attr in dir(mod):
if attr.startswith("test_"):
if attr == "test_incremental_ids":
continue
getattr(mod, attr)()
print(f"✓ {name}.{attr}")
print("All Python tests passed")

View file

@ -1,4 +1,9 @@
from kindle_zotero_importer.clippings import parse_clippings_text, _stable_id
from kindle_zotero_importer.clippings import (
Clipping,
_stable_id,
dedup_clippings,
parse_clippings_text,
)
def test_parse_and_id_stable():
txt = "My Book (Author)\n- Your Highlight at location 10-20 | Added on Monday, 6 October 2025 11:32:29\n\nSome text\n==========\n"
@ -15,3 +20,57 @@ def test_added_on_iso():
txt = "B\n- Your Highlight at location 1 | Added on Tuesday, 18 May 2021 14:18:13\n\nText\n==========\n"
c = parse_clippings_text(txt)[0]
assert c.added_on_iso.startswith("2021-05-18T14:18:13")
def _clipping(identifier, kind, text, location):
return Clipping(
id=identifier,
title="Book",
kind=kind,
text=text,
raw_detail="detail",
location=location,
)
def test_dedup_keeps_longest_progressive_highlight_revision():
short = _clipping("short", "highlight", "A selection long enough to compare", "53-54")
long = _clipping(
"long",
"highlight",
"A selection long enough to compare and its extended ending",
"53-55",
)
assert dedup_clippings([short, long]) == [long]
def test_dedup_preserves_unrelated_text_with_same_location_start():
short = _clipping(
"short",
"highlight",
"J. he is actually of obscure extraction and this is long enough",
"53-54",
)
corrected = _clipping(
"corrected",
"highlight",
"A. J. he is actually of obscure extraction and this is long enough",
"53-55",
)
assert dedup_clippings([short, corrected]) == [short, corrected]
def test_dedup_preserves_same_text_at_different_locations():
first = _clipping("first", "highlight", "A repeated sentence in a book", "10-11")
second = _clipping("second", "highlight", "A repeated sentence in a book", "80-81")
assert dedup_clippings([first, second]) == [first, second]
def test_dedup_never_collides_notes_with_highlights():
highlight = _clipping("highlight", "highlight", "Same meaningful text", "10-11")
note = _clipping("note", "note", "Same meaningful text", "11")
assert dedup_clippings([highlight, note]) == [highlight, note]

63
tests/test_collector.py Normal file
View file

@ -0,0 +1,63 @@
import os
from pathlib import Path
import sys
import tempfile
import types
os.environ.setdefault(
"KANZI_ANNOTATION_DB", str(Path(tempfile.gettempdir()) / "kanzi-test-import.db")
)
try:
import fastapi # noqa: F401
except ModuleNotFoundError:
class _FastAPI:
def __init__(self, **_kwargs):
pass
def get(self, *_args, **_kwargs):
return lambda function: function
post = get
class _HTTPException(Exception):
def __init__(self, status_code: int, detail: str):
super().__init__(detail)
self.status_code = status_code
self.detail = detail
fake_fastapi = types.ModuleType("fastapi")
fake_fastapi.FastAPI = _FastAPI
fake_fastapi.HTTPException = _HTTPException
fake_fastapi.Request = object
fake_responses = types.ModuleType("fastapi.responses")
fake_responses.HTMLResponse = str
sys.modules["fastapi"] = fake_fastapi
sys.modules["fastapi.responses"] = fake_responses
from collector import app as collector
from fastapi import HTTPException
class _Request:
def __init__(self, token: str | None = None):
self.headers = {"authorization": f"Bearer {token}"} if token else {}
def test_collector_issues_and_verifies_installation_credentials():
original_db = collector.DB
with tempfile.TemporaryDirectory() as temporary:
collector.DB = Path(temporary) / "annotations.db"
collector.init_db()
credentials = collector.register()
assert collector._authenticated_anon_id(
_Request(credentials["token"])
) == credentials["anonId"]
try:
collector._authenticated_anon_id(_Request("wrong"))
except HTTPException as error:
assert error.status_code == 401
else:
raise AssertionError("Invalid collector token was accepted")
collector.DB = original_db

View file

@ -4,7 +4,7 @@ def test_final_includes_kindle_id_tag():
positioned = {
"format":"x",
"items":[
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None},
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None,"note_ids":["note1"]},
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
]
@ -15,6 +15,7 @@ def test_final_includes_kindle_id_tag():
assert {"name":"kindle-import"} in tags
assert {"name":"kindle-id:abc123"} in tags
assert plan["annotations"][0]["clipping_id"]=="abc123"
assert plan["annotations"][0]["note_ids"] == ["note1"]
def test_final_skips_non_positioned():
positioned = {"format":"x","items":[{"status":"epub-text-not-found","clipping":{"id":"a"},"zotero":{},"annotation":{}}]}

View file

@ -1,5 +1,18 @@
import json, tempfile, pathlib
import json
from pathlib import Path
import tempfile
from kindle_zotero_importer import cli
from kindle_zotero_importer.cli import (
_add_previous_imported_comments,
_changed_note_clipping_ids,
_deletion_ids,
_merge_annotation_history,
_merge_positioned_history,
_unresolved_clipping_ids,
)
from kindle_zotero_importer.clippings import clippings_to_jsonable, load_clippings
from kindle_zotero_importer import pdf_position
def test_incremental_ids(tmp_path: Path):
# simulate previous final with one integrated id
@ -17,3 +30,216 @@ def test_incremental_ids(tmp_path: Path):
# changed: aaa text changed -> new id ddd, old aaa no longer in new -> deletion
new_ids2 = {"ddd","bbb","ccc"}
assert (prev_integrated - new_ids2) == {"aaa"}
def test_merge_annotation_history_preserves_old_and_replaces_changed():
old_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "2"}}
old_b = {"clipping_id": "b", "attachment_item_id": 1, "annotation": {"sortIndex": "1"}}
new_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "3"}}
merged = _merge_annotation_history(
{"all_annotations": [old_a, old_b]}, [new_a], []
)
assert merged == [old_b, new_a]
def test_merge_annotation_history_removes_deletions():
previous = {
"annotations": [
{"clipping_id": "a", "attachment_item_id": 1, "annotation": {}},
{"clipping_id": "b", "attachment_item_id": 1, "annotation": {}},
]
}
merged = _merge_annotation_history(previous, [], ["a"])
assert [annotation["clipping_id"] for annotation in merged] == ["b"]
def test_full_history_retains_current_unresolved_annotation():
previous = {
"all_annotations": [
{"clipping_id": "unresolved", "attachment_item_id": 1, "annotation": {}},
{"clipping_id": "removed", "attachment_item_id": 1, "annotation": {}},
]
}
merged = _merge_annotation_history(previous, [], ["removed"])
assert [annotation["clipping_id"] for annotation in merged] == ["unresolved"]
def test_deletions_include_removed_and_previous_failures():
assert _deletion_ids(
{"kept", "removed"},
{"kept"},
{"delete_failed_ids": ["retry"]},
) == {"removed", "retry"}
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
previous = {
"all_annotations": [
{"clipping_id": "a", "annotation": {"comment": "old"}}
]
}
_add_previous_imported_comments(current, previous)
assert current[0]["previous_imported_comment"] == "old"
def test_retry_unresolved_excludes_positioned_ignored_and_removed_entries():
previous = {
"items": [
{"clipping": {"id": "retry"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "done"}, "status": "positioned"},
{"clipping": {"id": "ignored"}, "status": "ignored-title"},
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
]
}
assert _unresolved_clipping_ids(previous, {"retry", "done", "ignored"}) == {
"retry"
}
def test_changed_note_ids_detect_add_replace_and_remove():
previous = [
{"clipping_id": "added", "note_ids": []},
{"clipping_id": "replaced", "note_ids": ["old"]},
{"clipping_id": "removed", "note_ids": ["old"]},
{"clipping_id": "same", "note_ids": ["same-note"]},
]
current = [
{"id": "added", "note_ids": ["new"]},
{"id": "replaced", "note_ids": ["new"]},
{"id": "removed"},
{"id": "same", "note_ids": ["same-note"]},
]
assert _changed_note_clipping_ids(current, previous) == {
"added",
"replaced",
"removed",
}
def test_positioned_history_preserves_unresolved_and_replaces_current():
previous = {
"items": [
{"clipping": {"id": "keep"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "replace"}, "status": "epub-text-not-found"},
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
]
}
current = {
"format": "plan",
"items": [{"clipping": {"id": "replace"}, "status": "positioned"}],
}
merged = _merge_positioned_history(previous, current, {"keep", "replace"})
assert {item["clipping"]["id"]: item["status"] for item in merged["items"]} == {
"keep": "pdf-text-not-found",
"replace": "positioned",
}
assert merged["status_counts"] == {"pdf-text-not-found": 1, "positioned": 1}
def test_pdf_position_reports_missing_tools():
original = pdf_position._missing_required_tools
pdf_position._missing_required_tools = lambda: ["pdftotext", "pdfinfo"]
try:
result = pdf_position.add_pdf_positions(
{
"items": [
{
"status": "ready-for-positioning",
"clipping": {"kind": "highlight", "text": "highlight"},
"zotero": {
"attachment": {"content_type": "application/pdf"}
},
}
]
}
)
finally:
pdf_position._missing_required_tools = original
assert result["items"][0]["status"] == "pdf-tools-missing:pdftotext,pdfinfo"
def test_pdf_position_preserves_unresolved_item_without_zotero_match():
item = {
"status": "unmatched-title",
"clipping": {"kind": "highlight", "text": "highlight"},
"zotero": None,
}
result = pdf_position.add_pdf_positions({"items": [item]})
assert result["items"] == [item]
def test_noop_incremental_writes_complete_artifacts():
with tempfile.TemporaryDirectory() as temporary:
workdir = Path(temporary)
source = workdir / "My Clippings.txt"
source.write_text(
"Book\n- Your Highlight at location 10-11 | Added on Tuesday, 18 May 2021 14:18:13\n\n"
"A highlight long enough to import\n==========\n",
encoding="utf-8",
)
clippings = clippings_to_jsonable(load_clippings(str(source)))
clipping_id = clippings["clippings"][0]["id"]
(workdir / "clippings.json").write_text(json.dumps(clippings), encoding="utf-8")
annotation = {
"clipping_id": clipping_id,
"attachment_item_id": 1,
"annotation": {"sortIndex": "1"},
}
(workdir / "import-plan.final.json").write_text(
json.dumps({"annotations": [annotation], "all_annotations": [annotation], "overrides_snapshot": {}}),
encoding="utf-8",
)
originals = (
cli.build_zotero_index,
cli.build_match_report,
cli.generate_override_skeleton,
cli.build_mismatch_review,
)
cli.build_zotero_index = lambda *_args: {"format": "index", "items": []}
cli.build_match_report = lambda *_args: {"format": "matches", "matches": []}
cli.generate_override_skeleton = lambda *_args: {"format": "overrides", "overrides": []}
cli.build_mismatch_review = lambda *_args: "review\n"
try:
result = cli.main(
[
"run",
str(source),
"--workdir",
str(workdir),
"--db",
str(workdir / "zotero.sqlite"),
"--storage-root",
str(workdir / "storage"),
]
)
finally:
(
cli.build_zotero_index,
cli.build_match_report,
cli.generate_override_skeleton,
cli.build_mismatch_review,
) = originals
assert result == 0
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
assert final_plan["annotations"] == []
assert final_plan["all_annotations"] == [annotation]
assert final_plan["delta_annotation_count"] == 0
assert (workdir / "import-plan.epub.json").exists()

View file

@ -1,14 +1,331 @@
import assert from 'assert';
// test the grouping logic for title variants sharing same candidate
function findOtherRows(rows, rowIndex, targetKey) {
const row = rows[rowIndex];
return rows.filter((r, idx) => idx !== rowIndex && (r.candidates||[]).some(c => (c.citation_key||c.key||String(c.item_id))===targetKey));
const assert = require('assert');
const fs = require('fs');
const vm = require('vm');
const bootstrapContext = {};
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
const plugin = bootstrapContext.KindleZoteroImporter;
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
assert.equal(
plugin.mergeComments('[r] this is a red note\n\nthis is a red note', ''),
'this is a red note'
);
assert.equal(plugin.mergeComments('[o]', ''), '');
assert.equal(plugin.mergeComments('[citation] keep this', ''), '[citation] keep this');
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'old note', annotationTags: [{ name: 'kindle-import' }] },
'replacement note',
'old note'
),
'replacement note'
);
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'old note\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
'replacement note',
'old note'
),
'manual addition\n\nreplacement note'
);
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'old first\n\nold second\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
'new note',
'old first\n\nold second'
),
'manual addition\n\nnew note'
);
assert.equal(
plugin.commentForExistingAnnotation(
{ annotationComment: 'manual note', annotationTags: [] },
'imported note'
),
'manual note\n\nimported note'
);
const base = {
annotationText: 'forged its might, should be enough to discredit',
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1255)',
}),
};
const extended = {
annotationText: 'forged its might, should be enough to discredit, even in the eyes of philosophers',
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1300)',
}),
};
assert.equal(plugin.areImportedDuplicates(base, extended), true);
assert.equal(
plugin.areImportedDuplicates(base, {
...base,
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
}),
}),
false
);
assert.equal(
plugin.areImportedDuplicates(base, {
...base,
annotationText: 'OCR-corrected text at the identical annotation position',
}),
true
);
assert.equal(
plugin.areImportedDuplicates(
{
annotationText: 'A multiline PDF selection long enough to compare',
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30]] }),
},
{
annotationText: 'A multiline PDF selection long enough to compare with an extended ending',
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30], [10, 40, 120, 50]] }),
}
const rows = [
{title:"gilbert-simondon-on-the-mode", candidates:[{citation_key:"chabot2013", key:"A"}, {citation_key:"simondon2017a"}]},
{title:"On the Mode (Univocal)", candidates:[{citation_key:"chabot2013"}, {citation_key:"simondon2017a"}]},
{title:"Other", candidates:[{citation_key:"foo"}]}
];
assert.equal(findOtherRows(rows, 0, "chabot2013").length, 1);
assert.equal(findOtherRows(rows, 0, "chabot2013")[0].title, "On the Mode (Univocal)");
console.log("manager logic ok");
),
true
);
const managerHtml = fs.readFileSync('plugin/manager.html', 'utf8');
const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
const managerContext = { TextEncoder };
vm.runInNewContext(scripts[0][1], managerContext);
const manager = managerContext.KindleManager;
manager.integratedPage = 1;
manager.integratedPageSize = 50;
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
assert.equal(page.totalPages, 3);
assert.deepEqual(Array.from(page.rows), Array.from({ length: 50 }, (_, index) => index + 50));
manager.mappingsPage = 99;
manager.mappingsPageSize = 50;
const finalPage = manager.paginate(Array.from({ length: 51 }, (_, index) => index), 'mappingsPage', 'mappingsPageSize');
assert.equal(manager.mappingsPage, 1);
assert.deepEqual(Array.from(finalPage.rows), [50]);
const collectorBatches = manager.collectorBatches('installation', Array.from(
{ length: 450 },
(_, index) => ({ clipping_id_hash: String(index), text: 'x'.repeat(2000) })
));
assert.equal(collectorBatches.length, 3);
assert.ok(collectorBatches.every(batch => batch.length <= 200));
assert.ok(collectorBatches.every(batch => new TextEncoder().encode(
JSON.stringify({ anonId: 'installation', data: batch })
).length <= 1500000));
assert.throws(
() => manager.collectorBatches('installation', [{ text: 'x'.repeat(1500000) }]),
/exceeds the collector request limit/
);
async function testRuntimeExtraction() {
const writes = [];
bootstrapContext.PathUtils = {
profileDir: '/profile',
join: (...parts) => parts.join('/').replaceAll('//', '/'),
parent: path => path.slice(0, path.lastIndexOf('/')),
};
bootstrapContext.IOUtils = {
makeDirectory: async () => {},
};
bootstrapContext.Zotero = {
File: { putContentsAsync: async (path, text) => writes.push([path, text]) },
};
bootstrapContext.fetch = async url => ({ ok: true, text: async () => url });
plugin.rootURI = 'jar:file:///kanzi.xpi!/';
await plugin.ensureRuntime();
assert.equal(plugin.runtimeProjectDir, '/profile/kanzi-runtime');
assert.equal(writes.length, 13);
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
}
async function testMaintenanceSweep() {
const annotation = (id, text, end, comment = '') => ({
id,
parentItemID: 10,
annotationText: text,
annotationComment: comment,
annotationPosition: JSON.stringify({
type: 'FragmentSelector',
value: `epubcfi(/6/12!/4/2/8,/1:795,/1:${end})`,
}),
saved: 0,
erased: 0,
saveTx: async function () { this.saved += 1; },
eraseTx: async function () { this.erased += 1; },
});
const short = annotation(1, 'A progressive highlight long enough to compare', 994);
const middle = annotation(2, 'A progressive highlight long enough to compare and then extended', 1093);
const longest = annotation(
3,
'A progressive highlight long enough to compare and then extended to its final form',
1141,
'[r] useful note\n\nuseful note'
);
const separate = annotation(4, 'A progressive highlight long enough to compare', 2050);
separate.annotationPosition = JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
});
const emptyComment = annotation(5, 'A separate highlight with a null comment', 3050, null);
emptyComment.annotationPosition = JSON.stringify({
type: 'FragmentSelector',
value: 'epubcfi(/6/12!/4/2/8,/1:3000,/1:3050)',
});
const results = { updatedComments: 0, removedDuplicates: 0 };
await plugin.maintainImportedAnnotations([short, middle, longest, separate, emptyComment], false, results);
assert.equal(results.removedDuplicates, 2);
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
assert.equal(short.erased, 1);
assert.equal(middle.erased, 1);
assert.equal(longest.erased, 0);
assert.equal(separate.erased, 0);
assert.equal(emptyComment.saved, 0);
assert.equal(longest.annotationComment, 'useful note');
assert.equal(results.updatedComments, 1);
}
async function testWriterReconciliation() {
let written = null;
bootstrapContext.Zotero = {
File: { putContentsAsync: async (_path, text) => { written = JSON.parse(text); } },
};
const plan = {
annotations: [{ clipping_id: 'ok' }, { clipping_id: 'failed' }],
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'ok' }, { clipping_id: 'failed' }],
};
await plugin.reconcileWriterState('/plan.json', plan, {
failed: [{ clipping_id: 'failed' }, { clipping_id: 'maintenance' }],
failedDeletionIds: ['delete-me'],
maintenanceDeletedIds: ['old'],
});
assert.deepEqual(Array.from(written.write_failed_ids), ['failed']);
assert.deepEqual(Array.from(written.annotations, entry => entry.clipping_id), ['ok']);
assert.deepEqual(Array.from(written.all_annotations, entry => entry.clipping_id), ['ok']);
assert.equal(written.annotation_count, 1);
assert.deepEqual(Array.from(written.delete_failed_ids), ['delete-me']);
}
async function testDeletionFailureIsRetried() {
bootstrapContext.Zotero = { logError: () => {} };
const annotation = {
annotationTags: [{ name: 'kindle-id:delete-me' }],
eraseTx: async () => { throw new Error('locked'); },
};
plugin.collectImportedAnnotations = async () => [annotation];
plugin.maintainImportedAnnotations = async () => {};
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['delete-me'],
}, false);
assert.deepEqual(Array.from(results.failedDeletionIds), ['delete-me']);
assert.equal(results.failed.length, 1);
}
async function testMaintenanceDeletionIsNotRepeated() {
bootstrapContext.Zotero = { logError: () => {} };
const annotation = {
annotationTags: [{ name: 'kindle-id:already-deleted' }],
eraseTx: async () => { throw new Error('must not be called again'); },
};
plugin.collectImportedAnnotations = async () => [annotation];
plugin.maintainImportedAnnotations = async (_annotations, _dryRun, results) => {
results.maintenanceDeletedIds.push('already-deleted');
};
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['already-deleted'],
}, false);
assert.deepEqual(Array.from(results.failedDeletionIds), []);
assert.equal(results.failed.length, 0);
}
async function testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor() {
bootstrapContext.Zotero = { logError: () => {} };
const active = {
parentItemID: 10,
annotationText: 'The longer OCR-corrected duplicate annotation text',
annotationComment: '',
annotationPosition: JSON.stringify({ pageIndex: 1, rects: [[1, 2, 3, 4]] }),
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
erased: 0,
eraseTx: async function () { this.erased += 1; },
};
const duplicate = {
...active,
annotationText: 'The duplicate annotation text',
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
erased: 0,
eraseTx: async function () { this.erased += 1; },
};
let collection = 0;
plugin.collectImportedAnnotations = async () => (++collection === 1 ? [active, duplicate] : [active]);
plugin.maintainImportedAnnotations = maintainImportedAnnotations;
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['same'],
}, false);
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
assert.equal(duplicate.erased, 1);
assert.equal(active.erased, 1);
assert.equal(results.deletedForIncremental, 1);
assert.equal(results.failed.length, 0);
}
async function testEnumerationFailurePersistsAllDeletions() {
bootstrapContext.Zotero = { logError: () => {} };
plugin.collectImportedAnnotations = async () => { throw new Error('enumeration failed'); };
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
deletions: ['retry-deletion'],
}, false);
assert.deepEqual(Array.from(results.failedDeletionIds), ['retry-deletion']);
assert.equal(results.failed[0].clipping_id, 'maintenance');
}
async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
const old = {
annotationTags: [{ name: 'kindle-id:replacement' }],
erased: 0,
eraseTx: async function () { this.erased += 1; },
};
bootstrapContext.Zotero = {
logError: () => {},
Items: { get: () => null },
};
plugin.collectImportedAnnotations = async () => [old];
plugin.maintainImportedAnnotations = async () => {};
const results = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [{ clipping_id: 'replacement', attachment_item_id: 99, annotation: {} }],
deletions: ['replacement'],
}, false);
assert.equal(old.erased, 0);
assert.deepEqual(Array.from(results.failedDeletionIds), ['replacement']);
assert.equal(results.failed[0].clipping_id, 'replacement');
}
testRuntimeExtraction()
.then(testMaintenanceSweep)
.then(testWriterReconciliation)
.then(testDeletionFailureIsRetried)
.then(testMaintenanceDeletionIsNotRepeated)
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
.then(testEnumerationFailurePersistsAllDeletions)
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
.then(() => {
console.log('manager and annotation logic ok');
}).catch(error => {
console.error(error);
process.exitCode = 1;
});

View file

@ -2,6 +2,15 @@
"addons": {
"kindlezoteroimporter@utkubilen.de": {
"updates": [
{
"version": "0.7.0",
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.7.0/kanzi.xpi",
"applications": {
"zotero": {
"strict_min_version": "6.999"
}
}
},
{
"version": "0.6.20",
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.6.20/kanzi.xpi",

View file

@ -1,156 +0,0 @@
// Run in Zotero via Tools > Developer > Run JavaScript.
// Set PLAN_PATH to the generated import-plan.final.json path.
// Keep DRY_RUN=true until the preview output looks correct.
const PLAN_PATH = "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer/import-plan.final.json";
const DRY_RUN = false;
const text = await Zotero.File.getContentsAsync(PLAN_PATH);
const plan = JSON.parse(text);
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
throw new Error(`Unsupported plan format: ${plan.format}`);
}
const results = {
dryRun: DRY_RUN,
total: plan.annotations.length,
created: 0,
skippedExisting: 0,
updatedComments: 0,
removedDuplicates: 0,
failed: [],
preview: [],
};
const existingByAttachment = new Map();
function normalizePosition(position) {
if (!position) {
return "";
}
if (typeof position === "string") {
try {
return JSON.stringify(JSON.parse(position));
} catch (_error) {
return position;
}
}
return JSON.stringify(position);
}
function annotationFingerprint(annotation) {
return `${annotation.text || ""}\u0000${normalizePosition(annotation.position)}`;
}
function existingAnnotationFingerprint(annotation) {
return `${annotation.annotationText || ""}\u0000${normalizePosition(annotation.annotationPosition)}`;
}
async function getExistingAnnotations(attachment) {
if (!existingByAttachment.has(attachment.id)) {
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
const byFingerprint = new Map();
for (const annotation of annotations) {
const fingerprint = existingAnnotationFingerprint(annotation);
const kept = byFingerprint.get(fingerprint);
if (!kept) {
byFingerprint.set(fingerprint, annotation);
continue;
}
const keptHasComment = Boolean(kept.annotationComment);
const currentHasComment = Boolean(annotation.annotationComment);
const duplicate = keptHasComment || !currentHasComment ? annotation : kept;
const replacement = duplicate === annotation ? kept : annotation;
byFingerprint.set(fingerprint, replacement);
if (!DRY_RUN) {
await duplicate.eraseTx();
}
results.removedDuplicates += 1;
}
existingByAttachment.set(attachment.id, byFingerprint);
}
return existingByAttachment.get(attachment.id);
}
function mergeComments(existingComment, newComment) {
const seen = new Set();
const merged = [];
for (const comment of [existingComment, newComment]) {
for (const part of String(comment || "").split(/\n\s*\n/)) {
const cleaned = part.trim();
if (!cleaned || seen.has(cleaned)) {
continue;
}
seen.add(cleaned);
merged.push(cleaned);
}
}
return merged.join("\n\n");
}
for (const entry of plan.annotations) {
try {
const attachment = Zotero.Items.get(entry.attachment_item_id);
if (!attachment) {
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
}
const annotation = {
key: Zotero.DataObjectUtilities.generateKey(),
...entry.annotation,
};
if (!annotation.sortIndex) {
delete annotation.sortIndex;
}
const existingAnnotations = await getExistingAnnotations(attachment);
const fingerprint = annotationFingerprint(annotation);
const existingAnnotation = existingAnnotations.get(fingerprint);
if (existingAnnotation) {
const mergedComment = mergeComments(existingAnnotation.annotationComment, annotation.comment);
if (mergedComment && existingAnnotation.saveTx && existingAnnotation.annotationComment !== mergedComment) {
existingAnnotation.annotationComment = mergedComment;
if (!DRY_RUN) {
await existingAnnotation.saveTx();
}
results.updatedComments += 1;
}
results.skippedExisting += 1;
continue;
}
if (DRY_RUN) {
results.preview.push({
clipping_id: entry.clipping_id,
attachment_item_id: entry.attachment_item_id,
type: annotation.type,
text: annotation.text.slice(0, 120),
pageLabel: annotation.pageLabel,
sortIndex: annotation.sortIndex,
position: annotation.position,
});
continue;
}
await Zotero.Annotations.saveFromJSON(attachment, annotation);
existingAnnotations.set(fingerprint, {
annotationComment: annotation.comment || "",
annotationPosition: annotation.position,
annotationText: annotation.text || "",
});
results.created += 1;
} catch (error) {
results.failed.push({
clipping_id: entry.clipping_id,
attachment_item_id: entry.attachment_item_id,
message: String(error),
});
}
}
return JSON.stringify(results, null, 2);