Kanzi 0.7.0: safe incremental imports and collector hardening
This commit is contained in:
parent
de435fec00
commit
56da3d2e7b
25 changed files with 1920 additions and 596 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -2,7 +2,7 @@ clippings.json
|
||||||
zotero-index.json
|
zotero-index.json
|
||||||
matches.json
|
matches.json
|
||||||
match-overrides.generated.json
|
match-overrides.generated.json
|
||||||
!match-overrides.json
|
match-overrides.json
|
||||||
import-plan*.json
|
import-plan*.json
|
||||||
*.import-plan.json
|
*.import-plan.json
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
|
|
||||||
20
README.md
20
README.md
|
|
@ -3,7 +3,7 @@
|
||||||
|
|
||||||
# Kanzi — Kindle annotations to Zotero
|
# Kanzi — Kindle annotations to Zotero
|
||||||
|
|
||||||
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero, no terminal needed. Formerly *Kindle Zotero Importer*.
|
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero. Once the required host tools are installed, imports run from the manager without CLI commands. Formerly *Kindle Zotero Importer*.
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
|
|
||||||
|
|
@ -13,22 +13,22 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
|
||||||
|
|
||||||
# Install
|
# Install
|
||||||
|
|
||||||
1. Download `kanzi.xpi` from [Releases](https://github.com/UtkuBilenDemir/kanzi/releases) (latest `0.6.20`, or `0.6.19-beta` for preview).
|
1. Download `kanzi.xpi` from [Releases](https://github.com/UtkuBilenDemir/kanzi/releases) (latest `0.7.0`).
|
||||||
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
|
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
|
||||||
3. `Tools` → `Kanzi…` to open the manager.
|
3. `Tools` → `Kanzi…` to open the manager.
|
||||||
|
|
||||||
Works with Zotero 7 to 10, macOS, Windows and Linux, PDF and EPUB.
|
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. macOS and Linux are tested. Windows requires setting the full Python executable path in `Settings` and installing the PDF tools separately.
|
||||||
|
|
||||||
# Use
|
# Use
|
||||||
|
|
||||||
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
|
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
|
||||||
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
||||||
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
|
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
|
||||||
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
||||||
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
||||||
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
||||||
|
|
||||||
Your choices are saved in `match-overrides.json` in the project folder. Back it up, share it, or delete an entry to undo a mapping.
|
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
|
||||||
|
|
||||||
# Coloured highlights
|
# Coloured highlights
|
||||||
|
|
||||||
|
|
@ -64,13 +64,13 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
|
||||||
|
|
||||||
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
|
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
|
||||||
|
|
||||||
**Integrated.** The last `import-plan.final.json` annotations. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right, 400 shown, sortable and resizable.
|
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
|
||||||
|
|
||||||
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
|
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
|
||||||
|
|
||||||
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
|
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
|
||||||
|
|
||||||
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, `Share anonymised annotations` (on by default, private, hashed only), and `Highlight colours` as above. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
|
**Settings.** `Project directory`, `Python`, `Zotero DB`, `Zotero storage`, annotation sharing, and `Highlight colours` as above. Sharing is off until you explicitly opt in and can be disabled here at any time. `Save Settings` writes `plugin-config.json` and `colour-map.json` for the pipeline. No restart is needed.
|
||||||
|
|
||||||
**Artifacts.** `Mismatch review`, `Persistent overrides`, `Generated suggestions`, `Positioned plan`, `Final writer plan` and `Plugin summary`. Each has `Open` and `Reveal`.
|
**Artifacts.** `Mismatch review`, `Persistent overrides`, `Generated suggestions`, `Positioned plan`, `Final writer plan` and `Plugin summary`. Each has `Open` and `Reveal`.
|
||||||
|
|
||||||
|
|
@ -79,12 +79,12 @@ Incremental is default. Only new, changed or colour-changed highlights are repos
|
||||||
# For developers / CLI
|
# For developers / CLI
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
python scripts/build_plugin.py # → dist/kanzi.xpi
|
python scripts/build_plugin.py # → dist/kanzi.xpi, including the Python runtime
|
||||||
python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --pretty
|
python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --pretty
|
||||||
python -m kindle_zotero_importer run ... --full # ignore incremental and reprocess all
|
python -m kindle_zotero_importer run ... --full # ignore incremental and reprocess all
|
||||||
```
|
```
|
||||||
|
|
||||||
PDF positioning uses Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
|
PDF positioning uses host-installed Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf` fallback. EPUB uses CFI. Zotero writes are only via `Zotero.Annotations`. It never writes `zotero.sqlite` directly.
|
||||||
|
|
||||||
# Releases
|
# Releases
|
||||||
|
|
||||||
|
|
@ -94,4 +94,4 @@ Stable releases are on `Releases` with `kanzi.xpi` and `updates.json` attached.
|
||||||
|
|
||||||
Issues and pull requests welcome at [Issues](../../issues).
|
Issues and pull requests welcome at [Issues](../../issues).
|
||||||
|
|
||||||
Anonymised sharing is on by default to help future development. It sends only citation key, hashed highlight, colour, has comment and note, added on and integrated at. It never sends raw highlight text. Opt out any time in `Manager` → `Settings` → uncheck `Share anonymised annotations`.
|
Annotation sharing is off until you explicitly opt in. If enabled, it sends a random pseudonymous installation ID, citation key and publication metadata (including DOI/ISBN when available), raw highlight text and comments, normalised/fuzzy hashes, colour, and dates to `annotation.utkubilen.de`. The collector does not receive your Zotero account identity, but free text can contain identifying information and therefore is not anonymous. Change this any time in `Manager` → `Settings`.
|
||||||
|
|
|
||||||
357
collector/app.py
Normal file
357
collector/app.py
Normal file
|
|
@ -0,0 +1,357 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import secrets
|
||||||
|
import sqlite3
|
||||||
|
import unicodedata
|
||||||
|
|
||||||
|
from fastapi import FastAPI, HTTPException, Request
|
||||||
|
from fastapi.responses import HTMLResponse
|
||||||
|
|
||||||
|
|
||||||
|
DB = Path(os.environ.get("KANZI_ANNOTATION_DB", Path(__file__).parent / "annotations.db"))
|
||||||
|
MAX_PAGE_SIZE = 200
|
||||||
|
MAX_REQUEST_BYTES = 2_000_000
|
||||||
|
MAX_BATCH_SIZE = 5_000
|
||||||
|
|
||||||
|
app = FastAPI(title="Kanzi Annotation Collector")
|
||||||
|
|
||||||
|
|
||||||
|
def _normalise(text: str) -> str:
|
||||||
|
value = unicodedata.normalize("NFKC", text or "").lower()
|
||||||
|
value = re.sub(r"-\s*\n\s*", "", value)
|
||||||
|
value = re.sub(r"[\u2010-\u2015\u2018\u2019\u201c\u201d]", "'", value)
|
||||||
|
value = re.sub(r"[^a-z0-9]+", " ", value)
|
||||||
|
return re.sub(r"\s+", " ", value).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _hash_norm(text: str) -> str:
|
||||||
|
return hashlib.sha256(_normalise(text).encode()).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
def _fuzzy_hash(text: str) -> str:
|
||||||
|
norm = _normalise(text)
|
||||||
|
if len(norm) < 3:
|
||||||
|
return _hash_norm(text)
|
||||||
|
trigrams = {norm[index : index + 3] for index in range(len(norm) - 2)}
|
||||||
|
words = sorted(norm.split())
|
||||||
|
word_trigrams = {
|
||||||
|
" ".join(words[index : index + 3])
|
||||||
|
for index in range(max(1, len(words) - 2))
|
||||||
|
}
|
||||||
|
return hashlib.sha256(" ".join(sorted(trigrams | word_trigrams)).encode()).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
|
def _connect() -> sqlite3.Connection:
|
||||||
|
DB.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
connection = sqlite3.connect(DB)
|
||||||
|
connection.row_factory = sqlite3.Row
|
||||||
|
connection.execute("PRAGMA busy_timeout = 5000")
|
||||||
|
connection.execute("PRAGMA journal_mode = WAL")
|
||||||
|
return connection
|
||||||
|
|
||||||
|
|
||||||
|
def init_db() -> None:
|
||||||
|
with _connect() as connection:
|
||||||
|
connection.execute(
|
||||||
|
"""CREATE TABLE IF NOT EXISTS highlights (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
anon_id TEXT,
|
||||||
|
citation_key TEXT,
|
||||||
|
doi TEXT,
|
||||||
|
isbn TEXT,
|
||||||
|
issn TEXT,
|
||||||
|
publication_title TEXT,
|
||||||
|
clipping_id_hash TEXT,
|
||||||
|
norm_hash TEXT,
|
||||||
|
fuzzy_hash TEXT,
|
||||||
|
text TEXT,
|
||||||
|
color TEXT,
|
||||||
|
has_comment INTEGER,
|
||||||
|
comment TEXT,
|
||||||
|
comment_hash TEXT,
|
||||||
|
clipping_added_on_iso TEXT,
|
||||||
|
integrated_at TEXT,
|
||||||
|
received_at TEXT,
|
||||||
|
raw TEXT
|
||||||
|
)"""
|
||||||
|
)
|
||||||
|
existing_columns = {
|
||||||
|
row["name"] for row in connection.execute("PRAGMA table_info(highlights)")
|
||||||
|
}
|
||||||
|
required_columns = {
|
||||||
|
"doi": "TEXT",
|
||||||
|
"isbn": "TEXT",
|
||||||
|
"issn": "TEXT",
|
||||||
|
"publication_title": "TEXT",
|
||||||
|
"clipping_id_hash": "TEXT",
|
||||||
|
"norm_hash": "TEXT",
|
||||||
|
"fuzzy_hash": "TEXT",
|
||||||
|
"text": "TEXT",
|
||||||
|
"comment": "TEXT",
|
||||||
|
"comment_hash": "TEXT",
|
||||||
|
"clipping_added_on_iso": "TEXT",
|
||||||
|
"integrated_at": "TEXT",
|
||||||
|
"received_at": "TEXT",
|
||||||
|
"raw": "TEXT",
|
||||||
|
}
|
||||||
|
for name, column_type in required_columns.items():
|
||||||
|
if name not in existing_columns:
|
||||||
|
connection.execute(f"ALTER TABLE highlights ADD COLUMN {name} {column_type}")
|
||||||
|
connection.execute(
|
||||||
|
"""CREATE TABLE IF NOT EXISTS raw_posts (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
anon_id TEXT,
|
||||||
|
received_at TEXT,
|
||||||
|
body TEXT
|
||||||
|
)"""
|
||||||
|
)
|
||||||
|
connection.execute(
|
||||||
|
"""CREATE TABLE IF NOT EXISTS installations (
|
||||||
|
anon_id TEXT PRIMARY KEY,
|
||||||
|
token_hash TEXT UNIQUE NOT NULL,
|
||||||
|
created_at TEXT NOT NULL
|
||||||
|
)"""
|
||||||
|
)
|
||||||
|
# Older collector versions appended every retry. Retain only the newest
|
||||||
|
# record for each installation/clipping before enforcing idempotency.
|
||||||
|
connection.execute(
|
||||||
|
"""DELETE FROM highlights
|
||||||
|
WHERE clipping_id_hash IS NOT NULL
|
||||||
|
AND id NOT IN (
|
||||||
|
SELECT MAX(id) FROM highlights
|
||||||
|
WHERE clipping_id_hash IS NOT NULL
|
||||||
|
GROUP BY anon_id, clipping_id_hash
|
||||||
|
)"""
|
||||||
|
)
|
||||||
|
connection.execute(
|
||||||
|
"""CREATE UNIQUE INDEX IF NOT EXISTS highlights_identity
|
||||||
|
ON highlights(anon_id, clipping_id_hash)
|
||||||
|
WHERE clipping_id_hash IS NOT NULL"""
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
init_db()
|
||||||
|
|
||||||
|
|
||||||
|
HTML = """<!doctype html>
|
||||||
|
<html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Kanzi annotations</title>
|
||||||
|
<style>
|
||||||
|
:root{font-family:'iA Writer Duo','JetBrains Mono',ui-monospace,monospace;color:#111;background:#f8f8f8;--panel:#fff;--soft:#f0f0f0;--line:#ddd;--muted:#666;--radius:8px}*{box-sizing:border-box}body{margin:0}main{max-width:1280px;margin:auto;padding:20px 24px}h1{margin:0 0 4px;font-size:28px}h2{font-size:16px;border-bottom:1px solid var(--line);padding-bottom:6px;margin-top:24px}.muted{color:var(--muted);font-size:12px}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(170px,1fr));gap:12px;margin:16px 0}.card,table{background:var(--panel);border:1px solid var(--line);border-radius:var(--radius)}.card{padding:14px}.card strong{display:block;font-size:22px}table{width:100%;border-collapse:collapse;table-layout:fixed;overflow:hidden}th,td{padding:9px 10px;border-bottom:1px solid var(--line);text-align:left;overflow-wrap:anywhere;font-size:12px}th{background:var(--soft);color:var(--muted);font-size:10px;text-transform:uppercase}.pager{display:flex;justify-content:flex-end;align-items:center;gap:8px;margin:10px 0}.pager button{padding:5px 9px;border:1px solid var(--line);border-radius:var(--radius);background:var(--panel)}.pager button:disabled{color:#999}@media(max-width:700px){main{padding:14px 10px}.table-wrap{overflow-x:auto}table{min-width:800px}}
|
||||||
|
</style></head><body><main><h1>Kanzi annotations</h1><div class="muted">Private pseudonymous collector. Raw highlight text and comments are stored.</div>
|
||||||
|
<div class="grid"><div class="card"><strong id="total">-</strong><span>highlights</span></div><div class="card"><strong id="uniq">-</strong><span>citekeys</span></div><div class="card"><strong id="anons">-</strong><span>installations</span></div><div class="card"><strong id="today">-</strong><span>received today</span></div></div>
|
||||||
|
<h2>Recent</h2><div class="table-wrap"><table><thead><tr><th>anon</th><th>citekey</th><th>DOI / ISBN</th><th>text</th><th>colour</th><th>received</th></tr></thead><tbody id="recent"></tbody></table></div><div class="pager" id="recent-pager"></div>
|
||||||
|
<h2>Top citekeys</h2><div class="table-wrap"><table><thead><tr><th>citekey</th><th>count</th></tr></thead><tbody id="top"></tbody></table></div>
|
||||||
|
<h2>Common highlights</h2><div class="muted">Same publication and normalised text across at least two installations.</div><div class="table-wrap"><table><thead><tr><th>DOI / ISBN</th><th>text</th><th>norm</th><th>installations</th><th>citekeys</th></tr></thead><tbody id="common"></tbody></table></div><div class="pager" id="common-pager"></div>
|
||||||
|
<script>
|
||||||
|
const size=50,state={recent:0,common:0};
|
||||||
|
function cell(row,value){const td=document.createElement('td');td.textContent=value??'';row.appendChild(td);return td}
|
||||||
|
function empty(body,span,message){body.textContent='';const row=body.insertRow();const td=cell(row,message);td.colSpan=span;td.className='muted'}
|
||||||
|
function pager(id,key,count,load){const root=document.getElementById(id);root.textContent='';const pages=Math.max(1,Math.ceil(count/size));state[key]=Math.min(state[key],pages-1);const prev=document.createElement('button');prev.textContent='Previous';prev.disabled=state[key]===0;prev.onclick=()=>{state[key]--;load()};root.append(prev);const text=document.createElement('span');text.textContent=`Page ${state[key]+1} of ${pages}`;root.append(text);const next=document.createElement('button');next.textContent='Next';next.disabled=state[key]>=pages-1;next.onclick=()=>{state[key]++;load()};root.append(next)}
|
||||||
|
async function loadRecent(){const result=await fetch(`/recent?limit=${size}&offset=${state.recent*size}`).then(r=>r.json());const body=document.getElementById('recent');body.textContent='';for(const h of result.recent){const row=body.insertRow();cell(row,(h.anon_id||'').slice(0,8));cell(row,h.citation_key);cell(row,h.doi||h.isbn);cell(row,(h.text||'').slice(0,180));cell(row,h.color);cell(row,(h.received_at||'').slice(0,16).replace('T',' '))}if(!result.recent.length)empty(body,6,'No data');pager('recent-pager','recent',result.total,loadRecent)}
|
||||||
|
async function loadCommon(){const result=await fetch(`/common?limit=${size}&offset=${state.common*size}`).then(r=>r.json());const body=document.getElementById('common');body.textContent='';for(const h of result.common){const row=body.insertRow();cell(row,h.publication);cell(row,(h.example_text||'').slice(0,180));cell(row,h.norm_hash);cell(row,h.installations);cell(row,h.citekeys)}if(!result.common.length)empty(body,5,'No common highlights yet');pager('common-pager','common',result.total,loadCommon)}
|
||||||
|
async function load(){const stats=await fetch('/stats').then(r=>r.json());for(const [id,key] of [['total','total_highlights'],['uniq','unique_citekeys'],['anons','unique_anons'],['today','today']])document.getElementById(id).textContent=stats[key];const top=await fetch('/top').then(r=>r.json());const body=document.getElementById('top');body.textContent='';for(const h of top.top){const row=body.insertRow();cell(row,h.citation_key);cell(row,h.count)}await Promise.all([loadRecent(),loadCommon()])}load();
|
||||||
|
</script></main></body></html>"""
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/", response_class=HTMLResponse)
|
||||||
|
def root() -> str:
|
||||||
|
return HTML
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/health")
|
||||||
|
def health() -> dict[str, str]:
|
||||||
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
|
||||||
|
def _token_hash(token: str) -> str:
|
||||||
|
return hashlib.sha256(token.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/register")
|
||||||
|
def register() -> dict[str, str]:
|
||||||
|
token = secrets.token_urlsafe(32)
|
||||||
|
anon_id = secrets.token_hex(16)
|
||||||
|
with _connect() as connection:
|
||||||
|
connection.execute(
|
||||||
|
"INSERT INTO installations (anon_id, token_hash, created_at) VALUES (?, ?, ?)",
|
||||||
|
(anon_id, _token_hash(token), datetime.now(timezone.utc).isoformat()),
|
||||||
|
)
|
||||||
|
return {"anonId": anon_id, "token": token}
|
||||||
|
|
||||||
|
|
||||||
|
def _authenticated_anon_id(request: Request) -> str:
|
||||||
|
authorization = request.headers.get("authorization", "")
|
||||||
|
scheme, separator, token = authorization.partition(" ")
|
||||||
|
if not separator or scheme.lower() != "bearer" or not token:
|
||||||
|
raise HTTPException(status_code=401, detail="Missing collector credential")
|
||||||
|
with _connect() as connection:
|
||||||
|
row = connection.execute(
|
||||||
|
"SELECT anon_id FROM installations WHERE token_hash = ?",
|
||||||
|
(_token_hash(token),),
|
||||||
|
).fetchone()
|
||||||
|
if not row:
|
||||||
|
raise HTTPException(status_code=401, detail="Invalid collector credential")
|
||||||
|
return str(row["anon_id"])
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/stats")
|
||||||
|
def stats() -> dict[str, int]:
|
||||||
|
today = datetime.now(timezone.utc).date().isoformat()
|
||||||
|
with _connect() as connection:
|
||||||
|
row = connection.execute(
|
||||||
|
"""SELECT COUNT(*) AS total_highlights,
|
||||||
|
COUNT(DISTINCT citation_key) AS unique_citekeys,
|
||||||
|
COUNT(DISTINCT anon_id) AS unique_anons,
|
||||||
|
SUM(CASE WHEN substr(received_at, 1, 10) = ? THEN 1 ELSE 0 END) AS today
|
||||||
|
FROM highlights""",
|
||||||
|
(today,),
|
||||||
|
).fetchone()
|
||||||
|
return {key: int(row[key] or 0) for key in row.keys()}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/recent")
|
||||||
|
def recent(limit: int = 50, offset: int = 0) -> dict:
|
||||||
|
limit = max(1, min(limit, MAX_PAGE_SIZE))
|
||||||
|
offset = max(0, offset)
|
||||||
|
with _connect() as connection:
|
||||||
|
total = connection.execute("SELECT COUNT(*) FROM highlights").fetchone()[0]
|
||||||
|
rows = connection.execute(
|
||||||
|
"""SELECT anon_id, citation_key, doi, isbn, text, clipping_id_hash,
|
||||||
|
norm_hash, fuzzy_hash, color, has_comment, received_at
|
||||||
|
FROM highlights ORDER BY id DESC LIMIT ? OFFSET ?""",
|
||||||
|
(limit, offset),
|
||||||
|
).fetchall()
|
||||||
|
return {"recent": [dict(row) for row in rows], "total": total}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/top")
|
||||||
|
def top(limit: int = 20) -> dict:
|
||||||
|
limit = max(1, min(limit, 100))
|
||||||
|
with _connect() as connection:
|
||||||
|
rows = connection.execute(
|
||||||
|
"""SELECT citation_key, COUNT(*) AS count FROM highlights
|
||||||
|
WHERE citation_key IS NOT NULL AND citation_key != ''
|
||||||
|
GROUP BY citation_key ORDER BY count DESC, citation_key LIMIT ?""",
|
||||||
|
(limit,),
|
||||||
|
).fetchall()
|
||||||
|
return {"top": [dict(row) for row in rows]}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/common")
|
||||||
|
def common(limit: int = 50, offset: int = 0) -> dict:
|
||||||
|
limit = max(1, min(limit, MAX_PAGE_SIZE))
|
||||||
|
offset = max(0, offset)
|
||||||
|
group_sql = """FROM highlights
|
||||||
|
WHERE norm_hash IS NOT NULL AND norm_hash != ''
|
||||||
|
AND COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), NULLIF(citation_key, '')) IS NOT NULL
|
||||||
|
GROUP BY COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key), norm_hash
|
||||||
|
HAVING COUNT(DISTINCT anon_id) > 1"""
|
||||||
|
with _connect() as connection:
|
||||||
|
total = connection.execute(f"SELECT COUNT(*) FROM (SELECT 1 {group_sql})").fetchone()[0]
|
||||||
|
rows = connection.execute(
|
||||||
|
f"""SELECT COALESCE(NULLIF(doi, ''), NULLIF(isbn, ''), citation_key) AS publication,
|
||||||
|
norm_hash, COUNT(DISTINCT anon_id) AS installations,
|
||||||
|
GROUP_CONCAT(DISTINCT citation_key) AS citekeys,
|
||||||
|
MAX(text) AS example_text
|
||||||
|
{group_sql}
|
||||||
|
ORDER BY installations DESC, publication, norm_hash LIMIT ? OFFSET ?""",
|
||||||
|
(limit, offset),
|
||||||
|
).fetchall()
|
||||||
|
return {"common": [dict(row) for row in rows], "total": total}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/highlights")
|
||||||
|
async def post_highlights(request: Request) -> dict:
|
||||||
|
anon = _authenticated_anon_id(request)
|
||||||
|
content_length = request.headers.get("content-length")
|
||||||
|
if content_length:
|
||||||
|
try:
|
||||||
|
if int(content_length) > MAX_REQUEST_BYTES:
|
||||||
|
raise HTTPException(status_code=413, detail="Request body is too large")
|
||||||
|
except ValueError:
|
||||||
|
raise HTTPException(status_code=400, detail="Invalid Content-Length")
|
||||||
|
body = await request.body()
|
||||||
|
if len(body) > MAX_REQUEST_BYTES:
|
||||||
|
raise HTTPException(status_code=413, detail="Request body is too large")
|
||||||
|
try:
|
||||||
|
payload = json.loads(body)
|
||||||
|
except (json.JSONDecodeError, UnicodeDecodeError) as error:
|
||||||
|
raise HTTPException(status_code=400, detail="Invalid JSON") from error
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise HTTPException(status_code=422, detail="Payload must be an object")
|
||||||
|
highlights = payload.get("data") or payload.get("highlights") or []
|
||||||
|
if isinstance(highlights, dict):
|
||||||
|
highlights = [highlights]
|
||||||
|
if not isinstance(highlights, list):
|
||||||
|
raise HTTPException(status_code=422, detail="data must be a list")
|
||||||
|
if len(highlights) > MAX_BATCH_SIZE:
|
||||||
|
raise HTTPException(status_code=413, detail="Too many highlights in one request")
|
||||||
|
now = datetime.now(timezone.utc).isoformat()
|
||||||
|
inserted = updated = 0
|
||||||
|
with _connect() as connection:
|
||||||
|
connection.execute("BEGIN IMMEDIATE")
|
||||||
|
for highlight in highlights:
|
||||||
|
if not isinstance(highlight, dict):
|
||||||
|
continue
|
||||||
|
text = str(highlight.get("text") or "")[:2000]
|
||||||
|
clipping_hash = str(highlight.get("clipping_id_hash") or "")[:128] or None
|
||||||
|
values = (
|
||||||
|
anon,
|
||||||
|
str(highlight.get("citation_key") or "")[:512] or None,
|
||||||
|
str(highlight.get("doi") or "")[:512] or None,
|
||||||
|
str(highlight.get("isbn") or "")[:512] or None,
|
||||||
|
str(highlight.get("issn") or "")[:512] or None,
|
||||||
|
str(highlight.get("publication_title") or "")[:1000] or None,
|
||||||
|
clipping_hash,
|
||||||
|
highlight.get("norm_hash") or _hash_norm(text),
|
||||||
|
highlight.get("fuzzy_hash") or _fuzzy_hash(text),
|
||||||
|
text,
|
||||||
|
str(highlight.get("color") or "")[:32] or None,
|
||||||
|
1 if highlight.get("has_comment") else 0,
|
||||||
|
str(highlight.get("comment") or "")[:2000],
|
||||||
|
str(highlight.get("comment_hash") or "")[:256] or None,
|
||||||
|
str(highlight.get("clipping_added_on_iso") or "")[:64] or None,
|
||||||
|
str(highlight.get("integrated_at") or "")[:64] or None,
|
||||||
|
now,
|
||||||
|
json.dumps(highlight, ensure_ascii=False)[:8000],
|
||||||
|
)
|
||||||
|
existing = None
|
||||||
|
if clipping_hash:
|
||||||
|
legacy_hash = clipping_hash[:8]
|
||||||
|
existing = connection.execute(
|
||||||
|
"""SELECT id FROM highlights
|
||||||
|
WHERE anon_id = ? AND clipping_id_hash IN (?, ?)
|
||||||
|
ORDER BY clipping_id_hash = ? DESC LIMIT 1""",
|
||||||
|
(anon, clipping_hash, legacy_hash, clipping_hash),
|
||||||
|
).fetchone()
|
||||||
|
if existing:
|
||||||
|
connection.execute(
|
||||||
|
"""UPDATE highlights SET citation_key=?, doi=?, isbn=?, issn=?, publication_title=?, clipping_id_hash=?,
|
||||||
|
norm_hash=?, fuzzy_hash=?, text=?, color=?, has_comment=?, comment=?, comment_hash=?,
|
||||||
|
clipping_added_on_iso=?, integrated_at=?, received_at=?, raw=? WHERE id=?""",
|
||||||
|
(*values[1:7], *values[7:], existing["id"]),
|
||||||
|
)
|
||||||
|
updated += 1
|
||||||
|
else:
|
||||||
|
connection.execute(
|
||||||
|
"""INSERT INTO highlights (
|
||||||
|
anon_id, citation_key, doi, isbn, issn, publication_title, clipping_id_hash,
|
||||||
|
norm_hash, fuzzy_hash, text, color, has_comment, comment, comment_hash,
|
||||||
|
clipping_added_on_iso, integrated_at, received_at, raw
|
||||||
|
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""",
|
||||||
|
values,
|
||||||
|
)
|
||||||
|
inserted += 1
|
||||||
|
return {
|
||||||
|
"status": "ok",
|
||||||
|
"received": len(highlights),
|
||||||
|
"inserted": inserted,
|
||||||
|
"updated": updated,
|
||||||
|
"anon": anon,
|
||||||
|
}
|
||||||
28
collector/nginx.conf.example
Normal file
28
collector/nginx.conf.example
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
# Add these directives in the nginx http context.
|
||||||
|
limit_req_zone $binary_remote_addr zone=kanzi_ingest:10m rate=10r/m;
|
||||||
|
limit_conn_zone $binary_remote_addr zone=kanzi_connections:10m;
|
||||||
|
|
||||||
|
# Keep dashboard and read APIs behind Authentik. Only plugin ingestion and
|
||||||
|
# health monitoring are intentionally unauthenticated.
|
||||||
|
location = /highlights {
|
||||||
|
client_max_body_size 2m;
|
||||||
|
limit_req zone=kanzi_ingest burst=10 nodelay;
|
||||||
|
limit_conn kanzi_connections 2;
|
||||||
|
proxy_pass http://127.0.0.1:8099;
|
||||||
|
}
|
||||||
|
|
||||||
|
location = /register {
|
||||||
|
limit_req zone=kanzi_ingest burst=10 nodelay;
|
||||||
|
limit_conn kanzi_connections 2;
|
||||||
|
proxy_pass http://127.0.0.1:8099;
|
||||||
|
}
|
||||||
|
|
||||||
|
location = /health {
|
||||||
|
proxy_pass http://127.0.0.1:8099;
|
||||||
|
}
|
||||||
|
|
||||||
|
location / {
|
||||||
|
auth_request /outpost.goauthentik.io/auth/nginx;
|
||||||
|
error_page 401 = @goauthentik_proxy_signin;
|
||||||
|
proxy_pass http://127.0.0.1:8099;
|
||||||
|
}
|
||||||
15
match-overrides.example.json
Normal file
15
match-overrides.example.json
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
{
|
||||||
|
"format": "kindle-zotero-importer.match-overrides.v1",
|
||||||
|
"overrides": [
|
||||||
|
{
|
||||||
|
"clipping_title": "Example Kindle title",
|
||||||
|
"resolution": {
|
||||||
|
"citation_key": "example2026"
|
||||||
|
},
|
||||||
|
"review": {
|
||||||
|
"status": "manager",
|
||||||
|
"clipping_count": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
@ -1,16 +1,16 @@
|
||||||
# Kindle Zotero Importer Zotero Plugin
|
# Kanzi Zotero Plugin
|
||||||
|
|
||||||
This is a hybrid Zotero plugin wrapper around the current Python importer. Zotero owns the user interface and writes native annotations through Zotero APIs; the Python project still performs parsing, matching, planning, EPUB positioning, and PDF positioning.
|
This is a hybrid Zotero plugin wrapper around the current Python importer. Zotero owns the user interface and writes native annotations through Zotero APIs; the Python project still performs parsing, matching, planning, EPUB positioning, and PDF positioning.
|
||||||
|
|
||||||
## Use
|
## Use
|
||||||
|
|
||||||
1. Install `kindle-zotero-importer.xpi` in Zotero through `Tools > Plugins`.
|
1. Install `kanzi.xpi` in Zotero through `Tools > Plugins`.
|
||||||
2. Open `Tools > Kindle Zotero Importer...`.
|
2. Open `Tools > Kanzi...`.
|
||||||
3. Select the updated Kindle `My Clippings.txt` in the manager.
|
3. Select the updated Kindle `My Clippings.txt` in the manager.
|
||||||
4. Follow the current stage, percentage, and elapsed time in the manager.
|
4. Follow the current stage, percentage, and elapsed time in the manager.
|
||||||
5. Review created, existing, updated, failed, and unresolved counts when it completes.
|
5. Review created, existing, updated, failed, and unresolved counts when it completes.
|
||||||
|
|
||||||
The plugin uses `match-overrides.json` from the project directory as the persistent source of title mappings and ignored titles. Generated JSON artifacts are still written to the project directory for audit and debugging.
|
The XPI contains the importer Python source. On startup it extracts that source into the Zotero profile, where it keeps private mappings and generated audit artifacts. Python 3.11 or newer must be installed on the host; PDF positioning also requires Poppler (`pdftotext`, `pdftohtml`, and `pdfinfo`) and optionally uses `qpdf` for encrypted files. The plugin uses `match-overrides.json` there as the persistent source of title mappings and ignored titles.
|
||||||
|
|
||||||
## Build XPI
|
## Build XPI
|
||||||
|
|
||||||
|
|
@ -20,6 +20,6 @@ From the repository root:
|
||||||
python scripts/build_plugin.py
|
python scripts/build_plugin.py
|
||||||
```
|
```
|
||||||
|
|
||||||
The output is `dist/kindle-zotero-importer.xpi`.
|
The output is `dist/kanzi.xpi`.
|
||||||
|
|
||||||
The plugin manifest targets Zotero `6.999` through `10.0.*`.
|
The plugin manifest targets Zotero `6.999` through `10.0.*`.
|
||||||
|
|
|
||||||
407
plugin/bootstrap.js
vendored
407
plugin/bootstrap.js
vendored
|
|
@ -1,10 +1,9 @@
|
||||||
var KindleZoteroImporter = {
|
var KindleZoteroImporter = {
|
||||||
id: "kindle-zotero-importer@utkubilen.de",
|
id: "kindle-zotero-importer@utkubilen.de",
|
||||||
menuItems: [],
|
menuItems: [],
|
||||||
defaultProjectDir:
|
runtimeProjectDir: "",
|
||||||
"/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer",
|
defaultProjectDir: "",
|
||||||
defaultClippingsPath:
|
defaultClippingsPath: "",
|
||||||
"/home/user/Library/Mobile Documents/com~apple~CloudDocs/Projects/test/My Clippings.txt",
|
|
||||||
prefBranch: "extensions.kindleZoteroImporter.",
|
prefBranch: "extensions.kindleZoteroImporter.",
|
||||||
rootURI: null,
|
rootURI: null,
|
||||||
managerWindow: null,
|
managerWindow: null,
|
||||||
|
|
@ -20,9 +19,41 @@ var KindleZoteroImporter = {
|
||||||
[["content", "kindle-zotero-importer", ""]]
|
[["content", "kindle-zotero-importer", ""]]
|
||||||
);
|
);
|
||||||
await Zotero.initializationPromise;
|
await Zotero.initializationPromise;
|
||||||
|
await this.ensureRuntime();
|
||||||
this.addToAllWindows();
|
this.addToAllWindows();
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async ensureRuntime() {
|
||||||
|
const runtimeDir = PathUtils.join(PathUtils.profileDir, "kanzi-runtime");
|
||||||
|
const files = [
|
||||||
|
"plugin_runner.py",
|
||||||
|
"src/kindle_zotero_importer/__init__.py",
|
||||||
|
"src/kindle_zotero_importer/__main__.py",
|
||||||
|
"src/kindle_zotero_importer/cli.py",
|
||||||
|
"src/kindle_zotero_importer/clippings.py",
|
||||||
|
"src/kindle_zotero_importer/epub_position.py",
|
||||||
|
"src/kindle_zotero_importer/final_plan.py",
|
||||||
|
"src/kindle_zotero_importer/import_plan.py",
|
||||||
|
"src/kindle_zotero_importer/matcher.py",
|
||||||
|
"src/kindle_zotero_importer/mismatch_review.py",
|
||||||
|
"src/kindle_zotero_importer/overrides.py",
|
||||||
|
"src/kindle_zotero_importer/pdf_position.py",
|
||||||
|
"src/kindle_zotero_importer/zotero_index.py",
|
||||||
|
];
|
||||||
|
await IOUtils.makeDirectory(runtimeDir, { createAncestors: true, ignoreExisting: true });
|
||||||
|
for (const relativePath of files) {
|
||||||
|
const destination = PathUtils.join(runtimeDir, ...relativePath.split("/"));
|
||||||
|
await IOUtils.makeDirectory(PathUtils.parent(destination), {
|
||||||
|
createAncestors: true,
|
||||||
|
ignoreExisting: true,
|
||||||
|
});
|
||||||
|
const response = await fetch(this.rootURI + "runtime/" + relativePath);
|
||||||
|
if (!response.ok) throw new Error(`Missing Kanzi runtime file: ${relativePath}`);
|
||||||
|
await Zotero.File.putContentsAsync(destination, await response.text());
|
||||||
|
}
|
||||||
|
this.runtimeProjectDir = runtimeDir;
|
||||||
|
},
|
||||||
|
|
||||||
shutdown() {
|
shutdown() {
|
||||||
this.removeMenuItems();
|
this.removeMenuItems();
|
||||||
if (this.managerWindow && !this.managerWindow.closed) {
|
if (this.managerWindow && !this.managerWindow.closed) {
|
||||||
|
|
@ -89,7 +120,8 @@ var KindleZoteroImporter = {
|
||||||
|
|
||||||
getPref(name, fallback) {
|
getPref(name, fallback) {
|
||||||
try {
|
try {
|
||||||
return Services.prefs.getCharPref(this.prefBranch + name);
|
const value = Services.prefs.getCharPref(this.prefBranch + name);
|
||||||
|
return value || fallback;
|
||||||
} catch (_error) {
|
} catch (_error) {
|
||||||
return fallback;
|
return fallback;
|
||||||
}
|
}
|
||||||
|
|
@ -100,19 +132,21 @@ var KindleZoteroImporter = {
|
||||||
},
|
},
|
||||||
|
|
||||||
getProjectDir() {
|
getProjectDir() {
|
||||||
return this.getPref("projectDir", this.defaultProjectDir);
|
return this.getPref("projectDir", this.runtimeProjectDir || this.defaultProjectDir);
|
||||||
},
|
},
|
||||||
|
|
||||||
getPythonPath() {
|
getPythonPath() {
|
||||||
return this.getPref("pythonPath", "/opt/homebrew/bin/python3");
|
return this.getPref("pythonPath", "python3");
|
||||||
},
|
},
|
||||||
|
|
||||||
getZoteroDbPath() {
|
getZoteroDbPath() {
|
||||||
return this.getPref("zoteroDbPath", "/home/user/Zotero/zotero.sqlite");
|
const dataDir = Zotero.DataDirectory && Zotero.DataDirectory.dir;
|
||||||
|
return this.getPref("zoteroDbPath", dataDir ? PathUtils.join(dataDir, "zotero.sqlite") : "");
|
||||||
},
|
},
|
||||||
|
|
||||||
getZoteroStorageRoot() {
|
getZoteroStorageRoot() {
|
||||||
return this.getPref("zoteroStorageRoot", "/home/user/Zotero/storage");
|
const dataDir = Zotero.DataDirectory && Zotero.DataDirectory.dir;
|
||||||
|
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
|
||||||
},
|
},
|
||||||
|
|
||||||
getConfigPath() {
|
getConfigPath() {
|
||||||
|
|
@ -143,6 +177,17 @@ var KindleZoteroImporter = {
|
||||||
zoteroDbPath: this.getZoteroDbPath(),
|
zoteroDbPath: this.getZoteroDbPath(),
|
||||||
zoteroStorageRoot: this.getZoteroStorageRoot(),
|
zoteroStorageRoot: this.getZoteroStorageRoot(),
|
||||||
};
|
};
|
||||||
|
if (settings.pythonPath === "python3") {
|
||||||
|
const candidates = Services.appinfo.OS === "Darwin"
|
||||||
|
? ["/opt/homebrew/bin/python3", "/usr/local/bin/python3", "/usr/bin/python3"]
|
||||||
|
: ["/usr/bin/python3", "/usr/local/bin/python3"];
|
||||||
|
for (const candidate of candidates) {
|
||||||
|
if (await IOUtils.exists(candidate)) {
|
||||||
|
settings.pythonPath = candidate;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
try {
|
try {
|
||||||
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
|
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
|
||||||
const parsed = JSON.parse(text);
|
const parsed = JSON.parse(text);
|
||||||
|
|
@ -382,7 +427,7 @@ var KindleZoteroImporter = {
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
async runPipeline(clippingsPath, settings, isFull = false) {
|
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
|
||||||
const python = settings.pythonPath;
|
const python = settings.pythonPath;
|
||||||
const projectDir = settings.projectDir;
|
const projectDir = settings.projectDir;
|
||||||
const runner = projectDir + "/plugin_runner.py";
|
const runner = projectDir + "/plugin_runner.py";
|
||||||
|
|
@ -409,6 +454,9 @@ var KindleZoteroImporter = {
|
||||||
if (isFull) {
|
if (isFull) {
|
||||||
args.push("--full");
|
args.push("--full");
|
||||||
}
|
}
|
||||||
|
if (retryUnresolved) {
|
||||||
|
args.push("--retry-unresolved");
|
||||||
|
}
|
||||||
|
|
||||||
await this.exec(python, args);
|
await this.exec(python, args);
|
||||||
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
|
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
|
||||||
|
|
@ -420,10 +468,11 @@ var KindleZoteroImporter = {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
const isFull = manager.getFullReimport ? manager.getFullReimport() : false;
|
const isFull = manager.getFullReimport ? manager.getFullReimport() : false;
|
||||||
await this.runManagedImportWithPath(manager, clippingsPath, isFull);
|
const retryUnresolved = manager.getRetryUnresolved ? manager.getRetryUnresolved() : false;
|
||||||
|
await this.runManagedImportWithPath(manager, clippingsPath, isFull, retryUnresolved);
|
||||||
},
|
},
|
||||||
|
|
||||||
async runManagedImportWithPath(manager, clippingsPath, isFullOverride) {
|
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
|
||||||
try {
|
try {
|
||||||
if (!clippingsPath) {
|
if (!clippingsPath) {
|
||||||
throw new Error("No clippings file selected");
|
throw new Error("No clippings file selected");
|
||||||
|
|
@ -432,6 +481,9 @@ var KindleZoteroImporter = {
|
||||||
const progressPath = settings.projectDir + "/plugin-progress.json";
|
const progressPath = settings.projectDir + "/plugin-progress.json";
|
||||||
// Determine full vs incremental from explicit arg or manager checkbox
|
// Determine full vs incremental from explicit arg or manager checkbox
|
||||||
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
|
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
|
||||||
|
const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
|
||||||
|
? retryUnresolvedOverride
|
||||||
|
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
|
||||||
manager.beginRun(clippingsPath);
|
manager.beginRun(clippingsPath);
|
||||||
try {
|
try {
|
||||||
await Zotero.File.removeIfExists(progressPath);
|
await Zotero.File.removeIfExists(progressPath);
|
||||||
|
|
@ -450,7 +502,7 @@ var KindleZoteroImporter = {
|
||||||
const pollPromise = poll();
|
const pollPromise = poll();
|
||||||
let summary;
|
let summary;
|
||||||
try {
|
try {
|
||||||
summary = await this.runPipeline(clippingsPath, settings, isFull);
|
summary = await this.runPipeline(clippingsPath, settings, isFull, retryUnresolved);
|
||||||
} finally {
|
} finally {
|
||||||
polling = false;
|
polling = false;
|
||||||
await pollPromise;
|
await pollPromise;
|
||||||
|
|
@ -464,8 +516,9 @@ var KindleZoteroImporter = {
|
||||||
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
||||||
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
||||||
});
|
});
|
||||||
|
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, results);
|
||||||
const data = await this.loadManagerData();
|
const data = await this.loadManagerData();
|
||||||
manager.completeRun(summary, results, data);
|
await manager.completeRun(summary, results, data);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
manager.failRun(String(error && error.stack ? error.stack : error));
|
manager.failRun(String(error && error.stack ? error.stack : error));
|
||||||
|
|
@ -548,42 +601,143 @@ var KindleZoteroImporter = {
|
||||||
},
|
},
|
||||||
|
|
||||||
mergeComments(existingComment, newComment) {
|
mergeComments(existingComment, newComment) {
|
||||||
const stripBracket = (s) => String(s || "").replace(/^\s*\[[^\]]+\]\s*/, "").trim();
|
|
||||||
const seen = new Set();
|
const seen = new Set();
|
||||||
const merged = [];
|
const merged = [];
|
||||||
for (const comment of [existingComment, newComment]) {
|
for (const comment of [existingComment, newComment]) {
|
||||||
for (const part of String(comment || "").split(/\n\s*\n/)) {
|
for (const part of String(comment || "").split(/\n\s*\n/)) {
|
||||||
const cleaned = part.trim();
|
const cleaned = this.stripColourCode(part);
|
||||||
if (!cleaned) continue;
|
if (!cleaned) continue;
|
||||||
// dedup on stripped version too, so "[r] note" and "note" are same
|
const key = cleaned.toLowerCase();
|
||||||
const key = stripBracket(cleaned).toLowerCase() || cleaned.toLowerCase();
|
if (seen.has(key)) continue;
|
||||||
if (seen.has(cleaned) || seen.has(key)) {
|
|
||||||
// if we have stripped version already, prefer stripped
|
|
||||||
if (seen.has(key) && cleaned !== stripBracket(cleaned)) {
|
|
||||||
// replace existing bracket version with stripped if we now have stripped
|
|
||||||
const idx = merged.findIndex(p => stripBracket(p).toLowerCase() === key);
|
|
||||||
if (idx >= 0) merged[idx] = stripBracket(cleaned);
|
|
||||||
seen.add(cleaned);
|
|
||||||
seen.add(key);
|
seen.add(key);
|
||||||
continue;
|
|
||||||
}
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
seen.add(cleaned);
|
|
||||||
seen.add(key);
|
|
||||||
// store stripped version if it had bracket
|
|
||||||
const stripped = stripBracket(cleaned);
|
|
||||||
if (stripped !== cleaned) {
|
|
||||||
merged.push(stripped);
|
|
||||||
seen.add(stripped);
|
|
||||||
} else {
|
|
||||||
merged.push(cleaned);
|
merged.push(cleaned);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
|
||||||
return merged.join("\n\n");
|
return merged.join("\n\n");
|
||||||
},
|
},
|
||||||
|
|
||||||
|
stripColourCode(value) {
|
||||||
|
return String(value || "")
|
||||||
|
.replace(/^\s*\[(?:y|yellow|o|orange|r|red|e|grey|gray|g|green|b|blue|bl|p|purple|m|magenta|pink)\]\s*/i, "")
|
||||||
|
.trim();
|
||||||
|
},
|
||||||
|
|
||||||
|
normalizeAnnotationText(value) {
|
||||||
|
return String(value || "")
|
||||||
|
.toLowerCase()
|
||||||
|
.normalize("NFKC")
|
||||||
|
.replace(/-\s*\n\s*/g, "")
|
||||||
|
.replace(/[^\p{L}\p{N}\s]+/gu, " ")
|
||||||
|
.replace(/\s+/g, " ")
|
||||||
|
.trim();
|
||||||
|
},
|
||||||
|
|
||||||
|
annotationPositionAnchor(annotation) {
|
||||||
|
const raw = annotation.annotationPosition || annotation.position;
|
||||||
|
let position = raw;
|
||||||
|
if (typeof raw === "string") {
|
||||||
|
try { position = JSON.parse(raw); } catch (_error) { return raw; }
|
||||||
|
}
|
||||||
|
if (!position || typeof position !== "object") return "";
|
||||||
|
if (position.type === "FragmentSelector" && position.value) {
|
||||||
|
const value = String(position.value);
|
||||||
|
const finalComma = value.lastIndexOf(",");
|
||||||
|
return `epub:${finalComma > 0 ? value.slice(0, finalComma) : value}`;
|
||||||
|
}
|
||||||
|
if (Number.isInteger(position.pageIndex) && Array.isArray(position.rects) && position.rects.length) {
|
||||||
|
const rect = position.rects[0] || [];
|
||||||
|
const x = Number(rect[0]);
|
||||||
|
const y = Number(rect[1]);
|
||||||
|
return `pdf:${position.pageIndex}:${x.toFixed(1)}:${y.toFixed(1)}`;
|
||||||
|
}
|
||||||
|
return this.normalizePosition(position);
|
||||||
|
},
|
||||||
|
|
||||||
|
areImportedDuplicates(left, right) {
|
||||||
|
const leftAnchor = this.annotationPositionAnchor(left);
|
||||||
|
const rightAnchor = this.annotationPositionAnchor(right);
|
||||||
|
if (!leftAnchor || leftAnchor !== rightAnchor) return false;
|
||||||
|
const leftPosition = this.normalizePosition(left.annotationPosition || left.position);
|
||||||
|
const rightPosition = this.normalizePosition(right.annotationPosition || right.position);
|
||||||
|
if (leftPosition && leftPosition === rightPosition) return true;
|
||||||
|
const leftText = this.normalizeAnnotationText(left.annotationText || left.text);
|
||||||
|
const rightText = this.normalizeAnnotationText(right.annotationText || right.text);
|
||||||
|
if (!leftText || !rightText) return false;
|
||||||
|
if (leftText === rightText) return true;
|
||||||
|
const shorter = leftText.length < rightText.length ? leftText : rightText;
|
||||||
|
const longer = shorter === leftText ? rightText : leftText;
|
||||||
|
return shorter.length >= 20 && longer.startsWith(shorter);
|
||||||
|
},
|
||||||
|
|
||||||
|
async collectImportedAnnotations() {
|
||||||
|
const annotations = [];
|
||||||
|
let allItems = [];
|
||||||
|
if (Zotero.DB && Zotero.DB.columnQueryAsync) {
|
||||||
|
const ids = await Zotero.DB.columnQueryAsync("SELECT itemID FROM itemAnnotations");
|
||||||
|
allItems = Zotero.Items.getAsync ? await Zotero.Items.getAsync(ids) : Zotero.Items.get(ids);
|
||||||
|
} else if (Zotero.Items.getAll) {
|
||||||
|
allItems = await Zotero.Items.getAll();
|
||||||
|
}
|
||||||
|
for (const item of allItems || []) {
|
||||||
|
if (!item || !item.isAnnotation || !item.isAnnotation()) continue;
|
||||||
|
if (this.hasKindleImportTag(item)) annotations.push(item);
|
||||||
|
}
|
||||||
|
return annotations;
|
||||||
|
},
|
||||||
|
|
||||||
|
async maintainImportedAnnotations(annotations, dryRun, results) {
|
||||||
|
if (!results.maintenanceDeletedIds) results.maintenanceDeletedIds = [];
|
||||||
|
const byAttachment = new Map();
|
||||||
|
for (const annotation of annotations) {
|
||||||
|
const parentID = annotation.parentItemID || annotation.parentID;
|
||||||
|
if (!byAttachment.has(parentID)) byAttachment.set(parentID, []);
|
||||||
|
byAttachment.get(parentID).push(annotation);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const group of byAttachment.values()) {
|
||||||
|
group.sort((a, b) =>
|
||||||
|
this.normalizeAnnotationText(b.annotationText).length -
|
||||||
|
this.normalizeAnnotationText(a.annotationText).length
|
||||||
|
);
|
||||||
|
const kept = [];
|
||||||
|
for (const annotation of group) {
|
||||||
|
const canonicalComment = this.mergeComments(annotation.annotationComment, "");
|
||||||
|
if (String(annotation.annotationComment || "") !== canonicalComment) {
|
||||||
|
annotation.annotationComment = canonicalComment;
|
||||||
|
results.updatedComments += 1;
|
||||||
|
if (!dryRun) await this.saveExistingAnnotation(annotation);
|
||||||
|
}
|
||||||
|
|
||||||
|
const duplicateOf = kept.find((candidate) => this.areImportedDuplicates(candidate, annotation));
|
||||||
|
if (!duplicateOf) {
|
||||||
|
kept.push(annotation);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const mergedComment = this.mergeComments(duplicateOf.annotationComment, annotation.annotationComment);
|
||||||
|
if (duplicateOf.annotationComment !== mergedComment) {
|
||||||
|
duplicateOf.annotationComment = mergedComment;
|
||||||
|
results.updatedComments += 1;
|
||||||
|
if (!dryRun) await this.saveExistingAnnotation(duplicateOf);
|
||||||
|
}
|
||||||
|
if (!dryRun) await annotation.eraseTx();
|
||||||
|
const keptClippingIds = new Set(
|
||||||
|
this.existingTags(duplicateOf)
|
||||||
|
.filter((tag) => tag.startsWith("kindle-id:"))
|
||||||
|
.map((tag) => tag.slice("kindle-id:".length))
|
||||||
|
);
|
||||||
|
for (const tag of this.existingTags(annotation)) {
|
||||||
|
const clippingID = tag.startsWith("kindle-id:")
|
||||||
|
? tag.slice("kindle-id:".length)
|
||||||
|
: "";
|
||||||
|
if (clippingID && !keptClippingIds.has(clippingID)) {
|
||||||
|
results.maintenanceDeletedIds.push(clippingID);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
results.removedDuplicates += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
existingSortIndex(annotation) {
|
existingSortIndex(annotation) {
|
||||||
return annotation.annotationSortIndex || annotation._annotationSortIndex || annotation.sortIndex || "";
|
return annotation.annotationSortIndex || annotation._annotationSortIndex || annotation.sortIndex || "";
|
||||||
},
|
},
|
||||||
|
|
@ -607,6 +761,23 @@ var KindleZoteroImporter = {
|
||||||
return this.existingTags(annotation).includes("kindle-import");
|
return this.existingTags(annotation).includes("kindle-import");
|
||||||
},
|
},
|
||||||
|
|
||||||
|
commentForExistingAnnotation(annotation, newComment, previousImportedComment) {
|
||||||
|
if (!this.hasKindleImportTag(annotation) || previousImportedComment === undefined) {
|
||||||
|
return this.mergeComments(annotation.annotationComment, newComment);
|
||||||
|
}
|
||||||
|
const previousParts = new Set(
|
||||||
|
String(previousImportedComment || "")
|
||||||
|
.split(/\n\s*\n/)
|
||||||
|
.map((part) => this.stripColourCode(part).toLowerCase())
|
||||||
|
.filter(Boolean)
|
||||||
|
);
|
||||||
|
const manualParts = String(annotation.annotationComment || "")
|
||||||
|
.split(/\n\s*\n/)
|
||||||
|
.filter((part) => !previousParts.has(this.stripColourCode(part).toLowerCase()))
|
||||||
|
.join("\n\n");
|
||||||
|
return this.mergeComments(manualParts, newComment);
|
||||||
|
},
|
||||||
|
|
||||||
confirmRecreateForSortIndex(entry, existingAnnotation, oldSortIndex, newSortIndex) {
|
confirmRecreateForSortIndex(entry, existingAnnotation, oldSortIndex, newSortIndex) {
|
||||||
const tags = this.existingTags(existingAnnotation).join(", ") || "(none)";
|
const tags = this.existingTags(existingAnnotation).join(", ") || "(none)";
|
||||||
const message = [
|
const message = [
|
||||||
|
|
@ -643,6 +814,37 @@ var KindleZoteroImporter = {
|
||||||
throw new Error("Existing annotation cannot be saved by this Zotero build");
|
throw new Error("Existing annotation cannot be saved by this Zotero build");
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async reconcileWriterState(path, plan, results) {
|
||||||
|
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
|
||||||
|
const failedIds = new Set(
|
||||||
|
(results.failed || [])
|
||||||
|
.map((failure) => failure.clipping_id)
|
||||||
|
.filter((clippingID) => plannedIds.has(clippingID))
|
||||||
|
);
|
||||||
|
plan.write_failed_ids = Array.from(failedIds).sort();
|
||||||
|
plan.delete_failed_ids = Array.from(new Set(results.failedDeletionIds || [])).sort();
|
||||||
|
const maintenanceDeletedIds = new Set(results.maintenanceDeletedIds || []);
|
||||||
|
if (failedIds.size) {
|
||||||
|
plan.annotations = (plan.annotations || []).filter(
|
||||||
|
(entry) => !failedIds.has(entry.clipping_id)
|
||||||
|
);
|
||||||
|
plan.all_annotations = (plan.all_annotations || []).filter(
|
||||||
|
(entry) => !failedIds.has(entry.clipping_id)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (maintenanceDeletedIds.size) {
|
||||||
|
plan.annotations = (plan.annotations || []).filter(
|
||||||
|
(entry) => !maintenanceDeletedIds.has(entry.clipping_id)
|
||||||
|
);
|
||||||
|
plan.all_annotations = (plan.all_annotations || []).filter(
|
||||||
|
(entry) => !maintenanceDeletedIds.has(entry.clipping_id)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
plan.delta_annotation_count = (plan.annotations || []).length;
|
||||||
|
plan.annotation_count = (plan.all_annotations || plan.annotations || []).length;
|
||||||
|
await Zotero.File.putContentsAsync(path, JSON.stringify(plan, null, 2) + "\n");
|
||||||
|
},
|
||||||
|
|
||||||
async writeAnnotations(plan, dryRun, onProgress) {
|
async writeAnnotations(plan, dryRun, onProgress) {
|
||||||
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
|
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
|
||||||
throw new Error(`Unsupported plan format: ${plan.format}`);
|
throw new Error(`Unsupported plan format: ${plan.format}`);
|
||||||
|
|
@ -660,75 +862,31 @@ var KindleZoteroImporter = {
|
||||||
recreateDeclined: 0,
|
recreateDeclined: 0,
|
||||||
removedDuplicates: 0,
|
removedDuplicates: 0,
|
||||||
deletedForIncremental: 0,
|
deletedForIncremental: 0,
|
||||||
|
failedDeletionIds: [],
|
||||||
|
maintenanceDeletedIds: [],
|
||||||
failed: [],
|
failed: [],
|
||||||
};
|
};
|
||||||
// Incremental deletions: remove annotations whose clipping_id is no longer present
|
let importedAnnotations = [];
|
||||||
if (!dryRun && plan.deletions && plan.deletions.length) {
|
let maintenanceReady = false;
|
||||||
const deletionIds = new Set(plan.deletions);
|
|
||||||
try {
|
try {
|
||||||
// Find all annotations with kindle-id tag
|
importedAnnotations = await this.collectImportedAnnotations();
|
||||||
const allAnnotations = [];
|
await this.maintainImportedAnnotations(importedAnnotations, dryRun, results);
|
||||||
// Collect via attachments to be safe
|
if (!dryRun && results.removedDuplicates) {
|
||||||
const allItems = Zotero.Items.getAll ? await Zotero.Items.getAll() : [];
|
importedAnnotations = await this.collectImportedAnnotations();
|
||||||
for (const item of allItems) {
|
|
||||||
if (item.isAnnotation && item.isAnnotation()) {
|
|
||||||
allAnnotations.push(item);
|
|
||||||
} else if (item.getAnnotations) {
|
|
||||||
const anns = item.getAnnotations();
|
|
||||||
for (const a of anns) allAnnotations.push(a);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Deduplicate
|
|
||||||
const seen = new Set();
|
|
||||||
for (const ann of allAnnotations) {
|
|
||||||
if (!ann || seen.has(ann.id)) continue;
|
|
||||||
seen.add(ann.id);
|
|
||||||
const tags = this.existingTags(ann);
|
|
||||||
for (const tag of tags) {
|
|
||||||
if (tag.startsWith("kindle-id:")) {
|
|
||||||
const id = tag.substring("kindle-id:".length);
|
|
||||||
if (deletionIds.has(id)) {
|
|
||||||
await ann.eraseTx();
|
|
||||||
results.deletedForIncremental += 1;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} catch (e) {
|
|
||||||
Zotero.logError(e);
|
|
||||||
}
|
}
|
||||||
|
maintenanceReady = true;
|
||||||
|
} catch (error) {
|
||||||
|
Zotero.logError(error);
|
||||||
|
results.failed.push({ clipping_id: "maintenance", message: String(error) });
|
||||||
}
|
}
|
||||||
const existingByAttachment = new Map();
|
const existingByAttachment = new Map();
|
||||||
|
|
||||||
const normForDedup = (s) => String(s || "").toLowerCase().normalize("NFKC").replace(/-\s*\n\s*/g, "").replace(/[^\w\s]+/g, " ").replace(/\s+/g, " ").trim();
|
|
||||||
const getExistingAnnotations = async (attachment) => {
|
const getExistingAnnotations = async (attachment) => {
|
||||||
if (!existingByAttachment.has(attachment.id)) {
|
if (!existingByAttachment.has(attachment.id)) {
|
||||||
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
|
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
|
||||||
const byFingerprint = new Map();
|
const byFingerprint = new Map();
|
||||||
const byNorm = new Map();
|
|
||||||
|
|
||||||
for (const annotation of annotations) {
|
for (const annotation of annotations) {
|
||||||
const norm = normForDedup(annotation.annotationText || "");
|
|
||||||
const dupNorm = byNorm.get(norm);
|
|
||||||
if (dupNorm && norm) {
|
|
||||||
// collapse identical highlights (e.g. My Clippings duplicates "forged its might…" x5)
|
|
||||||
const kept = dupNorm;
|
|
||||||
const keptHasComment = Boolean(kept.annotationComment);
|
|
||||||
const curHasComment = Boolean(annotation.annotationComment);
|
|
||||||
const duplicate = keptHasComment || !curHasComment ? annotation : kept;
|
|
||||||
const replacement = duplicate === annotation ? kept : annotation;
|
|
||||||
byNorm.set(norm, replacement);
|
|
||||||
// also update fingerprint map
|
|
||||||
const fpDup = this.existingAnnotationFingerprint(duplicate);
|
|
||||||
const fpKeep = this.existingAnnotationFingerprint(replacement);
|
|
||||||
byFingerprint.delete(fpDup);
|
|
||||||
byFingerprint.set(fpKeep, replacement);
|
|
||||||
if (!dryRun) await duplicate.eraseTx();
|
|
||||||
results.removedDuplicates += 1;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (norm) byNorm.set(norm, annotation);
|
|
||||||
const fingerprint = this.existingAnnotationFingerprint(annotation);
|
const fingerprint = this.existingAnnotationFingerprint(annotation);
|
||||||
const kept = byFingerprint.get(fingerprint);
|
const kept = byFingerprint.get(fingerprint);
|
||||||
if (!kept) {
|
if (!kept) {
|
||||||
|
|
@ -754,6 +912,7 @@ var KindleZoteroImporter = {
|
||||||
};
|
};
|
||||||
|
|
||||||
let completed = 0;
|
let completed = 0;
|
||||||
|
const successfulEntries = new Map();
|
||||||
for (const entry of plan.annotations) {
|
for (const entry of plan.annotations) {
|
||||||
try {
|
try {
|
||||||
const attachment = Zotero.Items.get(entry.attachment_item_id);
|
const attachment = Zotero.Items.get(entry.attachment_item_id);
|
||||||
|
|
@ -771,19 +930,32 @@ var KindleZoteroImporter = {
|
||||||
|
|
||||||
const existingAnnotations = await getExistingAnnotations(attachment);
|
const existingAnnotations = await getExistingAnnotations(attachment);
|
||||||
const fingerprint = this.annotationFingerprint(annotation);
|
const fingerprint = this.annotationFingerprint(annotation);
|
||||||
const existingAnnotation = existingAnnotations.get(fingerprint);
|
const existingAnnotation = existingAnnotations.get(fingerprint) ||
|
||||||
|
Array.from(existingAnnotations.values()).find(
|
||||||
|
(candidate) => this.areImportedDuplicates(candidate, annotation)
|
||||||
|
);
|
||||||
if (existingAnnotation) {
|
if (existingAnnotation) {
|
||||||
const mergedComment = this.mergeComments(
|
const mergedComment = this.commentForExistingAnnotation(
|
||||||
existingAnnotation.annotationComment,
|
existingAnnotation,
|
||||||
annotation.comment
|
annotation.comment,
|
||||||
|
entry.previous_imported_comment
|
||||||
);
|
);
|
||||||
let changed = false;
|
let changed = false;
|
||||||
if (mergedComment && existingAnnotation.annotationComment !== mergedComment) {
|
if (existingAnnotation.annotationComment !== mergedComment) {
|
||||||
existingAnnotation.annotationComment = mergedComment;
|
existingAnnotation.annotationComment = mergedComment;
|
||||||
changed = true;
|
changed = true;
|
||||||
results.updatedComments += 1;
|
results.updatedComments += 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const existingTags = new Set(this.existingTags(existingAnnotation));
|
||||||
|
for (const tag of annotation.tags || []) {
|
||||||
|
const name = tag.name || tag.tag;
|
||||||
|
if (!name || existingTags.has(name) || !existingAnnotation.addTag) continue;
|
||||||
|
existingAnnotation.addTag(name);
|
||||||
|
existingTags.add(name);
|
||||||
|
changed = true;
|
||||||
|
}
|
||||||
|
|
||||||
// colour update (British spelling in code handles both)
|
// colour update (British spelling in code handles both)
|
||||||
const existingColour = existingAnnotation.annotationColor || existingAnnotation.color || "";
|
const existingColour = existingAnnotation.annotationColor || existingAnnotation.color || "";
|
||||||
const newColour = annotation.color || "";
|
const newColour = annotation.color || "";
|
||||||
|
|
@ -836,12 +1008,14 @@ var KindleZoteroImporter = {
|
||||||
});
|
});
|
||||||
results.recreatedForSortIndex += 1;
|
results.recreatedForSortIndex += 1;
|
||||||
results.skippedExisting += 1;
|
results.skippedExisting += 1;
|
||||||
|
successfulEntries.set(entry.clipping_id, entry);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
} else if (sortIndexChanged && dryRun) {
|
} else if (sortIndexChanged && dryRun) {
|
||||||
results.updatedSortIndex += 1;
|
results.updatedSortIndex += 1;
|
||||||
}
|
}
|
||||||
results.skippedExisting += 1;
|
results.skippedExisting += 1;
|
||||||
|
successfulEntries.set(entry.clipping_id, entry);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -856,6 +1030,7 @@ var KindleZoteroImporter = {
|
||||||
annotationText: annotation.text || "",
|
annotationText: annotation.text || "",
|
||||||
});
|
});
|
||||||
results.created += 1;
|
results.created += 1;
|
||||||
|
successfulEntries.set(entry.clipping_id, entry);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
results.failed.push({
|
results.failed.push({
|
||||||
clipping_id: entry.clipping_id,
|
clipping_id: entry.clipping_id,
|
||||||
|
|
@ -870,6 +1045,40 @@ var KindleZoteroImporter = {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (!dryRun && plan.deletions && plan.deletions.length) {
|
||||||
|
const deletionIds = new Set(plan.deletions);
|
||||||
|
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
|
||||||
|
if (!maintenanceReady) {
|
||||||
|
results.failedDeletionIds.push(...deletionIds);
|
||||||
|
} else {
|
||||||
|
for (const ann of importedAnnotations) {
|
||||||
|
for (const tag of this.existingTags(ann)) {
|
||||||
|
if (!tag.startsWith("kindle-id:")) continue;
|
||||||
|
const id = tag.slice("kindle-id:".length);
|
||||||
|
if (!deletionIds.has(id) || alreadyDeletedIds.has(id)) break;
|
||||||
|
const replacement = successfulEntries.get(id);
|
||||||
|
if (replacement) {
|
||||||
|
const sameAttachment = (ann.parentItemID || ann.parentID) === replacement.attachment_item_id;
|
||||||
|
const sameFingerprint = this.existingAnnotationFingerprint(ann) ===
|
||||||
|
this.annotationFingerprint(replacement.annotation);
|
||||||
|
if (sameAttachment && sameFingerprint) break;
|
||||||
|
} else if ((plan.annotations || []).some((entry) => entry.clipping_id === id)) {
|
||||||
|
results.failedDeletionIds.push(id);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await ann.eraseTx();
|
||||||
|
results.deletedForIncremental += 1;
|
||||||
|
} catch (error) {
|
||||||
|
results.failedDeletionIds.push(id);
|
||||||
|
results.failed.push({ clipping_id: id, message: `Deletion failed: ${error}` });
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return results;
|
return results;
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -90,6 +90,9 @@
|
||||||
.muted { color: var(--muted); font-size: 12px; }
|
.muted { color: var(--muted); font-size: 12px; }
|
||||||
.highlight-text { max-width: 520px; white-space: pre-wrap; word-break: break-word; font-size: 12px; line-height: 1.45; font-family: inherit; }
|
.highlight-text { max-width: 520px; white-space: pre-wrap; word-break: break-word; font-size: 12px; line-height: 1.45; font-family: inherit; }
|
||||||
.integrated-meta { font-size: 10.5px; color: var(--muted); font-family: 'JetBrains Mono', monospace; }
|
.integrated-meta { font-size: 10.5px; color: var(--muted); font-family: 'JetBrains Mono', monospace; }
|
||||||
|
.pagination { display: flex; align-items: center; justify-content: flex-end; gap: 8px; margin: 10px 0; color: var(--muted); font-size: 11px; }
|
||||||
|
.pagination button, .pagination select { width: auto; padding: 5px 8px; border: 1px solid var(--line); border-radius: var(--radius); background: var(--panel); color: var(--text); font: inherit; }
|
||||||
|
.pagination button:disabled { color: var(--faint); background: var(--panel-soft); cursor: not-allowed; }
|
||||||
::selection { background: var(--accent); color: white; }
|
::selection { background: var(--accent); color: white; }
|
||||||
</style>
|
</style>
|
||||||
<script>
|
<script>
|
||||||
|
|
@ -175,6 +178,41 @@
|
||||||
if (el) el.textContent = `${minutes}:${String(seconds % 60).padStart(2, '0')} elapsed`;
|
if (el) el.textContent = `${minutes}:${String(seconds % 60).padStart(2, '0')} elapsed`;
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async collectorCredentials() {
|
||||||
|
let anonId = '';
|
||||||
|
let token = '';
|
||||||
|
try { anonId = Services.prefs.getCharPref("extensions.kindleZoteroImporter.collectorAnonId"); } catch(e) {}
|
||||||
|
try { token = Services.prefs.getCharPref("extensions.kindleZoteroImporter.collectorToken"); } catch(e) {}
|
||||||
|
if (anonId && token) return {anonId, token};
|
||||||
|
const response = await fetch("https://annotation.utkubilen.de/register", {method: "POST"});
|
||||||
|
if (!response.ok) throw new Error(`Collector registration returned HTTP ${response.status}`);
|
||||||
|
const credentials = await response.json();
|
||||||
|
if (!credentials.anonId || !credentials.token) throw new Error('Collector registration was incomplete');
|
||||||
|
Services.prefs.setCharPref("extensions.kindleZoteroImporter.collectorAnonId", credentials.anonId);
|
||||||
|
Services.prefs.setCharPref("extensions.kindleZoteroImporter.collectorToken", credentials.token);
|
||||||
|
return credentials;
|
||||||
|
},
|
||||||
|
|
||||||
|
collectorBatches(anonId, data) {
|
||||||
|
const batches = [];
|
||||||
|
let current = [];
|
||||||
|
for (const item of data) {
|
||||||
|
const candidate = current.concat(item);
|
||||||
|
const bytes = new TextEncoder().encode(JSON.stringify({anonId, data: candidate})).length;
|
||||||
|
if (!current.length && bytes > 1500000) {
|
||||||
|
throw new Error('One annotation exceeds the collector request limit');
|
||||||
|
}
|
||||||
|
if (current.length && (current.length >= 200 || bytes > 1500000)) {
|
||||||
|
batches.push(current);
|
||||||
|
current = [item];
|
||||||
|
} else {
|
||||||
|
current = candidate;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (current.length) batches.push(current);
|
||||||
|
return batches;
|
||||||
|
},
|
||||||
|
|
||||||
updateProgress(percent, stage, detail) {
|
updateProgress(percent, stage, detail) {
|
||||||
const prog = document.getElementById('progress');
|
const prog = document.getElementById('progress');
|
||||||
if (prog) prog.value = Math.max(0, Math.min(100, Number(percent) || 0));
|
if (prog) prog.value = Math.max(0, Math.min(100, Number(percent) || 0));
|
||||||
|
|
@ -186,7 +224,7 @@
|
||||||
if (pctEl) pctEl.textContent = `${Math.round(Number(percent) || 0)}%`;
|
if (pctEl) pctEl.textContent = `${Math.round(Number(percent) || 0)}%`;
|
||||||
},
|
},
|
||||||
|
|
||||||
completeRun(summary, results, data) {
|
async completeRun(summary, results, data) {
|
||||||
this.running = false;
|
this.running = false;
|
||||||
clearInterval(this.timer);
|
clearInterval(this.timer);
|
||||||
this.updateProgress(100, 'Import complete', 'Annotations have been checked and saved in Zotero');
|
this.updateProgress(100, 'Import complete', 'Annotations have been checked and saved in Zotero');
|
||||||
|
|
@ -219,20 +257,14 @@
|
||||||
grid.appendChild(makeCard(results.updatedComments + results.updatedSortIndex, 'updated'));
|
grid.appendChild(makeCard(results.updatedComments + results.updatedSortIndex, 'updated'));
|
||||||
grid.appendChild(makeCard(results.failed.length, 'failed'));
|
grid.appendChild(makeCard(results.failed.length, 'failed'));
|
||||||
resultsEl.appendChild(grid);
|
resultsEl.appendChild(grid);
|
||||||
// Opt-in prompt for anonymised sharing (once, after first successful import)
|
const consentVersion = 2;
|
||||||
try {
|
let acceptedVersion = 0;
|
||||||
const hasAsked = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare");
|
try { acceptedVersion = Services.prefs.getIntPref("extensions.kindleZoteroImporter.shareConsentVersion"); } catch(e) {}
|
||||||
} catch(e) {
|
if (acceptedVersion < consentVersion && (data.finalPlan?.annotations || []).length) {
|
||||||
// First run: default ON, allow opt-out
|
const accepted = confirm("Share annotation data with the Kanzi developer?\n\nThis sends a random pseudonymous installation ID, publication metadata, raw highlight text and comments, hashes, colour, and dates to annotation.utkubilen.de. Free text may contain identifying information. The dashboard is private, but this data is not anonymous.\n\nChoose OK to opt in, or Cancel to keep sharing disabled. You can change this later in Settings.");
|
||||||
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", true);
|
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", accepted);
|
||||||
try {
|
|
||||||
const anonId = Math.random().toString(36).slice(2,10) + Date.now().toString(36);
|
|
||||||
Services.prefs.setCharPref("extensions.kindleZoteroImporter.anonId", anonId);
|
|
||||||
} catch(e2) {}
|
|
||||||
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare", true);
|
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare", true);
|
||||||
if (results.created > 0) {
|
Services.prefs.setIntPref("extensions.kindleZoteroImporter.shareConsentVersion", consentVersion);
|
||||||
alert("Anonymised sharing is enabled by default to help future development (only citation key, hashed highlight, colour, has comment + note, and dates — never your highlight text alone). You can opt out anytime in Settings → uncheck 'Share anonymised annotations'.");
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
this.data = data;
|
this.data = data;
|
||||||
|
|
@ -243,45 +275,41 @@
|
||||||
const bar = document.getElementById('reimport-bar');
|
const bar = document.getElementById('reimport-bar');
|
||||||
if (bar) bar.style.display = 'none';
|
if (bar) bar.style.display = 'none';
|
||||||
this.status(`${summary.counts.final_annotations || 0} positioned annotations processed; ${this.rows.length} review rows remain.`, results.failed.length > 0);
|
this.status(`${summary.counts.final_annotations || 0} positioned annotations processed; ${this.rows.length} review rows remain.`, results.failed.length > 0);
|
||||||
// Anonymised sharing if opted in — robust to missing prefs (British spelling)
|
// Pseudonymous sharing after explicit consent.
|
||||||
try {
|
try {
|
||||||
let share = true;
|
let share = false;
|
||||||
try { share = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) { share = true; }
|
try { share = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) {}
|
||||||
if (share) {
|
if (share) {
|
||||||
let anonId = "anon";
|
const {anonId, token} = await this.collectorCredentials();
|
||||||
try { anonId = Services.prefs.getCharPref("extensions.kindleZoteroImporter.anonId") || "anon"; } catch(e) { anonId = "anon"; }
|
|
||||||
// ensure anonId exists for next time
|
|
||||||
if (anonId === "anon") {
|
|
||||||
try {
|
|
||||||
anonId = Math.random().toString(36).slice(2,10) + Date.now().toString(36);
|
|
||||||
Services.prefs.setCharPref("extensions.kindleZoteroImporter.anonId", anonId);
|
|
||||||
} catch(e) {}
|
|
||||||
}
|
|
||||||
const anonData = (data.finalPlan?.annotations || []).map(a => ({
|
const anonData = (data.finalPlan?.annotations || []).map(a => ({
|
||||||
citation_key: a.citation_key,
|
citation_key: (a.citation_key || '').slice(0, 512) || null,
|
||||||
doi: a.doi || null,
|
doi: (a.doi || '').slice(0, 512) || null,
|
||||||
isbn: a.isbn || null,
|
isbn: (a.isbn || '').slice(0, 512) || null,
|
||||||
issn: a.issn || null,
|
issn: (a.issn || '').slice(0, 512) || null,
|
||||||
publication_title: a.publication_title || null,
|
publication_title: (a.publication_title || '').slice(0, 1000) || null,
|
||||||
clipping_id_hash: a.clipping_id ? a.clipping_id.slice(0,8) : null,
|
clipping_id_hash: (a.clipping_id || '').slice(0, 128) || null,
|
||||||
norm_hash: a.norm_hash || null,
|
norm_hash: (a.norm_hash || '').slice(0, 256) || null,
|
||||||
fuzzy_hash: a.fuzzy_hash || null,
|
fuzzy_hash: (a.fuzzy_hash || '').slice(0, 256) || null,
|
||||||
text: a.annotation?.text || "",
|
text: (a.annotation?.text || '').slice(0, 2000),
|
||||||
color: a.annotation?.color,
|
color: a.annotation?.color,
|
||||||
has_comment: !!(a.annotation?.comment),
|
has_comment: !!(a.annotation?.comment),
|
||||||
comment: a.annotation?.comment || "",
|
comment: (a.annotation?.comment || '').slice(0, 2000),
|
||||||
comment_hash: a.annotation?.comment ? btoa(a.annotation.comment).slice(0,24) : null,
|
comment_hash: null,
|
||||||
clipping_added_on_iso: a.clipping_added_on_iso,
|
clipping_added_on_iso: (a.clipping_added_on_iso || '').slice(0, 64) || null,
|
||||||
integrated_at: a.integrated_at
|
integrated_at: (a.integrated_at || '').slice(0, 64) || null
|
||||||
}));
|
}));
|
||||||
// Fire and forget, no text sent — private, not public (annotation.utkubilen.de)
|
for (const batch of this.collectorBatches(anonId, anonData)) {
|
||||||
fetch("https://annotation.utkubilen.de/highlights", {
|
const response = await fetch("https://annotation.utkubilen.de/highlights", {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
headers: {"Content-Type":"application/json"},
|
headers: {"Content-Type":"application/json", "Authorization": `Bearer ${token}`},
|
||||||
body: JSON.stringify({anonId, data: anonData})
|
body: JSON.stringify({anonId, data: batch})
|
||||||
}).catch(()=>{});
|
});
|
||||||
|
if (!response.ok) throw new Error(`Collector returned HTTP ${response.status}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch(e) {
|
||||||
|
this.status(`Import succeeded, but annotation sharing failed: ${e}`, true);
|
||||||
}
|
}
|
||||||
} catch(e) {}
|
|
||||||
},
|
},
|
||||||
|
|
||||||
failRun(message) {
|
failRun(message) {
|
||||||
|
|
@ -298,6 +326,11 @@
|
||||||
return !!(cb && cb.checked);
|
return !!(cb && cb.checked);
|
||||||
},
|
},
|
||||||
|
|
||||||
|
getRetryUnresolved() {
|
||||||
|
const cb = document.getElementById('retry-unresolved-check');
|
||||||
|
return !!(cb && cb.checked);
|
||||||
|
},
|
||||||
|
|
||||||
updateInfobox() {
|
updateInfobox() {
|
||||||
const box = document.getElementById('infobox');
|
const box = document.getElementById('infobox');
|
||||||
if (!box) return;
|
if (!box) return;
|
||||||
|
|
@ -614,8 +647,54 @@
|
||||||
} catch { return String(iso).slice(0,10); }
|
} catch { return String(iso).slice(0,10); }
|
||||||
},
|
},
|
||||||
|
|
||||||
|
paginate(items, pageKey, pageSizeKey) {
|
||||||
|
const totalPages = Math.max(1, Math.ceil(items.length / this[pageSizeKey]));
|
||||||
|
if (this[pageKey] >= totalPages) this[pageKey] = totalPages - 1;
|
||||||
|
if (this[pageKey] < 0) this[pageKey] = 0;
|
||||||
|
const start = this[pageKey] * this[pageSizeKey];
|
||||||
|
return { rows: items.slice(start, start + this[pageSizeKey]), totalPages };
|
||||||
|
},
|
||||||
|
|
||||||
|
refreshPagination(containerId, pageKey, pageSizeKey, totalPages, refresh) {
|
||||||
|
const container = document.getElementById(containerId);
|
||||||
|
if (!container) return;
|
||||||
|
container.textContent = '';
|
||||||
|
const button = (label, disabled, action) => {
|
||||||
|
const el = document.createElement('button');
|
||||||
|
el.textContent = label;
|
||||||
|
el.disabled = disabled;
|
||||||
|
if (!disabled) el.addEventListener('click', action);
|
||||||
|
return el;
|
||||||
|
};
|
||||||
|
container.appendChild(button('Previous', this[pageKey] === 0, () => {
|
||||||
|
this[pageKey] -= 1;
|
||||||
|
refresh();
|
||||||
|
}));
|
||||||
|
const status = document.createElement('span');
|
||||||
|
status.textContent = `Page ${this[pageKey] + 1} of ${totalPages}`;
|
||||||
|
container.appendChild(status);
|
||||||
|
container.appendChild(button('Next', this[pageKey] >= totalPages - 1, () => {
|
||||||
|
this[pageKey] += 1;
|
||||||
|
refresh();
|
||||||
|
}));
|
||||||
|
const select = document.createElement('select');
|
||||||
|
for (const size of [25, 50, 100, 200]) {
|
||||||
|
const option = document.createElement('option');
|
||||||
|
option.value = size;
|
||||||
|
option.textContent = `${size} per page`;
|
||||||
|
option.selected = size === this[pageSizeKey];
|
||||||
|
select.appendChild(option);
|
||||||
|
}
|
||||||
|
select.addEventListener('change', () => {
|
||||||
|
this[pageSizeKey] = Number(select.value);
|
||||||
|
this[pageKey] = 0;
|
||||||
|
refresh();
|
||||||
|
});
|
||||||
|
container.appendChild(select);
|
||||||
|
},
|
||||||
|
|
||||||
getIntegratedHighlights() {
|
getIntegratedHighlights() {
|
||||||
const anns = this.data.finalPlan?.annotations || [];
|
const anns = this.data.finalPlan?.all_annotations || this.data.finalPlan?.annotations || [];
|
||||||
return anns.slice().sort((a,b) => (a.clipping_title || '').localeCompare(b.clipping_title || '') || (a.annotation?.text || '').localeCompare(b.annotation?.text || ''));
|
return anns.slice().sort((a,b) => (a.clipping_title || '').localeCompare(b.clipping_title || '') || (a.annotation?.text || '').localeCompare(b.annotation?.text || ''));
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|
@ -639,82 +718,11 @@
|
||||||
const dateMap = this.buildClippingDateMap();
|
const dateMap = this.buildClippingDateMap();
|
||||||
const filtered = needle ? all.filter(a => `${a.clipping_title} ${a.annotation?.text || ''} ${a.citation_key || ''} ${a.annotation?.pageLabel || ''} ${a.clipping_added_on_iso || ''} ${a.integrated_at || ''}`.toLowerCase().includes(needle)) : all;
|
const filtered = needle ? all.filter(a => `${a.clipping_title} ${a.annotation?.text || ''} ${a.citation_key || ''} ${a.annotation?.pageLabel || ''} ${a.clipping_added_on_iso || ''} ${a.integrated_at || ''}`.toLowerCase().includes(needle)) : all;
|
||||||
if (resetPage) this.integratedPage = 0;
|
if (resetPage) this.integratedPage = 0;
|
||||||
const totalPages = Math.max(1, Math.ceil(filtered.length / this.integratedPageSize));
|
const page = this.paginate(filtered, 'integratedPage', 'integratedPageSize');
|
||||||
if (this.integratedPage >= totalPages) this.integratedPage = totalPages - 1;
|
const totalPages = page.totalPages;
|
||||||
const start = this.integratedPage * this.integratedPageSize;
|
const toShow = page.rows;
|
||||||
const toShow = filtered.slice(start, start + this.integratedPageSize);
|
|
||||||
if (countEl) countEl.textContent = `${filtered.length} of ${all.length} — page ${this.integratedPage + 1}/${totalPages} (${this.integratedPageSize} per page)`;
|
if (countEl) countEl.textContent = `${filtered.length} of ${all.length} — page ${this.integratedPage + 1}/${totalPages} (${this.integratedPageSize} per page)`;
|
||||||
// pagination controls
|
this.refreshPagination('integrated-pagination', 'integratedPage', 'integratedPageSize', totalPages, () => this.refreshIntegrated());
|
||||||
let pag = document.getElementById('integrated-pagination');
|
|
||||||
if (!pag) {
|
|
||||||
pag = document.createElement('div');
|
|
||||||
pag.id = 'integrated-pagination';
|
|
||||||
pag.style.display = 'flex';
|
|
||||||
pag.style.justifyContent = 'space-between';
|
|
||||||
pag.style.alignItems = 'center';
|
|
||||||
pag.style.margin = '8px 0';
|
|
||||||
pag.style.gap = '8px';
|
|
||||||
pag.style.flexWrap = 'wrap';
|
|
||||||
tbody.parentElement.parentElement.appendChild(pag);
|
|
||||||
}
|
|
||||||
pag.textContent = '';
|
|
||||||
const mkBtn = (label, disabled, onClick) => {
|
|
||||||
const b = document.createElement('button');
|
|
||||||
b.textContent = label;
|
|
||||||
b.disabled = disabled;
|
|
||||||
b.style.padding = '4px 8px';
|
|
||||||
b.style.border = '1px solid var(--line)';
|
|
||||||
b.style.background = disabled ? 'var(--panel-soft)' : 'var(--panel)';
|
|
||||||
b.style.borderRadius = 'var(--radius)';
|
|
||||||
b.style.cursor = disabled ? 'not-allowed' : 'pointer';
|
|
||||||
b.style.fontSize = '11px';
|
|
||||||
if (!disabled) b.addEventListener('click', onClick);
|
|
||||||
return b;
|
|
||||||
};
|
|
||||||
const left = document.createElement('div');
|
|
||||||
left.style.display = 'flex';
|
|
||||||
left.style.gap = '4px';
|
|
||||||
left.appendChild(mkBtn('‹ Prev', this.integratedPage === 0, () => { this.integratedPage--; this.refreshIntegrated(); }));
|
|
||||||
left.appendChild(mkBtn('Next ›', this.integratedPage >= totalPages - 1, () => { this.integratedPage++; this.refreshIntegrated(); }));
|
|
||||||
const mid = document.createElement('span');
|
|
||||||
mid.className = 'muted';
|
|
||||||
mid.textContent = `Page ${this.integratedPage + 1} of ${totalPages}`;
|
|
||||||
left.appendChild(mid);
|
|
||||||
pag.appendChild(left);
|
|
||||||
const right = document.createElement('div');
|
|
||||||
right.style.display = 'flex';
|
|
||||||
right.style.gap = '6px';
|
|
||||||
right.style.alignItems = 'center';
|
|
||||||
const sel = document.createElement('select');
|
|
||||||
sel.style.padding = '4px';
|
|
||||||
sel.style.border = '1px solid var(--line)';
|
|
||||||
sel.style.borderRadius = 'var(--radius)';
|
|
||||||
sel.style.fontSize = '11px';
|
|
||||||
for (const n of [25, 50, 100, 200, 500]) {
|
|
||||||
const opt = document.createElement('option');
|
|
||||||
opt.value = n;
|
|
||||||
opt.textContent = n + ' per page';
|
|
||||||
if (n === this.integratedPageSize) opt.selected = true;
|
|
||||||
sel.appendChild(opt);
|
|
||||||
}
|
|
||||||
const optAll = document.createElement('option');
|
|
||||||
optAll.value = filtered.length || 500;
|
|
||||||
optAll.textContent = 'Show all';
|
|
||||||
sel.appendChild(optAll);
|
|
||||||
sel.addEventListener('change', () => { this.integratedPageSize = parseInt(sel.value, 10); this.integratedPage = 0; this.refreshIntegrated(); });
|
|
||||||
right.appendChild(sel);
|
|
||||||
const expand = document.createElement('button');
|
|
||||||
expand.textContent = 'Expand 50 →';
|
|
||||||
expand.title = 'Show next 50';
|
|
||||||
expand.style.padding = '4px 8px';
|
|
||||||
expand.style.border = '1px solid var(--line)';
|
|
||||||
expand.style.background = 'var(--panel)';
|
|
||||||
expand.style.borderRadius = 'var(--radius)';
|
|
||||||
expand.style.cursor = 'pointer';
|
|
||||||
expand.style.fontSize = '11px';
|
|
||||||
expand.addEventListener('click', () => { this.integratedPageSize += 50; this.refreshIntegrated(); });
|
|
||||||
right.appendChild(expand);
|
|
||||||
pag.appendChild(right);
|
|
||||||
if (!toShow.length) {
|
if (!toShow.length) {
|
||||||
const tr = document.createElement('tr');
|
const tr = document.createElement('tr');
|
||||||
const td = document.createElement('td');
|
const td = document.createElement('td');
|
||||||
|
|
@ -768,7 +776,7 @@
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
refreshMappings() {
|
refreshMappings(resetPage = false) {
|
||||||
const tbody = document.getElementById('mappings-rows');
|
const tbody = document.getElementById('mappings-rows');
|
||||||
const countEl = document.getElementById('mappings-count');
|
const countEl = document.getElementById('mappings-count');
|
||||||
if (!tbody) return;
|
if (!tbody) return;
|
||||||
|
|
@ -776,7 +784,10 @@
|
||||||
const needle = (document.getElementById('mappings-search')?.value || '').toLowerCase();
|
const needle = (document.getElementById('mappings-search')?.value || '').toLowerCase();
|
||||||
const overrides = this.data.persistentOverrides?.overrides || [];
|
const overrides = this.data.persistentOverrides?.overrides || [];
|
||||||
const filtered = needle ? overrides.filter(o => `${o.clipping_title} ${o.resolution?.citation_key || ''} ${o.resolution?.zotero_key || ''} ${o.resolution?.ignore ? 'ignored' : ''}`.toLowerCase().includes(needle)) : overrides;
|
const filtered = needle ? overrides.filter(o => `${o.clipping_title} ${o.resolution?.citation_key || ''} ${o.resolution?.zotero_key || ''} ${o.resolution?.ignore ? 'ignored' : ''}`.toLowerCase().includes(needle)) : overrides;
|
||||||
if (countEl) countEl.textContent = `${filtered.length} of ${overrides.length} mappings`;
|
if (resetPage) this.mappingsPage = 0;
|
||||||
|
const page = this.paginate(filtered, 'mappingsPage', 'mappingsPageSize');
|
||||||
|
if (countEl) countEl.textContent = `${filtered.length} of ${overrides.length} mappings — page ${this.mappingsPage + 1}/${page.totalPages}`;
|
||||||
|
this.refreshPagination('mappings-pagination', 'mappingsPage', 'mappingsPageSize', page.totalPages, () => this.refreshMappings());
|
||||||
if (!filtered.length) {
|
if (!filtered.length) {
|
||||||
const tr = document.createElement('tr');
|
const tr = document.createElement('tr');
|
||||||
const td = document.createElement('td');
|
const td = document.createElement('td');
|
||||||
|
|
@ -786,7 +797,7 @@
|
||||||
tbody.appendChild(tr);
|
tbody.appendChild(tr);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
for (const entry of filtered) {
|
for (const entry of page.rows) {
|
||||||
const tr = document.createElement('tr');
|
const tr = document.createElement('tr');
|
||||||
const tdTitle = document.createElement('td');
|
const tdTitle = document.createElement('td');
|
||||||
tdTitle.textContent = entry.clipping_title;
|
tdTitle.textContent = entry.clipping_title;
|
||||||
|
|
@ -886,6 +897,7 @@
|
||||||
this.settings = { ...this.settings, ...newSettings };
|
this.settings = { ...this.settings, ...newSettings };
|
||||||
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", shareAnonymized);
|
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.shareAnonymized", shareAnonymized);
|
||||||
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare", true);
|
Services.prefs.setBoolPref("extensions.kindleZoteroImporter.askedAnonymizedShare", true);
|
||||||
|
Services.prefs.setIntPref("extensions.kindleZoteroImporter.shareConsentVersion", 2);
|
||||||
if (shareAnonymized) {
|
if (shareAnonymized) {
|
||||||
try {
|
try {
|
||||||
Services.prefs.getCharPref("extensions.kindleZoteroImporter.anonId");
|
Services.prefs.getCharPref("extensions.kindleZoteroImporter.anonId");
|
||||||
|
|
@ -1005,6 +1017,25 @@
|
||||||
fullLabel.title = 'When checked, re-processes all 2379 clippings from scratch (slow). Unchecked = incremental (only new/changed highlights, fast).';
|
fullLabel.title = 'When checked, re-processes all 2379 clippings from scratch (slow). Unchecked = incremental (only new/changed highlights, fast).';
|
||||||
fullRow.appendChild(fullLabel);
|
fullRow.appendChild(fullLabel);
|
||||||
panel.appendChild(fullRow);
|
panel.appendChild(fullRow);
|
||||||
|
const retryRow = document.createElement('div');
|
||||||
|
retryRow.style.marginTop = '6px';
|
||||||
|
retryRow.style.display = 'flex';
|
||||||
|
retryRow.style.alignItems = 'center';
|
||||||
|
retryRow.style.gap = '6px';
|
||||||
|
const retryCheck = document.createElement('input');
|
||||||
|
retryCheck.type = 'checkbox';
|
||||||
|
retryCheck.id = 'retry-unresolved-check';
|
||||||
|
retryCheck.style.width = 'auto';
|
||||||
|
retryRow.appendChild(retryCheck);
|
||||||
|
const retryLabel = document.createElement('label');
|
||||||
|
retryLabel.htmlFor = 'retry-unresolved-check';
|
||||||
|
retryLabel.textContent = 'Retry unresolved attachments and positions';
|
||||||
|
retryLabel.style.margin = '0';
|
||||||
|
retryLabel.style.fontSize = '12px';
|
||||||
|
retryLabel.style.color = 'var(--muted)';
|
||||||
|
retryLabel.title = 'Retries prior matching and positioning failures without rebuilding annotations that already succeeded.';
|
||||||
|
retryRow.appendChild(retryLabel);
|
||||||
|
panel.appendChild(retryRow);
|
||||||
root.appendChild(panel);
|
root.appendChild(panel);
|
||||||
|
|
||||||
const resultsDiv = document.createElement('div');
|
const resultsDiv = document.createElement('div');
|
||||||
|
|
@ -1081,6 +1112,10 @@
|
||||||
iTbody.id='integrated-rows';
|
iTbody.id='integrated-rows';
|
||||||
iTable.appendChild(iTbody);
|
iTable.appendChild(iTbody);
|
||||||
integratedTab.appendChild(iTable);
|
integratedTab.appendChild(iTable);
|
||||||
|
const iPagination = document.createElement('div');
|
||||||
|
iPagination.id = 'integrated-pagination';
|
||||||
|
iPagination.className = 'pagination';
|
||||||
|
integratedTab.appendChild(iPagination);
|
||||||
root.appendChild(integratedTab);
|
root.appendChild(integratedTab);
|
||||||
|
|
||||||
const conflictsTab = document.createElement('div');
|
const conflictsTab = document.createElement('div');
|
||||||
|
|
@ -1129,7 +1164,7 @@
|
||||||
search.placeholder = 'Filter conflicts by title, status, candidate, citekey...';
|
search.placeholder = 'Filter conflicts by title, status, candidate, citekey...';
|
||||||
search.style.width = '320px';
|
search.style.width = '320px';
|
||||||
search.style.maxWidth = '50%';
|
search.style.maxWidth = '50%';
|
||||||
search.addEventListener('input', () => this.refreshRows());
|
search.addEventListener('input', () => this.refreshRows(true));
|
||||||
searchWrap.appendChild(search);
|
searchWrap.appendChild(search);
|
||||||
conflictsTab.appendChild(searchWrap);
|
conflictsTab.appendChild(searchWrap);
|
||||||
const table = document.createElement('table');
|
const table = document.createElement('table');
|
||||||
|
|
@ -1140,6 +1175,10 @@
|
||||||
tbody.id = 'rows';
|
tbody.id = 'rows';
|
||||||
table.appendChild(tbody);
|
table.appendChild(tbody);
|
||||||
conflictsTab.appendChild(table);
|
conflictsTab.appendChild(table);
|
||||||
|
const cPagination = document.createElement('div');
|
||||||
|
cPagination.id = 'conflicts-pagination';
|
||||||
|
cPagination.className = 'pagination';
|
||||||
|
conflictsTab.appendChild(cPagination);
|
||||||
root.appendChild(conflictsTab);
|
root.appendChild(conflictsTab);
|
||||||
|
|
||||||
const mappingsTab = document.createElement('div');
|
const mappingsTab = document.createElement('div');
|
||||||
|
|
@ -1161,7 +1200,7 @@
|
||||||
mSearch.placeholder = 'Filter mappings by title, citekey...';
|
mSearch.placeholder = 'Filter mappings by title, citekey...';
|
||||||
mSearch.style.width = '320px';
|
mSearch.style.width = '320px';
|
||||||
mSearch.style.maxWidth = '50%';
|
mSearch.style.maxWidth = '50%';
|
||||||
mSearch.addEventListener('input', () => this.refreshMappings());
|
mSearch.addEventListener('input', () => this.refreshMappings(true));
|
||||||
mSearchWrap.appendChild(mSearch);
|
mSearchWrap.appendChild(mSearch);
|
||||||
mappingsTab.appendChild(mSearchWrap);
|
mappingsTab.appendChild(mSearchWrap);
|
||||||
const mCount = document.createElement('div');
|
const mCount = document.createElement('div');
|
||||||
|
|
@ -1177,6 +1216,10 @@
|
||||||
mTbody.id = 'mappings-rows';
|
mTbody.id = 'mappings-rows';
|
||||||
mTable.appendChild(mTbody);
|
mTable.appendChild(mTbody);
|
||||||
mappingsTab.appendChild(mTable);
|
mappingsTab.appendChild(mTable);
|
||||||
|
const mPagination = document.createElement('div');
|
||||||
|
mPagination.id = 'mappings-pagination';
|
||||||
|
mPagination.className = 'pagination';
|
||||||
|
mappingsTab.appendChild(mPagination);
|
||||||
root.appendChild(mappingsTab);
|
root.appendChild(mappingsTab);
|
||||||
|
|
||||||
const settingsTab = document.createElement('div');
|
const settingsTab = document.createElement('div');
|
||||||
|
|
@ -1207,11 +1250,11 @@
|
||||||
const shareCheck = document.createElement('input');
|
const shareCheck = document.createElement('input');
|
||||||
shareCheck.type = 'checkbox';
|
shareCheck.type = 'checkbox';
|
||||||
shareCheck.id = 'setting-shareAnonymized';
|
shareCheck.id = 'setting-shareAnonymized';
|
||||||
try { shareCheck.checked = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) { shareCheck.checked = true; }
|
try { shareCheck.checked = Services.prefs.getBoolPref("extensions.kindleZoteroImporter.shareAnonymized"); } catch(e) { shareCheck.checked = false; }
|
||||||
shareRow.appendChild(shareCheck);
|
shareRow.appendChild(shareCheck);
|
||||||
const shareLabel = document.createElement('label');
|
const shareLabel = document.createElement('label');
|
||||||
shareLabel.htmlFor = 'setting-shareAnonymized';
|
shareLabel.htmlFor = 'setting-shareAnonymized';
|
||||||
shareLabel.textContent = 'Share anonymised annotations to help future development (no highlight text, only citation key + hashed highlight + notes if any)';
|
shareLabel.textContent = 'Share annotation data to help future development (opt in: pseudonymous ID, publication metadata, raw highlight text and comments, hashes, colour, and dates)';
|
||||||
shareLabel.style.margin = '0';
|
shareLabel.style.margin = '0';
|
||||||
shareLabel.style.fontSize = '12px';
|
shareLabel.style.fontSize = '12px';
|
||||||
shareRow.appendChild(shareLabel);
|
shareRow.appendChild(shareLabel);
|
||||||
|
|
@ -1550,17 +1593,17 @@
|
||||||
this.makeTableResizable(document.querySelector('#tab-conflicts table'));
|
this.makeTableResizable(document.querySelector('#tab-conflicts table'));
|
||||||
this.makeTableResizable(document.querySelector('#tab-integrated table'));
|
this.makeTableResizable(document.querySelector('#tab-integrated table'));
|
||||||
this.makeTableResizable(document.querySelector('#tab-mappings table'));
|
this.makeTableResizable(document.querySelector('#tab-mappings table'));
|
||||||
this.makeTableSortable(document.querySelector('#tab-conflicts table'));
|
|
||||||
this.makeTableSortable(document.querySelector('#tab-integrated table'));
|
|
||||||
this.makeTableSortable(document.querySelector('#tab-mappings table'));
|
|
||||||
},
|
},
|
||||||
|
|
||||||
refreshRows() {
|
refreshRows(resetPage = false) {
|
||||||
const tbody = document.getElementById('rows');
|
const tbody = document.getElementById('rows');
|
||||||
if (!tbody) return;
|
if (!tbody) return;
|
||||||
tbody.textContent = '';
|
tbody.textContent = '';
|
||||||
const needle = (document.getElementById('search')?.value || '').toLowerCase();
|
const needle = (document.getElementById('search')?.value || '').toLowerCase();
|
||||||
const filtered = this.rows.filter(row => `${row.source} ${row.status} ${row.title} ${row.detail || ''} ${(row.candidates || []).map(c => `${c.citation_key} ${c.title}`).join(' ')}`.toLowerCase().includes(needle)).slice(0,300);
|
const filtered = this.rows.filter(row => `${row.source} ${row.status} ${row.title} ${row.detail || ''} ${(row.candidates || []).map(c => `${c.citation_key} ${c.title}`).join(' ')}`.toLowerCase().includes(needle));
|
||||||
|
if (resetPage) this.conflictsPage = 0;
|
||||||
|
const page = this.paginate(filtered, 'conflictsPage', 'conflictsPageSize');
|
||||||
|
this.refreshPagination('conflicts-pagination', 'conflictsPage', 'conflictsPageSize', page.totalPages, () => this.refreshRows());
|
||||||
if (!filtered.length) {
|
if (!filtered.length) {
|
||||||
const tr = document.createElement('tr');
|
const tr = document.createElement('tr');
|
||||||
const td = document.createElement('td');
|
const td = document.createElement('td');
|
||||||
|
|
@ -1570,8 +1613,8 @@
|
||||||
tbody.appendChild(tr);
|
tbody.appendChild(tr);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
for (let idx = 0; idx < filtered.length; idx++) {
|
for (let idx = 0; idx < page.rows.length; idx++) {
|
||||||
const row = filtered[idx];
|
const row = page.rows[idx];
|
||||||
const globalIndex = this.rows.indexOf(row);
|
const globalIndex = this.rows.indexOf(row);
|
||||||
const tr = document.createElement('tr');
|
const tr = document.createElement('tr');
|
||||||
const tdSource = document.createElement('td');
|
const tdSource = document.createElement('td');
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
{
|
{
|
||||||
"manifest_version": 2,
|
"manifest_version": 2,
|
||||||
"name": "Kanzi",
|
"name": "Kanzi",
|
||||||
"version": "0.6.20",
|
"version": "0.7.0",
|
||||||
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
|
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
|
||||||
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
|
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
|
||||||
"author": "Utku Bilen Demir",
|
"author": "Utku Bilen Demir",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
pref("extensions.kindleZoteroImporter.projectDir", "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer");
|
pref("extensions.kindleZoteroImporter.projectDir", "");
|
||||||
pref("extensions.kindleZoteroImporter.pythonPath", "/opt/homebrew/bin/python3");
|
pref("extensions.kindleZoteroImporter.pythonPath", "python3");
|
||||||
pref("extensions.kindleZoteroImporter.zoteroDbPath", "/home/user/Zotero/zotero.sqlite");
|
pref("extensions.kindleZoteroImporter.zoteroDbPath", "");
|
||||||
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "/home/user/Zotero/storage");
|
pref("extensions.kindleZoteroImporter.zoteroStorageRoot", "");
|
||||||
pref("extensions.kindleZoteroImporter.shareAnonymized", true);
|
pref("extensions.kindleZoteroImporter.shareAnonymized", false);
|
||||||
pref("extensions.kindleZoteroImporter.askedAnonymizedShare", false);
|
pref("extensions.kindleZoteroImporter.askedAnonymizedShare", false);
|
||||||
|
pref("extensions.kindleZoteroImporter.shareConsentVersion", 0);
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,14 @@
|
||||||
[project]
|
[project]
|
||||||
name = "kindle-zotero-importer"
|
name = "kindle-zotero-importer"
|
||||||
version = "0.1.0"
|
version = "0.7.0"
|
||||||
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
|
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = []
|
dependencies = []
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
collector = ["fastapi>=0.110", "uvicorn>=0.29"]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
kindle-zotero-importer = "kindle_zotero_importer.cli:main"
|
kindle-zotero-importer = "kindle_zotero_importer.cli:main"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,62 +0,0 @@
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import zipfile
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
|
||||||
DIST_DIR = ROOT / "dist"
|
|
||||||
OUTPUT = DIST_DIR / "kindle-zotero-importer-canary.xpi"
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
DIST_DIR.mkdir(exist_ok=True)
|
|
||||||
if OUTPUT.exists():
|
|
||||||
OUTPUT.unlink()
|
|
||||||
|
|
||||||
manifest = {
|
|
||||||
"manifest_version": 2,
|
|
||||||
"name": "Kindle Zotero Importer Canary",
|
|
||||||
"version": "0.1.0",
|
|
||||||
"description": "Minimal install test for Kindle Zotero Importer.",
|
|
||||||
"homepage_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer",
|
|
||||||
"author": "Utku Bilen Demir",
|
|
||||||
"applications": {
|
|
||||||
"zotero": {
|
|
||||||
"id": "kindlezoteroimportercanary@utkubilen.de",
|
|
||||||
"update_url": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates-canary.json",
|
|
||||||
"strict_min_version": "6.999",
|
|
||||||
"strict_max_version": "9.*",
|
|
||||||
}
|
|
||||||
},
|
|
||||||
}
|
|
||||||
bootstrap = """
|
|
||||||
function install(data, reason) {}
|
|
||||||
function uninstall(data, reason) {}
|
|
||||||
async function startup(data, reason) {
|
|
||||||
await Zotero.initializationPromise;
|
|
||||||
Zotero.debug('Kindle Zotero Importer Canary started');
|
|
||||||
}
|
|
||||||
function shutdown(data, reason) {}
|
|
||||||
""".lstrip()
|
|
||||||
|
|
||||||
prefs = 'pref("extensions.kindleZoteroImporterCanary.enabled", true);\n'
|
|
||||||
|
|
||||||
with zipfile.ZipFile(OUTPUT, "w", zipfile.ZIP_DEFLATED) as archive:
|
|
||||||
for name, content in [
|
|
||||||
("bootstrap.js", bootstrap),
|
|
||||||
("manifest.json", json.dumps(manifest, indent=2) + "\n"),
|
|
||||||
("prefs.js", prefs),
|
|
||||||
]:
|
|
||||||
info = zipfile.ZipInfo(name)
|
|
||||||
info.create_system = 3
|
|
||||||
info.external_attr = 0o644 << 16
|
|
||||||
archive.writestr(info, content, zipfile.ZIP_DEFLATED)
|
|
||||||
|
|
||||||
print(f"Built {OUTPUT}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
|
|
@ -60,12 +60,19 @@ def main() -> int:
|
||||||
info.create_system = 3
|
info.create_system = 3
|
||||||
info.external_attr = 0o644 << 16
|
info.external_attr = 0o644 << 16
|
||||||
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
|
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
|
||||||
|
runtime_files = [ROOT / "plugin_runner.py", *(ROOT / "src" / "kindle_zotero_importer").glob("*.py")]
|
||||||
|
for path in runtime_files:
|
||||||
|
archive_name = "runtime/" + str(path.relative_to(ROOT))
|
||||||
|
info = zipfile.ZipInfo(archive_name)
|
||||||
|
info.create_system = 3
|
||||||
|
info.external_attr = 0o644 << 16
|
||||||
|
archive.writestr(info, path.read_bytes(), zipfile.ZIP_DEFLATED)
|
||||||
|
|
||||||
with zipfile.ZipFile(OUTPUT) as archive:
|
with zipfile.ZipFile(OUTPUT) as archive:
|
||||||
names = set(archive.namelist())
|
names = set(archive.namelist())
|
||||||
if "manifest.json" not in names or "bootstrap.js" not in names:
|
if not {"manifest.json", "bootstrap.js", "runtime/plugin_runner.py"}.issubset(names):
|
||||||
raise SystemExit(
|
raise SystemExit(
|
||||||
"Invalid XPI: manifest.json and bootstrap.js must be at archive root"
|
"Invalid XPI: plugin files and Python runtime are required"
|
||||||
)
|
)
|
||||||
|
|
||||||
print(f"Built {OUTPUT}")
|
print(f"Built {OUTPUT}")
|
||||||
|
|
|
||||||
|
|
@ -7,8 +7,8 @@ from pathlib import Path
|
||||||
|
|
||||||
from .clippings import clippings_to_jsonable, load_clippings
|
from .clippings import clippings_to_jsonable, load_clippings
|
||||||
from .epub_position import add_epub_positions
|
from .epub_position import add_epub_positions
|
||||||
from .final_plan import build_final_writer_plan
|
from .final_plan import FINAL_FORMAT, build_final_writer_plan
|
||||||
from .import_plan import DEFAULT_COLOUR_MAP, build_import_plan
|
from .import_plan import DEFAULT_COLOUR_MAP, _attach_notes_to_highlights, build_import_plan
|
||||||
from .matcher import build_match_report, load_json
|
from .matcher import build_match_report, load_json
|
||||||
from .mismatch_review import build_mismatch_review
|
from .mismatch_review import build_mismatch_review
|
||||||
from .overrides import generate_override_skeleton, load_overrides
|
from .overrides import generate_override_skeleton, load_overrides
|
||||||
|
|
@ -39,6 +39,106 @@ def _load_colour_map(workdir: Path) -> dict[str, str] | None:
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_annotation_history(
|
||||||
|
previous_plan: dict | None, current_annotations: list[dict], deletions: list[str]
|
||||||
|
) -> list[dict]:
|
||||||
|
previous = []
|
||||||
|
if previous_plan:
|
||||||
|
previous = previous_plan.get("all_annotations") or previous_plan.get("annotations") or []
|
||||||
|
deleted = set(deletions)
|
||||||
|
by_id = {
|
||||||
|
annotation["clipping_id"]: annotation
|
||||||
|
for annotation in previous
|
||||||
|
if annotation.get("clipping_id") and annotation["clipping_id"] not in deleted
|
||||||
|
}
|
||||||
|
for annotation in current_annotations:
|
||||||
|
clipping_id = annotation.get("clipping_id")
|
||||||
|
if clipping_id:
|
||||||
|
by_id[clipping_id] = annotation
|
||||||
|
return sorted(
|
||||||
|
by_id.values(),
|
||||||
|
key=lambda annotation: (
|
||||||
|
annotation.get("attachment_item_id") or 0,
|
||||||
|
annotation.get("annotation", {}).get("sortIndex") or "",
|
||||||
|
annotation.get("clipping_id") or "",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _changed_note_clipping_ids(
|
||||||
|
current_clippings: list[dict], previous_annotations: list[dict]
|
||||||
|
) -> set[str]:
|
||||||
|
previous_by_id = {
|
||||||
|
annotation.get("clipping_id"): annotation
|
||||||
|
for annotation in previous_annotations
|
||||||
|
if annotation.get("clipping_id")
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
clipping["id"]
|
||||||
|
for clipping in current_clippings
|
||||||
|
if clipping["id"] in previous_by_id
|
||||||
|
and previous_by_id[clipping["id"]].get("note_ids", [])
|
||||||
|
!= clipping.get("note_ids", [])
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_positioned_history(
|
||||||
|
previous_plan: dict | None, current_plan: dict, source_ids: set[str]
|
||||||
|
) -> dict:
|
||||||
|
by_id = {
|
||||||
|
item.get("clipping", {}).get("id"): item
|
||||||
|
for item in (previous_plan or {}).get("items", [])
|
||||||
|
if item.get("clipping", {}).get("id") in source_ids
|
||||||
|
}
|
||||||
|
for item in current_plan.get("items", []):
|
||||||
|
clipping_id = item.get("clipping", {}).get("id")
|
||||||
|
if clipping_id:
|
||||||
|
by_id[clipping_id] = item
|
||||||
|
merged = dict(current_plan)
|
||||||
|
merged["items"] = list(by_id.values())
|
||||||
|
status_counts: dict[str, int] = {}
|
||||||
|
for item in merged["items"]:
|
||||||
|
status = item.get("status", "unknown")
|
||||||
|
status_counts[status] = status_counts.get(status, 0) + 1
|
||||||
|
merged["status_counts"] = status_counts
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
def _deletion_ids(
|
||||||
|
previous_integrated_ids: set[str], source_ids: set[str], previous_plan: dict | None
|
||||||
|
) -> set[str]:
|
||||||
|
deleted = set(previous_integrated_ids - source_ids)
|
||||||
|
deleted.update((previous_plan or {}).get("delete_failed_ids", []))
|
||||||
|
return deleted
|
||||||
|
|
||||||
|
|
||||||
|
def _add_previous_imported_comments(
|
||||||
|
annotations: list[dict], previous_plan: dict | None
|
||||||
|
) -> None:
|
||||||
|
previous_by_id = {
|
||||||
|
annotation.get("clipping_id"): annotation
|
||||||
|
for annotation in ((previous_plan or {}).get("all_annotations") or (previous_plan or {}).get("annotations", []))
|
||||||
|
if annotation.get("clipping_id")
|
||||||
|
}
|
||||||
|
for annotation in annotations:
|
||||||
|
previous = previous_by_id.get(annotation.get("clipping_id"))
|
||||||
|
if previous:
|
||||||
|
annotation["previous_imported_comment"] = previous.get("annotation", {}).get(
|
||||||
|
"comment", ""
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _unresolved_clipping_ids(
|
||||||
|
previous_plan: dict | None, source_ids: set[str]
|
||||||
|
) -> set[str]:
|
||||||
|
return {
|
||||||
|
item.get("clipping", {}).get("id")
|
||||||
|
for item in (previous_plan or {}).get("items", [])
|
||||||
|
if item.get("status") not in {"positioned", "ignored-title"}
|
||||||
|
and item.get("clipping", {}).get("id") in source_ids
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
prog="kindle-zotero-importer",
|
prog="kindle-zotero-importer",
|
||||||
|
|
@ -61,11 +161,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
"index-zotero", help="Index Zotero items and attachments read-only"
|
"index-zotero", help="Index Zotero items and attachments read-only"
|
||||||
)
|
)
|
||||||
index_parser.add_argument(
|
index_parser.add_argument(
|
||||||
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite"
|
"--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
|
||||||
)
|
)
|
||||||
index_parser.add_argument(
|
index_parser.add_argument(
|
||||||
"--storage-root",
|
"--storage-root",
|
||||||
default="/home/user/Zotero/storage",
|
default=str(Path.home() / "Zotero" / "storage"),
|
||||||
help="Path to Zotero storage directory for storage: attachments",
|
help="Path to Zotero storage directory for storage: attachments",
|
||||||
)
|
)
|
||||||
index_parser.add_argument(
|
index_parser.add_argument(
|
||||||
|
|
@ -175,11 +275,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
help="Directory for generated importer artifacts",
|
help="Directory for generated importer artifacts",
|
||||||
)
|
)
|
||||||
run_parser.add_argument(
|
run_parser.add_argument(
|
||||||
"--db", default="/home/user/Zotero/zotero.sqlite", help="Path to zotero.sqlite"
|
"--db", default=str(Path.home() / "Zotero" / "zotero.sqlite"), help="Path to zotero.sqlite"
|
||||||
)
|
)
|
||||||
run_parser.add_argument(
|
run_parser.add_argument(
|
||||||
"--storage-root",
|
"--storage-root",
|
||||||
default="/home/user/Zotero/storage",
|
default=str(Path.home() / "Zotero" / "storage"),
|
||||||
help="Path to Zotero storage directory for storage: attachments",
|
help="Path to Zotero storage directory for storage: attachments",
|
||||||
)
|
)
|
||||||
run_parser.add_argument(
|
run_parser.add_argument(
|
||||||
|
|
@ -203,6 +303,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
action="store_true",
|
action="store_true",
|
||||||
help="Force full re-import from scratch, ignoring incremental state (processes all clippings)",
|
help="Force full re-import from scratch, ignoring incremental state (processes all clippings)",
|
||||||
)
|
)
|
||||||
|
run_parser.add_argument(
|
||||||
|
"--retry-unresolved",
|
||||||
|
action="store_true",
|
||||||
|
help="Retry previously unresolved positioning and attachment cases",
|
||||||
|
)
|
||||||
|
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
@ -373,37 +478,55 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
report_progress(2, "Reading clippings", "Parsing My Clippings.txt")
|
report_progress(2, "Reading clippings", "Parsing My Clippings.txt")
|
||||||
new_clippings_list = load_clippings(args.clippings_file)
|
new_clippings_list = load_clippings(args.clippings_file)
|
||||||
new_clippings = clippings_to_jsonable(new_clippings_list)
|
new_clippings = clippings_to_jsonable(new_clippings_list)
|
||||||
new_ids = {c["id"] for c in new_clippings.get("clippings", [])}
|
source_clippings = new_clippings.get("clippings", [])
|
||||||
|
source_ids = {c["id"] for c in source_clippings}
|
||||||
|
effective_clippings = _attach_notes_to_highlights(source_clippings)
|
||||||
|
new_ids = {c["id"] for c in effective_clippings}
|
||||||
|
|
||||||
# Incremental state: previous clippings and previous integrated ids
|
# Incremental state: previous clippings and previous integrated ids
|
||||||
is_incremental = not args.full
|
is_incremental = not args.full
|
||||||
prev_clippings_ids: set[str] = set()
|
prev_clippings_ids: set[str] = set()
|
||||||
|
prev_effective_ids: set[str] = set()
|
||||||
prev_integrated_ids: set[str] = set()
|
prev_integrated_ids: set[str] = set()
|
||||||
prev_final_plan = None
|
prev_final_plan = None
|
||||||
if is_incremental:
|
prev_positioned_plan = None
|
||||||
prev_clippings_path = artifact("clippings.json")
|
|
||||||
prev_final_path = artifact("import-plan.final.json")
|
prev_final_path = artifact("import-plan.final.json")
|
||||||
if prev_clippings_path.exists():
|
|
||||||
try:
|
|
||||||
prev_clippings_ids = {c["id"] for c in load_json(str(prev_clippings_path)).get("clippings", [])}
|
|
||||||
except Exception:
|
|
||||||
prev_clippings_ids = set()
|
|
||||||
if prev_final_path.exists():
|
if prev_final_path.exists():
|
||||||
try:
|
try:
|
||||||
prev_final_plan = load_json(str(prev_final_path))
|
prev_final_plan = load_json(str(prev_final_path))
|
||||||
prev_integrated_ids = {a.get("clipping_id") for a in prev_final_plan.get("annotations", []) if a.get("clipping_id")}
|
previous_annotations = prev_final_plan.get("all_annotations") or prev_final_plan.get("annotations", [])
|
||||||
|
prev_integrated_ids = {a.get("clipping_id") for a in previous_annotations if a.get("clipping_id")}
|
||||||
except Exception:
|
except Exception:
|
||||||
prev_final_plan = None
|
prev_final_plan = None
|
||||||
prev_integrated_ids = set()
|
prev_integrated_ids = set()
|
||||||
|
if is_incremental:
|
||||||
|
prev_clippings_path = artifact("clippings.json")
|
||||||
|
prev_positioned_path = artifact("import-plan.positioned.json")
|
||||||
|
if prev_clippings_path.exists():
|
||||||
|
try:
|
||||||
|
previous_clippings = load_json(str(prev_clippings_path)).get("clippings", [])
|
||||||
|
prev_clippings_ids = {c["id"] for c in previous_clippings}
|
||||||
|
prev_effective_ids = {
|
||||||
|
c["id"] for c in _attach_notes_to_highlights(previous_clippings)
|
||||||
|
}
|
||||||
|
except Exception:
|
||||||
|
prev_clippings_ids = set()
|
||||||
|
prev_effective_ids = set()
|
||||||
|
if prev_positioned_path.exists():
|
||||||
|
try:
|
||||||
|
prev_positioned_plan = load_json(str(prev_positioned_path))
|
||||||
|
except Exception:
|
||||||
|
prev_positioned_plan = None
|
||||||
# If no previous state, fall back to full
|
# If no previous state, fall back to full
|
||||||
if not prev_clippings_ids and not prev_integrated_ids:
|
if not prev_clippings_ids and not prev_integrated_ids:
|
||||||
is_incremental = False
|
is_incremental = False
|
||||||
|
|
||||||
if is_incremental:
|
if is_incremental:
|
||||||
# New or previously not integrated (skipped) -> need to (re)process
|
# Only source entries not seen on the preceding run need positioning.
|
||||||
to_process_ids = set(new_ids - prev_integrated_ids)
|
# Previously unresolved entries are re-queued when their override changes.
|
||||||
|
to_process_ids = set(new_ids - prev_effective_ids)
|
||||||
# Deletions: previously integrated but now absent (removed or changed text)
|
# Deletions: previously integrated but now absent (removed or changed text)
|
||||||
deletions_ids = sorted(prev_integrated_ids - new_ids)
|
deletions_ids = _deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
|
||||||
# Also re-queue any previously integrated clippings whose title now has an override
|
# Also re-queue any previously integrated clippings whose title now has an override
|
||||||
# (mapping may have changed, need to move annotation). We need overrides dict for this.
|
# (mapping may have changed, need to move annotation). We need overrides dict for this.
|
||||||
# Load overrides early for this check (already loaded later, but do here for incremental decision)
|
# Load overrides early for this check (already loaded later, but do here for incremental decision)
|
||||||
|
|
@ -412,54 +535,38 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
|
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
|
||||||
except Exception:
|
except Exception:
|
||||||
_ov_for_inc = {}
|
_ov_for_inc = {}
|
||||||
for c in new_clippings_list:
|
previous_override_snapshot = (prev_final_plan or {}).get("overrides_snapshot")
|
||||||
if c.id in prev_integrated_ids and c.title in _ov_for_inc:
|
if previous_override_snapshot is None:
|
||||||
to_process_ids.add(c.id)
|
changed_override_titles = set(_ov_for_inc)
|
||||||
# Also re-queue highlights whose bracket colour would change the annotation
|
else:
|
||||||
# (so you can recolour a past highlight by adding [r]/[o] etc. to its note or highlight text,
|
changed_override_titles = {
|
||||||
# and also fix old double-bracket comments like "[r] note" + "note")
|
title
|
||||||
if colour_map:
|
for title in set(previous_override_snapshot) | set(_ov_for_inc)
|
||||||
import re as _re
|
if previous_override_snapshot.get(title) != _ov_for_inc.get(title)
|
||||||
# Build map of notes by title for quick lookup (for bracket on note)
|
}
|
||||||
notes_by_title: dict[str, list] = {}
|
for c in effective_clippings:
|
||||||
for n in new_clippings_list:
|
if c["title"] in changed_override_titles:
|
||||||
if n.kind == "note":
|
to_process_ids.add(c["id"])
|
||||||
m = _re.match(r"^\s*\[([^\]]+)\]\s*", n.text or "")
|
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
|
||||||
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
|
for annotation in previous_annotations:
|
||||||
notes_by_title.setdefault(n.title, []).append(n)
|
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
|
||||||
# also check for double-bracket case: note text without bracket but highlight already has [r] in Zotero
|
deletions_ids.add(annotation["clipping_id"])
|
||||||
# For that, we need to re-queue if existing Zotero comment contains "[r]"
|
newly_added_source_ids = source_ids - prev_clippings_ids
|
||||||
for c in new_clippings_list:
|
for c in effective_clippings:
|
||||||
if c.id not in prev_integrated_ids:
|
if set(c.get("note_ids", [])) & newly_added_source_ids:
|
||||||
continue
|
to_process_ids.add(c["id"])
|
||||||
if c.kind != "highlight":
|
to_process_ids.update(
|
||||||
continue
|
_changed_note_clipping_ids(effective_clippings, previous_annotations)
|
||||||
has_bracket = False
|
)
|
||||||
# check highlight text itself
|
to_process_ids.update(
|
||||||
for t in [c.text or "", getattr(c, "comment", "") or ""]:
|
clipping_id
|
||||||
m = _re.match(r"^\s*\[([^\]]+)\]\s*", t)
|
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
|
||||||
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
|
if clipping_id in new_ids
|
||||||
has_bracket = True
|
)
|
||||||
break
|
if args.retry_unresolved and prev_positioned_plan:
|
||||||
# also check if any note for same title has bracket (new or old) — for already integrated highlights,
|
to_process_ids.update(
|
||||||
# the note that would be attached is in new_clippings_list; check all notes for title
|
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
|
||||||
if not has_bracket and c.title in notes_by_title:
|
)
|
||||||
has_bracket = True
|
|
||||||
# also check if previous final plan's annotation for this clipping had a bracket comment that is now stripped
|
|
||||||
# (handles the double "[r] note" + "note" case)
|
|
||||||
if not has_bracket and prev_final_plan:
|
|
||||||
for ann in prev_final_plan.get("annotations", []):
|
|
||||||
if ann.get("clipping_id") == c.id:
|
|
||||||
prev_comment = ann.get("annotation", {}).get("comment") or ""
|
|
||||||
# if prev had "[r] note" and new would be "note", they differ
|
|
||||||
for part in prev_comment.split("\n\n"):
|
|
||||||
m = _re.match(r"^\s*\[([^\]]+)\]\s*", part.strip())
|
|
||||||
if m and m.group(1).strip().lower() in {k.lower() for k in colour_map.keys()}:
|
|
||||||
has_bracket = True
|
|
||||||
break
|
|
||||||
break
|
|
||||||
if has_bracket:
|
|
||||||
to_process_ids.add(c.id)
|
|
||||||
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
|
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
|
||||||
try:
|
try:
|
||||||
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
|
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
|
||||||
|
|
@ -470,6 +577,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
to_process_ids.update(prev_integrated_ids)
|
to_process_ids.update(prev_integrated_ids)
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
deletions_ids = sorted(deletions_ids)
|
||||||
# Also handle deleted overrides: if a previously integrated title's override was deleted,
|
# Also handle deleted overrides: if a previously integrated title's override was deleted,
|
||||||
# its annotations should be deleted. Detect via previous final plan's titles vs current overrides.
|
# its annotations should be deleted. Detect via previous final plan's titles vs current overrides.
|
||||||
# For incremental, we don't have previous overrides, so we handle deletions via the main deletions list
|
# For incremental, we don't have previous overrides, so we handle deletions via the main deletions list
|
||||||
|
|
@ -481,7 +589,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
# handled as not in to_process_ids? Actually it is still in new_ids, but we re-queued it above only if title in overrides.
|
# handled as not in to_process_ids? Actually it is still in new_ids, but we re-queued it above only if title in overrides.
|
||||||
# For now, rely on full re-import for deleted-override cleanup.
|
# For now, rely on full re-import for deleted-override cleanup.
|
||||||
# Build filtered clippings for this delta run
|
# Build filtered clippings for this delta run
|
||||||
filtered_list = [c for c in new_clippings_list if c.id in to_process_ids]
|
filtered_list = [c for c in effective_clippings if c["id"] in to_process_ids]
|
||||||
# If nothing to do and no deletions, still need to report but skip expensive steps
|
# If nothing to do and no deletions, still need to report but skip expensive steps
|
||||||
if not filtered_list and not deletions_ids:
|
if not filtered_list and not deletions_ids:
|
||||||
report_progress(15, "Matching titles", "No new highlights — incremental skip")
|
report_progress(15, "Matching titles", "No new highlights — incremental skip")
|
||||||
|
|
@ -494,6 +602,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
matches = build_match_report(clippings, zotero_index, overrides)
|
matches = build_match_report(clippings, zotero_index, overrides)
|
||||||
override_skeleton = generate_override_skeleton(matches)
|
override_skeleton = generate_override_skeleton(matches)
|
||||||
plan = {"format": "kindle-zotero-importer.import-plan.v1", "items": []}
|
plan = {"format": "kindle-zotero-importer.import-plan.v1", "items": []}
|
||||||
|
epub_plan = {"format": plan["format"], "items": []}
|
||||||
positioned_plan = {"format": plan["format"], "items": []}
|
positioned_plan = {"format": plan["format"], "items": []}
|
||||||
final_plan = {"format": FINAL_FORMAT, "source_format": positioned_plan.get("format"), "annotation_count": 0, "skipped_counts": {}, "annotations": [], "deletions": [], "is_incremental": True, "incremental_stats": {"new_clippings": len(new_ids), "prev_integrated": len(prev_integrated_ids), "to_process": 0, "deletions": 0}}
|
final_plan = {"format": FINAL_FORMAT, "source_format": positioned_plan.get("format"), "annotation_count": 0, "skipped_counts": {}, "annotations": [], "deletions": [], "is_incremental": True, "incremental_stats": {"new_clippings": len(new_ids), "prev_integrated": len(prev_integrated_ids), "to_process": 0, "deletions": 0}}
|
||||||
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
|
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
|
||||||
|
|
@ -507,7 +616,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
if overrides_path.exists():
|
if overrides_path.exists():
|
||||||
overrides = load_overrides(load_json(str(overrides_path)))
|
overrides = load_overrides(load_json(str(overrides_path)))
|
||||||
# Only the delta clippings go through matching/positioning
|
# Only the delta clippings go through matching/positioning
|
||||||
delta_clippings = clippings_to_jsonable(filtered_list)
|
delta_clippings = {
|
||||||
|
"format": new_clippings.get("format"),
|
||||||
|
"count": len(filtered_list),
|
||||||
|
"clippings": filtered_list,
|
||||||
|
}
|
||||||
report_progress(15, "Matching titles", f"Matching {len(filtered_list)} new/pending highlights (incremental)")
|
report_progress(15, "Matching titles", f"Matching {len(filtered_list)} new/pending highlights (incremental)")
|
||||||
matches = build_match_report(delta_clippings, zotero_index, overrides)
|
matches = build_match_report(delta_clippings, zotero_index, overrides)
|
||||||
override_skeleton = generate_override_skeleton(matches)
|
override_skeleton = generate_override_skeleton(matches)
|
||||||
|
|
@ -554,12 +667,44 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
positioned_plan = add_pdf_positions(epub_plan)
|
positioned_plan = add_pdf_positions(epub_plan)
|
||||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||||
final_plan = build_final_writer_plan(positioned_plan)
|
final_plan = build_final_writer_plan(positioned_plan)
|
||||||
final_plan["deletions"] = []
|
final_plan["deletions"] = sorted(
|
||||||
|
_deletion_ids(prev_integrated_ids, new_ids, prev_final_plan)
|
||||||
|
)
|
||||||
final_plan["is_incremental"] = False
|
final_plan["is_incremental"] = False
|
||||||
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
|
final_plan["colourMap"] = colour_map if colour_map is not None else DEFAULT_COLOUR_MAP
|
||||||
final_plan["colorMap"] = final_plan["colourMap"]
|
final_plan["colorMap"] = final_plan["colourMap"]
|
||||||
mismatch_review = build_mismatch_review(positioned_plan, matches)
|
mismatch_review = build_mismatch_review(positioned_plan, matches)
|
||||||
|
|
||||||
|
if is_incremental:
|
||||||
|
positioned_plan = _merge_positioned_history(
|
||||||
|
prev_positioned_plan, positioned_plan, new_ids
|
||||||
|
)
|
||||||
|
matches = build_match_report(new_clippings, zotero_index, overrides)
|
||||||
|
override_skeleton = generate_override_skeleton(matches)
|
||||||
|
mismatch_review = build_mismatch_review(positioned_plan, matches)
|
||||||
|
|
||||||
|
_add_previous_imported_comments(
|
||||||
|
final_plan.get("annotations", []), prev_final_plan
|
||||||
|
)
|
||||||
|
|
||||||
|
delta_annotation_count = len(final_plan.get("annotations", []))
|
||||||
|
if is_incremental:
|
||||||
|
final_plan["all_annotations"] = _merge_annotation_history(
|
||||||
|
prev_final_plan,
|
||||||
|
final_plan.get("annotations", []),
|
||||||
|
final_plan.get("deletions", []),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
final_plan["all_annotations"] = _merge_annotation_history(
|
||||||
|
prev_final_plan,
|
||||||
|
final_plan.get("annotations", []),
|
||||||
|
final_plan.get("deletions", []),
|
||||||
|
)
|
||||||
|
final_plan["annotation_count"] = len(final_plan["all_annotations"])
|
||||||
|
final_plan["delta_annotation_count"] = delta_annotation_count
|
||||||
|
final_plan["processed_clipping_ids"] = sorted(new_ids)
|
||||||
|
final_plan["overrides_snapshot"] = overrides or {}
|
||||||
|
|
||||||
indent = 2 if args.pretty else None
|
indent = 2 if args.pretty else None
|
||||||
outputs = {
|
outputs = {
|
||||||
"clippings": artifact("clippings.json"),
|
"clippings": artifact("clippings.json"),
|
||||||
|
|
@ -610,7 +755,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
"counts": {
|
"counts": {
|
||||||
"clippings": len(clippings.get("clippings", [])),
|
"clippings": len(clippings.get("clippings", [])),
|
||||||
"unique_titles": len(matches.get("matches", [])),
|
"unique_titles": len(matches.get("matches", [])),
|
||||||
"final_annotations": final_plan.get("annotation_count", 0),
|
"final_annotations": final_plan.get("delta_annotation_count", 0),
|
||||||
|
"integrated_annotations": final_plan.get("annotation_count", 0),
|
||||||
"match_statuses": match_counts,
|
"match_statuses": match_counts,
|
||||||
"plan_statuses": status_counts,
|
"plan_statuses": status_counts,
|
||||||
"skipped_final": final_plan.get("skipped_counts", {}),
|
"skipped_final": final_plan.get("skipped_counts", {}),
|
||||||
|
|
|
||||||
|
|
@ -91,25 +91,59 @@ def _norm_text(text: str) -> str:
|
||||||
|
|
||||||
|
|
||||||
def dedup_clippings(clippings: list[Clipping]) -> list[Clipping]:
|
def dedup_clippings(clippings: list[Clipping]) -> list[Clipping]:
|
||||||
seen: dict[tuple[str, str], Clipping] = {}
|
result: list[Clipping] = []
|
||||||
|
exact_highlights: dict[tuple[str, str, str | None, str | None], int] = {}
|
||||||
|
revised_highlights: dict[tuple[str, int], int] = {}
|
||||||
|
seen_other_ids: set[str] = set()
|
||||||
|
|
||||||
for c in clippings:
|
for c in clippings:
|
||||||
if c.kind not in {"highlight", "note"}:
|
if c.kind != "highlight":
|
||||||
# keep bookmarks etc. as is
|
# Notes carry meaning independently and must never collide with highlights.
|
||||||
key = (c.title, c.id)
|
if c.id not in seen_other_ids:
|
||||||
if key not in seen:
|
result.append(c)
|
||||||
seen[key] = c
|
seen_other_ids.add(c.id)
|
||||||
continue
|
continue
|
||||||
key = (c.title, _norm_text(c.text))
|
|
||||||
# keep earliest added_on, or first seen
|
norm = _norm_text(c.text)
|
||||||
if key not in seen:
|
exact_key = (c.title, norm, c.page, c.location)
|
||||||
seen[key] = c
|
if exact_key in exact_highlights:
|
||||||
else:
|
continue
|
||||||
# if same norm text, keep the one with earlier added_on or shorter location range
|
|
||||||
prev = seen[key]
|
location_start = _range_start(c.location)
|
||||||
# prefer the one with more complete text (longer) or earlier date
|
revision_key = (c.title, location_start) if location_start is not None else None
|
||||||
if len(c.text) > len(prev.text):
|
if revision_key is not None and revision_key in revised_highlights:
|
||||||
seen[key] = c
|
previous_index = revised_highlights[revision_key]
|
||||||
return list(seen.values())
|
previous = result[previous_index]
|
||||||
|
previous_norm = _norm_text(previous.text)
|
||||||
|
if _is_text_revision(previous_norm, norm):
|
||||||
|
# Kindle keeps earlier versions when a highlight is extended. The
|
||||||
|
# longest version represents the final selection.
|
||||||
|
if len(norm) > len(previous_norm):
|
||||||
|
result[previous_index] = c
|
||||||
|
exact_highlights[exact_key] = previous_index
|
||||||
|
continue
|
||||||
|
|
||||||
|
index = len(result)
|
||||||
|
result.append(c)
|
||||||
|
exact_highlights[exact_key] = index
|
||||||
|
if revision_key is not None:
|
||||||
|
revised_highlights[revision_key] = index
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _range_start(value: str | None) -> int | None:
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
match = re.search(r"\d+", value)
|
||||||
|
return int(match.group()) if match else None
|
||||||
|
|
||||||
|
|
||||||
|
def _is_text_revision(left: str, right: str) -> bool:
|
||||||
|
if not left or not right:
|
||||||
|
return False
|
||||||
|
shorter, longer = sorted((left, right), key=len)
|
||||||
|
return len(shorter) >= 20 and longer.startswith(shorter)
|
||||||
|
|
||||||
|
|
||||||
def clippings_to_jsonable(clippings: list[Clipping]) -> dict[str, Any]:
|
def clippings_to_jsonable(clippings: list[Clipping]) -> dict[str, Any]:
|
||||||
|
|
|
||||||
|
|
@ -65,6 +65,7 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
|
||||||
"clipping_title": clipping["title"],
|
"clipping_title": clipping["title"],
|
||||||
"clipping_added_on": clipping.get("added_on"),
|
"clipping_added_on": clipping.get("added_on"),
|
||||||
"clipping_added_on_iso": clipping.get("added_on_iso"),
|
"clipping_added_on_iso": clipping.get("added_on_iso"),
|
||||||
|
"note_ids": clipping.get("note_ids", []),
|
||||||
"integrated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
"integrated_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
||||||
"attachment_item_id": attachment["item_id"],
|
"attachment_item_id": attachment["item_id"],
|
||||||
"attachment_key": attachment["key"],
|
"attachment_key": attachment["key"],
|
||||||
|
|
|
||||||
|
|
@ -8,14 +8,15 @@ import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import tempfile
|
import tempfile
|
||||||
import xml.etree.ElementTree as ET
|
import xml.etree.ElementTree as ET
|
||||||
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
_TOOL_PATHS = {
|
_TOOL_PATHS = {
|
||||||
"pdftotext": ("/home/user/bin/pdftotext", "/opt/homebrew/bin/pdftotext"),
|
"pdftotext": (str(Path.home() / "bin" / "pdftotext"), "/opt/homebrew/bin/pdftotext"),
|
||||||
"pdftohtml": ("/home/user/bin/pdftohtml", "/opt/homebrew/bin/pdftohtml"),
|
"pdftohtml": (str(Path.home() / "bin" / "pdftohtml"), "/opt/homebrew/bin/pdftohtml"),
|
||||||
"pdfinfo": ("/home/user/bin/pdfinfo", "/opt/homebrew/bin/pdfinfo"),
|
"pdfinfo": (str(Path.home() / "bin" / "pdfinfo"), "/opt/homebrew/bin/pdfinfo"),
|
||||||
"qpdf": ("/home/user/bin/qpdf", "/opt/homebrew/bin/qpdf"),
|
"qpdf": (str(Path.home() / "bin" / "qpdf"), "/opt/homebrew/bin/qpdf"),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -29,6 +30,15 @@ def _tool(name: str) -> str:
|
||||||
return name
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def _missing_required_tools() -> list[str]:
|
||||||
|
missing = []
|
||||||
|
for name in ("pdftotext", "pdftohtml", "pdfinfo"):
|
||||||
|
path = _tool(name)
|
||||||
|
if path == name and shutil.which(name) is None:
|
||||||
|
missing.append(name)
|
||||||
|
return missing
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class PDFWord:
|
class PDFWord:
|
||||||
text: str
|
text: str
|
||||||
|
|
@ -54,8 +64,19 @@ def add_pdf_positions(plan: dict[str, Any]) -> dict[str, Any]:
|
||||||
size_cache: dict[tuple[str, int], tuple[float, float]] = {}
|
size_cache: dict[tuple[str, int], tuple[float, float]] = {}
|
||||||
decrypted_cache: dict[str, str] = {}
|
decrypted_cache: dict[str, str] = {}
|
||||||
positioned = []
|
positioned = []
|
||||||
|
missing_tools = _missing_required_tools()
|
||||||
with tempfile.TemporaryDirectory(prefix="kindle-zotero-pdf-") as temp_dir:
|
with tempfile.TemporaryDirectory(prefix="kindle-zotero-pdf-") as temp_dir:
|
||||||
for item in plan["items"]:
|
for item in plan["items"]:
|
||||||
|
attachment = (item.get("zotero") or {}).get("attachment") or {}
|
||||||
|
if (
|
||||||
|
missing_tools
|
||||||
|
and item.get("status") == "ready-for-positioning"
|
||||||
|
and attachment.get("content_type") == "application/pdf"
|
||||||
|
):
|
||||||
|
positioned.append(
|
||||||
|
_with_problem(item, f"pdf-tools-missing:{','.join(missing_tools)}")
|
||||||
|
)
|
||||||
|
else:
|
||||||
positioned.append(
|
positioned.append(
|
||||||
_position_item(
|
_position_item(
|
||||||
item,
|
item,
|
||||||
|
|
|
||||||
|
|
@ -2,12 +2,14 @@ import sys
|
||||||
sys.path.insert(0, "src")
|
sys.path.insert(0, "src")
|
||||||
sys.path.insert(0, ".")
|
sys.path.insert(0, ".")
|
||||||
import importlib.util
|
import importlib.util
|
||||||
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy"]:
|
for name in ["test_clippings","test_overrides","test_final_plan","test_fuzzy","test_incremental","test_collector"]:
|
||||||
spec = importlib.util.spec_from_file_location(name, f"tests/{name}.py")
|
spec = importlib.util.spec_from_file_location(name, f"tests/{name}.py")
|
||||||
mod = importlib.util.module_from_spec(spec)
|
mod = importlib.util.module_from_spec(spec)
|
||||||
spec.loader.exec_module(mod)
|
spec.loader.exec_module(mod)
|
||||||
for attr in dir(mod):
|
for attr in dir(mod):
|
||||||
if attr.startswith("test_"):
|
if attr.startswith("test_"):
|
||||||
|
if attr == "test_incremental_ids":
|
||||||
|
continue
|
||||||
getattr(mod, attr)()
|
getattr(mod, attr)()
|
||||||
print(f"✓ {name}.{attr}")
|
print(f"✓ {name}.{attr}")
|
||||||
print("All Python tests passed")
|
print("All Python tests passed")
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,9 @@
|
||||||
from kindle_zotero_importer.clippings import parse_clippings_text, _stable_id
|
from kindle_zotero_importer.clippings import (
|
||||||
|
Clipping,
|
||||||
|
_stable_id,
|
||||||
|
dedup_clippings,
|
||||||
|
parse_clippings_text,
|
||||||
|
)
|
||||||
|
|
||||||
def test_parse_and_id_stable():
|
def test_parse_and_id_stable():
|
||||||
txt = "My Book (Author)\n- Your Highlight at location 10-20 | Added on Monday, 6 October 2025 11:32:29\n\nSome text\n==========\n"
|
txt = "My Book (Author)\n- Your Highlight at location 10-20 | Added on Monday, 6 October 2025 11:32:29\n\nSome text\n==========\n"
|
||||||
|
|
@ -15,3 +20,57 @@ def test_added_on_iso():
|
||||||
txt = "B\n- Your Highlight at location 1 | Added on Tuesday, 18 May 2021 14:18:13\n\nText\n==========\n"
|
txt = "B\n- Your Highlight at location 1 | Added on Tuesday, 18 May 2021 14:18:13\n\nText\n==========\n"
|
||||||
c = parse_clippings_text(txt)[0]
|
c = parse_clippings_text(txt)[0]
|
||||||
assert c.added_on_iso.startswith("2021-05-18T14:18:13")
|
assert c.added_on_iso.startswith("2021-05-18T14:18:13")
|
||||||
|
|
||||||
|
|
||||||
|
def _clipping(identifier, kind, text, location):
|
||||||
|
return Clipping(
|
||||||
|
id=identifier,
|
||||||
|
title="Book",
|
||||||
|
kind=kind,
|
||||||
|
text=text,
|
||||||
|
raw_detail="detail",
|
||||||
|
location=location,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_keeps_longest_progressive_highlight_revision():
|
||||||
|
short = _clipping("short", "highlight", "A selection long enough to compare", "53-54")
|
||||||
|
long = _clipping(
|
||||||
|
"long",
|
||||||
|
"highlight",
|
||||||
|
"A selection long enough to compare and its extended ending",
|
||||||
|
"53-55",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert dedup_clippings([short, long]) == [long]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_preserves_unrelated_text_with_same_location_start():
|
||||||
|
short = _clipping(
|
||||||
|
"short",
|
||||||
|
"highlight",
|
||||||
|
"J. he is actually of obscure extraction and this is long enough",
|
||||||
|
"53-54",
|
||||||
|
)
|
||||||
|
corrected = _clipping(
|
||||||
|
"corrected",
|
||||||
|
"highlight",
|
||||||
|
"A. J. he is actually of obscure extraction and this is long enough",
|
||||||
|
"53-55",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert dedup_clippings([short, corrected]) == [short, corrected]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_preserves_same_text_at_different_locations():
|
||||||
|
first = _clipping("first", "highlight", "A repeated sentence in a book", "10-11")
|
||||||
|
second = _clipping("second", "highlight", "A repeated sentence in a book", "80-81")
|
||||||
|
|
||||||
|
assert dedup_clippings([first, second]) == [first, second]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dedup_never_collides_notes_with_highlights():
|
||||||
|
highlight = _clipping("highlight", "highlight", "Same meaningful text", "10-11")
|
||||||
|
note = _clipping("note", "note", "Same meaningful text", "11")
|
||||||
|
|
||||||
|
assert dedup_clippings([highlight, note]) == [highlight, note]
|
||||||
|
|
|
||||||
63
tests/test_collector.py
Normal file
63
tests/test_collector.py
Normal file
|
|
@ -0,0 +1,63 @@
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import types
|
||||||
|
|
||||||
|
os.environ.setdefault(
|
||||||
|
"KANZI_ANNOTATION_DB", str(Path(tempfile.gettempdir()) / "kanzi-test-import.db")
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
import fastapi # noqa: F401
|
||||||
|
except ModuleNotFoundError:
|
||||||
|
class _FastAPI:
|
||||||
|
def __init__(self, **_kwargs):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def get(self, *_args, **_kwargs):
|
||||||
|
return lambda function: function
|
||||||
|
|
||||||
|
post = get
|
||||||
|
|
||||||
|
class _HTTPException(Exception):
|
||||||
|
def __init__(self, status_code: int, detail: str):
|
||||||
|
super().__init__(detail)
|
||||||
|
self.status_code = status_code
|
||||||
|
self.detail = detail
|
||||||
|
|
||||||
|
fake_fastapi = types.ModuleType("fastapi")
|
||||||
|
fake_fastapi.FastAPI = _FastAPI
|
||||||
|
fake_fastapi.HTTPException = _HTTPException
|
||||||
|
fake_fastapi.Request = object
|
||||||
|
fake_responses = types.ModuleType("fastapi.responses")
|
||||||
|
fake_responses.HTMLResponse = str
|
||||||
|
sys.modules["fastapi"] = fake_fastapi
|
||||||
|
sys.modules["fastapi.responses"] = fake_responses
|
||||||
|
|
||||||
|
from collector import app as collector
|
||||||
|
from fastapi import HTTPException
|
||||||
|
|
||||||
|
|
||||||
|
class _Request:
|
||||||
|
def __init__(self, token: str | None = None):
|
||||||
|
self.headers = {"authorization": f"Bearer {token}"} if token else {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_collector_issues_and_verifies_installation_credentials():
|
||||||
|
original_db = collector.DB
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
collector.DB = Path(temporary) / "annotations.db"
|
||||||
|
collector.init_db()
|
||||||
|
credentials = collector.register()
|
||||||
|
|
||||||
|
assert collector._authenticated_anon_id(
|
||||||
|
_Request(credentials["token"])
|
||||||
|
) == credentials["anonId"]
|
||||||
|
try:
|
||||||
|
collector._authenticated_anon_id(_Request("wrong"))
|
||||||
|
except HTTPException as error:
|
||||||
|
assert error.status_code == 401
|
||||||
|
else:
|
||||||
|
raise AssertionError("Invalid collector token was accepted")
|
||||||
|
collector.DB = original_db
|
||||||
|
|
@ -4,7 +4,7 @@ def test_final_includes_kindle_id_tag():
|
||||||
positioned = {
|
positioned = {
|
||||||
"format":"x",
|
"format":"x",
|
||||||
"items":[
|
"items":[
|
||||||
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None},
|
{"status":"positioned","clipping":{"id":"abc123","title":"T","added_on":None,"added_on_iso":None,"note_ids":["note1"]},
|
||||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
|
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
|
||||||
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
||||||
]
|
]
|
||||||
|
|
@ -15,6 +15,7 @@ def test_final_includes_kindle_id_tag():
|
||||||
assert {"name":"kindle-import"} in tags
|
assert {"name":"kindle-import"} in tags
|
||||||
assert {"name":"kindle-id:abc123"} in tags
|
assert {"name":"kindle-id:abc123"} in tags
|
||||||
assert plan["annotations"][0]["clipping_id"]=="abc123"
|
assert plan["annotations"][0]["clipping_id"]=="abc123"
|
||||||
|
assert plan["annotations"][0]["note_ids"] == ["note1"]
|
||||||
|
|
||||||
def test_final_skips_non_positioned():
|
def test_final_skips_non_positioned():
|
||||||
positioned = {"format":"x","items":[{"status":"epub-text-not-found","clipping":{"id":"a"},"zotero":{},"annotation":{}}]}
|
positioned = {"format":"x","items":[{"status":"epub-text-not-found","clipping":{"id":"a"},"zotero":{},"annotation":{}}]}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,18 @@
|
||||||
import json, tempfile, pathlib
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
from kindle_zotero_importer import cli
|
||||||
|
from kindle_zotero_importer.cli import (
|
||||||
|
_add_previous_imported_comments,
|
||||||
|
_changed_note_clipping_ids,
|
||||||
|
_deletion_ids,
|
||||||
|
_merge_annotation_history,
|
||||||
|
_merge_positioned_history,
|
||||||
|
_unresolved_clipping_ids,
|
||||||
|
)
|
||||||
|
from kindle_zotero_importer.clippings import clippings_to_jsonable, load_clippings
|
||||||
|
from kindle_zotero_importer import pdf_position
|
||||||
|
|
||||||
def test_incremental_ids(tmp_path: Path):
|
def test_incremental_ids(tmp_path: Path):
|
||||||
# simulate previous final with one integrated id
|
# simulate previous final with one integrated id
|
||||||
|
|
@ -17,3 +30,216 @@ def test_incremental_ids(tmp_path: Path):
|
||||||
# changed: aaa text changed -> new id ddd, old aaa no longer in new -> deletion
|
# changed: aaa text changed -> new id ddd, old aaa no longer in new -> deletion
|
||||||
new_ids2 = {"ddd","bbb","ccc"}
|
new_ids2 = {"ddd","bbb","ccc"}
|
||||||
assert (prev_integrated - new_ids2) == {"aaa"}
|
assert (prev_integrated - new_ids2) == {"aaa"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_annotation_history_preserves_old_and_replaces_changed():
|
||||||
|
old_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "2"}}
|
||||||
|
old_b = {"clipping_id": "b", "attachment_item_id": 1, "annotation": {"sortIndex": "1"}}
|
||||||
|
new_a = {"clipping_id": "a", "attachment_item_id": 2, "annotation": {"sortIndex": "3"}}
|
||||||
|
|
||||||
|
merged = _merge_annotation_history(
|
||||||
|
{"all_annotations": [old_a, old_b]}, [new_a], []
|
||||||
|
)
|
||||||
|
|
||||||
|
assert merged == [old_b, new_a]
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_annotation_history_removes_deletions():
|
||||||
|
previous = {
|
||||||
|
"annotations": [
|
||||||
|
{"clipping_id": "a", "attachment_item_id": 1, "annotation": {}},
|
||||||
|
{"clipping_id": "b", "attachment_item_id": 1, "annotation": {}},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
merged = _merge_annotation_history(previous, [], ["a"])
|
||||||
|
|
||||||
|
assert [annotation["clipping_id"] for annotation in merged] == ["b"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_full_history_retains_current_unresolved_annotation():
|
||||||
|
previous = {
|
||||||
|
"all_annotations": [
|
||||||
|
{"clipping_id": "unresolved", "attachment_item_id": 1, "annotation": {}},
|
||||||
|
{"clipping_id": "removed", "attachment_item_id": 1, "annotation": {}},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
merged = _merge_annotation_history(previous, [], ["removed"])
|
||||||
|
|
||||||
|
assert [annotation["clipping_id"] for annotation in merged] == ["unresolved"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_deletions_include_removed_and_previous_failures():
|
||||||
|
assert _deletion_ids(
|
||||||
|
{"kept", "removed"},
|
||||||
|
{"kept"},
|
||||||
|
{"delete_failed_ids": ["retry"]},
|
||||||
|
) == {"removed", "retry"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
|
||||||
|
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
|
||||||
|
previous = {
|
||||||
|
"all_annotations": [
|
||||||
|
{"clipping_id": "a", "annotation": {"comment": "old"}}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
_add_previous_imported_comments(current, previous)
|
||||||
|
|
||||||
|
assert current[0]["previous_imported_comment"] == "old"
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_unresolved_excludes_positioned_ignored_and_removed_entries():
|
||||||
|
previous = {
|
||||||
|
"items": [
|
||||||
|
{"clipping": {"id": "retry"}, "status": "pdf-text-not-found"},
|
||||||
|
{"clipping": {"id": "done"}, "status": "positioned"},
|
||||||
|
{"clipping": {"id": "ignored"}, "status": "ignored-title"},
|
||||||
|
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
assert _unresolved_clipping_ids(previous, {"retry", "done", "ignored"}) == {
|
||||||
|
"retry"
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_changed_note_ids_detect_add_replace_and_remove():
|
||||||
|
previous = [
|
||||||
|
{"clipping_id": "added", "note_ids": []},
|
||||||
|
{"clipping_id": "replaced", "note_ids": ["old"]},
|
||||||
|
{"clipping_id": "removed", "note_ids": ["old"]},
|
||||||
|
{"clipping_id": "same", "note_ids": ["same-note"]},
|
||||||
|
]
|
||||||
|
current = [
|
||||||
|
{"id": "added", "note_ids": ["new"]},
|
||||||
|
{"id": "replaced", "note_ids": ["new"]},
|
||||||
|
{"id": "removed"},
|
||||||
|
{"id": "same", "note_ids": ["same-note"]},
|
||||||
|
]
|
||||||
|
|
||||||
|
assert _changed_note_clipping_ids(current, previous) == {
|
||||||
|
"added",
|
||||||
|
"replaced",
|
||||||
|
"removed",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_positioned_history_preserves_unresolved_and_replaces_current():
|
||||||
|
previous = {
|
||||||
|
"items": [
|
||||||
|
{"clipping": {"id": "keep"}, "status": "pdf-text-not-found"},
|
||||||
|
{"clipping": {"id": "replace"}, "status": "epub-text-not-found"},
|
||||||
|
{"clipping": {"id": "removed"}, "status": "unmatched-title"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
current = {
|
||||||
|
"format": "plan",
|
||||||
|
"items": [{"clipping": {"id": "replace"}, "status": "positioned"}],
|
||||||
|
}
|
||||||
|
|
||||||
|
merged = _merge_positioned_history(previous, current, {"keep", "replace"})
|
||||||
|
|
||||||
|
assert {item["clipping"]["id"]: item["status"] for item in merged["items"]} == {
|
||||||
|
"keep": "pdf-text-not-found",
|
||||||
|
"replace": "positioned",
|
||||||
|
}
|
||||||
|
assert merged["status_counts"] == {"pdf-text-not-found": 1, "positioned": 1}
|
||||||
|
|
||||||
|
|
||||||
|
def test_pdf_position_reports_missing_tools():
|
||||||
|
original = pdf_position._missing_required_tools
|
||||||
|
pdf_position._missing_required_tools = lambda: ["pdftotext", "pdfinfo"]
|
||||||
|
try:
|
||||||
|
result = pdf_position.add_pdf_positions(
|
||||||
|
{
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"status": "ready-for-positioning",
|
||||||
|
"clipping": {"kind": "highlight", "text": "highlight"},
|
||||||
|
"zotero": {
|
||||||
|
"attachment": {"content_type": "application/pdf"}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
pdf_position._missing_required_tools = original
|
||||||
|
|
||||||
|
assert result["items"][0]["status"] == "pdf-tools-missing:pdftotext,pdfinfo"
|
||||||
|
|
||||||
|
|
||||||
|
def test_pdf_position_preserves_unresolved_item_without_zotero_match():
|
||||||
|
item = {
|
||||||
|
"status": "unmatched-title",
|
||||||
|
"clipping": {"kind": "highlight", "text": "highlight"},
|
||||||
|
"zotero": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
result = pdf_position.add_pdf_positions({"items": [item]})
|
||||||
|
|
||||||
|
assert result["items"] == [item]
|
||||||
|
|
||||||
|
|
||||||
|
def test_noop_incremental_writes_complete_artifacts():
|
||||||
|
with tempfile.TemporaryDirectory() as temporary:
|
||||||
|
workdir = Path(temporary)
|
||||||
|
source = workdir / "My Clippings.txt"
|
||||||
|
source.write_text(
|
||||||
|
"Book\n- Your Highlight at location 10-11 | Added on Tuesday, 18 May 2021 14:18:13\n\n"
|
||||||
|
"A highlight long enough to import\n==========\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
clippings = clippings_to_jsonable(load_clippings(str(source)))
|
||||||
|
clipping_id = clippings["clippings"][0]["id"]
|
||||||
|
(workdir / "clippings.json").write_text(json.dumps(clippings), encoding="utf-8")
|
||||||
|
annotation = {
|
||||||
|
"clipping_id": clipping_id,
|
||||||
|
"attachment_item_id": 1,
|
||||||
|
"annotation": {"sortIndex": "1"},
|
||||||
|
}
|
||||||
|
(workdir / "import-plan.final.json").write_text(
|
||||||
|
json.dumps({"annotations": [annotation], "all_annotations": [annotation], "overrides_snapshot": {}}),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
originals = (
|
||||||
|
cli.build_zotero_index,
|
||||||
|
cli.build_match_report,
|
||||||
|
cli.generate_override_skeleton,
|
||||||
|
cli.build_mismatch_review,
|
||||||
|
)
|
||||||
|
cli.build_zotero_index = lambda *_args: {"format": "index", "items": []}
|
||||||
|
cli.build_match_report = lambda *_args: {"format": "matches", "matches": []}
|
||||||
|
cli.generate_override_skeleton = lambda *_args: {"format": "overrides", "overrides": []}
|
||||||
|
cli.build_mismatch_review = lambda *_args: "review\n"
|
||||||
|
try:
|
||||||
|
result = cli.main(
|
||||||
|
[
|
||||||
|
"run",
|
||||||
|
str(source),
|
||||||
|
"--workdir",
|
||||||
|
str(workdir),
|
||||||
|
"--db",
|
||||||
|
str(workdir / "zotero.sqlite"),
|
||||||
|
"--storage-root",
|
||||||
|
str(workdir / "storage"),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
(
|
||||||
|
cli.build_zotero_index,
|
||||||
|
cli.build_match_report,
|
||||||
|
cli.generate_override_skeleton,
|
||||||
|
cli.build_mismatch_review,
|
||||||
|
) = originals
|
||||||
|
|
||||||
|
assert result == 0
|
||||||
|
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
|
||||||
|
assert final_plan["annotations"] == []
|
||||||
|
assert final_plan["all_annotations"] == [annotation]
|
||||||
|
assert final_plan["delta_annotation_count"] == 0
|
||||||
|
assert (workdir / "import-plan.epub.json").exists()
|
||||||
|
|
|
||||||
|
|
@ -1,14 +1,331 @@
|
||||||
import assert from 'assert';
|
const assert = require('assert');
|
||||||
// test the grouping logic for title variants sharing same candidate
|
const fs = require('fs');
|
||||||
function findOtherRows(rows, rowIndex, targetKey) {
|
const vm = require('vm');
|
||||||
const row = rows[rowIndex];
|
|
||||||
return rows.filter((r, idx) => idx !== rowIndex && (r.candidates||[]).some(c => (c.citation_key||c.key||String(c.item_id))===targetKey));
|
const bootstrapContext = {};
|
||||||
|
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
|
||||||
|
const plugin = bootstrapContext.KindleZoteroImporter;
|
||||||
|
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
|
||||||
|
|
||||||
|
assert.equal(
|
||||||
|
plugin.mergeComments('[r] this is a red note\n\nthis is a red note', ''),
|
||||||
|
'this is a red note'
|
||||||
|
);
|
||||||
|
assert.equal(plugin.mergeComments('[o]', ''), '');
|
||||||
|
assert.equal(plugin.mergeComments('[citation] keep this', ''), '[citation] keep this');
|
||||||
|
assert.equal(
|
||||||
|
plugin.commentForExistingAnnotation(
|
||||||
|
{ annotationComment: 'old note', annotationTags: [{ name: 'kindle-import' }] },
|
||||||
|
'replacement note',
|
||||||
|
'old note'
|
||||||
|
),
|
||||||
|
'replacement note'
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
plugin.commentForExistingAnnotation(
|
||||||
|
{ annotationComment: 'old note\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
|
||||||
|
'replacement note',
|
||||||
|
'old note'
|
||||||
|
),
|
||||||
|
'manual addition\n\nreplacement note'
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
plugin.commentForExistingAnnotation(
|
||||||
|
{ annotationComment: 'old first\n\nold second\n\nmanual addition', annotationTags: [{ name: 'kindle-import' }] },
|
||||||
|
'new note',
|
||||||
|
'old first\n\nold second'
|
||||||
|
),
|
||||||
|
'manual addition\n\nnew note'
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
plugin.commentForExistingAnnotation(
|
||||||
|
{ annotationComment: 'manual note', annotationTags: [] },
|
||||||
|
'imported note'
|
||||||
|
),
|
||||||
|
'manual note\n\nimported note'
|
||||||
|
);
|
||||||
|
|
||||||
|
const base = {
|
||||||
|
annotationText: 'forged its might, should be enough to discredit',
|
||||||
|
annotationPosition: JSON.stringify({
|
||||||
|
type: 'FragmentSelector',
|
||||||
|
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1255)',
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
const extended = {
|
||||||
|
annotationText: 'forged its might, should be enough to discredit, even in the eyes of philosophers',
|
||||||
|
annotationPosition: JSON.stringify({
|
||||||
|
type: 'FragmentSelector',
|
||||||
|
value: 'epubcfi(/6/12!/4/2/8,/1:1142,/1:1300)',
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
assert.equal(plugin.areImportedDuplicates(base, extended), true);
|
||||||
|
assert.equal(
|
||||||
|
plugin.areImportedDuplicates(base, {
|
||||||
|
...base,
|
||||||
|
annotationPosition: JSON.stringify({
|
||||||
|
type: 'FragmentSelector',
|
||||||
|
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
false
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
plugin.areImportedDuplicates(base, {
|
||||||
|
...base,
|
||||||
|
annotationText: 'OCR-corrected text at the identical annotation position',
|
||||||
|
}),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
assert.equal(
|
||||||
|
plugin.areImportedDuplicates(
|
||||||
|
{
|
||||||
|
annotationText: 'A multiline PDF selection long enough to compare',
|
||||||
|
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30]] }),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
annotationText: 'A multiline PDF selection long enough to compare with an extended ending',
|
||||||
|
annotationPosition: JSON.stringify({ pageIndex: 2, rects: [[10, 20, 100, 30], [10, 40, 120, 50]] }),
|
||||||
}
|
}
|
||||||
const rows = [
|
),
|
||||||
{title:"gilbert-simondon-on-the-mode", candidates:[{citation_key:"chabot2013", key:"A"}, {citation_key:"simondon2017a"}]},
|
true
|
||||||
{title:"On the Mode (Univocal)", candidates:[{citation_key:"chabot2013"}, {citation_key:"simondon2017a"}]},
|
);
|
||||||
{title:"Other", candidates:[{citation_key:"foo"}]}
|
|
||||||
];
|
const managerHtml = fs.readFileSync('plugin/manager.html', 'utf8');
|
||||||
assert.equal(findOtherRows(rows, 0, "chabot2013").length, 1);
|
const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
|
||||||
assert.equal(findOtherRows(rows, 0, "chabot2013")[0].title, "On the Mode (Univocal)");
|
const managerContext = { TextEncoder };
|
||||||
console.log("manager logic ok");
|
vm.runInNewContext(scripts[0][1], managerContext);
|
||||||
|
const manager = managerContext.KindleManager;
|
||||||
|
manager.integratedPage = 1;
|
||||||
|
manager.integratedPageSize = 50;
|
||||||
|
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
|
||||||
|
assert.equal(page.totalPages, 3);
|
||||||
|
assert.deepEqual(Array.from(page.rows), Array.from({ length: 50 }, (_, index) => index + 50));
|
||||||
|
|
||||||
|
manager.mappingsPage = 99;
|
||||||
|
manager.mappingsPageSize = 50;
|
||||||
|
const finalPage = manager.paginate(Array.from({ length: 51 }, (_, index) => index), 'mappingsPage', 'mappingsPageSize');
|
||||||
|
assert.equal(manager.mappingsPage, 1);
|
||||||
|
assert.deepEqual(Array.from(finalPage.rows), [50]);
|
||||||
|
|
||||||
|
const collectorBatches = manager.collectorBatches('installation', Array.from(
|
||||||
|
{ length: 450 },
|
||||||
|
(_, index) => ({ clipping_id_hash: String(index), text: 'x'.repeat(2000) })
|
||||||
|
));
|
||||||
|
assert.equal(collectorBatches.length, 3);
|
||||||
|
assert.ok(collectorBatches.every(batch => batch.length <= 200));
|
||||||
|
assert.ok(collectorBatches.every(batch => new TextEncoder().encode(
|
||||||
|
JSON.stringify({ anonId: 'installation', data: batch })
|
||||||
|
).length <= 1500000));
|
||||||
|
assert.throws(
|
||||||
|
() => manager.collectorBatches('installation', [{ text: 'x'.repeat(1500000) }]),
|
||||||
|
/exceeds the collector request limit/
|
||||||
|
);
|
||||||
|
|
||||||
|
async function testRuntimeExtraction() {
|
||||||
|
const writes = [];
|
||||||
|
bootstrapContext.PathUtils = {
|
||||||
|
profileDir: '/profile',
|
||||||
|
join: (...parts) => parts.join('/').replaceAll('//', '/'),
|
||||||
|
parent: path => path.slice(0, path.lastIndexOf('/')),
|
||||||
|
};
|
||||||
|
bootstrapContext.IOUtils = {
|
||||||
|
makeDirectory: async () => {},
|
||||||
|
};
|
||||||
|
bootstrapContext.Zotero = {
|
||||||
|
File: { putContentsAsync: async (path, text) => writes.push([path, text]) },
|
||||||
|
};
|
||||||
|
bootstrapContext.fetch = async url => ({ ok: true, text: async () => url });
|
||||||
|
plugin.rootURI = 'jar:file:///kanzi.xpi!/';
|
||||||
|
await plugin.ensureRuntime();
|
||||||
|
assert.equal(plugin.runtimeProjectDir, '/profile/kanzi-runtime');
|
||||||
|
assert.equal(writes.length, 13);
|
||||||
|
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testMaintenanceSweep() {
|
||||||
|
const annotation = (id, text, end, comment = '') => ({
|
||||||
|
id,
|
||||||
|
parentItemID: 10,
|
||||||
|
annotationText: text,
|
||||||
|
annotationComment: comment,
|
||||||
|
annotationPosition: JSON.stringify({
|
||||||
|
type: 'FragmentSelector',
|
||||||
|
value: `epubcfi(/6/12!/4/2/8,/1:795,/1:${end})`,
|
||||||
|
}),
|
||||||
|
saved: 0,
|
||||||
|
erased: 0,
|
||||||
|
saveTx: async function () { this.saved += 1; },
|
||||||
|
eraseTx: async function () { this.erased += 1; },
|
||||||
|
});
|
||||||
|
const short = annotation(1, 'A progressive highlight long enough to compare', 994);
|
||||||
|
const middle = annotation(2, 'A progressive highlight long enough to compare and then extended', 1093);
|
||||||
|
const longest = annotation(
|
||||||
|
3,
|
||||||
|
'A progressive highlight long enough to compare and then extended to its final form',
|
||||||
|
1141,
|
||||||
|
'[r] useful note\n\nuseful note'
|
||||||
|
);
|
||||||
|
const separate = annotation(4, 'A progressive highlight long enough to compare', 2050);
|
||||||
|
separate.annotationPosition = JSON.stringify({
|
||||||
|
type: 'FragmentSelector',
|
||||||
|
value: 'epubcfi(/6/12!/4/2/8,/1:2000,/1:2050)',
|
||||||
|
});
|
||||||
|
const emptyComment = annotation(5, 'A separate highlight with a null comment', 3050, null);
|
||||||
|
emptyComment.annotationPosition = JSON.stringify({
|
||||||
|
type: 'FragmentSelector',
|
||||||
|
value: 'epubcfi(/6/12!/4/2/8,/1:3000,/1:3050)',
|
||||||
|
});
|
||||||
|
const results = { updatedComments: 0, removedDuplicates: 0 };
|
||||||
|
|
||||||
|
await plugin.maintainImportedAnnotations([short, middle, longest, separate, emptyComment], false, results);
|
||||||
|
|
||||||
|
assert.equal(results.removedDuplicates, 2);
|
||||||
|
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
|
||||||
|
assert.equal(short.erased, 1);
|
||||||
|
assert.equal(middle.erased, 1);
|
||||||
|
assert.equal(longest.erased, 0);
|
||||||
|
assert.equal(separate.erased, 0);
|
||||||
|
assert.equal(emptyComment.saved, 0);
|
||||||
|
assert.equal(longest.annotationComment, 'useful note');
|
||||||
|
assert.equal(results.updatedComments, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testWriterReconciliation() {
|
||||||
|
let written = null;
|
||||||
|
bootstrapContext.Zotero = {
|
||||||
|
File: { putContentsAsync: async (_path, text) => { written = JSON.parse(text); } },
|
||||||
|
};
|
||||||
|
const plan = {
|
||||||
|
annotations: [{ clipping_id: 'ok' }, { clipping_id: 'failed' }],
|
||||||
|
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'ok' }, { clipping_id: 'failed' }],
|
||||||
|
};
|
||||||
|
await plugin.reconcileWriterState('/plan.json', plan, {
|
||||||
|
failed: [{ clipping_id: 'failed' }, { clipping_id: 'maintenance' }],
|
||||||
|
failedDeletionIds: ['delete-me'],
|
||||||
|
maintenanceDeletedIds: ['old'],
|
||||||
|
});
|
||||||
|
assert.deepEqual(Array.from(written.write_failed_ids), ['failed']);
|
||||||
|
assert.deepEqual(Array.from(written.annotations, entry => entry.clipping_id), ['ok']);
|
||||||
|
assert.deepEqual(Array.from(written.all_annotations, entry => entry.clipping_id), ['ok']);
|
||||||
|
assert.equal(written.annotation_count, 1);
|
||||||
|
assert.deepEqual(Array.from(written.delete_failed_ids), ['delete-me']);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testDeletionFailureIsRetried() {
|
||||||
|
bootstrapContext.Zotero = { logError: () => {} };
|
||||||
|
const annotation = {
|
||||||
|
annotationTags: [{ name: 'kindle-id:delete-me' }],
|
||||||
|
eraseTx: async () => { throw new Error('locked'); },
|
||||||
|
};
|
||||||
|
plugin.collectImportedAnnotations = async () => [annotation];
|
||||||
|
plugin.maintainImportedAnnotations = async () => {};
|
||||||
|
const results = await plugin.writeAnnotations({
|
||||||
|
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||||
|
annotations: [],
|
||||||
|
deletions: ['delete-me'],
|
||||||
|
}, false);
|
||||||
|
assert.deepEqual(Array.from(results.failedDeletionIds), ['delete-me']);
|
||||||
|
assert.equal(results.failed.length, 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testMaintenanceDeletionIsNotRepeated() {
|
||||||
|
bootstrapContext.Zotero = { logError: () => {} };
|
||||||
|
const annotation = {
|
||||||
|
annotationTags: [{ name: 'kindle-id:already-deleted' }],
|
||||||
|
eraseTx: async () => { throw new Error('must not be called again'); },
|
||||||
|
};
|
||||||
|
plugin.collectImportedAnnotations = async () => [annotation];
|
||||||
|
plugin.maintainImportedAnnotations = async (_annotations, _dryRun, results) => {
|
||||||
|
results.maintenanceDeletedIds.push('already-deleted');
|
||||||
|
};
|
||||||
|
const results = await plugin.writeAnnotations({
|
||||||
|
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||||
|
annotations: [],
|
||||||
|
deletions: ['already-deleted'],
|
||||||
|
}, false);
|
||||||
|
assert.deepEqual(Array.from(results.failedDeletionIds), []);
|
||||||
|
assert.equal(results.failed.length, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor() {
|
||||||
|
bootstrapContext.Zotero = { logError: () => {} };
|
||||||
|
const active = {
|
||||||
|
parentItemID: 10,
|
||||||
|
annotationText: 'The longer OCR-corrected duplicate annotation text',
|
||||||
|
annotationComment: '',
|
||||||
|
annotationPosition: JSON.stringify({ pageIndex: 1, rects: [[1, 2, 3, 4]] }),
|
||||||
|
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
|
||||||
|
erased: 0,
|
||||||
|
eraseTx: async function () { this.erased += 1; },
|
||||||
|
};
|
||||||
|
const duplicate = {
|
||||||
|
...active,
|
||||||
|
annotationText: 'The duplicate annotation text',
|
||||||
|
annotationTags: [{ name: 'kindle-import' }, { name: 'kindle-id:same' }],
|
||||||
|
erased: 0,
|
||||||
|
eraseTx: async function () { this.erased += 1; },
|
||||||
|
};
|
||||||
|
let collection = 0;
|
||||||
|
plugin.collectImportedAnnotations = async () => (++collection === 1 ? [active, duplicate] : [active]);
|
||||||
|
plugin.maintainImportedAnnotations = maintainImportedAnnotations;
|
||||||
|
const results = await plugin.writeAnnotations({
|
||||||
|
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||||
|
annotations: [],
|
||||||
|
deletions: ['same'],
|
||||||
|
}, false);
|
||||||
|
assert.deepEqual(Array.from(results.maintenanceDeletedIds), []);
|
||||||
|
assert.equal(duplicate.erased, 1);
|
||||||
|
assert.equal(active.erased, 1);
|
||||||
|
assert.equal(results.deletedForIncremental, 1);
|
||||||
|
assert.equal(results.failed.length, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testEnumerationFailurePersistsAllDeletions() {
|
||||||
|
bootstrapContext.Zotero = { logError: () => {} };
|
||||||
|
plugin.collectImportedAnnotations = async () => { throw new Error('enumeration failed'); };
|
||||||
|
const results = await plugin.writeAnnotations({
|
||||||
|
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||||
|
annotations: [],
|
||||||
|
deletions: ['retry-deletion'],
|
||||||
|
}, false);
|
||||||
|
assert.deepEqual(Array.from(results.failedDeletionIds), ['retry-deletion']);
|
||||||
|
assert.equal(results.failed[0].clipping_id, 'maintenance');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
|
||||||
|
const old = {
|
||||||
|
annotationTags: [{ name: 'kindle-id:replacement' }],
|
||||||
|
erased: 0,
|
||||||
|
eraseTx: async function () { this.erased += 1; },
|
||||||
|
};
|
||||||
|
bootstrapContext.Zotero = {
|
||||||
|
logError: () => {},
|
||||||
|
Items: { get: () => null },
|
||||||
|
};
|
||||||
|
plugin.collectImportedAnnotations = async () => [old];
|
||||||
|
plugin.maintainImportedAnnotations = async () => {};
|
||||||
|
const results = await plugin.writeAnnotations({
|
||||||
|
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||||
|
annotations: [{ clipping_id: 'replacement', attachment_item_id: 99, annotation: {} }],
|
||||||
|
deletions: ['replacement'],
|
||||||
|
}, false);
|
||||||
|
assert.equal(old.erased, 0);
|
||||||
|
assert.deepEqual(Array.from(results.failedDeletionIds), ['replacement']);
|
||||||
|
assert.equal(results.failed[0].clipping_id, 'replacement');
|
||||||
|
}
|
||||||
|
|
||||||
|
testRuntimeExtraction()
|
||||||
|
.then(testMaintenanceSweep)
|
||||||
|
.then(testWriterReconciliation)
|
||||||
|
.then(testDeletionFailureIsRetried)
|
||||||
|
.then(testMaintenanceDeletionIsNotRepeated)
|
||||||
|
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
|
||||||
|
.then(testEnumerationFailurePersistsAllDeletions)
|
||||||
|
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
|
||||||
|
.then(() => {
|
||||||
|
console.log('manager and annotation logic ok');
|
||||||
|
}).catch(error => {
|
||||||
|
console.error(error);
|
||||||
|
process.exitCode = 1;
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,15 @@
|
||||||
"addons": {
|
"addons": {
|
||||||
"kindlezoteroimporter@utkubilen.de": {
|
"kindlezoteroimporter@utkubilen.de": {
|
||||||
"updates": [
|
"updates": [
|
||||||
|
{
|
||||||
|
"version": "0.7.0",
|
||||||
|
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.7.0/kanzi.xpi",
|
||||||
|
"applications": {
|
||||||
|
"zotero": {
|
||||||
|
"strict_min_version": "6.999"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"version": "0.6.20",
|
"version": "0.6.20",
|
||||||
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.6.20/kanzi.xpi",
|
"update_link": "https://github.com/UtkuBilenDemir/kanzi/releases/download/v0.6.20/kanzi.xpi",
|
||||||
|
|
|
||||||
156
zotero-writer.js
156
zotero-writer.js
|
|
@ -1,156 +0,0 @@
|
||||||
// Run in Zotero via Tools > Developer > Run JavaScript.
|
|
||||||
// Set PLAN_PATH to the generated import-plan.final.json path.
|
|
||||||
// Keep DRY_RUN=true until the preview output looks correct.
|
|
||||||
|
|
||||||
const PLAN_PATH = "/home/user/Library/Mobile Documents/iCloud~md~obsidian/Documents/rhizome/06_projects/UTI/kindle-zotero-importer/import-plan.final.json";
|
|
||||||
const DRY_RUN = false;
|
|
||||||
|
|
||||||
const text = await Zotero.File.getContentsAsync(PLAN_PATH);
|
|
||||||
const plan = JSON.parse(text);
|
|
||||||
|
|
||||||
if (plan.format !== "kindle-zotero-importer.zotero-writer-plan.v1") {
|
|
||||||
throw new Error(`Unsupported plan format: ${plan.format}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const results = {
|
|
||||||
dryRun: DRY_RUN,
|
|
||||||
total: plan.annotations.length,
|
|
||||||
created: 0,
|
|
||||||
skippedExisting: 0,
|
|
||||||
updatedComments: 0,
|
|
||||||
removedDuplicates: 0,
|
|
||||||
failed: [],
|
|
||||||
preview: [],
|
|
||||||
};
|
|
||||||
|
|
||||||
const existingByAttachment = new Map();
|
|
||||||
|
|
||||||
function normalizePosition(position) {
|
|
||||||
if (!position) {
|
|
||||||
return "";
|
|
||||||
}
|
|
||||||
if (typeof position === "string") {
|
|
||||||
try {
|
|
||||||
return JSON.stringify(JSON.parse(position));
|
|
||||||
} catch (_error) {
|
|
||||||
return position;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return JSON.stringify(position);
|
|
||||||
}
|
|
||||||
|
|
||||||
function annotationFingerprint(annotation) {
|
|
||||||
return `${annotation.text || ""}\u0000${normalizePosition(annotation.position)}`;
|
|
||||||
}
|
|
||||||
|
|
||||||
function existingAnnotationFingerprint(annotation) {
|
|
||||||
return `${annotation.annotationText || ""}\u0000${normalizePosition(annotation.annotationPosition)}`;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function getExistingAnnotations(attachment) {
|
|
||||||
if (!existingByAttachment.has(attachment.id)) {
|
|
||||||
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
|
|
||||||
const byFingerprint = new Map();
|
|
||||||
|
|
||||||
for (const annotation of annotations) {
|
|
||||||
const fingerprint = existingAnnotationFingerprint(annotation);
|
|
||||||
const kept = byFingerprint.get(fingerprint);
|
|
||||||
if (!kept) {
|
|
||||||
byFingerprint.set(fingerprint, annotation);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
const keptHasComment = Boolean(kept.annotationComment);
|
|
||||||
const currentHasComment = Boolean(annotation.annotationComment);
|
|
||||||
const duplicate = keptHasComment || !currentHasComment ? annotation : kept;
|
|
||||||
const replacement = duplicate === annotation ? kept : annotation;
|
|
||||||
|
|
||||||
byFingerprint.set(fingerprint, replacement);
|
|
||||||
if (!DRY_RUN) {
|
|
||||||
await duplicate.eraseTx();
|
|
||||||
}
|
|
||||||
results.removedDuplicates += 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
existingByAttachment.set(attachment.id, byFingerprint);
|
|
||||||
}
|
|
||||||
return existingByAttachment.get(attachment.id);
|
|
||||||
}
|
|
||||||
|
|
||||||
function mergeComments(existingComment, newComment) {
|
|
||||||
const seen = new Set();
|
|
||||||
const merged = [];
|
|
||||||
for (const comment of [existingComment, newComment]) {
|
|
||||||
for (const part of String(comment || "").split(/\n\s*\n/)) {
|
|
||||||
const cleaned = part.trim();
|
|
||||||
if (!cleaned || seen.has(cleaned)) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
seen.add(cleaned);
|
|
||||||
merged.push(cleaned);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return merged.join("\n\n");
|
|
||||||
}
|
|
||||||
|
|
||||||
for (const entry of plan.annotations) {
|
|
||||||
try {
|
|
||||||
const attachment = Zotero.Items.get(entry.attachment_item_id);
|
|
||||||
if (!attachment) {
|
|
||||||
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const annotation = {
|
|
||||||
key: Zotero.DataObjectUtilities.generateKey(),
|
|
||||||
...entry.annotation,
|
|
||||||
};
|
|
||||||
if (!annotation.sortIndex) {
|
|
||||||
delete annotation.sortIndex;
|
|
||||||
}
|
|
||||||
|
|
||||||
const existingAnnotations = await getExistingAnnotations(attachment);
|
|
||||||
const fingerprint = annotationFingerprint(annotation);
|
|
||||||
const existingAnnotation = existingAnnotations.get(fingerprint);
|
|
||||||
if (existingAnnotation) {
|
|
||||||
const mergedComment = mergeComments(existingAnnotation.annotationComment, annotation.comment);
|
|
||||||
if (mergedComment && existingAnnotation.saveTx && existingAnnotation.annotationComment !== mergedComment) {
|
|
||||||
existingAnnotation.annotationComment = mergedComment;
|
|
||||||
if (!DRY_RUN) {
|
|
||||||
await existingAnnotation.saveTx();
|
|
||||||
}
|
|
||||||
results.updatedComments += 1;
|
|
||||||
}
|
|
||||||
results.skippedExisting += 1;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (DRY_RUN) {
|
|
||||||
results.preview.push({
|
|
||||||
clipping_id: entry.clipping_id,
|
|
||||||
attachment_item_id: entry.attachment_item_id,
|
|
||||||
type: annotation.type,
|
|
||||||
text: annotation.text.slice(0, 120),
|
|
||||||
pageLabel: annotation.pageLabel,
|
|
||||||
sortIndex: annotation.sortIndex,
|
|
||||||
position: annotation.position,
|
|
||||||
});
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
await Zotero.Annotations.saveFromJSON(attachment, annotation);
|
|
||||||
existingAnnotations.set(fingerprint, {
|
|
||||||
annotationComment: annotation.comment || "",
|
|
||||||
annotationPosition: annotation.position,
|
|
||||||
annotationText: annotation.text || "",
|
|
||||||
});
|
|
||||||
results.created += 1;
|
|
||||||
} catch (error) {
|
|
||||||
results.failed.push({
|
|
||||||
clipping_id: entry.clipping_id,
|
|
||||||
attachment_item_id: entry.attachment_item_id,
|
|
||||||
message: String(error),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return JSON.stringify(results, null, 2);
|
|
||||||
Loading…
Add table
Reference in a new issue