kindle-zotero_importer/README.md

Kindle Zotero Importer

Import Kindle My Clippings.txt highlights into Zotero as native annotations — directly inside Zotero, no terminal needed.

The hybrid Zotero plugin (plugin/) owns the UI and writes annotations through Zotero's native APIs. The Python pipeline (src/kindle_zotero_importer/) does parsing, matching, and EPUB/PDF positioning.

What it does now (0.6.4)

• One Tools entry: Tools → Kindle Zotero Importer… opens a manager window (chrome://kindle-zotero-importer/content/manager.html).
• File picker + staged progress: choose your cumulative My Clippings.txt → live stage, percent, elapsed, plugin-progress.json (Reading clippings → Indexing Zotero → Matching → Building plan → Positioning EPUB → Positioning PDF → Finalizing → Saving).
• Incremental by default: hashes each clipping sha256(title|raw_detail|text)[:16] (src/kindle_zotero_importer/clippings.py:98). Only new/changed ids (new_ids - prev_integrated_ids) go through expensive pdftohtml/pdftotext (src/kindle_zotero_importer/pdf_position.py:120,197). Already integrated kindle-id:<id> tags (src/kindle_zotero_importer/final_plan.py:44) are skipped. Check Full re-import from scratch to ignore incremental and re-process all 2379.
• Integrated tab: last import-plan.final.json annotations (975 in current artifacts) with Kindle Title | Citekey | Highlight Text | Added On | Integrated | Page, filterable, newest Integrated first.
• Conflicts tab: unresolved title matches (matched/ignored filtered out). Each row shows Source | Status | Count | Kindle Title | Candidates / Detail (N suggestion(s) — top: citekey (%)) and an explicit candidate-list — each candidate citekey · title (score%) with its own Use button (saveCandidateAt). Free-form or enter any citekey / Zotero key / ID + Use Custom + Ignore Title. If you map chabot2013 for one simondon variant, it offers to apply the same mapping to other variants sharing that candidate and hides them immediately.
• Mappings tab: persistent match-overrides.json (38 entries) with Kindle Title | Resolution | Status | Count | Date | Action, sorted by updated_at newest first (plugin/bootstrap.js:288 created_at/updated_at ISO), filterable, Delete per row returns title to Conflicts after next Re-import.
• Re-import bar: after any Use/Use Custom/Ignore or Delete, a Re-import with saved overrides bar appears reusing lastClippingsPath (plugin/bootstrap.js:734 runManagedImportWithPath) or prompting for file. Explains why re-import is needed (new citation_key → new attachment → new epubcfi/pdf rect/sortIndex).
• Settings tab: editable Project directory, Python, Zotero DB, Zotero storage (setting-* ids) + Save Settings → plugin/bootstrap.js:315 saveSettingsFromManager writes prefs + plugin-config.json.
• Artifacts tab: per-row Open/Reveal for docs/mismatch-review.md, match-overrides.json, import-plan.*.json, plugin-summary.json.
• Theme: black #0a0a0a over white #ffffff iA Writer Duo monospace, table-layout:fixed with draggable div.resizer (plugin/manager.html:270) and sortable th (▲/▼) for all tables, selectable text.

Install

python scripts/build_plugin.py
# → dist/kindle-zotero-importer.xpi  (manifest 0.6.4, Zotero 6.999–10.0.*)

In Zotero 10: Tools → Plugins → gear → Install Plugin From File… → dist/kindle-zotero-importer.xpi → restart → Tools → Kindle Zotero Importer….

plugin/manifest.json:11 update_url https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates.json enables auto-update.

Use

1. Tools → Kindle Zotero Importer… → Choose My Clippings.txt (cumulative file).
2. Keep manager open for Positioning … (incremental) progress and Result (created/already present/updated/failed/deletedForIncremental).
3. Conflicts → pick Use per candidate or Use Custom with any citation_key (deleuze1987), 8-char Zotero key (JTDWDKRH), or numeric item_id; use Ignore Title for titles to skip.
4. Re-import with saved overrides (uses last file, or prompts) → re-matches/positions only the delta and writes kindle-import + kindle-id:<hash> tagged annotations.
5. Integrated to verify highlights (Added On Kindle date, Integrated UTC now, Citekey always visible), Mappings to review/delete established overrides.

For title variants (gilbert-simondon… vs On the Mode… (Univocal) both candidate simondon2017a/chabot2013), mapping one offers to apply to the others sharing that candidate — or map them individually; they clear from Conflicts immediately and after Re-import are matched in matches.json and absent from match-overrides.generated.json.

For full rebuild, check Full re-import from scratch in the run-panel.

CLI (for debugging)

python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --db ~/Zotero/zotero.sqlite --storage-root ~/Zotero/storage --overrides match-overrides.json --summary-output plugin-summary.json --pretty
python -m kindle_zotero_importer run ... --full   # ignore incremental
PYTHONPATH=src python -m kindle_zotero_importer parse "/path/to/My Clippings.txt" --pretty | head

--progress-output plugin-progress.json drives the manager progress bar.

PDF positioning uses Poppler (pdftotext, pdftohtml, pdfinfo) plus qpdf --decrypt fallback; EPUB uses epubcfi.

Safety Rule

Never write directly to zotero.sqlite. Python is read-only for indexing; writes are only via Zotero.Annotations.saveFromJSON / eraseTx in plugin/bootstrap.js:975 writeAnnotations.

Releases — including beta

GitHub Releases are built from dist/kindle-zotero-importer.xpi:

python scripts/build_plugin.py
# tag and push
git tag v0.6.4 && git push origin v0.6.4
# GitHub → Releases → Draft a new release → Tag v0.6.4 → Title 0.6.4 → Attach dist/kindle-zotero-importer.xpi
# Generate updates.json:
# {
#   "addons": {
#     "kindlezoteroimporter@utkubilen.de": {
#       "updates": [{
#         "version": "0.6.4",
#         "update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.4/kindle-zotero-importer.xpi",
#         "applications": {"zotero": {"strict_min_version": "6.999"}}
#       }]
#     }
#   }
# }
# upload updates.json to the same release (update_url points to /releases/latest/download/updates.json)

For a beta/pre-release: on the GitHub Release form check Set as a pre-release and use a tag like v0.6.4-beta.1 with version 0.6.4-beta.1 in plugin/manifest.json:4 and updates.json. Zotero will offer it as an update only to users on that channel; stable v0.6.4 stays latest. You can also mark This is a pre-release without changing update_url — strict_max_version 10.0.* already allows beta testing in Zotero 10.

Donations

Zotero has no built-in plugin donation. Add your sponsor link to README.md and plugin/manifest.json:6 homepage_url, and to the manager infobox (plugin/manager.html:485). Recommended: GitHub Sponsors (https://github.com/sponsors/UtkuBilenDemir) or Ko-fi/PayPal/OpenCollective. The Donate button in Plugins manager comes from aboutURL if you add "aboutURL": "https://github.com/sponsors/…" to manifest.json.

Project layout

• plugin/ — hybrid bootstrap plugin (bootstrap.js, manager.html, manifest.json, prefs.js)
• src/kindle_zotero_importer/ — clippings.py, zotero_index.py, matcher.py, import_plan.py, epub_position.py, pdf_position.py, final_plan.py, cli.py
• scripts/build_plugin.py — reproducible XPI builder
• match-overrides.json — persistent title → citation_key/zotero_key/ignore mappings (now with created_at/updated_at, sorted newest first)