diff --git a/README.md b/README.md index 3d43919..16a32c2 100644 --- a/README.md +++ b/README.md @@ -1,116 +1,94 @@ # Kindle Zotero Importer -Prototype importer for turning Kindle `My Clippings.txt` entries into a JSON import plan that Zotero can later write as native annotations. +Import Kindle `My Clippings.txt` highlights into Zotero as native annotations — directly inside Zotero, no terminal needed. -The project intentionally keeps Zotero writes out of Python. Python parses, matches, and plans. Zotero JavaScript, and eventually a Zotero plugin, will perform annotation creation through Zotero's own APIs. +The hybrid Zotero plugin (`plugin/`) owns the UI and writes annotations through Zotero's native APIs. The Python pipeline (`src/kindle_zotero_importer/`) does parsing, matching, and EPUB/PDF positioning. -## Current Milestone +## What it does now (0.6.4) -Parse Kindle clippings and emit a stable JSON preview: +- **One Tools entry**: `Tools → Kindle Zotero Importer…` opens a manager window (`chrome://kindle-zotero-importer/content/manager.html`). +- **File picker + staged progress**: choose your cumulative `My Clippings.txt` → live stage, percent, elapsed, `plugin-progress.json` (`Reading clippings → Indexing Zotero → Matching → Building plan → Positioning EPUB → Positioning PDF → Finalizing → Saving`). +- **Incremental by default**: hashes each clipping `sha256(title|raw_detail|text)[:16]` (`src/kindle_zotero_importer/clippings.py:98`). Only new/changed `id`s (`new_ids - prev_integrated_ids`) go through expensive `pdftohtml`/`pdftotext` (`src/kindle_zotero_importer/pdf_position.py:120,197`). Already integrated `kindle-id:` tags (`src/kindle_zotero_importer/final_plan.py:44`) are skipped. Check `Full re-import from scratch` to ignore incremental and re-process all 2379. +- **Integrated tab**: last `import-plan.final.json` annotations (`975` in current artifacts) with `Kindle Title | Citekey | Highlight Text | Added On | Integrated | Page`, filterable, newest `Integrated` first. +- **Conflicts tab**: unresolved title matches (`matched`/`ignored` filtered out). Each row shows `Source | Status | Count | Kindle Title | Candidates / Detail` (`N suggestion(s) — top: citekey (%)`) and an explicit `candidate-list` — each candidate `citekey · title (score%)` with its own `Use` button (`saveCandidateAt`). Free-form `or enter any citekey / Zotero key / ID` + `Use Custom` + `Ignore Title`. If you map `chabot2013` for one `simondon` variant, it offers to apply the same mapping to other variants sharing that candidate and hides them immediately. +- **Mappings tab**: persistent `match-overrides.json` (`38` entries) with `Kindle Title | Resolution | Status | Count | Date | Action`, sorted by `updated_at` newest first (`plugin/bootstrap.js:288` `created_at`/`updated_at` ISO), filterable, `Delete` per row returns title to `Conflicts` after next `Re-import`. +- **Re-import bar**: after any `Use`/`Use Custom`/`Ignore` or `Delete`, a `Re-import with saved overrides` bar appears reusing `lastClippingsPath` (`plugin/bootstrap.js:734` `runManagedImportWithPath`) or prompting for file. Explains why re-import is needed (new `citation_key` → new attachment → new `epubcfi`/`pdf rect`/`sortIndex`). +- **Settings tab**: editable `Project directory`, `Python`, `Zotero DB`, `Zotero storage` (`setting-*` ids) + `Save Settings` → `plugin/bootstrap.js:315` `saveSettingsFromManager` writes prefs + `plugin-config.json`. +- **Artifacts tab**: per-row `Open`/`Reveal` for `docs/mismatch-review.md`, `match-overrides.json`, `import-plan.*.json`, `plugin-summary.json`. +- **Theme**: black `#0a0a0a` over white `#ffffff` `iA Writer Duo` monospace, `table-layout:fixed` with draggable `div.resizer` (`plugin/manager.html:270`) and sortable `th` (`▲/▼`) for all tables, selectable text. -```sh -python -m kindle_zotero_importer parse "/path/to/My Clippings.txt" --output clippings.json -``` - -Index Zotero items and PDF/EPUB attachments read-only: - -```sh -python -m kindle_zotero_importer index-zotero --output zotero-index.json -``` - -Match parsed Kindle titles to Zotero items: - -```sh -python -m kindle_zotero_importer match clippings.json zotero-index.json --overrides match-overrides.json --output matches.json -``` - -Generate reusable manual overrides for ambiguous/unmatched titles: - -```sh -python -m kindle_zotero_importer generate-overrides matches.json --output match-overrides.generated.json --pretty -``` - -Generated overrides are designed for batch editing: unresolved titles are sorted by clipping count, unmatched titles default to `ignore: true`, and already-entered overrides are kept at the end as an audit trail. Replace `ignore` with a confirmed `citation_key`, `zotero_key`, or `zotero_item_id` to import a title instead of discarding it. - -Apply reviewed overrides: - -```sh -python -m kindle_zotero_importer match clippings.json zotero-index.json --overrides match-overrides.json --output matches.json -``` - -Keep confirmed mappings in `match-overrides.json`. The `match-overrides.generated.json` file is only a disposable review skeleton and can be regenerated from the current `matches.json` at any time. - -To permanently skip a Kindle title that should not be imported, add an ignore override: - -```json -{ - "clipping_title": "Example Kindle Title", - "resolution": { - "ignore": true - } -} -``` - -If a matched Zotero item has multiple usable PDF/EPUB attachments, add the selected attachment to the same override entry: - -```json -{ - "clipping_title": "Example Kindle Title", - "resolution": { - "citation_key": "example2024", - "attachment_key": "ABC12345" - } -} -``` - -Use either `attachment_key` or `attachment_item_id`. Attachment choices are listed in `docs/mismatch-review.md` when an item is ambiguous. - -For cumulative Kindle exports, rerun the whole pipeline against the latest `My Clippings.txt`. Previously confirmed overrides in `match-overrides.json` will be applied automatically to old and new clippings. - -Review unresolved cases in `docs/mismatch-review.md` after each run. Prioritize: - -1. `matched-title-no-attachment`: add/link the missing PDF or EPUB in Zotero. -2. `unmatched-title`: add a confirmed title mapping to `match-overrides.json`. -3. position failures: improve text/position matching or handle manually. - -Regenerate the copy-friendly review report: - -```sh -python -m kindle_zotero_importer review-mismatches import-plan.positioned.json matches.json --output docs/mismatch-review.md -``` - -Each reviewed title includes a YAML-style block with standalone keys such as `clipping_title`, `citation_key`, `zotero_item_id`, `zotero_key`, and `override_entry` for easy copying. - -Generate a preliminary import plan for matched titles: - -```sh -python -m kindle_zotero_importer plan clippings.json zotero-index.json matches.json --output import-plan.json --pretty -``` - -Position annotations and export the Zotero writer plan: - -```sh -python -m kindle_zotero_importer position-epub import-plan.json --output import-plan.epub.json --pretty -python -m kindle_zotero_importer position-pdf import-plan.epub.json --output import-plan.positioned.json --pretty -python -m kindle_zotero_importer finalize import-plan.positioned.json --output import-plan.final.json --pretty -``` - -PDF positioning uses Poppler tools (`pdftotext`, `pdftohtml`, `pdfinfo`). If a PDF permits viewing but blocks text copying, the importer retries extraction through a temporary `qpdf --decrypt` copy. The Zotero attachment itself is not modified. - -## Safety Rule - -Do not write directly to `zotero.sqlite`. Use read-only SQLite access for indexing and Zotero's JavaScript API for writes. - -## Zotero Plugin Wrapper - -The `plugin/` directory contains a hybrid Zotero plugin with one user-facing Tools action. It lets the user choose `My Clippings.txt`, runs the importer with visible progress, and writes positioned annotations through Zotero's native APIs. Unresolved cases are reported after the import; intermediate JSON artifacts are implementation details. - -Build the plugin package: +## Install ```sh python scripts/build_plugin.py +# → dist/kindle-zotero-importer.xpi (manifest 0.6.4, Zotero 6.999–10.0.*) ``` -Install `dist/kindle-zotero-importer.xpi` in Zotero via `Tools > Plugins`. Open `Tools > Kindle Zotero Importer...`, choose `My Clippings.txt`, and keep the manager open to see the current stage, percentage, elapsed time, and final annotation counts. +In Zotero 10: `Tools → Plugins → gear → Install Plugin From File…` → `dist/kindle-zotero-importer.xpi` → restart → `Tools → Kindle Zotero Importer…`. -The XPI manifest targets Zotero `6.999` through `10.0.*`. +`plugin/manifest.json:11` `update_url` `https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates.json` enables auto-update. + +## Use + +1. `Tools → Kindle Zotero Importer…` → `Choose My Clippings.txt` (cumulative file). +2. Keep manager open for `Positioning … (incremental)` progress and `Result` (`created`/`already present`/`updated`/`failed`/`deletedForIncremental`). +3. `Conflicts` → pick `Use` per candidate or `Use Custom` with any `citation_key` (`deleuze1987`), 8-char `Zotero key` (`JTDWDKRH`), or numeric `item_id`; use `Ignore Title` for titles to skip. +4. `Re-import with saved overrides` (uses last file, or prompts) → re-matches/positions only the delta and writes `kindle-import` + `kindle-id:` tagged annotations. +5. `Integrated` to verify highlights (`Added On` Kindle date, `Integrated` UTC now, `Citekey` always visible), `Mappings` to review/delete established overrides. + +For title variants (`gilbert-simondon…` vs `On the Mode… (Univocal)` both candidate `simondon2017a`/`chabot2013`), mapping one offers to apply to the others sharing that candidate — or map them individually; they clear from `Conflicts` immediately and after `Re-import` are `matched` in `matches.json` and absent from `match-overrides.generated.json`. + +For full rebuild, check `Full re-import from scratch` in the run-panel. + +## CLI (for debugging) + +```sh +python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --db ~/Zotero/zotero.sqlite --storage-root ~/Zotero/storage --overrides match-overrides.json --summary-output plugin-summary.json --pretty +python -m kindle_zotero_importer run ... --full # ignore incremental +PYTHONPATH=src python -m kindle_zotero_importer parse "/path/to/My Clippings.txt" --pretty | head +``` + +`--progress-output plugin-progress.json` drives the manager progress bar. + +PDF positioning uses Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf --decrypt` fallback; EPUB uses `epubcfi`. + +## Safety Rule + +Never write directly to `zotero.sqlite`. Python is read-only for indexing; writes are only via `Zotero.Annotations.saveFromJSON` / `eraseTx` in `plugin/bootstrap.js:975` `writeAnnotations`. + +## Releases — including beta + +GitHub Releases are built from `dist/kindle-zotero-importer.xpi`: + +```sh +python scripts/build_plugin.py +# tag and push +git tag v0.6.4 && git push origin v0.6.4 +# GitHub → Releases → Draft a new release → Tag v0.6.4 → Title 0.6.4 → Attach dist/kindle-zotero-importer.xpi +# Generate updates.json: +# { +# "addons": { +# "kindlezoteroimporter@utkubilen.de": { +# "updates": [{ +# "version": "0.6.4", +# "update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.4/kindle-zotero-importer.xpi", +# "applications": {"zotero": {"strict_min_version": "6.999"}} +# }] +# } +# } +# } +# upload updates.json to the same release (update_url points to /releases/latest/download/updates.json) +``` + +For a **beta/pre-release**: on the GitHub Release form check `Set as a pre-release` and use a tag like `v0.6.4-beta.1` with `version` `0.6.4-beta.1` in `plugin/manifest.json:4` and `updates.json`. Zotero will offer it as an update only to users on that channel; stable `v0.6.4` stays `latest`. You can also mark `This is a pre-release` without changing `update_url` — `strict_max_version` `10.0.*` already allows beta testing in Zotero 10. + +## Donations + +Zotero has no built-in plugin donation. Add your sponsor link to `README.md` and `plugin/manifest.json:6` `homepage_url`, and to the manager `infobox` (`plugin/manager.html:485`). Recommended: GitHub Sponsors (`https://github.com/sponsors/UtkuBilenDemir`) or Ko-fi/PayPal/OpenCollective. The `Donate` button in `Plugins` manager comes from `aboutURL` if you add `"aboutURL": "https://github.com/sponsors/…"` to `manifest.json`. + +## Project layout + +- `plugin/` — hybrid bootstrap plugin (`bootstrap.js`, `manager.html`, `manifest.json`, `prefs.js`) +- `src/kindle_zotero_importer/` — `clippings.py`, `zotero_index.py`, `matcher.py`, `import_plan.py`, `epub_position.py`, `pdf_position.py`, `final_plan.py`, `cli.py` +- `scripts/build_plugin.py` — reproducible XPI builder +- `match-overrides.json` — persistent title → `citation_key`/`zotero_key`/`ignore` mappings (now with `created_at`/`updated_at`, sorted newest first)