docs: update README to 0.6.4 actual plugin (manager, incremental, Mappings)
- describe Tools -> Manager, 6 tabs, incremental, Full re-import, candidate-list - update Install and Use, CLI, Releases/beta, Donations
This commit is contained in:
Utku Bilen Demir
parent
bfe54e3953
commit
152f8235fc
1 changed files with 82 additions and 104 deletions
186
README.md
186
README.md
|
|
@ -1,116 +1,94 @@
|
|||
# Kindle Zotero Importer
|
||||
|
||||
Prototype importer for turning Kindle `My Clippings.txt` entries into a JSON import plan that Zotero can later write as native annotations.
|
||||
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations — directly inside Zotero, no terminal needed.
|
||||
|
||||
The project intentionally keeps Zotero writes out of Python. Python parses, matches, and plans. Zotero JavaScript, and eventually a Zotero plugin, will perform annotation creation through Zotero's own APIs.
|
||||
The hybrid Zotero plugin (`plugin/`) owns the UI and writes annotations through Zotero's native APIs. The Python pipeline (`src/kindle_zotero_importer/`) does parsing, matching, and EPUB/PDF positioning.
|
||||
|
||||
## Current Milestone
|
||||
## What it does now (0.6.4)
|
||||
|
||||
Parse Kindle clippings and emit a stable JSON preview:
|
||||
- **One Tools entry**: `Tools → Kindle Zotero Importer…` opens a manager window (`chrome://kindle-zotero-importer/content/manager.html`).
|
||||
- **File picker + staged progress**: choose your cumulative `My Clippings.txt` → live stage, percent, elapsed, `plugin-progress.json` (`Reading clippings → Indexing Zotero → Matching → Building plan → Positioning EPUB → Positioning PDF → Finalizing → Saving`).
|
||||
- **Incremental by default**: hashes each clipping `sha256(title|raw_detail|text)[:16]` (`src/kindle_zotero_importer/clippings.py:98`). Only new/changed `id`s (`new_ids - prev_integrated_ids`) go through expensive `pdftohtml`/`pdftotext` (`src/kindle_zotero_importer/pdf_position.py:120,197`). Already integrated `kindle-id:<id>` tags (`src/kindle_zotero_importer/final_plan.py:44`) are skipped. Check `Full re-import from scratch` to ignore incremental and re-process all 2379.
|
||||
- **Integrated tab**: last `import-plan.final.json` annotations (`975` in current artifacts) with `Kindle Title | Citekey | Highlight Text | Added On | Integrated | Page`, filterable, newest `Integrated` first.
|
||||
- **Conflicts tab**: unresolved title matches (`matched`/`ignored` filtered out). Each row shows `Source | Status | Count | Kindle Title | Candidates / Detail` (`N suggestion(s) — top: citekey (%)`) and an explicit `candidate-list` — each candidate `citekey · title (score%)` with its own `Use` button (`saveCandidateAt`). Free-form `or enter any citekey / Zotero key / ID` + `Use Custom` + `Ignore Title`. If you map `chabot2013` for one `simondon` variant, it offers to apply the same mapping to other variants sharing that candidate and hides them immediately.
|
||||
- **Mappings tab**: persistent `match-overrides.json` (`38` entries) with `Kindle Title | Resolution | Status | Count | Date | Action`, sorted by `updated_at` newest first (`plugin/bootstrap.js:288` `created_at`/`updated_at` ISO), filterable, `Delete` per row returns title to `Conflicts` after next `Re-import`.
|
||||
- **Re-import bar**: after any `Use`/`Use Custom`/`Ignore` or `Delete`, a `Re-import with saved overrides` bar appears reusing `lastClippingsPath` (`plugin/bootstrap.js:734` `runManagedImportWithPath`) or prompting for file. Explains why re-import is needed (new `citation_key` → new attachment → new `epubcfi`/`pdf rect`/`sortIndex`).
|
||||
- **Settings tab**: editable `Project directory`, `Python`, `Zotero DB`, `Zotero storage` (`setting-*` ids) + `Save Settings` → `plugin/bootstrap.js:315` `saveSettingsFromManager` writes prefs + `plugin-config.json`.
|
||||
- **Artifacts tab**: per-row `Open`/`Reveal` for `docs/mismatch-review.md`, `match-overrides.json`, `import-plan.*.json`, `plugin-summary.json`.
|
||||
- **Theme**: black `#0a0a0a` over white `#ffffff` `iA Writer Duo` monospace, `table-layout:fixed` with draggable `div.resizer` (`plugin/manager.html:270`) and sortable `th` (`▲/▼`) for all tables, selectable text.
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer parse "/path/to/My Clippings.txt" --output clippings.json
|
||||
```
|
||||
|
||||
Index Zotero items and PDF/EPUB attachments read-only:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer index-zotero --output zotero-index.json
|
||||
```
|
||||
|
||||
Match parsed Kindle titles to Zotero items:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer match clippings.json zotero-index.json --overrides match-overrides.json --output matches.json
|
||||
```
|
||||
|
||||
Generate reusable manual overrides for ambiguous/unmatched titles:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer generate-overrides matches.json --output match-overrides.generated.json --pretty
|
||||
```
|
||||
|
||||
Generated overrides are designed for batch editing: unresolved titles are sorted by clipping count, unmatched titles default to `ignore: true`, and already-entered overrides are kept at the end as an audit trail. Replace `ignore` with a confirmed `citation_key`, `zotero_key`, or `zotero_item_id` to import a title instead of discarding it.
|
||||
|
||||
Apply reviewed overrides:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer match clippings.json zotero-index.json --overrides match-overrides.json --output matches.json
|
||||
```
|
||||
|
||||
Keep confirmed mappings in `match-overrides.json`. The `match-overrides.generated.json` file is only a disposable review skeleton and can be regenerated from the current `matches.json` at any time.
|
||||
|
||||
To permanently skip a Kindle title that should not be imported, add an ignore override:
|
||||
|
||||
```json
|
||||
{
|
||||
"clipping_title": "Example Kindle Title",
|
||||
"resolution": {
|
||||
"ignore": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If a matched Zotero item has multiple usable PDF/EPUB attachments, add the selected attachment to the same override entry:
|
||||
|
||||
```json
|
||||
{
|
||||
"clipping_title": "Example Kindle Title",
|
||||
"resolution": {
|
||||
"citation_key": "example2024",
|
||||
"attachment_key": "ABC12345"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use either `attachment_key` or `attachment_item_id`. Attachment choices are listed in `docs/mismatch-review.md` when an item is ambiguous.
|
||||
|
||||
For cumulative Kindle exports, rerun the whole pipeline against the latest `My Clippings.txt`. Previously confirmed overrides in `match-overrides.json` will be applied automatically to old and new clippings.
|
||||
|
||||
Review unresolved cases in `docs/mismatch-review.md` after each run. Prioritize:
|
||||
|
||||
1. `matched-title-no-attachment`: add/link the missing PDF or EPUB in Zotero.
|
||||
2. `unmatched-title`: add a confirmed title mapping to `match-overrides.json`.
|
||||
3. position failures: improve text/position matching or handle manually.
|
||||
|
||||
Regenerate the copy-friendly review report:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer review-mismatches import-plan.positioned.json matches.json --output docs/mismatch-review.md
|
||||
```
|
||||
|
||||
Each reviewed title includes a YAML-style block with standalone keys such as `clipping_title`, `citation_key`, `zotero_item_id`, `zotero_key`, and `override_entry` for easy copying.
|
||||
|
||||
Generate a preliminary import plan for matched titles:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer plan clippings.json zotero-index.json matches.json --output import-plan.json --pretty
|
||||
```
|
||||
|
||||
Position annotations and export the Zotero writer plan:
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer position-epub import-plan.json --output import-plan.epub.json --pretty
|
||||
python -m kindle_zotero_importer position-pdf import-plan.epub.json --output import-plan.positioned.json --pretty
|
||||
python -m kindle_zotero_importer finalize import-plan.positioned.json --output import-plan.final.json --pretty
|
||||
```
|
||||
|
||||
PDF positioning uses Poppler tools (`pdftotext`, `pdftohtml`, `pdfinfo`). If a PDF permits viewing but blocks text copying, the importer retries extraction through a temporary `qpdf --decrypt` copy. The Zotero attachment itself is not modified.
|
||||
|
||||
## Safety Rule
|
||||
|
||||
Do not write directly to `zotero.sqlite`. Use read-only SQLite access for indexing and Zotero's JavaScript API for writes.
|
||||
|
||||
## Zotero Plugin Wrapper
|
||||
|
||||
The `plugin/` directory contains a hybrid Zotero plugin with one user-facing Tools action. It lets the user choose `My Clippings.txt`, runs the importer with visible progress, and writes positioned annotations through Zotero's native APIs. Unresolved cases are reported after the import; intermediate JSON artifacts are implementation details.
|
||||
|
||||
Build the plugin package:
|
||||
## Install
|
||||
|
||||
```sh
|
||||
python scripts/build_plugin.py
|
||||
# → dist/kindle-zotero-importer.xpi (manifest 0.6.4, Zotero 6.999–10.0.*)
|
||||
```
|
||||
|
||||
Install `dist/kindle-zotero-importer.xpi` in Zotero via `Tools > Plugins`. Open `Tools > Kindle Zotero Importer...`, choose `My Clippings.txt`, and keep the manager open to see the current stage, percentage, elapsed time, and final annotation counts.
|
||||
In Zotero 10: `Tools → Plugins → gear → Install Plugin From File…` → `dist/kindle-zotero-importer.xpi` → restart → `Tools → Kindle Zotero Importer…`.
|
||||
|
||||
The XPI manifest targets Zotero `6.999` through `10.0.*`.
|
||||
`plugin/manifest.json:11` `update_url` `https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/latest/download/updates.json` enables auto-update.
|
||||
|
||||
## Use
|
||||
|
||||
1. `Tools → Kindle Zotero Importer…` → `Choose My Clippings.txt` (cumulative file).
|
||||
2. Keep manager open for `Positioning … (incremental)` progress and `Result` (`created`/`already present`/`updated`/`failed`/`deletedForIncremental`).
|
||||
3. `Conflicts` → pick `Use` per candidate or `Use Custom` with any `citation_key` (`deleuze1987`), 8-char `Zotero key` (`JTDWDKRH`), or numeric `item_id`; use `Ignore Title` for titles to skip.
|
||||
4. `Re-import with saved overrides` (uses last file, or prompts) → re-matches/positions only the delta and writes `kindle-import` + `kindle-id:<hash>` tagged annotations.
|
||||
5. `Integrated` to verify highlights (`Added On` Kindle date, `Integrated` UTC now, `Citekey` always visible), `Mappings` to review/delete established overrides.
|
||||
|
||||
For title variants (`gilbert-simondon…` vs `On the Mode… (Univocal)` both candidate `simondon2017a`/`chabot2013`), mapping one offers to apply to the others sharing that candidate — or map them individually; they clear from `Conflicts` immediately and after `Re-import` are `matched` in `matches.json` and absent from `match-overrides.generated.json`.
|
||||
|
||||
For full rebuild, check `Full re-import from scratch` in the run-panel.
|
||||
|
||||
## CLI (for debugging)
|
||||
|
||||
```sh
|
||||
python -m kindle_zotero_importer run "/path/to/My Clippings.txt" --workdir . --db ~/Zotero/zotero.sqlite --storage-root ~/Zotero/storage --overrides match-overrides.json --summary-output plugin-summary.json --pretty
|
||||
python -m kindle_zotero_importer run ... --full # ignore incremental
|
||||
PYTHONPATH=src python -m kindle_zotero_importer parse "/path/to/My Clippings.txt" --pretty | head
|
||||
```
|
||||
|
||||
`--progress-output plugin-progress.json` drives the manager progress bar.
|
||||
|
||||
PDF positioning uses Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`) plus `qpdf --decrypt` fallback; EPUB uses `epubcfi`.
|
||||
|
||||
## Safety Rule
|
||||
|
||||
Never write directly to `zotero.sqlite`. Python is read-only for indexing; writes are only via `Zotero.Annotations.saveFromJSON` / `eraseTx` in `plugin/bootstrap.js:975` `writeAnnotations`.
|
||||
|
||||
## Releases — including beta
|
||||
|
||||
GitHub Releases are built from `dist/kindle-zotero-importer.xpi`:
|
||||
|
||||
```sh
|
||||
python scripts/build_plugin.py
|
||||
# tag and push
|
||||
git tag v0.6.4 && git push origin v0.6.4
|
||||
# GitHub → Releases → Draft a new release → Tag v0.6.4 → Title 0.6.4 → Attach dist/kindle-zotero-importer.xpi
|
||||
# Generate updates.json:
|
||||
# {
|
||||
# "addons": {
|
||||
# "kindlezoteroimporter@utkubilen.de": {
|
||||
# "updates": [{
|
||||
# "version": "0.6.4",
|
||||
# "update_link": "https://github.com/UtkuBilenDemir/kindle-zotero_importer/releases/download/v0.6.4/kindle-zotero-importer.xpi",
|
||||
# "applications": {"zotero": {"strict_min_version": "6.999"}}
|
||||
# }]
|
||||
# }
|
||||
# }
|
||||
# }
|
||||
# upload updates.json to the same release (update_url points to /releases/latest/download/updates.json)
|
||||
```
|
||||
|
||||
For a **beta/pre-release**: on the GitHub Release form check `Set as a pre-release` and use a tag like `v0.6.4-beta.1` with `version` `0.6.4-beta.1` in `plugin/manifest.json:4` and `updates.json`. Zotero will offer it as an update only to users on that channel; stable `v0.6.4` stays `latest`. You can also mark `This is a pre-release` without changing `update_url` — `strict_max_version` `10.0.*` already allows beta testing in Zotero 10.
|
||||
|
||||
## Donations
|
||||
|
||||
Zotero has no built-in plugin donation. Add your sponsor link to `README.md` and `plugin/manifest.json:6` `homepage_url`, and to the manager `infobox` (`plugin/manager.html:485`). Recommended: GitHub Sponsors (`https://github.com/sponsors/UtkuBilenDemir`) or Ko-fi/PayPal/OpenCollective. The `Donate` button in `Plugins` manager comes from `aboutURL` if you add `"aboutURL": "https://github.com/sponsors/…"` to `manifest.json`.
|
||||
|
||||
## Project layout
|
||||
|
||||
- `plugin/` — hybrid bootstrap plugin (`bootstrap.js`, `manager.html`, `manifest.json`, `prefs.js`)
|
||||
- `src/kindle_zotero_importer/` — `clippings.py`, `zotero_index.py`, `matcher.py`, `import_plan.py`, `epub_position.py`, `pdf_position.py`, `final_plan.py`, `cli.py`
|
||||
- `scripts/build_plugin.py` — reproducible XPI builder
|
||||
- `match-overrides.json` — persistent title → `citation_key`/`zotero_key`/`ignore` mappings (now with `created_at`/`updated_at`, sorted newest first)
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue