Compare commits

..

No commits in common. "main" and "v0.7.0" have entirely different histories.
main ... v0.7.0

19 changed files with 156 additions and 2820 deletions

View file

@ -1,7 +1,7 @@
> [!CAUTION]
> **This project has been heavily vibecoded.**
# Kanzi — `K`indle `an`notations to `Z`otero `I`mporter
# Kanzi — Kindle annotations to Zotero
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero. Once the required host tools are installed, imports run from the manager without CLI commands. Formerly *Kindle Zotero Importer*.
@ -17,39 +17,16 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
3. `Tools` → `Kanzi…` to open the manager.
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. Kanzi detects standard Python installations automatically on macOS, Linux and Windows.
## Windows
Open PowerShell and install the host tools:
```powershell
winget install --id Python.Python.3.13 --exact
winget install --id oschwartz10612.Poppler --exact
winget install --id QPDF.QPDF --exact # Optional: handles copy-protected PDFs
```
Fully quit and reopen Zotero after installation so it receives the updated `PATH`, then install `kanzi.xpi` and open `Tools` → `Kanzi…`. Kanzi checks standard per-user and system Python locations as well as `PATH`; you should not need to enter an executable path manually. EPUB-only imports require Python but not Poppler.
To verify the installation in a new PowerShell window:
```powershell
python --version
pdftotext -v
pdftohtml -v
pdfinfo -v
qpdf --version # Optional
```
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. macOS and Linux are tested. Windows requires setting the full Python executable path in `Settings` and installing the PDF tools separately.
# Use
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
2. **Review before writing.** New highlights are grouped by title and selected by default. Untick individual highlights or whole titles, then press `Integrate selected`. Skipped highlights return on the next run. `Cancel` and closing the manager write nothing to Zotero and do not mark pending highlights as integrated.
3. **Check Integrated.** After integration, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
4. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows one expandable row for that title with up to three suggestions `citekey · title (score%)`. Expand it to inspect individual highlights and positioning problems. Click `Use` on the right candidate, enter a citation key, Zotero key or item ID under `Use Custom`, or choose `Ignore Title`. Tables can be sorted and resized.
5. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new, changed, deferred or removed highlights are processed. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
6. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
7. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
@ -85,11 +62,11 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
`Tools` → `Kanzi…` opens the manager.
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. After positioning, the pending review lets you select exactly what is written to Zotero.
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
**Conflicts.** One row per unresolved Kindle title, combining title-match and attachment/position statuses. Expand a row to inspect its highlights and positioning problems. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. Columns are sortable and resizable.
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.

506
plugin/bootstrap.js vendored
View file

@ -1,4 +1,4 @@
var Kanzi = {
var KindleZoteroImporter = {
id: "kindle-zotero-importer@utkubilen.de",
menuItems: [],
runtimeProjectDir: "",
@ -16,7 +16,7 @@ var Kanzi = {
);
this.chromeHandle = addonManagerStartup.registerChrome(
Services.io.newURI(this.rootURI + "manifest.json"),
[["content", "kanzi", ""]]
[["content", "kindle-zotero-importer", ""]]
);
await Zotero.initializationPromise;
await this.ensureRuntime();
@ -81,7 +81,7 @@ var Kanzi = {
}
const doc = win.document;
if (doc.getElementById("kanzi-manager")) {
if (doc.getElementById("kindle-zotero-importer-manager")) {
return;
}
@ -92,7 +92,7 @@ var Kanzi = {
}
const managerItem = doc.createXULElement("menuitem");
managerItem.setAttribute("id", "kanzi-manager");
managerItem.setAttribute("id", "kindle-zotero-importer-manager");
managerItem.setAttribute("label", "Kanzi...");
managerItem.addEventListener("command", () => this.openManager());
popup.appendChild(managerItem);
@ -136,14 +136,7 @@ var Kanzi = {
},
getPythonPath() {
const fallback = this.getDefaultPython();
const value = this.getPref("pythonPath", fallback);
// Existing Windows installs may have "python3" stored from the old
// Unix-only default; that binary name does not exist on Windows.
if (this.isWindows() && value === "python3") {
return "python";
}
return value || fallback;
return this.getPref("pythonPath", "python3");
},
getZoteroDbPath() {
@ -156,115 +149,12 @@ var Kanzi = {
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
},
isWindows() {
try {
return Services.appinfo.OS === "WINNT";
} catch (_error) {
return false;
}
},
getDefaultPython() {
return this.isWindows() ? "python" : "python3";
},
getEnvironmentVariable(name) {
try {
return Cc["@mozilla.org/process/environment;1"]
.getService(Ci.nsIEnvironment)
.get(name) || "";
} catch (_error) {
return "";
}
},
joinWindowsPath(base, ...parts) {
if (!base) {
return "";
}
return [String(base).replace(/[\\/]+$/, ""), ...parts]
.map((part, index) => index ? String(part).replace(/^[\\/]+|[\\/]+$/g, "") : part)
.join("\\");
},
getWindowsPythonCandidates(configuredPath) {
const localAppData = this.getEnvironmentVariable("LOCALAPPDATA");
const programFiles = this.getEnvironmentVariable("ProgramFiles");
const programFilesX86 = this.getEnvironmentVariable("ProgramFiles(x86)");
const userProfile = this.getEnvironmentVariable("USERPROFILE");
const systemDrive = this.getEnvironmentVariable("SystemDrive") || "C:";
const versions = ["314", "313", "312", "311"];
const candidates = [configuredPath];
for (const version of versions) {
candidates.push(
this.joinWindowsPath(localAppData, "Programs", "Python", `Python${version}`, "python.exe"),
this.joinWindowsPath(programFiles, `Python${version}`, "python.exe"),
this.joinWindowsPath(programFilesX86, `Python${version}-32`, "python.exe"),
this.joinWindowsPath(systemDrive, `Python${version}`, "python.exe")
);
}
candidates.push(
this.joinWindowsPath(localAppData, "Python", "bin", "python.exe"),
this.joinWindowsPath(userProfile, "scoop", "apps", "python", "current", "python.exe"),
this.joinWindowsPath(userProfile, "miniconda3", "python.exe"),
this.joinWindowsPath(userProfile, "anaconda3", "python.exe")
);
for (const directory of this.getEnvironmentVariable("PATH").split(";")) {
const cleanDirectory = directory.trim().replace(/^"|"$/g, "");
if (cleanDirectory) {
candidates.push(
this.joinWindowsPath(cleanDirectory, "python.exe"),
this.joinWindowsPath(cleanDirectory, "python3.exe")
);
}
}
return [...new Set(candidates.filter(Boolean))];
},
async resolvePythonPath(configuredPath) {
let candidates;
if (this.isWindows()) {
candidates = this.getWindowsPythonCandidates(configuredPath);
} else if (Services.appinfo.OS === "Darwin") {
candidates = [configuredPath];
candidates.push(
"/opt/homebrew/bin/python3",
"/opt/homebrew/opt/python@3.14/bin/python3.14",
"/opt/homebrew/opt/python@3.13/bin/python3.13",
"/opt/homebrew/opt/python@3.12/bin/python3.12",
"/opt/homebrew/opt/python@3.11/bin/python3.11",
"/usr/local/bin/python3",
"/usr/local/opt/python@3.14/bin/python3.14",
"/usr/local/opt/python@3.13/bin/python3.13",
"/usr/local/opt/python@3.12/bin/python3.12",
"/usr/local/opt/python@3.11/bin/python3.11",
"/usr/bin/python3"
);
} else {
candidates = [configuredPath, "/usr/bin/python3", "/usr/local/bin/python3"];
}
for (const candidate of candidates) {
if (!candidate || (!candidate.includes("/") && !candidate.includes("\\"))) continue;
try {
if (await IOUtils.exists(candidate)) return candidate;
} catch (_error) {}
}
return configuredPath || this.getDefaultPython();
},
joinPath(...parts) {
if (typeof PathUtils !== "undefined" && PathUtils.join) {
return PathUtils.join(...parts);
}
return parts.join("/").replaceAll("//", "/");
},
getConfigPath() {
return this.joinPath(this.getProjectDir(), "plugin-config.json");
return this.getProjectDir() + "/plugin-config.json";
},
artifactPath(relativePath) {
return this.joinPath(this.getProjectDir(), ...String(relativePath).split("/"));
return this.getProjectDir() + "/" + relativePath;
},
async openArtifact(relativePath) {
@ -287,6 +177,17 @@ var Kanzi = {
zoteroDbPath: this.getZoteroDbPath(),
zoteroStorageRoot: this.getZoteroStorageRoot(),
};
if (settings.pythonPath === "python3") {
const candidates = Services.appinfo.OS === "Darwin"
? ["/opt/homebrew/bin/python3", "/usr/local/bin/python3", "/usr/bin/python3"]
: ["/usr/bin/python3", "/usr/local/bin/python3"];
for (const candidate of candidates) {
if (await IOUtils.exists(candidate)) {
settings.pythonPath = candidate;
break;
}
}
}
try {
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
const parsed = JSON.parse(text);
@ -299,13 +200,12 @@ var Kanzi = {
if (parsed.colorMap) settings.colourMap = parsed.colorMap;
// also check colour-map.json for Python pipeline
try {
const cmapText = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
const cmapText = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json");
settings.colourMap = JSON.parse(cmapText);
} catch (_e2) {}
} catch (_error) {
// The config file is optional; defaults/preferences are enough for first run.
}
settings.pythonPath = await this.resolvePythonPath(settings.pythonPath);
return settings;
},
@ -367,8 +267,8 @@ var Kanzi = {
const data = await this.loadManagerData();
const io = { plugin: this, settings, data };
this.managerWindow = this.getMainWindow().openDialog(
"chrome://kanzi/content/manager.html",
"kanzi-manager-window",
"chrome://kindle-zotero-importer/content/manager.html",
"kindle-zotero-importer-manager-window",
"chrome,centerscreen,resizable",
io
);
@ -376,7 +276,6 @@ var Kanzi = {
async loadManagerData() {
return {
clippings: await this.readJsonArtifact("clippings.json"),
matches: await this.readJsonArtifact("matches.json"),
generatedOverrides: await this.readJsonArtifact("match-overrides.generated.json"),
positionedPlan: await this.readJsonArtifact("import-plan.positioned.json"),
@ -406,17 +305,11 @@ var Kanzi = {
const index = payload.overrides.findIndex(
(entry) => entry.clipping_title === resolution.clipping_title
);
const clippings = await this.readJsonArtifact("clippings.json");
const sourceCount = this.countClippingsForTitle(clippings, resolution.clipping_title);
const clippingCount = sourceCount || resolution.clipping_count || resolution.review?.clipping_count || 0;
const nowIso = new Date().toISOString();
const entry = {
clipping_title: resolution.clipping_title,
resolution: resolution.resolution,
review: {
...(resolution.review || { status: "manager" }),
clipping_count: clippingCount,
},
review: resolution.review || { status: "manager", clipping_count: resolution.clipping_count || 0 },
created_at: nowIso,
updated_at: nowIso,
};
@ -436,13 +329,6 @@ var Kanzi = {
return entry;
},
countClippingsForTitle(payload, clippingTitle) {
return (payload?.clippings || []).reduce(
(count, clipping) => count + (clipping.title === clippingTitle ? 1 : 0),
0
);
},
async deleteOverride(clippingTitle) {
if (!clippingTitle) {
throw new Error("Missing clipping title for delete");
@ -460,23 +346,6 @@ var Kanzi = {
return true;
},
async discardClip(clippingId) {
if (!clippingId) {
throw new Error("Missing clipping id for discard");
}
const path = this.artifactPath("import-plan.final.json");
const plan = JSON.parse(await Zotero.File.getContentsAsync(path));
const discarded = new Set(plan.discarded_clipping_ids || []);
discarded.add(clippingId);
plan.discarded_clipping_ids = [...discarded].sort();
// Also drop from deferred so it is never re-queued.
if (Array.isArray(plan.deferred_clipping_ids)) {
plan.deferred_clipping_ids = plan.deferred_clipping_ids.filter((id) => id !== clippingId);
}
await Zotero.File.putContentsAsync(path, JSON.stringify(plan, null, 2) + "\n");
return true;
},
async saveSettingsFromManager(newSettings) {
const toSave = {
projectDir: newSettings.projectDir || this.getProjectDir(),
@ -489,7 +358,7 @@ var Kanzi = {
// also write colour-map.json for Python pipeline (mirrors plugin-config)
try {
await Zotero.File.putContentsAsync(
this.joinPath(this.getProjectDir(), "colour-map.json"),
this.getProjectDir() + "/colour-map.json",
JSON.stringify(newSettings.colourMap, null, 2) + "\n"
);
} catch (_e) {}
@ -507,7 +376,7 @@ var Kanzi = {
async loadColourMap() {
try {
const text = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
const text = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json");
return JSON.parse(text);
} catch (_e) {}
try {
@ -547,7 +416,7 @@ var Kanzi = {
if (result !== fp.returnOK) {
return null;
}
return this.filePickerResultToPath(fp.file);
return fp.file;
} catch (error) {
Zotero.logError(error);
return this.promptForPath(
@ -558,32 +427,12 @@ var Kanzi = {
}
},
filePickerResultToPath(file) {
if (!file) {
return null;
}
if (typeof file === "string") {
return file;
}
// Mozilla FilePicker returns an nsIFile; Python needs a native string path.
if (typeof file.path === "string" && file.path) {
return file.path;
}
try {
const str = String(file);
if (str && str !== "[object Object]") {
return str;
}
} catch (_error) {}
return null;
},
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
const python = settings.pythonPath || this.getDefaultPython();
const python = settings.pythonPath;
const projectDir = settings.projectDir;
const runner = this.joinPath(projectDir, "plugin_runner.py");
const summaryOutput = this.joinPath(projectDir, "plugin-summary.json");
const progressOutput = this.joinPath(projectDir, "plugin-progress.json");
const runner = projectDir + "/plugin_runner.py";
const summaryOutput = projectDir + "/plugin-summary.json";
const progressOutput = projectDir + "/plugin-progress.json";
const args = [
runner,
"run",
@ -609,34 +458,12 @@ var Kanzi = {
args.push("--retry-unresolved");
}
let execResult = null;
try {
execResult = await this.exec(python, args);
} catch (error) {
const installHint = this.isWindows()
? `Install Python 3.11 or newer, fully restart Zotero, and let Kanzi detect python.exe. `
: `Install Python 3.11 or newer and check the Python path in Kanzi Settings. `;
throw new Error(
`Python could not run the Kanzi pipeline (${python}): ${error && error.message ? error.message : error}. ` +
installHint
);
}
try {
await this.exec(python, args);
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
} catch (error) {
const detail = execResult ? ` Python output: ${JSON.stringify(execResult).slice(0, 1000)}.` : "";
throw new Error(
`Python pipeline did not write ${summaryOutput}.${detail} ` +
`Python: ${python}. Runner: ${runner}. ` +
`Make sure Python 3 is installed and reachable, then check the project directory is writable.`
);
}
},
async runManagedImport(manager) {
const clippingsPath = await this.pickFile("Select Kindle My Clippings.txt", "*.txt");
// NOTE: writing now happens only after the user reviews the pending
// highlights and presses Integrate (see integratePending).
if (!clippingsPath) {
return;
}
@ -647,23 +474,17 @@ var Kanzi = {
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
try {
clippingsPath = this.filePickerResultToPath(clippingsPath) || clippingsPath;
if (!clippingsPath) {
throw new Error("No clippings file selected");
}
const settings = await this.getRuntimeSettings();
const progressPath = this.joinPath(settings.projectDir, "plugin-progress.json");
const progressPath = settings.projectDir + "/plugin-progress.json";
// Determine full vs incremental from explicit arg or manager checkbox
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
? retryUnresolvedOverride
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
manager.beginRun(clippingsPath);
const finalPlanPath = this.joinPath(settings.projectDir, "import-plan.final.json");
let previousFinalPlan = null;
try {
previousFinalPlan = JSON.parse(await Zotero.File.getContentsAsync(finalPlanPath));
} catch (_backupError) {}
try {
await Zotero.File.removeIfExists(progressPath);
} catch (_error) {}
@ -690,142 +511,20 @@ var Kanzi = {
const finalPlan = JSON.parse(
await Zotero.File.getContentsAsync(summary.outputs.final_plan)
);
const pendingCount = (finalPlan.annotations || []).length;
const pendingDeletions = (finalPlan.deletions || []).length;
if (!pendingCount && !pendingDeletions) {
// Nothing new — run the maintenance sweep and finish as before.
manager.updateProgress(96, "Writing annotations", "Checking existing annotations");
const emptyResults = await this.writeAnnotations(finalPlan, false, (completed, total) => {
manager.updateProgress(96, "Writing annotations", "Saving annotations in Zotero");
const results = await this.writeAnnotations(finalPlan, false, (completed, total) => {
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
});
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, emptyResults);
const emptyData = await this.loadManagerData();
await manager.completeRun(summary, emptyResults, emptyData);
return;
}
// Pause before writing: let the user review and select what to integrate.
const fullDeltaIds = (finalPlan.annotations || []).map((a) => a.clipping_id);
// The pipeline has already written its generated history. Replace it with
// a safe ledger until Integrate is pressed, including if the window closes.
const deferredPlan = this.deferPendingPlan(previousFinalPlan, finalPlan, fullDeltaIds);
await Zotero.File.putContentsAsync(
summary.outputs.final_plan,
JSON.stringify(deferredPlan, null, 2) + "\n"
);
this._pending = {
summary,
finalPlan,
finalPlanPath: summary.outputs.final_plan,
fullDeltaIds,
previousFinalPlan,
};
manager.updateProgress(
90,
"Review pending",
`${pendingCount} highlight(s) ready — select and press Integrate`
);
manager.showPendingReview(summary, finalPlan);
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, results);
const data = await this.loadManagerData();
await manager.completeRun(summary, results, data);
} catch (error) {
Zotero.logError(error);
manager.failRun(String(error && error.stack ? error.stack : error));
}
},
deferPendingPlan(previousPlan, generatedPlan, deferredIds) {
const plan = previousPlan
? { ...previousPlan }
: {
...generatedPlan,
annotations: [],
all_annotations: [],
deletions: [],
annotation_count: 0,
delta_annotation_count: 0,
};
const deferred = new Set(plan.deferred_clipping_ids || []);
for (const id of deferredIds || []) deferred.add(id);
plan.deferred_clipping_ids = [...deferred].sort();
return plan;
},
// Pure helper (tested): narrow a pipeline plan to the user's selection.
filterPlanForSelection(finalPlan, fullDeltaIds, selectedIds, previousPlan = null) {
const selected = new Set(selectedIds || []);
const unselected = new Set(
(fullDeltaIds || []).filter((id) => !selected.has(id))
);
const plan = { ...finalPlan };
plan.annotations = (finalPlan.annotations || []).filter((a) =>
selected.has(a.clipping_id)
);
if (unselected.size && Array.isArray(finalPlan.all_annotations)) {
plan.all_annotations = finalPlan.all_annotations.filter(
(a) => !unselected.has(a.clipping_id)
);
const present = new Set(plan.all_annotations.map((a) => a.clipping_id));
for (const annotation of previousPlan?.all_annotations || previousPlan?.annotations || []) {
if (unselected.has(annotation.clipping_id) && !present.has(annotation.clipping_id)) {
plan.all_annotations.push(annotation);
present.add(annotation.clipping_id);
}
}
}
const deferred = new Set(previousPlan?.deferred_clipping_ids || []);
for (const id of selected) deferred.delete(id);
for (const id of unselected) deferred.add(id);
plan.deferred_clipping_ids = [...deferred].sort();
plan.delta_annotation_count = plan.annotations.length;
if (Array.isArray(plan.all_annotations)) {
plan.annotation_count = plan.all_annotations.length;
}
return plan;
},
async integratePending(manager, selectedIds) {
const pending = this._pending;
if (!pending) {
throw new Error("No pending review to integrate");
}
try {
const plan = this.filterPlanForSelection(
pending.finalPlan,
pending.fullDeltaIds,
selectedIds,
pending.previousFinalPlan
);
manager.updateProgress(96, "Writing annotations", "Saving selected annotations in Zotero");
const results = await this.writeAnnotations(plan, false, (completed, total) => {
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
});
await this.reconcileWriterState(pending.finalPlanPath, plan, results);
this._pending = null;
const data = await this.loadManagerData();
await manager.completeRun(pending.summary, results, data);
} catch (error) {
Zotero.logError(error);
manager.failRun(String(error && error.stack ? error.stack : error));
}
},
async cancelPendingReview(manager) {
// The deferred ledger was written before the review appeared, so closing
// or cancelling never records pending highlights as integrated.
this._pending = null;
manager.hidePendingReview();
manager.cancelRun("Review cancelled — nothing was written. Run again when ready.");
try {
const data = await this.loadManagerData();
manager.data = data;
manager.rows = manager.conflictRows();
manager.refreshRows(true);
manager.refreshTitleOverview();
manager.refreshIntegrated(true);
manager.refreshMappings(true);
} catch (_reloadError) {}
},
async exec(command, args) {
if (!Zotero.Utilities || !Zotero.Utilities.Internal || !Zotero.Utilities.Internal.exec) {
throw new Error("Zotero.Utilities.Internal.exec is unavailable in this Zotero build");
@ -1115,117 +814,6 @@ var Kanzi = {
throw new Error("Existing annotation cannot be saved by this Zotero build");
},
async saveRelatedItem(item) {
const options = { skipDateModifiedUpdate: true };
if (item.save) {
await item.save(options);
return;
}
if (item.saveTx) {
await item.saveTx(options);
return;
}
throw new Error("Related item cannot be saved by this Zotero build");
},
async addRelatedItemRelations(entries, dryRun, results) {
const pairs = new Map();
for (const entry of entries || []) {
for (const target of entry.related_items || []) {
const sourceID = Number(entry.parent_item_id);
const targetID = Number(target.item_id);
if (!sourceID || !targetID || sourceID === targetID) continue;
const pairKey = [sourceID, targetID].sort((a, b) => a - b).join(":");
if (!pairs.has(pairKey)) {
pairs.set(pairKey, { sourceID, targetID, citationKey: target.citation_key || "" });
}
}
}
for (const pair of pairs.values()) {
try {
const source = Zotero.Items.get(pair.sourceID);
const target = Zotero.Items.get(pair.targetID);
if (!source || !target) {
throw new Error(`Related item pair not found: ${pair.sourceID} -> ${pair.targetID}`);
}
if (dryRun) {
results.relationsPlanned += 1;
continue;
}
if (!source.addRelatedItem || !target.addRelatedItem) {
throw new Error("This Zotero build does not support addRelatedItem()");
}
let changed = false;
const addBothDirections = async () => {
const sourceChanged = source.addRelatedItem(target);
const targetChanged = target.addRelatedItem(source);
if (sourceChanged) await this.saveRelatedItem(source);
if (targetChanged) await this.saveRelatedItem(target);
changed = sourceChanged || targetChanged;
};
if (Zotero.DB?.executeTransaction) {
await Zotero.DB.executeTransaction(addBothDirections);
} else {
await addBothDirections();
}
if (changed) results.relationsAdded += 1;
else results.relationsSkipped += 1;
} catch (error) {
Zotero.logError(error);
results.relationFailures.push({
parent_item_id: pair.sourceID,
related_item_id: pair.targetID,
citation_key: pair.citationKey,
message: String(error),
});
}
}
},
async backfillRelationLinks(entries, dryRun, results) {
// Appends missing `Related: @key <link>` blocks to pre-existing
// annotations. Cross-library targets can never become Related items,
// so the comment link is the only visible connection for them.
for (const entry of entries || []) {
const targets = entry.related_items || [];
if (!targets.length || !entry.clipping_id || !entry.attachment_item_id) continue;
const block = "Related: " + targets.map(
(target) => `@${target.citation_key} ${target.link || ""}`.trim()
).join(" · ");
try {
const attachment = Zotero.Items.get(entry.attachment_item_id);
if (!attachment) {
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
}
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
const tag = `kindle-id:${entry.clipping_id}`;
const existing = (annotations || []).find((annotation) =>
this.existingTags(annotation).includes(tag)
);
if (!existing) continue;
const comment = String(existing.annotationComment || "");
if (comment.includes(block)) continue;
if (dryRun) {
results.commentLinksPlanned += 1;
continue;
}
existing.annotationComment = comment ? `${comment}\n\n${block}` : block;
await this.saveExistingAnnotation(existing);
results.commentLinksAdded += 1;
} catch (error) {
Zotero.logError(error);
results.relationFailures.push({
parent_item_id: entry.parent_item_id,
related_item_id: (targets[0] || {}).item_id,
citation_key: (targets[0] || {}).citation_key,
message: `comment link: ${error}`,
});
}
}
},
async reconcileWriterState(path, plan, results) {
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
const failedIds = new Set(
@ -1273,12 +861,6 @@ var Kanzi = {
blockedRecreateMissingTag: 0,
recreateDeclined: 0,
removedDuplicates: 0,
relationsAdded: 0,
relationsSkipped: 0,
relationsPlanned: 0,
commentLinksAdded: 0,
commentLinksPlanned: 0,
relationFailures: [],
deletedForIncremental: 0,
failedDeletionIds: [],
maintenanceDeletedIds: [],
@ -1463,14 +1045,6 @@ var Kanzi = {
}
}
const currentIDs = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
const relationEntries = (plan.all_annotations || []).filter(
(entry) => !currentIDs.has(entry.clipping_id)
);
relationEntries.push(...successfulEntries.values());
await this.addRelatedItemRelations(relationEntries, dryRun, results);
await this.backfillRelationLinks(relationEntries, dryRun, results);
if (!dryRun && plan.deletions && plan.deletions.length) {
const deletionIds = new Set(plan.deletions);
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
@ -1572,7 +1146,7 @@ function uninstall(_data, _reason) {}
async function startup(data, _reason) {
try {
await Kanzi.startup(data);
await KindleZoteroImporter.startup(data);
} catch (error) {
Zotero.logError(error);
}
@ -1580,7 +1154,7 @@ async function startup(data, _reason) {
function shutdown(_data, _reason) {
try {
Kanzi.shutdown();
KindleZoteroImporter.shutdown();
} catch (error) {
Zotero.logError(error);
}
@ -1588,7 +1162,7 @@ function shutdown(_data, _reason) {
function onMainWindowLoad({ window }) {
try {
Kanzi.onMainWindowLoad(window);
KindleZoteroImporter.onMainWindowLoad(window);
} catch (error) {
Zotero.logError(error);
}
@ -1596,7 +1170,7 @@ function onMainWindowLoad({ window }) {
function onMainWindowUnload({ window }) {
try {
Kanzi.onMainWindowUnload(window);
KindleZoteroImporter.onMainWindowUnload(window);
} catch (error) {
Zotero.logError(error);
}

File diff suppressed because it is too large Load diff

View file

@ -1,7 +1,7 @@
{
"manifest_version": 2,
"name": "Kanzi",
"version": "0.7.1",
"version": "0.7.0",
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
"author": "Utku Bilen Demir",

View file

@ -1,6 +1,6 @@
[project]
name = "kindle-zotero-importer"
version = "0.7.1"
version = "0.7.0"
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
readme = "README.md"
requires-python = ">=3.11"

View file

@ -1,7 +1,6 @@
from __future__ import annotations
import argparse
import copy
import json
import sys
from pathlib import Path
@ -9,12 +8,7 @@ from pathlib import Path
from .clippings import clippings_to_jsonable, load_clippings
from .epub_position import add_epub_positions
from .final_plan import FINAL_FORMAT, build_final_writer_plan
from .import_plan import (
DEFAULT_COLOUR_MAP,
_attach_notes_to_highlights,
_related_items_for_citekeys,
build_import_plan,
)
from .import_plan import DEFAULT_COLOUR_MAP, _attach_notes_to_highlights, build_import_plan
from .matcher import build_match_report, load_json
from .mismatch_review import build_mismatch_review
from .overrides import generate_override_skeleton, load_overrides
@ -71,20 +65,6 @@ def _merge_annotation_history(
)
def _refresh_history_relations(
annotations: list[dict], zotero_index: dict
) -> None:
"""Resolve citekeys for old annotations without re-positioning them."""
items_by_id = {item["item_id"]: item for item in zotero_index.get("items", [])}
for entry in annotations:
annotation = entry.get("annotation") or {}
entry["related_items"] = _related_items_for_citekeys(
(annotation.get("text") or "", annotation.get("comment") or ""),
entry.get("parent_item_id"),
items_by_id,
)
def _changed_note_clipping_ids(
current_clippings: list[dict], previous_annotations: list[dict]
) -> set[str]:
@ -132,137 +112,6 @@ def _deletion_ids(
return deleted
def _deferred_clipping_ids(
previous_plan: dict | None, current_ids: set[str]
) -> set[str]:
return {
clipping_id
for clipping_id in (previous_plan or {}).get("deferred_clipping_ids", [])
if clipping_id in current_ids
}
def _discarded_clipping_ids(
previous_plan: dict | None, current_ids: set[str]
) -> set[str]:
"""Clips the user permanently discarded in Conflicts (e.g. standalone notes)."""
return {
clipping_id
for clipping_id in (previous_plan or {}).get("discarded_clipping_ids", [])
if clipping_id in current_ids
}
def _changed_override_titles(
previous_snapshot: dict | None, current_overrides: dict
) -> set[str]:
if previous_snapshot is None:
return set(current_overrides)
return {
title
for title in set(previous_snapshot) | set(current_overrides)
if previous_snapshot.get(title) != current_overrides.get(title)
}
def _clipping_ids_for_titles(
clippings: list[dict], titles: set[str]
) -> set[str]:
return {clipping["id"] for clipping in clippings if clipping["title"] in titles}
def _reattempt_failed_with_alternate_attachments(positioned_plan: dict) -> dict:
"""Retry text/rect failures against another attachment of the same item.
`_choose_attachment` commits to one file (e.g. the sole EPUB) without
comparing text. When that file lacks the quote but a sibling PDF/EPUB
has it (e.g. Dostoyevsky EPUB vs PDF), re-anchor the clone there and
keep the winner, tagged `positioned-via-alternate-attachment`.
Partial shingle anchors are also upgraded when a sibling holds the
full quote.
"""
retryable = {"epub-text-not-found", "pdf-text-not-found", "pdf-rects-not-found"}
retries: list[dict] = []
upgrade_ids: set[str] = set()
for item in positioned_plan.get("items", []):
if item.get("status") in retryable:
pass
elif item.get("status") == "positioned" and "partial-text-match" in (
item.get("problems") or []
):
clipping_id = (item.get("clipping") or {}).get("id")
if clipping_id:
upgrade_ids.add(clipping_id)
else:
continue
zotero = item.get("zotero") or {}
current = zotero.get("attachment") or {}
choices = zotero.get("attachment_choices") or []
alternates = [
choice
for choice in choices
if choice.get("item_id") != current.get("item_id")
]
if not alternates:
continue
alternates.sort(
key=lambda choice: 0
if choice.get("content_type") != current.get("content_type")
else 1
)
clone = copy.deepcopy(item)
clone_zotero = dict(clone.get("zotero") or {})
clone_zotero["attachment"] = alternates[0]
clone["zotero"] = clone_zotero
clone["status"] = "ready-for-positioning"
clone["problems"] = []
retries.append(clone)
if not retries:
return positioned_plan
mini = {"format": positioned_plan.get("format"), "items": retries}
mini = add_epub_positions(mini)
mini = add_pdf_positions(mini)
winners = {}
for attempt in mini.get("items", []):
clipping_id = (attempt.get("clipping") or {}).get("id")
if clipping_id and attempt.get("status") == "positioned":
winners[clipping_id] = attempt
if not winners:
return positioned_plan
merged = []
for item in positioned_plan.get("items", []):
clipping_id = (item.get("clipping") or {}).get("id")
if clipping_id in winners:
won = winners[clipping_id]
if clipping_id in upgrade_ids and "partial-text-match" in (
won.get("problems") or []
):
merged.append(item)
continue
won = dict(won)
carried = [
problem
for problem in item.get("problems", [])
if problem != "partial-text-match"
or "partial-text-match" in (won.get("problems") or [])
]
won["problems"] = [
*carried,
"positioned-via-alternate-attachment",
]
merged.append(won)
else:
merged.append(item)
updated = dict(positioned_plan)
updated["items"] = merged
status_counts: dict[str, int] = {}
for entry in merged:
status = entry.get("status", "unknown")
status_counts[status] = status_counts.get(status, 0) + 1
updated["status_counts"] = status_counts
return updated
def _add_previous_imported_comments(
annotations: list[dict], previous_plan: dict | None
) -> None:
@ -285,12 +134,7 @@ def _unresolved_clipping_ids(
return {
item.get("clipping", {}).get("id")
for item in (previous_plan or {}).get("items", [])
if item.get("status") not in {
"positioned",
"ignored-title",
"pdf-position-skipped-non-highlight",
"epub-position-skipped-non-highlight",
}
if item.get("status") not in {"positioned", "ignored-title"}
and item.get("clipping", {}).get("id") in source_ids
}
@ -691,12 +535,18 @@ def main(argv: list[str] | None = None) -> int:
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
except Exception:
_ov_for_inc = {}
changed_override_titles = _changed_override_titles(
(prev_final_plan or {}).get("overrides_snapshot"), _ov_for_inc
)
to_process_ids.update(
_clipping_ids_for_titles(effective_clippings, changed_override_titles)
)
previous_override_snapshot = (prev_final_plan or {}).get("overrides_snapshot")
if previous_override_snapshot is None:
changed_override_titles = set(_ov_for_inc)
else:
changed_override_titles = {
title
for title in set(previous_override_snapshot) | set(_ov_for_inc)
if previous_override_snapshot.get(title) != _ov_for_inc.get(title)
}
for c in effective_clippings:
if c["title"] in changed_override_titles:
to_process_ids.add(c["id"])
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
for annotation in previous_annotations:
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
@ -713,15 +563,10 @@ def main(argv: list[str] | None = None) -> int:
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
if clipping_id in new_ids
)
to_process_ids.update(_deferred_clipping_ids(prev_final_plan, new_ids))
if args.retry_unresolved and prev_positioned_plan:
to_process_ids.update(
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
)
# Permanently discarded clips (user action in Conflicts) never re-queue.
to_process_ids.difference_update(
_discarded_clipping_ids(prev_final_plan, new_ids)
)
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
try:
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
@ -787,7 +632,6 @@ def main(argv: list[str] | None = None) -> int:
epub_plan = add_epub_positions(plan)
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
positioned_plan = add_pdf_positions(epub_plan)
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan)
# Attach deletions for writer
@ -821,7 +665,6 @@ def main(argv: list[str] | None = None) -> int:
epub_plan = add_epub_positions(plan)
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
positioned_plan = add_pdf_positions(epub_plan)
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan)
final_plan["deletions"] = sorted(
@ -857,14 +700,10 @@ def main(argv: list[str] | None = None) -> int:
final_plan.get("annotations", []),
final_plan.get("deletions", []),
)
_refresh_history_relations(final_plan["all_annotations"], zotero_index)
final_plan["annotation_count"] = len(final_plan["all_annotations"])
final_plan["delta_annotation_count"] = delta_annotation_count
final_plan["processed_clipping_ids"] = sorted(new_ids)
final_plan["overrides_snapshot"] = overrides or {}
final_plan["discarded_clipping_ids"] = sorted(
_discarded_clipping_ids(prev_final_plan, new_ids)
)
indent = 2 if args.pretty else None
outputs = {

View file

@ -4,7 +4,6 @@ from dataclasses import dataclass
from html import unescape
from html.parser import HTMLParser
import posixpath
import unicodedata
import zipfile
import xml.etree.ElementTree as ET
from typing import Any
@ -69,8 +68,6 @@ def _position_item(
annotation["sortIndex"] = position["sortIndex"]
updated["annotation"] = annotation
updated["status"] = "positioned"
if position.get("partial"):
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
return updated
@ -107,42 +104,11 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
return None
match_start = normalized_haystack.find(normalized_quote)
raw_start: int | None = None
raw_end: int | None = None
partial = False
if match_start >= 0:
if match_start < 0:
return None
match_end = match_start + len(normalized_quote)
raw_start = haystack_map[match_start]
raw_end = haystack_map[match_end - 1] + 1
else:
stripped_quote, _ = _strip_with_map(normalized_quote)
if stripped_quote and len(stripped_quote) >= 20:
stripped_haystack, stripped_map = _strip_with_map(normalized_haystack)
strip_start = stripped_haystack.find(stripped_quote)
strip_len = len(stripped_quote)
if strip_start < 0:
hit = _partial_stripped_range(stripped_haystack, stripped_quote)
if hit is not None:
strip_start, strip_end = hit
strip_len = strip_end - strip_start
partial = True
if strip_start >= 0:
# stripped_map points into normalized_haystack;
# convert via haystack_map back to raw haystack offsets.
norm_start = stripped_map[strip_start]
norm_end = stripped_map[strip_start + strip_len - 1]
raw_start = haystack_map[norm_start]
raw_end = haystack_map[norm_end] + 1
else:
span = _shingle_span(normalized_haystack, _quote_words(normalized_quote))
if span is not None:
raw_start = haystack_map[span[0]]
raw_end = haystack_map[span[1] - 1] + 1
partial = True
else:
partial = False
if raw_start is None or raw_end is None:
return None
# Account for the spaces inserted between text nodes in haystack construction.
adjusted_nodes = _nodes_for_joined_text(text_nodes)
@ -173,7 +139,6 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
"value": value,
},
"sortIndex": f"{spine_step:05d}|{raw_start:08d}",
"partial": partial,
}
@ -248,11 +213,6 @@ def _clean_text(text: str) -> str:
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text)
text = (
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
)
normalized = []
mapping = []
last_was_space = False
@ -272,58 +232,6 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
return "".join(normalized), mapping
def _strip_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text).casefold()
stripped: list[str] = []
mapping: list[int] = []
for index, char in enumerate(text):
if char.isalnum():
stripped.append(char)
mapping.append(index)
return "".join(stripped), mapping
def _partial_stripped_range(
haystack: str, quote: str, min_len: int = 30
) -> tuple[int, int] | None:
if len(quote) < min_len * 2:
return None
mid = len(quote) // 2
for part in (quote[:mid], quote[mid:]):
if len(part) < min_len:
continue
start = haystack.find(part)
if start >= 0:
return start, start + len(part)
return None
def _quote_words(normalized: str) -> list[str]:
words: list[str] = []
for token in normalized.split():
cleaned = "".join(char for char in token if char.isalnum())
if cleaned:
words.append(cleaned)
return words
def _shingle_span(
normalized_text: str, quote_words: list[str], width: int = 6
) -> tuple[int, int] | None:
first: int | None = None
last: int | None = None
for start in range(len(quote_words) - width + 1):
needle = " ".join(quote_words[start : start + width])
at = normalized_text.find(needle)
if at >= 0:
end = at + len(needle)
first = at if first is None else min(first, at)
last = end if last is None else max(last, end)
if first is None or last is None:
return None
return first, last
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
nodes = []
cursor = 0

View file

@ -71,7 +71,6 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
"attachment_key": attachment["key"],
"parent_item_id": item["zotero"]["parent_item_id"],
"parent_key": item["zotero"]["parent_key"],
"related_items": item["zotero"].get("related_items", []),
"citation_key": item["zotero"].get("citation_key"),
"doi": item["zotero"].get("doi"),
"isbn": item["zotero"].get("isbn"),

View file

@ -124,12 +124,6 @@ def _plan_clipping(
zotero_item, overrides_by_title.get(clipping["title"]) or {}
)
status = "ready-for-positioning" if attachment else attachment_status
stub = _annotation_stub(clipping, colour_map) if attachment else None
related_items = (
_append_citekey_links(stub, clipping, zotero_item, items_by_id)
if stub
else []
)
return {
"clipping": clipping,
"status": status,
@ -144,96 +138,17 @@ def _plan_clipping(
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
"url": zotero_item["fields"].get("url"),
"publication_title": zotero_item["fields"].get("publicationTitle"),
"related_items": related_items,
"attachment": attachment,
"attachment_choices": zotero_item.get("attachments", []),
"expected_attachment_type": _expected_attachment_type(
clipping, zotero_item.get("attachments", [])
),
},
"annotation": stub,
"annotation": _annotation_stub(clipping, colour_map) if attachment else None,
"problems": problems,
}
def _append_citekey_links(
stub: dict[str, Any],
clipping: dict[str, Any],
parent_item: dict[str, Any],
items_by_id: dict[int, dict[str, Any]],
) -> list[dict[str, Any]]:
"""Turn `@citekey` mentions into clickable Zotero item links.
Kindle-side `@key` tokens (Obsidian-style) resolve against the library
index; matches to *other* items are appended to the annotation comment
as `zotero://select` links, which Zotero renders clickable. Self-links
to the parent item are skipped.
"""
related_items = _related_items_for_citekeys(
(clipping.get("text") or "", clipping.get("comment") or ""),
parent_item.get("item_id"),
items_by_id,
)
links = [
f"@{target['citation_key']} {target['link']}" for target in related_items
]
if not links:
return []
existing = stub.get("comment") or ""
block = "Related: " + " · ".join(links)
if block not in existing:
stub["comment"] = f"{existing}\n\n{block}" if existing else block
return related_items
def _select_link(item: dict[str, Any]) -> str:
"""Library-aware `zotero://select` link for an indexed item.
Group-library targets need the `/groups/{groupID}/items/{key}` form;
personal-library (or unknown) targets use `/library/items/{key}`.
"""
group_id = item.get("group_id")
if group_id:
return f"zotero://select/groups/{group_id}/items/{item['key']}"
return f"zotero://select/library/items/{item['key']}"
def _related_items_for_citekeys(
sources: tuple[str, ...],
parent_item_id: int | None,
items_by_id: dict[int, dict[str, Any]],
) -> list[dict[str, Any]]:
tokens = {
token.casefold()
for source in sources
for token in re.findall(r"@([A-Za-z0-9_][\w:.-]*)", source or "")
}
if not tokens:
return []
by_citekey = {
str(item.get("fields", {}).get("citationKey", "")).casefold(): item
for item in items_by_id.values()
if item.get("fields", {}).get("citationKey")
}
related = []
seen_item_ids = set()
for token in sorted(tokens):
target = by_citekey.get(token)
target_id = target.get("item_id") if target else None
if not target or target_id == parent_item_id or target_id in seen_item_ids:
continue
seen_item_ids.add(target_id)
related.append(
{
"item_id": target_id,
"key": target["key"],
"citation_key": target["fields"]["citationKey"],
"link": _select_link(target),
}
)
return related
def _choose_attachment(
zotero_item: dict[str, Any],
override: dict[str, Any] | None = None,

View file

@ -7,7 +7,6 @@ import re
import shutil
import subprocess
import tempfile
import unicodedata
import xml.etree.ElementTree as ET
from pathlib import Path
from typing import Any
@ -127,12 +126,7 @@ def _position_item(
text_pages, item["clipping"]["text"], item["clipping"].get("page")
)
if page_index is None:
failed = _with_problem(item, "pdf-text-not-found")
hint = _page_hint_text(text_pages, item["clipping"].get("page"))
if hint:
failed = dict(failed)
failed["page_hint"] = hint
return failed
return _with_problem(item, "pdf-text-not-found")
page_xml = xml_cache.setdefault(
(extraction_path, page_index),
extract_pdf_page_xml(extraction_path, page_index),
@ -158,8 +152,6 @@ def _position_item(
annotation["sortIndex"] = position.get("sortIndex")
updated["annotation"] = annotation
updated["status"] = "positioned"
if position.get("partial"):
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
return updated
@ -235,37 +227,6 @@ def find_pdf_text_page(
normalized_page, _ = _normalize_with_map(pages[index])
if normalized_quote in normalized_page:
return index
# Fallback: alphanumeric-only search tolerates PDF extraction artifacts
# such as "o f" for "of", hyphenation, and punctuation differences.
stripped_quote, _ = _strip_with_map(quote)
if stripped_quote and len(stripped_quote) >= 20:
for index in page_order:
stripped_page, _ = _strip_with_map(pages[index])
if stripped_quote in stripped_page:
return index
# Partial fallback: long highlights may span pages or contain a
# divergent middle (edition differences). Anchor to either half so
# Zotero-style word search behaviour is approximated.
for index in page_order:
stripped_page, _ = _strip_with_map(pages[index])
if _partial_stripped_hit(stripped_page, stripped_quote):
return index
# Shingle fallback: scattered OCR substitutions (e.g. Kindle
# "yadsınaraz" vs PDF "yadsınamaz") break every long substring but
# leave most 6-word runs intact. Rank pages by shingle hits.
quote_words = _quote_words(normalized_quote)
if len(quote_words) >= 6:
required = 2 if len(quote_words) >= 12 else 1
best_index: int | None = None
best_hits = 0
for index in page_order:
normalized_page, _ = _normalize_with_map(pages[index])
hits = _shingle_hits(normalized_page, quote_words)
if hits > best_hits:
best_hits = hits
best_index = index
if best_index is not None and best_hits >= required:
return best_index
return None
@ -329,52 +290,25 @@ def find_pdf_rects(
text = " ".join(word.text for word in page.words)
normalized_text, text_map = _normalize_with_map(text)
start = normalized_text.find(normalized_quote)
raw_start: int | None = None
raw_end: int | None = None
partial = False
if start >= 0:
if start < 0:
return None
end = start + len(normalized_quote)
raw_start = text_map[start]
raw_end = text_map[end - 1] + 1
else:
stripped_quote, _ = _strip_with_map(quote)
if stripped_quote and len(stripped_quote) >= 20:
stripped_text, stripped_map = _strip_with_map(text)
strip_start = stripped_text.find(stripped_quote)
if strip_start >= 0:
strip_end = strip_start + len(stripped_quote)
raw_start = stripped_map[strip_start]
raw_end = stripped_map[strip_end - 1] + 1
else:
hit = _partial_stripped_range(stripped_text, stripped_quote)
if hit is not None:
strip_start, strip_end = hit
raw_start = stripped_map[strip_start]
raw_end = stripped_map[strip_end - 1] + 1
partial = True
else:
span = _shingle_span(normalized_text, _quote_words(normalized_quote))
if span is not None:
raw_start = text_map[span[0]]
raw_end = text_map[span[1] - 1] + 1
partial = True
if raw_start is None or raw_end is None:
return None
matched_words = [
word for word in page.words if word.end >= raw_start and word.start <= raw_end
]
if not matched_words:
return None
size_key = (path, page.index)
if size_key not in size_cache:
size_cache[size_key] = _pdf_page_size(path, page.index + 1, page.width, page.height)
pdf_width, pdf_height = size_cache[size_key]
pdf_width, pdf_height = size_cache.setdefault(
(path, page.index),
_pdf_page_size(path, page.index + 1, page.width, page.height),
)
rects = _word_rects(matched_words, page, pdf_width, pdf_height)
return {
"pageIndex": page.index,
"rects": rects,
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
"partial": partial,
}
@ -427,11 +361,6 @@ def _pdf_page_size(
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text)
text = (
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
)
normalized = []
mapping = []
last_was_space = False
@ -451,112 +380,6 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
return "".join(normalized), mapping
def _strip_with_map(text: str) -> tuple[str, list[int]]:
"""Alphanumeric-only folding with index mapping.
Removes spaces and punctuation so PDF artifacts like "o f" for "of",
hyphenated line breaks, and curly-quote/dash variants still match.
Mapping points back into the input string for rect recovery.
"""
text = unicodedata.normalize("NFKC", text).casefold()
stripped: list[str] = []
mapping: list[int] = []
for index, char in enumerate(text):
if char.isalnum():
stripped.append(char)
mapping.append(index)
return "".join(stripped), mapping
def _partial_stripped_range(
haystack: str, quote: str, min_len: int = 30
) -> tuple[int, int] | None:
"""Best half-quote hit for long highlights spanning pages/editions."""
if len(quote) < min_len * 2:
return None
mid = len(quote) // 2
for part in (quote[:mid], quote[mid:]):
if len(part) < min_len:
continue
start = haystack.find(part)
if start >= 0:
return start, start + len(part)
return None
def _partial_stripped_hit(haystack: str, quote: str, min_len: int = 30) -> bool:
return _partial_stripped_range(haystack, quote, min_len) is not None
def _quote_words(normalized: str, width: int = 6) -> list[str]:
del width
words: list[str] = []
for token in normalized.split():
cleaned = "".join(char for char in token if char.isalnum())
if cleaned:
words.append(cleaned)
return words
def _shingle_hits(normalized_page: str, quote_words: list[str], width: int = 6) -> int:
hits = 0
seen: set[tuple[str, ...]] = set()
for start in range(len(quote_words) - width + 1):
shingle = tuple(quote_words[start : start + width])
if shingle in seen:
continue
seen.add(shingle)
if " ".join(shingle) in normalized_page:
hits += 1
return hits
def _shingle_span(
normalized_text: str, quote_words: list[str], width: int = 6
) -> tuple[int, int] | None:
first: int | None = None
last: int | None = None
for start in range(len(quote_words) - width + 1):
needle = " ".join(quote_words[start : start + width])
at = normalized_text.find(needle)
if at >= 0:
end = at + len(needle)
first = at if first is None else min(first, at)
last = end if last is None else max(last, end)
if first is None or last is None:
return None
return first, last
def _page_hint_text(
pages: list[str], kindle_page: str | None, radius: int = 1, limit: int = 1200
) -> dict[str, Any] | None:
"""Snippet of the PDF pages around the Kindle page hint.
Lets the user eyeball which part a `pdf-text-not-found` highlight
belongs to and place it manually. Truncated to keep artifacts small.
"""
if not kindle_page or not kindle_page.isdigit():
return None
center = int(kindle_page) - 1
if center < 0 or center >= len(pages):
return None
parts = []
for index in range(max(0, center - radius), min(len(pages), center + radius + 1)):
snippet = " ".join(pages[index].split())
if snippet:
parts.append(f"[PDF p.{index + 1}] {snippet[:limit]}")
if not parts:
return None
return {
"kindle_page": kindle_page,
"pdf_pages": list(
range(max(1, center - radius + 1), min(len(pages), center + radius + 1) + 1)
),
"text": "\n\n".join(parts)[: limit * 2 + 64],
}
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
updated = dict(item)
updated["status"] = problem

View file

@ -34,8 +34,6 @@ class ZoteroItem:
key: str
item_type: str
title: str | None
library_id: int | None = None
group_id: int | None = None
creators: list[str] = field(default_factory=list)
fields: dict[str, str] = field(default_factory=dict)
attachments: list[ZoteroAttachment] = field(default_factory=list)
@ -49,19 +47,15 @@ def build_zotero_index(db_path: str, storage_root: str | None = None) -> dict[st
creators = _load_creators(connection)
attachments = _load_attachments(connection, storage_root)
groups = _load_groups(connection)
indexed_items: list[ZoteroItem] = []
for row in items:
fields = _metadata_for_item(row)
library_id = row["libraryID"]
indexed_items.append(
ZoteroItem(
item_id=row["itemID"],
key=row["key"],
item_type=row["typeName"],
title=fields.get("title"),
library_id=library_id,
group_id=groups.get(library_id),
creators=creators.get(row["itemID"], []),
fields=fields,
attachments=attachments.get(row["itemID"], []),
@ -86,7 +80,6 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
SELECT
i.itemID,
i.key,
i.libraryID,
it.typeName,
{field_columns}
FROM items i
@ -102,14 +95,6 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
return list(connection.execute(query))
def _load_groups(connection: sqlite3.Connection) -> dict[int, int]:
try:
rows = connection.execute("SELECT groupID, libraryID FROM groups")
except Exception:
return {}
return {row["libraryID"]: row["groupID"] for row in rows}
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
rows = connection.execute(
"""

View file

@ -1,15 +0,0 @@
Kanzi Dummy EPUB
- Your Highlight on page 1 | location 100-120 | Added on Tuesday, 18 May 2021 14:18:13
This is a test highlight long enough to import for EPUB.
==========
Kanzi Dummy PDF
- Your Highlight on page 2 | location 200-220 | Added on Tuesday, 18 May 2021 14:20:00
This is a test highlight long enough to import for PDF.
==========
Unknown Book Title XYZ
- Your Highlight on page 1 | location 10-11 | Added on Tuesday, 18 May 2021 14:18:13
This should land in Conflicts for manual mapping.
==========

View file

@ -1,36 +0,0 @@
Kanzi Windows smoke test (Parallels VM)
COPY THESE 3 FILES INTO THE VM:
- My Clippings.txt
- kanzi-dummy.epub
- kanzi-dummy.pdf
- dist/kanzi.xpi (from repo root, same cross-platform build)
1) PowerShell:
winget install --id Python.Python.3.13 --exact
winget install --id oschwartz10612.Poppler --exact
winget install --id QPDF.QPDF --exact
python --version
pdftotext -v
pdfinfo -v
qpdf --version
2) Fully quit + reopen Zotero (picks up PATH).
3) Fresh Zotero, sync off. Create:
- Book "Kanzi Dummy EPUB" + attach kanzi-dummy.epub
- Book "Kanzi Dummy PDF" + attach kanzi-dummy.pdf
4) Tools -> Kanzi -> Choose My Clippings.txt (the copied one).
Leave Python path empty to test auto-detect.
EXPECT:
- 2 highlights position and await review, 1 title in Conflicts
("Unknown Book Title XYZ").
- Summary shows new Status by Kindle Title table with source counts.
- Save a mapping for the unknown title -> Re-import processes it.
- Table headers sort and persist across pages.
IF PYTHON NOT FOUND:
Settings shows detected path hint; paste full python.exe path,
Save Settings, retry.

Binary file not shown.

View file

@ -1,33 +0,0 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
endobj
4 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
endobj
5 0 obj
<< /Length 134 >>
stream
BT /F1 12 Tf 72 720 Td (Kanzi Dummy PDF) Tj ET
BT /F1 12 Tf 72 700 Td (This is a test highlight long enough to import for PDF.) Tj ET
endstream
endobj
xref
0 6
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000241 00000 n
0000000311 00000 n
trailer
<< /Size 6 /Root 1 0 R >>
startxref
495
%%EOF

View file

@ -1,93 +0,0 @@
"""Generate minimal EPUB + PDF fixtures for the Windows smoke test.
Run: python3 tests/fixtures/windows-smoke/make_fixtures.py
Outputs kanzi-dummy.epub and kanzi-dummy.pdf next to this script.
Both contain the highlight sentences from 'My Clippings.txt' so
EPUB CFI and PDF text positioning can succeed.
"""
from __future__ import annotations
from pathlib import Path
import zipfile
HERE = Path(__file__).resolve().parent
EPUB_TEXT = "This is a test highlight long enough to import for EPUB."
PDF_TEXT = "This is a test highlight long enough to import for PDF."
def make_epub(path: Path) -> None:
chapter = f"""<?xml version="1.0" encoding="utf-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head><title>Kanzi Dummy</title></head>
<body>
<h1>Kanzi Dummy EPUB</h1>
<p>Some intro text before the highlight.</p>
<p>{EPUB_TEXT}</p>
<p>Some trailing text after the highlight.</p>
</body>
</html>
"""
container = """<?xml version="1.0" encoding="utf-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
</container>
"""
opf = """<?xml version="1.0" encoding="utf-8"?>
<package version="3.0" unique-identifier="uid" xmlns="http://www.idpf.org/2007/opf">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Kanzi Dummy EPUB</dc:title><dc:identifier id="uid">kanzi-dummy-epub</dc:identifier><dc:language>en</dc:language></metadata>
<manifest><item id="ch1" href="chapter1.xhtml" media-type="application/xhtml+xml"/></manifest>
<spine><itemref idref="ch1"/></spine>
</package>
"""
with zipfile.ZipFile(path, "w") as z:
z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
z.writestr("META-INF/container.xml", container, compress_type=zipfile.ZIP_DEFLATED)
z.writestr("OEBPS/content.opf", opf, compress_type=zipfile.ZIP_DEFLATED)
z.writestr("OEBPS/chapter1.xhtml", chapter, compress_type=zipfile.ZIP_DEFLATED)
def _pdf_escape(text: str) -> str:
return text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
def make_pdf(path: Path) -> None:
# Minimal one-page PDF with Helvetica text. pdftotext extracts it fine.
content = (
f"BT /F1 12 Tf 72 720 Td ({_pdf_escape('Kanzi Dummy PDF')}) Tj ET\n"
f"BT /F1 12 Tf 72 700 Td ({_pdf_escape(PDF_TEXT)}) Tj ET\n"
)
objects: list[bytes] = []
objects.append(b"<< /Type /Catalog /Pages 2 0 R >>")
objects.append(b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
objects.append(
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"
)
objects.append(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
objects.append(f"<< /Length {len(content.encode('latin-1'))} >>\nstream\n{content}endstream".encode("latin-1"))
out = bytearray(b"%PDF-1.4\n")
offsets = [0]
for i, body in enumerate(objects, start=1):
offsets.append(len(out))
out += f"{i} 0 obj\n".encode("ascii") + body + b"\nendobj\n"
xref_pos = len(out)
out += f"xref\n0 {len(objects) + 1}\n".encode("ascii")
out += b"0000000000 65535 f \n"
for off in offsets[1:]:
out += f"{off:010d} 00000 n \n".encode("ascii")
out += (
f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
f"startxref\n{xref_pos}\n%%EOF\n"
).encode("ascii")
path.write_bytes(bytes(out))
def main() -> None:
make_epub(HERE / "kanzi-dummy.epub")
make_pdf(HERE / "kanzi-dummy.pdf")
print("wrote kanzi-dummy.epub + kanzi-dummy.pdf")
if __name__ == "__main__":
main()

View file

@ -28,206 +28,10 @@ def test_pdf_fallback_tag():
"format":"x",
"items":[
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1","related_items":[{"item_id":20,"key":"P2","citation_key":"c2"}]},
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
]
}
plan = build_final_writer_plan(positioned)
tags = plan["annotations"][0]["annotation"]["tags"]
assert any(t["name"].startswith("kindle-id:") for t in tags)
assert plan["annotations"][0]["related_items"][0]["item_id"] == 20
def test_pdf_tolerates_of_split_artifact():
from kindle_zotero_importer.pdf_position import find_pdf_text_page, find_pdf_rects, PDFPageXML, PDFWord
quote = "nothing other than idolatry of the machine and which"
page_text = "nothing other than idolatry o f the machine and which"
assert find_pdf_text_page([page_text], quote) == 0
words = []
cursor = 0
for w in page_text.split():
words.append(PDFWord(text=w, left=0.0, top=0.0, width=10.0, height=10.0, start=cursor, end=cursor + len(w)))
cursor += len(w) + 1
page = PDFPageXML(index=0, width=100.0, height=100.0, words=words)
rects = find_pdf_rects("dummy.pdf", page, quote, {("dummy.pdf", 0): (100.0, 100.0)})
assert rects is not None
assert rects["pageIndex"] == 0
assert rects["rects"]
def test_pdf_tolerates_punctuation_and_quotes():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
quote = "This, then, gives rise to “intemperate technicism”."
page = "this then gives rise to intemperate technicism"
assert find_pdf_text_page([page], quote) == 0
def test_pdf_partial_half_anchors_spanning_text():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
first = "A" * 70
second = "Z" * 70
quote = first + second
page = "xx " + first.lower() + " yy"
assert find_pdf_text_page([page], quote) == 0
def test_pdf_shingle_fallback_scattered_ocr_errors():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
quote = (
"Gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
"buna inanmamasım sağlayacak güç ve yeteneği bulur mucize yadsınaraz "
"dururda bile olsa boyun eğmez"
)
page = (
"gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
"buna inanmaması sağlayacak güç ve yeteneği bulur mucize yadsınamaz "
"dururda bile olsa boyun eğmez"
)
assert find_pdf_text_page([page], quote) == 0
def test_page_hint_snippet_around_kindle_page():
from kindle_zotero_importer.pdf_position import _page_hint_text
pages = ["", "intro words here", "target page words here", "trailing words"]
hint = _page_hint_text(pages, "3")
assert hint is not None
assert hint["kindle_page"] == "3"
assert "target page words" in hint["text"]
assert _page_hint_text(pages, None) is None
assert _page_hint_text(pages, "99") is None
def test_citekey_mentions_become_zotero_links():
from kindle_zotero_importer.import_plan import build_import_plan
clippings_payload = {
"format": "x",
"clippings": [
{
"id": "c1",
"title": "Book A",
"kind": "highlight",
"text": "As argued by @bookb2020, this follows.",
"page": None,
"location": "10",
}
],
}
zotero_index = {
"format": "x",
"items": [
{
"item_id": 1,
"key": "PARENT01",
"title": "Book A",
"fields": {"citationKey": "booka2020"},
"attachments": [
{
"item_id": 11,
"key": "ATTACH01",
"title": "pdf",
"content_type": "application/pdf",
"path": "x.pdf",
"resolved_path": "x.pdf",
}
],
},
{
"item_id": 2,
"key": "TARGET02",
"title": "Book B",
"fields": {"citationKey": "bookb2020"},
"attachments": [],
},
],
}
match_report = {
"format": "x",
"matches": [
{
"clipping_title": "Book A",
"status": "matched",
"candidates": [{"item_id": 1}],
}
],
}
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
comment = plan["items"][0]["annotation"]["comment"] or ""
assert "@bookb2020" in comment
assert "zotero://select/library/items/TARGET02" in comment
assert plan["items"][0]["zotero"]["related_items"] == [
{
"item_id": 2,
"key": "TARGET02",
"citation_key": "bookb2020",
"link": "zotero://select/library/items/TARGET02",
}
]
# self-links are skipped
clippings_payload["clippings"][0]["text"] = "See @booka2020 here."
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
assert "zotero://select" not in (plan["items"][0]["annotation"]["comment"] or "")
assert plan["items"][0]["zotero"]["related_items"] == []
def test_historical_annotations_gain_citekey_relations():
from kindle_zotero_importer.cli import _refresh_history_relations
history = [
{
"clipping_id": "old",
"parent_item_id": 1,
"annotation": {
"text": "An old highlight",
"comment": "body without organs @deleuze1983",
},
}
]
zotero_index = {
"items": [
{
"item_id": 1,
"key": "SOURCE01",
"fields": {"citationKey": "simondon2017a"},
},
{
"item_id": 2,
"key": "TARGET02",
"fields": {"citationKey": "deleuze1983"},
},
]
}
_refresh_history_relations(history, zotero_index)
assert history[0]["related_items"] == [
{
"item_id": 2,
"key": "TARGET02",
"citation_key": "deleuze1983",
"link": "zotero://select/library/items/TARGET02",
}
]
def test_group_library_targets_use_group_select_links():
from kindle_zotero_importer.import_plan import _select_link
assert (
_select_link({"key": "ABC123", "group_id": 6644344})
== "zotero://select/groups/6644344/items/ABC123"
)
assert (
_select_link({"key": "ABC123", "group_id": None})
== "zotero://select/library/items/ABC123"
)
assert (
_select_link({"key": "ABC123"})
== "zotero://select/library/items/ABC123"
)
def test_epub_tolerates_extraction_artifacts():
from kindle_zotero_importer.epub_position import find_epub_cfi, TextNode
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="idolatry o f the machine", start=0, end=24)]
quote = "idolatry of the machine and which"
# quote is longer than node, so exact fails; use full haystack variant
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="nothing other than idolatry o f the machine and which", start=0, end=53)]
result = find_epub_cfi(nodes, quote)
assert result is not None
assert result["selector"]["value"].startswith("epubcfi(")

View file

@ -6,11 +6,7 @@ from kindle_zotero_importer import cli
from kindle_zotero_importer.cli import (
_add_previous_imported_comments,
_changed_note_clipping_ids,
_changed_override_titles,
_clipping_ids_for_titles,
_deferred_clipping_ids,
_deletion_ids,
_discarded_clipping_ids,
_merge_annotation_history,
_merge_positioned_history,
_unresolved_clipping_ids,
@ -82,114 +78,6 @@ def test_deletions_include_removed_and_previous_failures():
) == {"removed", "retry"}
def test_deferred_clippings_are_reprocessed_while_they_remain_in_source():
previous = {"deferred_clipping_ids": ["retry", "removed"]}
assert _deferred_clipping_ids(previous, {"retry", "current"}) == {"retry"}
def test_discarded_clips_never_requeue_and_skips_are_not_unresolved():
previous = {"discarded_clipping_ids": ["gone", "removed"]}
assert _discarded_clipping_ids(previous, {"gone", "current"}) == {"gone"}
plan = {
"items": [
{"clipping": {"id": "note"}, "status": "pdf-position-skipped-non-highlight"},
{"clipping": {"id": "fail"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "done"}, "status": "positioned"},
]
}
assert _unresolved_clipping_ids(plan, {"note", "fail", "done"}) == {"fail"}
def test_alternate_attachment_retry_prefers_pdf_when_epub_misses():
from kindle_zotero_importer.cli import _reattempt_failed_with_alternate_attachments
from kindle_zotero_importer.epub_position import add_epub_positions
from kindle_zotero_importer.import_plan import build_import_plan
from kindle_zotero_importer.pdf_position import add_pdf_positions
fixtures = Path(__file__).parent / "fixtures" / "windows-smoke"
epub_path = str(fixtures / "kanzi-dummy.epub")
pdf_path = str(fixtures / "kanzi-dummy.pdf")
quote = "This is a test highlight long enough to import for PDF."
clippings_payload = {
"format": "kindle-zotero-importer.clippings.v1",
"clippings": [
{
"id": "q1",
"title": "Dummy Book",
"kind": "highlight",
"text": quote,
"page": None,
"location": "100",
}
],
}
zotero_index = {
"format": "index",
"items": [
{
"item_id": 1,
"key": "ABC12345",
"title": "Dummy Book",
"fields": {},
"attachments": [
{
"item_id": 11,
"key": "EPUBKEY1",
"title": "epub",
"content_type": "application/epub+zip",
"path": epub_path,
"resolved_path": epub_path,
},
{
"item_id": 12,
"key": "PDFKEY123",
"title": "pdf",
"content_type": "application/pdf",
"path": pdf_path,
"resolved_path": pdf_path,
},
],
}
],
}
match_report = {
"format": "matches",
"matches": [
{
"clipping_title": "Dummy Book",
"status": "matched",
"candidates": [{"item_id": 1}],
}
],
}
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
assert plan["items"][0]["zotero"]["attachment"]["content_type"] == "application/epub+zip"
positioned = add_pdf_positions(add_epub_positions(plan))
assert positioned["items"][0]["status"] == "positioned"
assert "partial-text-match" in positioned["items"][0]["problems"]
rescued = _reattempt_failed_with_alternate_attachments(positioned)
assert rescued["items"][0]["status"] == "positioned"
assert "positioned-via-alternate-attachment" in rescued["items"][0]["problems"]
assert "partial-text-match" not in rescued["items"][0]["problems"]
def test_changed_override_requeues_every_clipping_for_exact_title():
previous = {"Shared title": {"citation_key": "old"}}
current = {"Shared title": {"citation_key": "new"}}
clippings = [
{"id": "first", "title": "Shared title"},
{"id": "second", "title": "Shared title"},
{"id": "third", "title": "Different title"},
]
changed_titles = _changed_override_titles(previous, current)
assert changed_titles == {"Shared title"}
assert _clipping_ids_for_titles(clippings, changed_titles) == {"first", "second"}
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
previous = {
@ -352,6 +240,6 @@ def test_noop_incremental_writes_complete_artifacts():
assert result == 0
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
assert final_plan["annotations"] == []
assert final_plan["all_annotations"] == [{**annotation, "related_items": []}]
assert final_plan["all_annotations"] == [annotation]
assert final_plan["delta_annotation_count"] == 0
assert (workdir / "import-plan.epub.json").exists()

View file

@ -4,7 +4,7 @@ const vm = require('vm');
const bootstrapContext = {};
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
const plugin = bootstrapContext.Kanzi;
const plugin = bootstrapContext.KindleZoteroImporter;
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
assert.equal(
@ -96,47 +96,6 @@ const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
const managerContext = { TextEncoder };
vm.runInNewContext(scripts[0][1], managerContext);
const manager = managerContext.KindleManager;
assert.equal(plugin.countClippingsForTitle({ clippings: [
{ title: 'Shared title' },
{ title: 'Shared title' },
{ title: 'Different title' },
] }, 'Shared title'), 2);
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
{ id: 'n1', title: 'Shared title', kind: 'note', text: 'note' },
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
] },
matches: { matches: [{
clipping_title: 'Shared title',
clipping_count: 3,
status: 'matched',
candidates: [],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [{
clipping: { id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
status: 'pdf-text-not-found',
problems: ['not found'],
}] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const conflictRows = manager.conflictRows();
assert.equal(conflictRows.length, 1);
assert.equal(conflictRows[0].count, 3);
assert.equal(conflictRows[0].clippings.length, 3);
assert.equal(conflictRows[0].highlights, 2);
assert.equal(conflictRows[0].notes, 1);
manager.sortState.conflicts = { column: 'count', direction: 'desc' };
const sortedRows = manager.sortRows(
[{ title: 'small', count: 2 }, { title: 'large', count: 20 }, { title: 'middle', count: 10 }],
'conflicts',
{ count: row => row.count }
);
assert.deepEqual(Array.from(sortedRows, row => row.title), ['large', 'middle', 'small']);
manager.integratedPage = 1;
manager.integratedPageSize = 50;
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
@ -184,34 +143,6 @@ async function testRuntimeExtraction() {
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
}
async function testPythonPathRecovery() {
bootstrapContext.Services = { appinfo: { OS: 'Darwin' } };
bootstrapContext.IOUtils = {
exists: async path => path === '/opt/homebrew/opt/python@3.13/bin/python3.13',
};
assert.equal(
await plugin.resolvePythonPath('/opt/homebrew/bin/python3'),
'/opt/homebrew/opt/python@3.13/bin/python3.13'
);
bootstrapContext.IOUtils.exists = async path => path === '/custom/python3';
assert.equal(await plugin.resolvePythonPath('/custom/python3'), '/custom/python3');
bootstrapContext.Services.appinfo.OS = 'WINNT';
const windowsEnvironment = {
LOCALAPPDATA: 'C:\\Users\\test\\AppData\\Local',
ProgramFiles: 'C:\\Program Files',
'ProgramFiles(x86)': 'C:\\Program Files (x86)',
USERPROFILE: 'C:\\Users\\test',
SystemDrive: 'C:',
PATH: 'C:\\Tools\\Python;C:\\Windows\\System32',
};
plugin.getEnvironmentVariable = name => windowsEnvironment[name] || '';
const expected = 'C:\\Users\\test\\AppData\\Local\\Programs\\Python\\Python313\\python.exe';
bootstrapContext.IOUtils.exists = async path => path === expected;
assert.equal(await plugin.resolvePythonPath('python'), expected);
assert.ok(plugin.getWindowsPythonCandidates('python').includes('C:\\Tools\\Python\\python.exe'));
}
async function testMaintenanceSweep() {
const annotation = (id, text, end, comment = '') => ({
id,
@ -384,317 +315,7 @@ async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
assert.equal(results.failed[0].clipping_id, 'replacement');
}
async function testCitekeyRelationsAreBidirectionalAndAdditive() {
const originalZotero = bootstrapContext.Zotero;
const originalCollect = plugin.collectImportedAnnotations;
const originalMaintain = plugin.maintainImportedAnnotations;
const item = (id, key) => ({
id,
key,
related: new Set(),
saved: 0,
addRelatedItem(other) {
if (this.related.has(other.key)) return false;
this.related.add(other.key);
return true;
},
async save(options) {
assert.equal(options.skipDateModifiedUpdate, true);
this.saved += 1;
},
});
const source = item(1, 'SOURCE01');
const target = item(2, 'TARGET02');
bootstrapContext.Zotero = {
Items: { get: id => id === 1 ? source : id === 2 ? target : id === 11 ? attachment : null },
DB: { executeTransaction: async callback => callback() },
logError: () => {},
};
const entries = [
{
clipping_id: 'history-1',
parent_item_id: 1,
attachment_item_id: 11,
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
},
{
parent_item_id: 1,
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
},
];
const existingAnnotation = {
annotationComment: 'an old note',
annotationTags: [{ name: 'kindle-id:history-1' }],
saved: 0,
saveTx: async function () { this.saved += 1; },
};
const attachment = { getAnnotations: () => [existingAnnotation] };
const results = {
relationsAdded: 0,
relationsSkipped: 0,
relationsPlanned: 0,
relationFailures: [],
};
try {
plugin.collectImportedAnnotations = async () => [];
plugin.maintainImportedAnnotations = async () => {};
const writerResults = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
all_annotations: entries,
deletions: [],
}, false);
assert.deepEqual(Array.from(source.related), ['TARGET02']);
assert.deepEqual(Array.from(target.related), ['SOURCE01']);
assert.equal(source.saved, 1);
assert.equal(target.saved, 1);
assert.equal(writerResults.relationsAdded, 1);
assert.equal(writerResults.relationFailures.length, 0);
assert.equal(writerResults.commentLinksAdded, 1);
assert.ok(existingAnnotation.annotationComment.includes(
'Related: @target2020 zotero://select/library/items/TARGET02'
));
assert.equal(existingAnnotation.saved, 1);
await plugin.addRelatedItemRelations(entries, false, results);
assert.equal(source.saved, 1);
assert.equal(target.saved, 1);
assert.equal(results.relationsSkipped, 1);
} finally {
bootstrapContext.Zotero = originalZotero;
plugin.collectImportedAnnotations = originalCollect;
plugin.maintainImportedAnnotations = originalMaintain;
}
}
function testPendingReviewGrouping() {
const groups = manager.groupPendingAnnotations([
{ clipping_id: 'a1', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'spice', pageLabel: '12', color: '#ffd400' } },
{ clipping_id: 'a2', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'worm', pageLabel: '', color: '#ff6666' } },
{ clipping_id: 'b1', clipping_title: 'Unknown Book', citation_key: '', annotation: { text: 'mystery', pageLabel: '', color: '#ffd400' } },
]);
assert.equal(groups.length, 2);
assert.equal(groups[0].items.length, 2);
assert.equal(groups[0].citationKey, 'herbert2020');
assert.equal(groups[1].items.length, 1);
assert.deepEqual(manager.groupPendingAnnotations([]), []);
}
function testFilterPlanForSelection() {
const plan = {
annotations: [{ clipping_id: 'keep' }, { clipping_id: 'skip' }],
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'keep' }, { clipping_id: 'skip' }],
};
const previous = { deferred_clipping_ids: ['older'] };
const filtered = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep'], previous);
assert.deepEqual(Array.from(filtered.annotations, a => a.clipping_id), ['keep']);
assert.deepEqual(Array.from(filtered.all_annotations, a => a.clipping_id), ['old', 'keep']);
assert.deepEqual(Array.from(filtered.deferred_clipping_ids), ['older', 'skip']);
// Original plan is not mutated.
assert.equal(plan.annotations.length, 2);
const all = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep', 'skip'], previous);
assert.equal(all.annotations.length, 2);
assert.equal(all.all_annotations.length, 3);
assert.deepEqual(Array.from(all.deferred_clipping_ids), ['older']);
const priorAnnotation = { clipping_id: 'skip', annotation: { text: 'existing' } };
const preserved = plugin.filterPlanForSelection(
plan,
['keep', 'skip'],
['keep'],
{ all_annotations: [priorAnnotation] }
);
assert.equal(preserved.all_annotations.find(a => a.clipping_id === 'skip'), priorAnnotation);
const deferred = plugin.deferPendingPlan(
{ all_annotations: [{ clipping_id: 'old' }], deferred_clipping_ids: ['earlier'] },
plan,
['keep', 'skip']
);
assert.deepEqual(Array.from(deferred.deferred_clipping_ids), ['earlier', 'keep', 'skip']);
assert.deepEqual(Array.from(deferred.all_annotations, a => a.clipping_id), ['old']);
}
function testTitleCountsAreAuthoritative() {
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
{ id: 'h3', title: 'Shared title', kind: 'highlight', text: 'third' },
] },
matches: { matches: [{
clipping_title: 'Shared title',
clipping_count: 99,
status: 'ambiguous',
candidates: [{ citation_key: 'shared2020', key: '', item_id: 1, title: 'Shared', score: 0.9 }],
}] },
generatedOverrides: { overrides: [{
clipping_title: 'Shared title',
review: { status: 'review', clipping_count: 1 },
candidates: [],
}] },
positionedPlan: { items: [{
clipping: { id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
status: 'pdf-text-not-found',
problems: ['not found'],
}] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const rows = manager.conflictRows();
assert.equal(rows.length, 1);
// Source has 3 clippings; misleading match (99) and override (1) counts must not win.
assert.equal(rows[0].count, 3);
assert.equal(rows[0].clippings.length, 3);
// Only 1 of 3 has position problems.
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 3 });
// Without clippings.json, fall back to conflict counts.
manager.data = {
clippings: null,
matches: { matches: [{
clipping_title: 'Lonely title',
clipping_count: 5,
status: 'ambiguous',
candidates: [],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const fallback = manager.conflictRows();
assert.equal(fallback.length, 1);
assert.equal(fallback[0].count, 5);
}
function testMappedTargetAndDiscardedClips() {
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' },
{ id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' },
] },
matches: { matches: [{
clipping_title: 'Mapped title',
clipping_count: 2,
status: 'matched',
candidates: [{ citation_key: 'mapped2020', key: '', item_id: 7, title: 'Mapped', score: 1 }],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [
{ clipping: { id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' }, status: 'pdf-text-not-found', problems: ['pdf-text-not-found'] },
{ clipping: { id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' }, status: 'pdf-position-skipped-non-highlight', problems: ['pdf-position-skipped-non-highlight'] },
] },
finalPlan: { all_annotations: [], discarded_clipping_ids: [] },
persistentOverrides: { overrides: [{ clipping_title: 'Mapped title', resolution: { citation_key: 'mapped2020' } }] },
};
let rows = manager.conflictRows();
assert.equal(rows.length, 1);
assert.equal(rows[0].mappedTarget, 'mapped2020');
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 2, total: 2 });
manager.data.finalPlan.discarded_clipping_ids = ['n1'];
rows = manager.conflictRows();
assert.equal(rows.length, 1);
assert.equal(rows[0].count, 1);
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 1 });
}
async function testDiscardClipPersists() {
const originalZotero = bootstrapContext.Zotero;
let written = null;
const plan = { discarded_clipping_ids: [], deferred_clipping_ids: ['x1'] };
bootstrapContext.Zotero = {
File: {
getContentsAsync: async () => JSON.stringify(plan),
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
},
};
const originalArtifactPath = plugin.artifactPath;
plugin.artifactPath = rel => `/proj/${rel}`;
try {
await plugin.discardClip('x1');
assert.deepEqual(written.discarded_clipping_ids, ['x1']);
assert.deepEqual(written.deferred_clipping_ids, []);
} finally {
plugin.artifactPath = originalArtifactPath;
bootstrapContext.Zotero = originalZotero;
}
}
function testTitleOverviewAndSorting() {
manager.data = {
clippings: { clippings: [
{ id: 'a1', title: 'Alpha', kind: 'highlight', text: 'one' },
{ id: 'a2', title: 'Alpha', kind: 'highlight', text: 'two' },
{ id: 'b1', title: 'Beta', kind: 'highlight', text: 'three' },
] },
matches: { matches: [
{ clipping_title: 'Alpha', clipping_count: 2, status: 'matched', candidates: [{ citation_key: 'alpha2020' }] },
{ clipping_title: 'Beta', clipping_count: 1, status: 'unmatched', candidates: [] },
] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [] },
finalPlan: {
all_annotations: [{ clipping_id: 'a1', clipping_title: 'Alpha' }],
deferred_clipping_ids: ['b1'],
},
persistentOverrides: { overrides: [] },
};
const overview = manager.titleOverviewRows();
assert.equal(overview.length, 2);
const alpha = overview.find(r => r.title === 'Alpha');
const beta = overview.find(r => r.title === 'Beta');
assert.equal(alpha.integrated, 1);
assert.equal(beta.pending, 1);
assert.equal(beta.status, 'pending review');
manager.sortState.titles = { column: 'title', direction: 'desc' };
const sorted = manager.sortRows(overview, 'titles', { title: row => row.title });
assert.deepEqual(Array.from(sorted, row => row.title), ['Beta', 'Alpha']);
}
async function testSaveOverrideUsesSourceCount() {
const originalRead = plugin.readJsonArtifact;
const originalArtifactPath = plugin.artifactPath;
const originalZotero = bootstrapContext.Zotero;
let written = null;
plugin.artifactPath = rel => `/proj/${rel}`;
plugin.readJsonArtifact = async rel => rel === 'clippings.json'
? { clippings: [{ title: 'Shared title' }, { title: 'Shared title' }, { title: 'Shared title' }] }
: null;
bootstrapContext.Zotero = {
File: {
getContentsAsync: async () => JSON.stringify({ format: 'kindle-zotero-importer.match-overrides.v1', overrides: [] }),
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
},
};
try {
const entry = await plugin.saveOverride({
clipping_title: 'Shared title',
clipping_count: 1,
resolution: { citation_key: 'shared2020' },
review: { status: 'pdf-text-not-found', clipping_count: 1 },
});
assert.equal(entry.review.clipping_count, 3);
assert.equal(written.overrides[0].review.clipping_count, 3);
} finally {
plugin.readJsonArtifact = originalRead;
plugin.artifactPath = originalArtifactPath;
bootstrapContext.Zotero = originalZotero;
}
}
testTitleCountsAreAuthoritative();
testTitleOverviewAndSorting();
testMappedTargetAndDiscardedClips();
testPendingReviewGrouping();
testFilterPlanForSelection();
testRuntimeExtraction()
.then(testPythonPathRecovery)
.then(testSaveOverrideUsesSourceCount)
.then(testDiscardClipPersists)
.then(testMaintenanceSweep)
.then(testWriterReconciliation)
.then(testDeletionFailureIsRetried)
@ -702,7 +323,6 @@ testRuntimeExtraction()
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
.then(testEnumerationFailurePersistsAllDeletions)
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
.then(testCitekeyRelationsAreBidirectionalAndAdditive)
.then(() => {
console.log('manager and annotation logic ok');
}).catch(error => {