Kanzi 0.7.1: safer imports and resilient positioning

This commit is contained in:
Utku Bilen Demir 2026-10-15 14:01:05 +02:00
parent 495f50989d
commit aa3fc5ebc5
19 changed files with 2823 additions and 159 deletions

View file

@ -17,16 +17,39 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero. 2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
3. `Tools` → `Kanzi…` to open the manager. 3. `Tools` → `Kanzi…` to open the manager.
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. macOS and Linux are tested. Windows requires setting the full Python executable path in `Settings` and installing the PDF tools separately. Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. Kanzi detects standard Python installations automatically on macOS, Linux and Windows.
## Windows
Open PowerShell and install the host tools:
```powershell
winget install --id Python.Python.3.13 --exact
winget install --id oschwartz10612.Poppler --exact
winget install --id QPDF.QPDF --exact # Optional: handles copy-protected PDFs
```
Fully quit and reopen Zotero after installation so it receives the updated `PATH`, then install `kanzi.xpi` and open `Tools` → `Kanzi…`. Kanzi checks standard per-user and system Python locations as well as `PATH`; you should not need to enter an executable path manually. EPUB-only imports require Python but not Poppler.
To verify the installation in a new PowerShell window:
```powershell
python --version
pdftotext -v
pdftohtml -v
pdfinfo -v
qpdf --version # Optional
```
# Use # Use
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress. 1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything. 2. **Review before writing.** New highlights are grouped by title and selected by default. Untick individual highlights or whole titles, then press `Integrate selected`. Skipped highlights return on the next run. `Cancel` and closing the manager write nothing to Zotero and do not mark pending highlights as integrated.
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate. 3. **Check Integrated.** After integration, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything. 4. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows one expandable row for that title with up to three suggestions `citekey · title (score%)`. Expand it to inspect individual highlights and positioning problems. Click `Use` on the right candidate, enter a citation key, Zotero key or item ID under `Use Custom`, or choose `Ignore Title`. Tables can be sorted and resized.
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching. 5. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new, changed, deferred or removed highlights are processed. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`. 6. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
7. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles. Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
@ -62,11 +85,11 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
`Tools` → `Kanzi…` opens the manager. `Tools` → `Kanzi…` opens the manager.
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. **Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. After positioning, the pending review lets you select exactly what is written to Zotero.
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated. **Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate. **Conflicts.** One row per unresolved Kindle title, combining title-match and attachment/position statuses. Expand a row to inspect its highlights and positioning problems. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. Columns are sortable and resizable.
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable. **Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.

506
plugin/bootstrap.js vendored
View file

@ -1,4 +1,4 @@
var KindleZoteroImporter = { var Kanzi = {
id: "kindle-zotero-importer@utkubilen.de", id: "kindle-zotero-importer@utkubilen.de",
menuItems: [], menuItems: [],
runtimeProjectDir: "", runtimeProjectDir: "",
@ -16,7 +16,7 @@ var KindleZoteroImporter = {
); );
this.chromeHandle = addonManagerStartup.registerChrome( this.chromeHandle = addonManagerStartup.registerChrome(
Services.io.newURI(this.rootURI + "manifest.json"), Services.io.newURI(this.rootURI + "manifest.json"),
[["content", "kindle-zotero-importer", ""]] [["content", "kanzi", ""]]
); );
await Zotero.initializationPromise; await Zotero.initializationPromise;
await this.ensureRuntime(); await this.ensureRuntime();
@ -81,7 +81,7 @@ var KindleZoteroImporter = {
} }
const doc = win.document; const doc = win.document;
if (doc.getElementById("kindle-zotero-importer-manager")) { if (doc.getElementById("kanzi-manager")) {
return; return;
} }
@ -92,7 +92,7 @@ var KindleZoteroImporter = {
} }
const managerItem = doc.createXULElement("menuitem"); const managerItem = doc.createXULElement("menuitem");
managerItem.setAttribute("id", "kindle-zotero-importer-manager"); managerItem.setAttribute("id", "kanzi-manager");
managerItem.setAttribute("label", "Kanzi..."); managerItem.setAttribute("label", "Kanzi...");
managerItem.addEventListener("command", () => this.openManager()); managerItem.addEventListener("command", () => this.openManager());
popup.appendChild(managerItem); popup.appendChild(managerItem);
@ -136,7 +136,14 @@ var KindleZoteroImporter = {
}, },
getPythonPath() { getPythonPath() {
return this.getPref("pythonPath", "python3"); const fallback = this.getDefaultPython();
const value = this.getPref("pythonPath", fallback);
// Existing Windows installs may have "python3" stored from the old
// Unix-only default; that binary name does not exist on Windows.
if (this.isWindows() && value === "python3") {
return "python";
}
return value || fallback;
}, },
getZoteroDbPath() { getZoteroDbPath() {
@ -149,12 +156,115 @@ var KindleZoteroImporter = {
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : ""); return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
}, },
isWindows() {
try {
return Services.appinfo.OS === "WINNT";
} catch (_error) {
return false;
}
},
getDefaultPython() {
return this.isWindows() ? "python" : "python3";
},
getEnvironmentVariable(name) {
try {
return Cc["@mozilla.org/process/environment;1"]
.getService(Ci.nsIEnvironment)
.get(name) || "";
} catch (_error) {
return "";
}
},
joinWindowsPath(base, ...parts) {
if (!base) {
return "";
}
return [String(base).replace(/[\\/]+$/, ""), ...parts]
.map((part, index) => index ? String(part).replace(/^[\\/]+|[\\/]+$/g, "") : part)
.join("\\");
},
getWindowsPythonCandidates(configuredPath) {
const localAppData = this.getEnvironmentVariable("LOCALAPPDATA");
const programFiles = this.getEnvironmentVariable("ProgramFiles");
const programFilesX86 = this.getEnvironmentVariable("ProgramFiles(x86)");
const userProfile = this.getEnvironmentVariable("USERPROFILE");
const systemDrive = this.getEnvironmentVariable("SystemDrive") || "C:";
const versions = ["314", "313", "312", "311"];
const candidates = [configuredPath];
for (const version of versions) {
candidates.push(
this.joinWindowsPath(localAppData, "Programs", "Python", `Python${version}`, "python.exe"),
this.joinWindowsPath(programFiles, `Python${version}`, "python.exe"),
this.joinWindowsPath(programFilesX86, `Python${version}-32`, "python.exe"),
this.joinWindowsPath(systemDrive, `Python${version}`, "python.exe")
);
}
candidates.push(
this.joinWindowsPath(localAppData, "Python", "bin", "python.exe"),
this.joinWindowsPath(userProfile, "scoop", "apps", "python", "current", "python.exe"),
this.joinWindowsPath(userProfile, "miniconda3", "python.exe"),
this.joinWindowsPath(userProfile, "anaconda3", "python.exe")
);
for (const directory of this.getEnvironmentVariable("PATH").split(";")) {
const cleanDirectory = directory.trim().replace(/^"|"$/g, "");
if (cleanDirectory) {
candidates.push(
this.joinWindowsPath(cleanDirectory, "python.exe"),
this.joinWindowsPath(cleanDirectory, "python3.exe")
);
}
}
return [...new Set(candidates.filter(Boolean))];
},
async resolvePythonPath(configuredPath) {
let candidates;
if (this.isWindows()) {
candidates = this.getWindowsPythonCandidates(configuredPath);
} else if (Services.appinfo.OS === "Darwin") {
candidates = [configuredPath];
candidates.push(
"/opt/homebrew/bin/python3",
"/opt/homebrew/opt/python@3.14/bin/python3.14",
"/opt/homebrew/opt/python@3.13/bin/python3.13",
"/opt/homebrew/opt/python@3.12/bin/python3.12",
"/opt/homebrew/opt/python@3.11/bin/python3.11",
"/usr/local/bin/python3",
"/usr/local/opt/python@3.14/bin/python3.14",
"/usr/local/opt/python@3.13/bin/python3.13",
"/usr/local/opt/python@3.12/bin/python3.12",
"/usr/local/opt/python@3.11/bin/python3.11",
"/usr/bin/python3"
);
} else {
candidates = [configuredPath, "/usr/bin/python3", "/usr/local/bin/python3"];
}
for (const candidate of candidates) {
if (!candidate || (!candidate.includes("/") && !candidate.includes("\\"))) continue;
try {
if (await IOUtils.exists(candidate)) return candidate;
} catch (_error) {}
}
return configuredPath || this.getDefaultPython();
},
joinPath(...parts) {
if (typeof PathUtils !== "undefined" && PathUtils.join) {
return PathUtils.join(...parts);
}
return parts.join("/").replaceAll("//", "/");
},
getConfigPath() { getConfigPath() {
return this.getProjectDir() + "/plugin-config.json"; return this.joinPath(this.getProjectDir(), "plugin-config.json");
}, },
artifactPath(relativePath) { artifactPath(relativePath) {
return this.getProjectDir() + "/" + relativePath; return this.joinPath(this.getProjectDir(), ...String(relativePath).split("/"));
}, },
async openArtifact(relativePath) { async openArtifact(relativePath) {
@ -177,17 +287,6 @@ var KindleZoteroImporter = {
zoteroDbPath: this.getZoteroDbPath(), zoteroDbPath: this.getZoteroDbPath(),
zoteroStorageRoot: this.getZoteroStorageRoot(), zoteroStorageRoot: this.getZoteroStorageRoot(),
}; };
if (settings.pythonPath === "python3") {
const candidates = Services.appinfo.OS === "Darwin"
? ["/opt/homebrew/bin/python3", "/usr/local/bin/python3", "/usr/bin/python3"]
: ["/usr/bin/python3", "/usr/local/bin/python3"];
for (const candidate of candidates) {
if (await IOUtils.exists(candidate)) {
settings.pythonPath = candidate;
break;
}
}
}
try { try {
const text = await Zotero.File.getContentsAsync(this.getConfigPath()); const text = await Zotero.File.getContentsAsync(this.getConfigPath());
const parsed = JSON.parse(text); const parsed = JSON.parse(text);
@ -200,12 +299,13 @@ var KindleZoteroImporter = {
if (parsed.colorMap) settings.colourMap = parsed.colorMap; if (parsed.colorMap) settings.colourMap = parsed.colorMap;
// also check colour-map.json for Python pipeline // also check colour-map.json for Python pipeline
try { try {
const cmapText = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json"); const cmapText = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
settings.colourMap = JSON.parse(cmapText); settings.colourMap = JSON.parse(cmapText);
} catch (_e2) {} } catch (_e2) {}
} catch (_error) { } catch (_error) {
// The config file is optional; defaults/preferences are enough for first run. // The config file is optional; defaults/preferences are enough for first run.
} }
settings.pythonPath = await this.resolvePythonPath(settings.pythonPath);
return settings; return settings;
}, },
@ -267,8 +367,8 @@ var KindleZoteroImporter = {
const data = await this.loadManagerData(); const data = await this.loadManagerData();
const io = { plugin: this, settings, data }; const io = { plugin: this, settings, data };
this.managerWindow = this.getMainWindow().openDialog( this.managerWindow = this.getMainWindow().openDialog(
"chrome://kindle-zotero-importer/content/manager.html", "chrome://kanzi/content/manager.html",
"kindle-zotero-importer-manager-window", "kanzi-manager-window",
"chrome,centerscreen,resizable", "chrome,centerscreen,resizable",
io io
); );
@ -276,6 +376,7 @@ var KindleZoteroImporter = {
async loadManagerData() { async loadManagerData() {
return { return {
clippings: await this.readJsonArtifact("clippings.json"),
matches: await this.readJsonArtifact("matches.json"), matches: await this.readJsonArtifact("matches.json"),
generatedOverrides: await this.readJsonArtifact("match-overrides.generated.json"), generatedOverrides: await this.readJsonArtifact("match-overrides.generated.json"),
positionedPlan: await this.readJsonArtifact("import-plan.positioned.json"), positionedPlan: await this.readJsonArtifact("import-plan.positioned.json"),
@ -305,11 +406,17 @@ var KindleZoteroImporter = {
const index = payload.overrides.findIndex( const index = payload.overrides.findIndex(
(entry) => entry.clipping_title === resolution.clipping_title (entry) => entry.clipping_title === resolution.clipping_title
); );
const clippings = await this.readJsonArtifact("clippings.json");
const sourceCount = this.countClippingsForTitle(clippings, resolution.clipping_title);
const clippingCount = sourceCount || resolution.clipping_count || resolution.review?.clipping_count || 0;
const nowIso = new Date().toISOString(); const nowIso = new Date().toISOString();
const entry = { const entry = {
clipping_title: resolution.clipping_title, clipping_title: resolution.clipping_title,
resolution: resolution.resolution, resolution: resolution.resolution,
review: resolution.review || { status: "manager", clipping_count: resolution.clipping_count || 0 }, review: {
...(resolution.review || { status: "manager" }),
clipping_count: clippingCount,
},
created_at: nowIso, created_at: nowIso,
updated_at: nowIso, updated_at: nowIso,
}; };
@ -329,6 +436,13 @@ var KindleZoteroImporter = {
return entry; return entry;
}, },
countClippingsForTitle(payload, clippingTitle) {
return (payload?.clippings || []).reduce(
(count, clipping) => count + (clipping.title === clippingTitle ? 1 : 0),
0
);
},
async deleteOverride(clippingTitle) { async deleteOverride(clippingTitle) {
if (!clippingTitle) { if (!clippingTitle) {
throw new Error("Missing clipping title for delete"); throw new Error("Missing clipping title for delete");
@ -346,6 +460,23 @@ var KindleZoteroImporter = {
return true; return true;
}, },
async discardClip(clippingId) {
if (!clippingId) {
throw new Error("Missing clipping id for discard");
}
const path = this.artifactPath("import-plan.final.json");
const plan = JSON.parse(await Zotero.File.getContentsAsync(path));
const discarded = new Set(plan.discarded_clipping_ids || []);
discarded.add(clippingId);
plan.discarded_clipping_ids = [...discarded].sort();
// Also drop from deferred so it is never re-queued.
if (Array.isArray(plan.deferred_clipping_ids)) {
plan.deferred_clipping_ids = plan.deferred_clipping_ids.filter((id) => id !== clippingId);
}
await Zotero.File.putContentsAsync(path, JSON.stringify(plan, null, 2) + "\n");
return true;
},
async saveSettingsFromManager(newSettings) { async saveSettingsFromManager(newSettings) {
const toSave = { const toSave = {
projectDir: newSettings.projectDir || this.getProjectDir(), projectDir: newSettings.projectDir || this.getProjectDir(),
@ -358,7 +489,7 @@ var KindleZoteroImporter = {
// also write colour-map.json for Python pipeline (mirrors plugin-config) // also write colour-map.json for Python pipeline (mirrors plugin-config)
try { try {
await Zotero.File.putContentsAsync( await Zotero.File.putContentsAsync(
this.getProjectDir() + "/colour-map.json", this.joinPath(this.getProjectDir(), "colour-map.json"),
JSON.stringify(newSettings.colourMap, null, 2) + "\n" JSON.stringify(newSettings.colourMap, null, 2) + "\n"
); );
} catch (_e) {} } catch (_e) {}
@ -376,7 +507,7 @@ var KindleZoteroImporter = {
async loadColourMap() { async loadColourMap() {
try { try {
const text = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json"); const text = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
return JSON.parse(text); return JSON.parse(text);
} catch (_e) {} } catch (_e) {}
try { try {
@ -416,7 +547,7 @@ var KindleZoteroImporter = {
if (result !== fp.returnOK) { if (result !== fp.returnOK) {
return null; return null;
} }
return fp.file; return this.filePickerResultToPath(fp.file);
} catch (error) { } catch (error) {
Zotero.logError(error); Zotero.logError(error);
return this.promptForPath( return this.promptForPath(
@ -427,12 +558,32 @@ var KindleZoteroImporter = {
} }
}, },
filePickerResultToPath(file) {
if (!file) {
return null;
}
if (typeof file === "string") {
return file;
}
// Mozilla FilePicker returns an nsIFile; Python needs a native string path.
if (typeof file.path === "string" && file.path) {
return file.path;
}
try {
const str = String(file);
if (str && str !== "[object Object]") {
return str;
}
} catch (_error) {}
return null;
},
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) { async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
const python = settings.pythonPath; const python = settings.pythonPath || this.getDefaultPython();
const projectDir = settings.projectDir; const projectDir = settings.projectDir;
const runner = projectDir + "/plugin_runner.py"; const runner = this.joinPath(projectDir, "plugin_runner.py");
const summaryOutput = projectDir + "/plugin-summary.json"; const summaryOutput = this.joinPath(projectDir, "plugin-summary.json");
const progressOutput = projectDir + "/plugin-progress.json"; const progressOutput = this.joinPath(projectDir, "plugin-progress.json");
const args = [ const args = [
runner, runner,
"run", "run",
@ -458,12 +609,34 @@ var KindleZoteroImporter = {
args.push("--retry-unresolved"); args.push("--retry-unresolved");
} }
await this.exec(python, args); let execResult = null;
try {
execResult = await this.exec(python, args);
} catch (error) {
const installHint = this.isWindows()
? `Install Python 3.11 or newer, fully restart Zotero, and let Kanzi detect python.exe. `
: `Install Python 3.11 or newer and check the Python path in Kanzi Settings. `;
throw new Error(
`Python could not run the Kanzi pipeline (${python}): ${error && error.message ? error.message : error}. ` +
installHint
);
}
try {
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput)); return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
} catch (error) {
const detail = execResult ? ` Python output: ${JSON.stringify(execResult).slice(0, 1000)}.` : "";
throw new Error(
`Python pipeline did not write ${summaryOutput}.${detail} ` +
`Python: ${python}. Runner: ${runner}. ` +
`Make sure Python 3 is installed and reachable, then check the project directory is writable.`
);
}
}, },
async runManagedImport(manager) { async runManagedImport(manager) {
const clippingsPath = await this.pickFile("Select Kindle My Clippings.txt", "*.txt"); const clippingsPath = await this.pickFile("Select Kindle My Clippings.txt", "*.txt");
// NOTE: writing now happens only after the user reviews the pending
// highlights and presses Integrate (see integratePending).
if (!clippingsPath) { if (!clippingsPath) {
return; return;
} }
@ -474,17 +647,23 @@ var KindleZoteroImporter = {
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) { async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
try { try {
clippingsPath = this.filePickerResultToPath(clippingsPath) || clippingsPath;
if (!clippingsPath) { if (!clippingsPath) {
throw new Error("No clippings file selected"); throw new Error("No clippings file selected");
} }
const settings = await this.getRuntimeSettings(); const settings = await this.getRuntimeSettings();
const progressPath = settings.projectDir + "/plugin-progress.json"; const progressPath = this.joinPath(settings.projectDir, "plugin-progress.json");
// Determine full vs incremental from explicit arg or manager checkbox // Determine full vs incremental from explicit arg or manager checkbox
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false); const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
const retryUnresolved = typeof retryUnresolvedOverride === "boolean" const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
? retryUnresolvedOverride ? retryUnresolvedOverride
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false); : (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
manager.beginRun(clippingsPath); manager.beginRun(clippingsPath);
const finalPlanPath = this.joinPath(settings.projectDir, "import-plan.final.json");
let previousFinalPlan = null;
try {
previousFinalPlan = JSON.parse(await Zotero.File.getContentsAsync(finalPlanPath));
} catch (_backupError) {}
try { try {
await Zotero.File.removeIfExists(progressPath); await Zotero.File.removeIfExists(progressPath);
} catch (_error) {} } catch (_error) {}
@ -511,20 +690,142 @@ var KindleZoteroImporter = {
const finalPlan = JSON.parse( const finalPlan = JSON.parse(
await Zotero.File.getContentsAsync(summary.outputs.final_plan) await Zotero.File.getContentsAsync(summary.outputs.final_plan)
); );
manager.updateProgress(96, "Writing annotations", "Saving annotations in Zotero"); const pendingCount = (finalPlan.annotations || []).length;
const results = await this.writeAnnotations(finalPlan, false, (completed, total) => { const pendingDeletions = (finalPlan.deletions || []).length;
if (!pendingCount && !pendingDeletions) {
// Nothing new — run the maintenance sweep and finish as before.
manager.updateProgress(96, "Writing annotations", "Checking existing annotations");
const emptyResults = await this.writeAnnotations(finalPlan, false, (completed, total) => {
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100; const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`); manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
}); });
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, results); await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, emptyResults);
const data = await this.loadManagerData(); const emptyData = await this.loadManagerData();
await manager.completeRun(summary, results, data); await manager.completeRun(summary, emptyResults, emptyData);
return;
}
// Pause before writing: let the user review and select what to integrate.
const fullDeltaIds = (finalPlan.annotations || []).map((a) => a.clipping_id);
// The pipeline has already written its generated history. Replace it with
// a safe ledger until Integrate is pressed, including if the window closes.
const deferredPlan = this.deferPendingPlan(previousFinalPlan, finalPlan, fullDeltaIds);
await Zotero.File.putContentsAsync(
summary.outputs.final_plan,
JSON.stringify(deferredPlan, null, 2) + "\n"
);
this._pending = {
summary,
finalPlan,
finalPlanPath: summary.outputs.final_plan,
fullDeltaIds,
previousFinalPlan,
};
manager.updateProgress(
90,
"Review pending",
`${pendingCount} highlight(s) ready — select and press Integrate`
);
manager.showPendingReview(summary, finalPlan);
} catch (error) { } catch (error) {
Zotero.logError(error); Zotero.logError(error);
manager.failRun(String(error && error.stack ? error.stack : error)); manager.failRun(String(error && error.stack ? error.stack : error));
} }
}, },
deferPendingPlan(previousPlan, generatedPlan, deferredIds) {
const plan = previousPlan
? { ...previousPlan }
: {
...generatedPlan,
annotations: [],
all_annotations: [],
deletions: [],
annotation_count: 0,
delta_annotation_count: 0,
};
const deferred = new Set(plan.deferred_clipping_ids || []);
for (const id of deferredIds || []) deferred.add(id);
plan.deferred_clipping_ids = [...deferred].sort();
return plan;
},
// Pure helper (tested): narrow a pipeline plan to the user's selection.
filterPlanForSelection(finalPlan, fullDeltaIds, selectedIds, previousPlan = null) {
const selected = new Set(selectedIds || []);
const unselected = new Set(
(fullDeltaIds || []).filter((id) => !selected.has(id))
);
const plan = { ...finalPlan };
plan.annotations = (finalPlan.annotations || []).filter((a) =>
selected.has(a.clipping_id)
);
if (unselected.size && Array.isArray(finalPlan.all_annotations)) {
plan.all_annotations = finalPlan.all_annotations.filter(
(a) => !unselected.has(a.clipping_id)
);
const present = new Set(plan.all_annotations.map((a) => a.clipping_id));
for (const annotation of previousPlan?.all_annotations || previousPlan?.annotations || []) {
if (unselected.has(annotation.clipping_id) && !present.has(annotation.clipping_id)) {
plan.all_annotations.push(annotation);
present.add(annotation.clipping_id);
}
}
}
const deferred = new Set(previousPlan?.deferred_clipping_ids || []);
for (const id of selected) deferred.delete(id);
for (const id of unselected) deferred.add(id);
plan.deferred_clipping_ids = [...deferred].sort();
plan.delta_annotation_count = plan.annotations.length;
if (Array.isArray(plan.all_annotations)) {
plan.annotation_count = plan.all_annotations.length;
}
return plan;
},
async integratePending(manager, selectedIds) {
const pending = this._pending;
if (!pending) {
throw new Error("No pending review to integrate");
}
try {
const plan = this.filterPlanForSelection(
pending.finalPlan,
pending.fullDeltaIds,
selectedIds,
pending.previousFinalPlan
);
manager.updateProgress(96, "Writing annotations", "Saving selected annotations in Zotero");
const results = await this.writeAnnotations(plan, false, (completed, total) => {
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
});
await this.reconcileWriterState(pending.finalPlanPath, plan, results);
this._pending = null;
const data = await this.loadManagerData();
await manager.completeRun(pending.summary, results, data);
} catch (error) {
Zotero.logError(error);
manager.failRun(String(error && error.stack ? error.stack : error));
}
},
async cancelPendingReview(manager) {
// The deferred ledger was written before the review appeared, so closing
// or cancelling never records pending highlights as integrated.
this._pending = null;
manager.hidePendingReview();
manager.cancelRun("Review cancelled — nothing was written. Run again when ready.");
try {
const data = await this.loadManagerData();
manager.data = data;
manager.rows = manager.conflictRows();
manager.refreshRows(true);
manager.refreshTitleOverview();
manager.refreshIntegrated(true);
manager.refreshMappings(true);
} catch (_reloadError) {}
},
async exec(command, args) { async exec(command, args) {
if (!Zotero.Utilities || !Zotero.Utilities.Internal || !Zotero.Utilities.Internal.exec) { if (!Zotero.Utilities || !Zotero.Utilities.Internal || !Zotero.Utilities.Internal.exec) {
throw new Error("Zotero.Utilities.Internal.exec is unavailable in this Zotero build"); throw new Error("Zotero.Utilities.Internal.exec is unavailable in this Zotero build");
@ -814,6 +1115,117 @@ var KindleZoteroImporter = {
throw new Error("Existing annotation cannot be saved by this Zotero build"); throw new Error("Existing annotation cannot be saved by this Zotero build");
}, },
async saveRelatedItem(item) {
const options = { skipDateModifiedUpdate: true };
if (item.save) {
await item.save(options);
return;
}
if (item.saveTx) {
await item.saveTx(options);
return;
}
throw new Error("Related item cannot be saved by this Zotero build");
},
async addRelatedItemRelations(entries, dryRun, results) {
const pairs = new Map();
for (const entry of entries || []) {
for (const target of entry.related_items || []) {
const sourceID = Number(entry.parent_item_id);
const targetID = Number(target.item_id);
if (!sourceID || !targetID || sourceID === targetID) continue;
const pairKey = [sourceID, targetID].sort((a, b) => a - b).join(":");
if (!pairs.has(pairKey)) {
pairs.set(pairKey, { sourceID, targetID, citationKey: target.citation_key || "" });
}
}
}
for (const pair of pairs.values()) {
try {
const source = Zotero.Items.get(pair.sourceID);
const target = Zotero.Items.get(pair.targetID);
if (!source || !target) {
throw new Error(`Related item pair not found: ${pair.sourceID} -> ${pair.targetID}`);
}
if (dryRun) {
results.relationsPlanned += 1;
continue;
}
if (!source.addRelatedItem || !target.addRelatedItem) {
throw new Error("This Zotero build does not support addRelatedItem()");
}
let changed = false;
const addBothDirections = async () => {
const sourceChanged = source.addRelatedItem(target);
const targetChanged = target.addRelatedItem(source);
if (sourceChanged) await this.saveRelatedItem(source);
if (targetChanged) await this.saveRelatedItem(target);
changed = sourceChanged || targetChanged;
};
if (Zotero.DB?.executeTransaction) {
await Zotero.DB.executeTransaction(addBothDirections);
} else {
await addBothDirections();
}
if (changed) results.relationsAdded += 1;
else results.relationsSkipped += 1;
} catch (error) {
Zotero.logError(error);
results.relationFailures.push({
parent_item_id: pair.sourceID,
related_item_id: pair.targetID,
citation_key: pair.citationKey,
message: String(error),
});
}
}
},
async backfillRelationLinks(entries, dryRun, results) {
// Appends missing `Related: @key <link>` blocks to pre-existing
// annotations. Cross-library targets can never become Related items,
// so the comment link is the only visible connection for them.
for (const entry of entries || []) {
const targets = entry.related_items || [];
if (!targets.length || !entry.clipping_id || !entry.attachment_item_id) continue;
const block = "Related: " + targets.map(
(target) => `@${target.citation_key} ${target.link || ""}`.trim()
).join(" · ");
try {
const attachment = Zotero.Items.get(entry.attachment_item_id);
if (!attachment) {
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
}
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
const tag = `kindle-id:${entry.clipping_id}`;
const existing = (annotations || []).find((annotation) =>
this.existingTags(annotation).includes(tag)
);
if (!existing) continue;
const comment = String(existing.annotationComment || "");
if (comment.includes(block)) continue;
if (dryRun) {
results.commentLinksPlanned += 1;
continue;
}
existing.annotationComment = comment ? `${comment}\n\n${block}` : block;
await this.saveExistingAnnotation(existing);
results.commentLinksAdded += 1;
} catch (error) {
Zotero.logError(error);
results.relationFailures.push({
parent_item_id: entry.parent_item_id,
related_item_id: (targets[0] || {}).item_id,
citation_key: (targets[0] || {}).citation_key,
message: `comment link: ${error}`,
});
}
}
},
async reconcileWriterState(path, plan, results) { async reconcileWriterState(path, plan, results) {
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id)); const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
const failedIds = new Set( const failedIds = new Set(
@ -861,6 +1273,12 @@ var KindleZoteroImporter = {
blockedRecreateMissingTag: 0, blockedRecreateMissingTag: 0,
recreateDeclined: 0, recreateDeclined: 0,
removedDuplicates: 0, removedDuplicates: 0,
relationsAdded: 0,
relationsSkipped: 0,
relationsPlanned: 0,
commentLinksAdded: 0,
commentLinksPlanned: 0,
relationFailures: [],
deletedForIncremental: 0, deletedForIncremental: 0,
failedDeletionIds: [], failedDeletionIds: [],
maintenanceDeletedIds: [], maintenanceDeletedIds: [],
@ -1045,6 +1463,14 @@ var KindleZoteroImporter = {
} }
} }
const currentIDs = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
const relationEntries = (plan.all_annotations || []).filter(
(entry) => !currentIDs.has(entry.clipping_id)
);
relationEntries.push(...successfulEntries.values());
await this.addRelatedItemRelations(relationEntries, dryRun, results);
await this.backfillRelationLinks(relationEntries, dryRun, results);
if (!dryRun && plan.deletions && plan.deletions.length) { if (!dryRun && plan.deletions && plan.deletions.length) {
const deletionIds = new Set(plan.deletions); const deletionIds = new Set(plan.deletions);
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds); const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
@ -1146,7 +1572,7 @@ function uninstall(_data, _reason) {}
async function startup(data, _reason) { async function startup(data, _reason) {
try { try {
await KindleZoteroImporter.startup(data); await Kanzi.startup(data);
} catch (error) { } catch (error) {
Zotero.logError(error); Zotero.logError(error);
} }
@ -1154,7 +1580,7 @@ async function startup(data, _reason) {
function shutdown(_data, _reason) { function shutdown(_data, _reason) {
try { try {
KindleZoteroImporter.shutdown(); Kanzi.shutdown();
} catch (error) { } catch (error) {
Zotero.logError(error); Zotero.logError(error);
} }
@ -1162,7 +1588,7 @@ function shutdown(_data, _reason) {
function onMainWindowLoad({ window }) { function onMainWindowLoad({ window }) {
try { try {
KindleZoteroImporter.onMainWindowLoad(window); Kanzi.onMainWindowLoad(window);
} catch (error) { } catch (error) {
Zotero.logError(error); Zotero.logError(error);
} }
@ -1170,7 +1596,7 @@ function onMainWindowLoad({ window }) {
function onMainWindowUnload({ window }) { function onMainWindowUnload({ window }) {
try { try {
KindleZoteroImporter.onMainWindowUnload(window); Kanzi.onMainWindowUnload(window);
} catch (error) { } catch (error) {
Zotero.logError(error); Zotero.logError(error);
} }

File diff suppressed because it is too large Load diff

View file

@ -1,7 +1,7 @@
{ {
"manifest_version": 2, "manifest_version": 2,
"name": "Kanzi", "name": "Kanzi",
"version": "0.7.0", "version": "0.7.1",
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.", "description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi", "homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
"author": "Utku Bilen Demir", "author": "Utku Bilen Demir",

View file

@ -1,6 +1,6 @@
[project] [project]
name = "kindle-zotero-importer" name = "kindle-zotero-importer"
version = "0.7.0" version = "0.7.1"
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files." description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"

View file

@ -1,6 +1,7 @@
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import copy
import json import json
import sys import sys
from pathlib import Path from pathlib import Path
@ -8,7 +9,12 @@ from pathlib import Path
from .clippings import clippings_to_jsonable, load_clippings from .clippings import clippings_to_jsonable, load_clippings
from .epub_position import add_epub_positions from .epub_position import add_epub_positions
from .final_plan import FINAL_FORMAT, build_final_writer_plan from .final_plan import FINAL_FORMAT, build_final_writer_plan
from .import_plan import DEFAULT_COLOUR_MAP, _attach_notes_to_highlights, build_import_plan from .import_plan import (
DEFAULT_COLOUR_MAP,
_attach_notes_to_highlights,
_related_items_for_citekeys,
build_import_plan,
)
from .matcher import build_match_report, load_json from .matcher import build_match_report, load_json
from .mismatch_review import build_mismatch_review from .mismatch_review import build_mismatch_review
from .overrides import generate_override_skeleton, load_overrides from .overrides import generate_override_skeleton, load_overrides
@ -65,6 +71,20 @@ def _merge_annotation_history(
) )
def _refresh_history_relations(
annotations: list[dict], zotero_index: dict
) -> None:
"""Resolve citekeys for old annotations without re-positioning them."""
items_by_id = {item["item_id"]: item for item in zotero_index.get("items", [])}
for entry in annotations:
annotation = entry.get("annotation") or {}
entry["related_items"] = _related_items_for_citekeys(
(annotation.get("text") or "", annotation.get("comment") or ""),
entry.get("parent_item_id"),
items_by_id,
)
def _changed_note_clipping_ids( def _changed_note_clipping_ids(
current_clippings: list[dict], previous_annotations: list[dict] current_clippings: list[dict], previous_annotations: list[dict]
) -> set[str]: ) -> set[str]:
@ -112,6 +132,137 @@ def _deletion_ids(
return deleted return deleted
def _deferred_clipping_ids(
previous_plan: dict | None, current_ids: set[str]
) -> set[str]:
return {
clipping_id
for clipping_id in (previous_plan or {}).get("deferred_clipping_ids", [])
if clipping_id in current_ids
}
def _discarded_clipping_ids(
previous_plan: dict | None, current_ids: set[str]
) -> set[str]:
"""Clips the user permanently discarded in Conflicts (e.g. standalone notes)."""
return {
clipping_id
for clipping_id in (previous_plan or {}).get("discarded_clipping_ids", [])
if clipping_id in current_ids
}
def _changed_override_titles(
previous_snapshot: dict | None, current_overrides: dict
) -> set[str]:
if previous_snapshot is None:
return set(current_overrides)
return {
title
for title in set(previous_snapshot) | set(current_overrides)
if previous_snapshot.get(title) != current_overrides.get(title)
}
def _clipping_ids_for_titles(
clippings: list[dict], titles: set[str]
) -> set[str]:
return {clipping["id"] for clipping in clippings if clipping["title"] in titles}
def _reattempt_failed_with_alternate_attachments(positioned_plan: dict) -> dict:
"""Retry text/rect failures against another attachment of the same item.
`_choose_attachment` commits to one file (e.g. the sole EPUB) without
comparing text. When that file lacks the quote but a sibling PDF/EPUB
has it (e.g. Dostoyevsky EPUB vs PDF), re-anchor the clone there and
keep the winner, tagged `positioned-via-alternate-attachment`.
Partial shingle anchors are also upgraded when a sibling holds the
full quote.
"""
retryable = {"epub-text-not-found", "pdf-text-not-found", "pdf-rects-not-found"}
retries: list[dict] = []
upgrade_ids: set[str] = set()
for item in positioned_plan.get("items", []):
if item.get("status") in retryable:
pass
elif item.get("status") == "positioned" and "partial-text-match" in (
item.get("problems") or []
):
clipping_id = (item.get("clipping") or {}).get("id")
if clipping_id:
upgrade_ids.add(clipping_id)
else:
continue
zotero = item.get("zotero") or {}
current = zotero.get("attachment") or {}
choices = zotero.get("attachment_choices") or []
alternates = [
choice
for choice in choices
if choice.get("item_id") != current.get("item_id")
]
if not alternates:
continue
alternates.sort(
key=lambda choice: 0
if choice.get("content_type") != current.get("content_type")
else 1
)
clone = copy.deepcopy(item)
clone_zotero = dict(clone.get("zotero") or {})
clone_zotero["attachment"] = alternates[0]
clone["zotero"] = clone_zotero
clone["status"] = "ready-for-positioning"
clone["problems"] = []
retries.append(clone)
if not retries:
return positioned_plan
mini = {"format": positioned_plan.get("format"), "items": retries}
mini = add_epub_positions(mini)
mini = add_pdf_positions(mini)
winners = {}
for attempt in mini.get("items", []):
clipping_id = (attempt.get("clipping") or {}).get("id")
if clipping_id and attempt.get("status") == "positioned":
winners[clipping_id] = attempt
if not winners:
return positioned_plan
merged = []
for item in positioned_plan.get("items", []):
clipping_id = (item.get("clipping") or {}).get("id")
if clipping_id in winners:
won = winners[clipping_id]
if clipping_id in upgrade_ids and "partial-text-match" in (
won.get("problems") or []
):
merged.append(item)
continue
won = dict(won)
carried = [
problem
for problem in item.get("problems", [])
if problem != "partial-text-match"
or "partial-text-match" in (won.get("problems") or [])
]
won["problems"] = [
*carried,
"positioned-via-alternate-attachment",
]
merged.append(won)
else:
merged.append(item)
updated = dict(positioned_plan)
updated["items"] = merged
status_counts: dict[str, int] = {}
for entry in merged:
status = entry.get("status", "unknown")
status_counts[status] = status_counts.get(status, 0) + 1
updated["status_counts"] = status_counts
return updated
def _add_previous_imported_comments( def _add_previous_imported_comments(
annotations: list[dict], previous_plan: dict | None annotations: list[dict], previous_plan: dict | None
) -> None: ) -> None:
@ -134,7 +285,12 @@ def _unresolved_clipping_ids(
return { return {
item.get("clipping", {}).get("id") item.get("clipping", {}).get("id")
for item in (previous_plan or {}).get("items", []) for item in (previous_plan or {}).get("items", [])
if item.get("status") not in {"positioned", "ignored-title"} if item.get("status") not in {
"positioned",
"ignored-title",
"pdf-position-skipped-non-highlight",
"epub-position-skipped-non-highlight",
}
and item.get("clipping", {}).get("id") in source_ids and item.get("clipping", {}).get("id") in source_ids
} }
@ -535,18 +691,12 @@ def main(argv: list[str] | None = None) -> int:
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {} _ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
except Exception: except Exception:
_ov_for_inc = {} _ov_for_inc = {}
previous_override_snapshot = (prev_final_plan or {}).get("overrides_snapshot") changed_override_titles = _changed_override_titles(
if previous_override_snapshot is None: (prev_final_plan or {}).get("overrides_snapshot"), _ov_for_inc
changed_override_titles = set(_ov_for_inc) )
else: to_process_ids.update(
changed_override_titles = { _clipping_ids_for_titles(effective_clippings, changed_override_titles)
title )
for title in set(previous_override_snapshot) | set(_ov_for_inc)
if previous_override_snapshot.get(title) != _ov_for_inc.get(title)
}
for c in effective_clippings:
if c["title"] in changed_override_titles:
to_process_ids.add(c["id"])
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", []) previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
for annotation in previous_annotations: for annotation in previous_annotations:
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"): if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
@ -563,10 +713,15 @@ def main(argv: list[str] | None = None) -> int:
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", []) for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
if clipping_id in new_ids if clipping_id in new_ids
) )
to_process_ids.update(_deferred_clipping_ids(prev_final_plan, new_ids))
if args.retry_unresolved and prev_positioned_plan: if args.retry_unresolved and prev_positioned_plan:
to_process_ids.update( to_process_ids.update(
_unresolved_clipping_ids(prev_positioned_plan, new_ids) _unresolved_clipping_ids(prev_positioned_plan, new_ids)
) )
# Permanently discarded clips (user action in Conflicts) never re-queue.
to_process_ids.difference_update(
_discarded_clipping_ids(prev_final_plan, new_ids)
)
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours) # If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
try: try:
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap") prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
@ -632,6 +787,7 @@ def main(argv: list[str] | None = None) -> int:
epub_plan = add_epub_positions(plan) epub_plan = add_epub_positions(plan)
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)") report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
positioned_plan = add_pdf_positions(epub_plan) positioned_plan = add_pdf_positions(epub_plan)
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report") report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan) final_plan = build_final_writer_plan(positioned_plan)
# Attach deletions for writer # Attach deletions for writer
@ -665,6 +821,7 @@ def main(argv: list[str] | None = None) -> int:
epub_plan = add_epub_positions(plan) epub_plan = add_epub_positions(plan)
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage") report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
positioned_plan = add_pdf_positions(epub_plan) positioned_plan = add_pdf_positions(epub_plan)
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report") report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
final_plan = build_final_writer_plan(positioned_plan) final_plan = build_final_writer_plan(positioned_plan)
final_plan["deletions"] = sorted( final_plan["deletions"] = sorted(
@ -700,10 +857,14 @@ def main(argv: list[str] | None = None) -> int:
final_plan.get("annotations", []), final_plan.get("annotations", []),
final_plan.get("deletions", []), final_plan.get("deletions", []),
) )
_refresh_history_relations(final_plan["all_annotations"], zotero_index)
final_plan["annotation_count"] = len(final_plan["all_annotations"]) final_plan["annotation_count"] = len(final_plan["all_annotations"])
final_plan["delta_annotation_count"] = delta_annotation_count final_plan["delta_annotation_count"] = delta_annotation_count
final_plan["processed_clipping_ids"] = sorted(new_ids) final_plan["processed_clipping_ids"] = sorted(new_ids)
final_plan["overrides_snapshot"] = overrides or {} final_plan["overrides_snapshot"] = overrides or {}
final_plan["discarded_clipping_ids"] = sorted(
_discarded_clipping_ids(prev_final_plan, new_ids)
)
indent = 2 if args.pretty else None indent = 2 if args.pretty else None
outputs = { outputs = {

View file

@ -4,6 +4,7 @@ from dataclasses import dataclass
from html import unescape from html import unescape
from html.parser import HTMLParser from html.parser import HTMLParser
import posixpath import posixpath
import unicodedata
import zipfile import zipfile
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from typing import Any from typing import Any
@ -68,6 +69,8 @@ def _position_item(
annotation["sortIndex"] = position["sortIndex"] annotation["sortIndex"] = position["sortIndex"]
updated["annotation"] = annotation updated["annotation"] = annotation
updated["status"] = "positioned" updated["status"] = "positioned"
if position.get("partial"):
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
return updated return updated
@ -104,11 +107,42 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
return None return None
match_start = normalized_haystack.find(normalized_quote) match_start = normalized_haystack.find(normalized_quote)
if match_start < 0: raw_start: int | None = None
return None raw_end: int | None = None
partial = False
if match_start >= 0:
match_end = match_start + len(normalized_quote) match_end = match_start + len(normalized_quote)
raw_start = haystack_map[match_start] raw_start = haystack_map[match_start]
raw_end = haystack_map[match_end - 1] + 1 raw_end = haystack_map[match_end - 1] + 1
else:
stripped_quote, _ = _strip_with_map(normalized_quote)
if stripped_quote and len(stripped_quote) >= 20:
stripped_haystack, stripped_map = _strip_with_map(normalized_haystack)
strip_start = stripped_haystack.find(stripped_quote)
strip_len = len(stripped_quote)
if strip_start < 0:
hit = _partial_stripped_range(stripped_haystack, stripped_quote)
if hit is not None:
strip_start, strip_end = hit
strip_len = strip_end - strip_start
partial = True
if strip_start >= 0:
# stripped_map points into normalized_haystack;
# convert via haystack_map back to raw haystack offsets.
norm_start = stripped_map[strip_start]
norm_end = stripped_map[strip_start + strip_len - 1]
raw_start = haystack_map[norm_start]
raw_end = haystack_map[norm_end] + 1
else:
span = _shingle_span(normalized_haystack, _quote_words(normalized_quote))
if span is not None:
raw_start = haystack_map[span[0]]
raw_end = haystack_map[span[1] - 1] + 1
partial = True
else:
partial = False
if raw_start is None or raw_end is None:
return None
# Account for the spaces inserted between text nodes in haystack construction. # Account for the spaces inserted between text nodes in haystack construction.
adjusted_nodes = _nodes_for_joined_text(text_nodes) adjusted_nodes = _nodes_for_joined_text(text_nodes)
@ -139,6 +173,7 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
"value": value, "value": value,
}, },
"sortIndex": f"{spine_step:05d}|{raw_start:08d}", "sortIndex": f"{spine_step:05d}|{raw_start:08d}",
"partial": partial,
} }
@ -213,6 +248,11 @@ def _clean_text(text: str) -> str:
def _normalize_with_map(text: str) -> tuple[str, list[int]]: def _normalize_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text)
text = (
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
)
normalized = [] normalized = []
mapping = [] mapping = []
last_was_space = False last_was_space = False
@ -232,6 +272,58 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
return "".join(normalized), mapping return "".join(normalized), mapping
def _strip_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text).casefold()
stripped: list[str] = []
mapping: list[int] = []
for index, char in enumerate(text):
if char.isalnum():
stripped.append(char)
mapping.append(index)
return "".join(stripped), mapping
def _partial_stripped_range(
haystack: str, quote: str, min_len: int = 30
) -> tuple[int, int] | None:
if len(quote) < min_len * 2:
return None
mid = len(quote) // 2
for part in (quote[:mid], quote[mid:]):
if len(part) < min_len:
continue
start = haystack.find(part)
if start >= 0:
return start, start + len(part)
return None
def _quote_words(normalized: str) -> list[str]:
words: list[str] = []
for token in normalized.split():
cleaned = "".join(char for char in token if char.isalnum())
if cleaned:
words.append(cleaned)
return words
def _shingle_span(
normalized_text: str, quote_words: list[str], width: int = 6
) -> tuple[int, int] | None:
first: int | None = None
last: int | None = None
for start in range(len(quote_words) - width + 1):
needle = " ".join(quote_words[start : start + width])
at = normalized_text.find(needle)
if at >= 0:
end = at + len(needle)
first = at if first is None else min(first, at)
last = end if last is None else max(last, end)
if first is None or last is None:
return None
return first, last
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]: def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
nodes = [] nodes = []
cursor = 0 cursor = 0

View file

@ -71,6 +71,7 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
"attachment_key": attachment["key"], "attachment_key": attachment["key"],
"parent_item_id": item["zotero"]["parent_item_id"], "parent_item_id": item["zotero"]["parent_item_id"],
"parent_key": item["zotero"]["parent_key"], "parent_key": item["zotero"]["parent_key"],
"related_items": item["zotero"].get("related_items", []),
"citation_key": item["zotero"].get("citation_key"), "citation_key": item["zotero"].get("citation_key"),
"doi": item["zotero"].get("doi"), "doi": item["zotero"].get("doi"),
"isbn": item["zotero"].get("isbn"), "isbn": item["zotero"].get("isbn"),

View file

@ -124,6 +124,12 @@ def _plan_clipping(
zotero_item, overrides_by_title.get(clipping["title"]) or {} zotero_item, overrides_by_title.get(clipping["title"]) or {}
) )
status = "ready-for-positioning" if attachment else attachment_status status = "ready-for-positioning" if attachment else attachment_status
stub = _annotation_stub(clipping, colour_map) if attachment else None
related_items = (
_append_citekey_links(stub, clipping, zotero_item, items_by_id)
if stub
else []
)
return { return {
"clipping": clipping, "clipping": clipping,
"status": status, "status": status,
@ -138,17 +144,96 @@ def _plan_clipping(
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None, "issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
"url": zotero_item["fields"].get("url"), "url": zotero_item["fields"].get("url"),
"publication_title": zotero_item["fields"].get("publicationTitle"), "publication_title": zotero_item["fields"].get("publicationTitle"),
"related_items": related_items,
"attachment": attachment, "attachment": attachment,
"attachment_choices": zotero_item.get("attachments", []), "attachment_choices": zotero_item.get("attachments", []),
"expected_attachment_type": _expected_attachment_type( "expected_attachment_type": _expected_attachment_type(
clipping, zotero_item.get("attachments", []) clipping, zotero_item.get("attachments", [])
), ),
}, },
"annotation": _annotation_stub(clipping, colour_map) if attachment else None, "annotation": stub,
"problems": problems, "problems": problems,
} }
def _append_citekey_links(
stub: dict[str, Any],
clipping: dict[str, Any],
parent_item: dict[str, Any],
items_by_id: dict[int, dict[str, Any]],
) -> list[dict[str, Any]]:
"""Turn `@citekey` mentions into clickable Zotero item links.
Kindle-side `@key` tokens (Obsidian-style) resolve against the library
index; matches to *other* items are appended to the annotation comment
as `zotero://select` links, which Zotero renders clickable. Self-links
to the parent item are skipped.
"""
related_items = _related_items_for_citekeys(
(clipping.get("text") or "", clipping.get("comment") or ""),
parent_item.get("item_id"),
items_by_id,
)
links = [
f"@{target['citation_key']} {target['link']}" for target in related_items
]
if not links:
return []
existing = stub.get("comment") or ""
block = "Related: " + " · ".join(links)
if block not in existing:
stub["comment"] = f"{existing}\n\n{block}" if existing else block
return related_items
def _select_link(item: dict[str, Any]) -> str:
"""Library-aware `zotero://select` link for an indexed item.
Group-library targets need the `/groups/{groupID}/items/{key}` form;
personal-library (or unknown) targets use `/library/items/{key}`.
"""
group_id = item.get("group_id")
if group_id:
return f"zotero://select/groups/{group_id}/items/{item['key']}"
return f"zotero://select/library/items/{item['key']}"
def _related_items_for_citekeys(
sources: tuple[str, ...],
parent_item_id: int | None,
items_by_id: dict[int, dict[str, Any]],
) -> list[dict[str, Any]]:
tokens = {
token.casefold()
for source in sources
for token in re.findall(r"@([A-Za-z0-9_][\w:.-]*)", source or "")
}
if not tokens:
return []
by_citekey = {
str(item.get("fields", {}).get("citationKey", "")).casefold(): item
for item in items_by_id.values()
if item.get("fields", {}).get("citationKey")
}
related = []
seen_item_ids = set()
for token in sorted(tokens):
target = by_citekey.get(token)
target_id = target.get("item_id") if target else None
if not target or target_id == parent_item_id or target_id in seen_item_ids:
continue
seen_item_ids.add(target_id)
related.append(
{
"item_id": target_id,
"key": target["key"],
"citation_key": target["fields"]["citationKey"],
"link": _select_link(target),
}
)
return related
def _choose_attachment( def _choose_attachment(
zotero_item: dict[str, Any], zotero_item: dict[str, Any],
override: dict[str, Any] | None = None, override: dict[str, Any] | None = None,

View file

@ -7,6 +7,7 @@ import re
import shutil import shutil
import subprocess import subprocess
import tempfile import tempfile
import unicodedata
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@ -126,7 +127,12 @@ def _position_item(
text_pages, item["clipping"]["text"], item["clipping"].get("page") text_pages, item["clipping"]["text"], item["clipping"].get("page")
) )
if page_index is None: if page_index is None:
return _with_problem(item, "pdf-text-not-found") failed = _with_problem(item, "pdf-text-not-found")
hint = _page_hint_text(text_pages, item["clipping"].get("page"))
if hint:
failed = dict(failed)
failed["page_hint"] = hint
return failed
page_xml = xml_cache.setdefault( page_xml = xml_cache.setdefault(
(extraction_path, page_index), (extraction_path, page_index),
extract_pdf_page_xml(extraction_path, page_index), extract_pdf_page_xml(extraction_path, page_index),
@ -152,6 +158,8 @@ def _position_item(
annotation["sortIndex"] = position.get("sortIndex") annotation["sortIndex"] = position.get("sortIndex")
updated["annotation"] = annotation updated["annotation"] = annotation
updated["status"] = "positioned" updated["status"] = "positioned"
if position.get("partial"):
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
return updated return updated
@ -227,6 +235,37 @@ def find_pdf_text_page(
normalized_page, _ = _normalize_with_map(pages[index]) normalized_page, _ = _normalize_with_map(pages[index])
if normalized_quote in normalized_page: if normalized_quote in normalized_page:
return index return index
# Fallback: alphanumeric-only search tolerates PDF extraction artifacts
# such as "o f" for "of", hyphenation, and punctuation differences.
stripped_quote, _ = _strip_with_map(quote)
if stripped_quote and len(stripped_quote) >= 20:
for index in page_order:
stripped_page, _ = _strip_with_map(pages[index])
if stripped_quote in stripped_page:
return index
# Partial fallback: long highlights may span pages or contain a
# divergent middle (edition differences). Anchor to either half so
# Zotero-style word search behaviour is approximated.
for index in page_order:
stripped_page, _ = _strip_with_map(pages[index])
if _partial_stripped_hit(stripped_page, stripped_quote):
return index
# Shingle fallback: scattered OCR substitutions (e.g. Kindle
# "yadsınaraz" vs PDF "yadsınamaz") break every long substring but
# leave most 6-word runs intact. Rank pages by shingle hits.
quote_words = _quote_words(normalized_quote)
if len(quote_words) >= 6:
required = 2 if len(quote_words) >= 12 else 1
best_index: int | None = None
best_hits = 0
for index in page_order:
normalized_page, _ = _normalize_with_map(pages[index])
hits = _shingle_hits(normalized_page, quote_words)
if hits > best_hits:
best_hits = hits
best_index = index
if best_index is not None and best_hits >= required:
return best_index
return None return None
@ -290,25 +329,52 @@ def find_pdf_rects(
text = " ".join(word.text for word in page.words) text = " ".join(word.text for word in page.words)
normalized_text, text_map = _normalize_with_map(text) normalized_text, text_map = _normalize_with_map(text)
start = normalized_text.find(normalized_quote) start = normalized_text.find(normalized_quote)
if start < 0: raw_start: int | None = None
return None raw_end: int | None = None
partial = False
if start >= 0:
end = start + len(normalized_quote) end = start + len(normalized_quote)
raw_start = text_map[start] raw_start = text_map[start]
raw_end = text_map[end - 1] + 1 raw_end = text_map[end - 1] + 1
else:
stripped_quote, _ = _strip_with_map(quote)
if stripped_quote and len(stripped_quote) >= 20:
stripped_text, stripped_map = _strip_with_map(text)
strip_start = stripped_text.find(stripped_quote)
if strip_start >= 0:
strip_end = strip_start + len(stripped_quote)
raw_start = stripped_map[strip_start]
raw_end = stripped_map[strip_end - 1] + 1
else:
hit = _partial_stripped_range(stripped_text, stripped_quote)
if hit is not None:
strip_start, strip_end = hit
raw_start = stripped_map[strip_start]
raw_end = stripped_map[strip_end - 1] + 1
partial = True
else:
span = _shingle_span(normalized_text, _quote_words(normalized_quote))
if span is not None:
raw_start = text_map[span[0]]
raw_end = text_map[span[1] - 1] + 1
partial = True
if raw_start is None or raw_end is None:
return None
matched_words = [ matched_words = [
word for word in page.words if word.end >= raw_start and word.start <= raw_end word for word in page.words if word.end >= raw_start and word.start <= raw_end
] ]
if not matched_words: if not matched_words:
return None return None
pdf_width, pdf_height = size_cache.setdefault( size_key = (path, page.index)
(path, page.index), if size_key not in size_cache:
_pdf_page_size(path, page.index + 1, page.width, page.height), size_cache[size_key] = _pdf_page_size(path, page.index + 1, page.width, page.height)
) pdf_width, pdf_height = size_cache[size_key]
rects = _word_rects(matched_words, page, pdf_width, pdf_height) rects = _word_rects(matched_words, page, pdf_width, pdf_height)
return { return {
"pageIndex": page.index, "pageIndex": page.index,
"rects": rects, "rects": rects,
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}", "sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
"partial": partial,
} }
@ -361,6 +427,11 @@ def _pdf_page_size(
def _normalize_with_map(text: str) -> tuple[str, list[int]]: def _normalize_with_map(text: str) -> tuple[str, list[int]]:
text = unicodedata.normalize("NFKC", text)
text = (
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
)
normalized = [] normalized = []
mapping = [] mapping = []
last_was_space = False last_was_space = False
@ -380,6 +451,112 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
return "".join(normalized), mapping return "".join(normalized), mapping
def _strip_with_map(text: str) -> tuple[str, list[int]]:
"""Alphanumeric-only folding with index mapping.
Removes spaces and punctuation so PDF artifacts like "o f" for "of",
hyphenated line breaks, and curly-quote/dash variants still match.
Mapping points back into the input string for rect recovery.
"""
text = unicodedata.normalize("NFKC", text).casefold()
stripped: list[str] = []
mapping: list[int] = []
for index, char in enumerate(text):
if char.isalnum():
stripped.append(char)
mapping.append(index)
return "".join(stripped), mapping
def _partial_stripped_range(
haystack: str, quote: str, min_len: int = 30
) -> tuple[int, int] | None:
"""Best half-quote hit for long highlights spanning pages/editions."""
if len(quote) < min_len * 2:
return None
mid = len(quote) // 2
for part in (quote[:mid], quote[mid:]):
if len(part) < min_len:
continue
start = haystack.find(part)
if start >= 0:
return start, start + len(part)
return None
def _partial_stripped_hit(haystack: str, quote: str, min_len: int = 30) -> bool:
return _partial_stripped_range(haystack, quote, min_len) is not None
def _quote_words(normalized: str, width: int = 6) -> list[str]:
del width
words: list[str] = []
for token in normalized.split():
cleaned = "".join(char for char in token if char.isalnum())
if cleaned:
words.append(cleaned)
return words
def _shingle_hits(normalized_page: str, quote_words: list[str], width: int = 6) -> int:
hits = 0
seen: set[tuple[str, ...]] = set()
for start in range(len(quote_words) - width + 1):
shingle = tuple(quote_words[start : start + width])
if shingle in seen:
continue
seen.add(shingle)
if " ".join(shingle) in normalized_page:
hits += 1
return hits
def _shingle_span(
normalized_text: str, quote_words: list[str], width: int = 6
) -> tuple[int, int] | None:
first: int | None = None
last: int | None = None
for start in range(len(quote_words) - width + 1):
needle = " ".join(quote_words[start : start + width])
at = normalized_text.find(needle)
if at >= 0:
end = at + len(needle)
first = at if first is None else min(first, at)
last = end if last is None else max(last, end)
if first is None or last is None:
return None
return first, last
def _page_hint_text(
pages: list[str], kindle_page: str | None, radius: int = 1, limit: int = 1200
) -> dict[str, Any] | None:
"""Snippet of the PDF pages around the Kindle page hint.
Lets the user eyeball which part a `pdf-text-not-found` highlight
belongs to and place it manually. Truncated to keep artifacts small.
"""
if not kindle_page or not kindle_page.isdigit():
return None
center = int(kindle_page) - 1
if center < 0 or center >= len(pages):
return None
parts = []
for index in range(max(0, center - radius), min(len(pages), center + radius + 1)):
snippet = " ".join(pages[index].split())
if snippet:
parts.append(f"[PDF p.{index + 1}] {snippet[:limit]}")
if not parts:
return None
return {
"kindle_page": kindle_page,
"pdf_pages": list(
range(max(1, center - radius + 1), min(len(pages), center + radius + 1) + 1)
),
"text": "\n\n".join(parts)[: limit * 2 + 64],
}
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]: def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
updated = dict(item) updated = dict(item)
updated["status"] = problem updated["status"] = problem

View file

@ -34,6 +34,8 @@ class ZoteroItem:
key: str key: str
item_type: str item_type: str
title: str | None title: str | None
library_id: int | None = None
group_id: int | None = None
creators: list[str] = field(default_factory=list) creators: list[str] = field(default_factory=list)
fields: dict[str, str] = field(default_factory=dict) fields: dict[str, str] = field(default_factory=dict)
attachments: list[ZoteroAttachment] = field(default_factory=list) attachments: list[ZoteroAttachment] = field(default_factory=list)
@ -47,15 +49,19 @@ def build_zotero_index(db_path: str, storage_root: str | None = None) -> dict[st
creators = _load_creators(connection) creators = _load_creators(connection)
attachments = _load_attachments(connection, storage_root) attachments = _load_attachments(connection, storage_root)
groups = _load_groups(connection)
indexed_items: list[ZoteroItem] = [] indexed_items: list[ZoteroItem] = []
for row in items: for row in items:
fields = _metadata_for_item(row) fields = _metadata_for_item(row)
library_id = row["libraryID"]
indexed_items.append( indexed_items.append(
ZoteroItem( ZoteroItem(
item_id=row["itemID"], item_id=row["itemID"],
key=row["key"], key=row["key"],
item_type=row["typeName"], item_type=row["typeName"],
title=fields.get("title"), title=fields.get("title"),
library_id=library_id,
group_id=groups.get(library_id),
creators=creators.get(row["itemID"], []), creators=creators.get(row["itemID"], []),
fields=fields, fields=fields,
attachments=attachments.get(row["itemID"], []), attachments=attachments.get(row["itemID"], []),
@ -80,6 +86,7 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
SELECT SELECT
i.itemID, i.itemID,
i.key, i.key,
i.libraryID,
it.typeName, it.typeName,
{field_columns} {field_columns}
FROM items i FROM items i
@ -95,6 +102,14 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
return list(connection.execute(query)) return list(connection.execute(query))
def _load_groups(connection: sqlite3.Connection) -> dict[int, int]:
try:
rows = connection.execute("SELECT groupID, libraryID FROM groups")
except Exception:
return {}
return {row["libraryID"]: row["groupID"] for row in rows}
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]: def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
rows = connection.execute( rows = connection.execute(
""" """

View file

@ -0,0 +1,15 @@
Kanzi Dummy EPUB
- Your Highlight on page 1 | location 100-120 | Added on Tuesday, 18 May 2021 14:18:13
This is a test highlight long enough to import for EPUB.
==========
Kanzi Dummy PDF
- Your Highlight on page 2 | location 200-220 | Added on Tuesday, 18 May 2021 14:20:00
This is a test highlight long enough to import for PDF.
==========
Unknown Book Title XYZ
- Your Highlight on page 1 | location 10-11 | Added on Tuesday, 18 May 2021 14:18:13
This should land in Conflicts for manual mapping.
==========

View file

@ -0,0 +1,36 @@
Kanzi Windows smoke test (Parallels VM)
COPY THESE 3 FILES INTO THE VM:
- My Clippings.txt
- kanzi-dummy.epub
- kanzi-dummy.pdf
- dist/kanzi.xpi (from repo root, same cross-platform build)
1) PowerShell:
winget install --id Python.Python.3.13 --exact
winget install --id oschwartz10612.Poppler --exact
winget install --id QPDF.QPDF --exact
python --version
pdftotext -v
pdfinfo -v
qpdf --version
2) Fully quit + reopen Zotero (picks up PATH).
3) Fresh Zotero, sync off. Create:
- Book "Kanzi Dummy EPUB" + attach kanzi-dummy.epub
- Book "Kanzi Dummy PDF" + attach kanzi-dummy.pdf
4) Tools -> Kanzi -> Choose My Clippings.txt (the copied one).
Leave Python path empty to test auto-detect.
EXPECT:
- 2 highlights position and await review, 1 title in Conflicts
("Unknown Book Title XYZ").
- Summary shows new Status by Kindle Title table with source counts.
- Save a mapping for the unknown title -> Re-import processes it.
- Table headers sort and persist across pages.
IF PYTHON NOT FOUND:
Settings shows detected path hint; paste full python.exe path,
Save Settings, retry.

Binary file not shown.

View file

@ -0,0 +1,33 @@
%PDF-1.4
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
2 0 obj
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
endobj
3 0 obj
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
endobj
4 0 obj
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
endobj
5 0 obj
<< /Length 134 >>
stream
BT /F1 12 Tf 72 720 Td (Kanzi Dummy PDF) Tj ET
BT /F1 12 Tf 72 700 Td (This is a test highlight long enough to import for PDF.) Tj ET
endstream
endobj
xref
0 6
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000241 00000 n
0000000311 00000 n
trailer
<< /Size 6 /Root 1 0 R >>
startxref
495
%%EOF

View file

@ -0,0 +1,93 @@
"""Generate minimal EPUB + PDF fixtures for the Windows smoke test.
Run: python3 tests/fixtures/windows-smoke/make_fixtures.py
Outputs kanzi-dummy.epub and kanzi-dummy.pdf next to this script.
Both contain the highlight sentences from 'My Clippings.txt' so
EPUB CFI and PDF text positioning can succeed.
"""
from __future__ import annotations
from pathlib import Path
import zipfile
HERE = Path(__file__).resolve().parent
EPUB_TEXT = "This is a test highlight long enough to import for EPUB."
PDF_TEXT = "This is a test highlight long enough to import for PDF."
def make_epub(path: Path) -> None:
chapter = f"""<?xml version="1.0" encoding="utf-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head><title>Kanzi Dummy</title></head>
<body>
<h1>Kanzi Dummy EPUB</h1>
<p>Some intro text before the highlight.</p>
<p>{EPUB_TEXT}</p>
<p>Some trailing text after the highlight.</p>
</body>
</html>
"""
container = """<?xml version="1.0" encoding="utf-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
</container>
"""
opf = """<?xml version="1.0" encoding="utf-8"?>
<package version="3.0" unique-identifier="uid" xmlns="http://www.idpf.org/2007/opf">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Kanzi Dummy EPUB</dc:title><dc:identifier id="uid">kanzi-dummy-epub</dc:identifier><dc:language>en</dc:language></metadata>
<manifest><item id="ch1" href="chapter1.xhtml" media-type="application/xhtml+xml"/></manifest>
<spine><itemref idref="ch1"/></spine>
</package>
"""
with zipfile.ZipFile(path, "w") as z:
z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
z.writestr("META-INF/container.xml", container, compress_type=zipfile.ZIP_DEFLATED)
z.writestr("OEBPS/content.opf", opf, compress_type=zipfile.ZIP_DEFLATED)
z.writestr("OEBPS/chapter1.xhtml", chapter, compress_type=zipfile.ZIP_DEFLATED)
def _pdf_escape(text: str) -> str:
return text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
def make_pdf(path: Path) -> None:
# Minimal one-page PDF with Helvetica text. pdftotext extracts it fine.
content = (
f"BT /F1 12 Tf 72 720 Td ({_pdf_escape('Kanzi Dummy PDF')}) Tj ET\n"
f"BT /F1 12 Tf 72 700 Td ({_pdf_escape(PDF_TEXT)}) Tj ET\n"
)
objects: list[bytes] = []
objects.append(b"<< /Type /Catalog /Pages 2 0 R >>")
objects.append(b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
objects.append(
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"
)
objects.append(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
objects.append(f"<< /Length {len(content.encode('latin-1'))} >>\nstream\n{content}endstream".encode("latin-1"))
out = bytearray(b"%PDF-1.4\n")
offsets = [0]
for i, body in enumerate(objects, start=1):
offsets.append(len(out))
out += f"{i} 0 obj\n".encode("ascii") + body + b"\nendobj\n"
xref_pos = len(out)
out += f"xref\n0 {len(objects) + 1}\n".encode("ascii")
out += b"0000000000 65535 f \n"
for off in offsets[1:]:
out += f"{off:010d} 00000 n \n".encode("ascii")
out += (
f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
f"startxref\n{xref_pos}\n%%EOF\n"
).encode("ascii")
path.write_bytes(bytes(out))
def main() -> None:
make_epub(HERE / "kanzi-dummy.epub")
make_pdf(HERE / "kanzi-dummy.pdf")
print("wrote kanzi-dummy.epub + kanzi-dummy.pdf")
if __name__ == "__main__":
main()

View file

@ -28,10 +28,206 @@ def test_pdf_fallback_tag():
"format":"x", "format":"x",
"items":[ "items":[
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None}, {"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"}, "zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1","related_items":[{"item_id":20,"key":"P2","citation_key":"c2"}]},
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}} "annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
] ]
} }
plan = build_final_writer_plan(positioned) plan = build_final_writer_plan(positioned)
tags = plan["annotations"][0]["annotation"]["tags"] tags = plan["annotations"][0]["annotation"]["tags"]
assert any(t["name"].startswith("kindle-id:") for t in tags) assert any(t["name"].startswith("kindle-id:") for t in tags)
assert plan["annotations"][0]["related_items"][0]["item_id"] == 20
def test_pdf_tolerates_of_split_artifact():
from kindle_zotero_importer.pdf_position import find_pdf_text_page, find_pdf_rects, PDFPageXML, PDFWord
quote = "nothing other than idolatry of the machine and which"
page_text = "nothing other than idolatry o f the machine and which"
assert find_pdf_text_page([page_text], quote) == 0
words = []
cursor = 0
for w in page_text.split():
words.append(PDFWord(text=w, left=0.0, top=0.0, width=10.0, height=10.0, start=cursor, end=cursor + len(w)))
cursor += len(w) + 1
page = PDFPageXML(index=0, width=100.0, height=100.0, words=words)
rects = find_pdf_rects("dummy.pdf", page, quote, {("dummy.pdf", 0): (100.0, 100.0)})
assert rects is not None
assert rects["pageIndex"] == 0
assert rects["rects"]
def test_pdf_tolerates_punctuation_and_quotes():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
quote = "This, then, gives rise to “intemperate technicism”."
page = "this then gives rise to intemperate technicism"
assert find_pdf_text_page([page], quote) == 0
def test_pdf_partial_half_anchors_spanning_text():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
first = "A" * 70
second = "Z" * 70
quote = first + second
page = "xx " + first.lower() + " yy"
assert find_pdf_text_page([page], quote) == 0
def test_pdf_shingle_fallback_scattered_ocr_errors():
from kindle_zotero_importer.pdf_position import find_pdf_text_page
quote = (
"Gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
"buna inanmamasım sağlayacak güç ve yeteneği bulur mucize yadsınaraz "
"dururda bile olsa boyun eğmez"
)
page = (
"gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
"buna inanmaması sağlayacak güç ve yeteneği bulur mucize yadsınamaz "
"dururda bile olsa boyun eğmez"
)
assert find_pdf_text_page([page], quote) == 0
def test_page_hint_snippet_around_kindle_page():
from kindle_zotero_importer.pdf_position import _page_hint_text
pages = ["", "intro words here", "target page words here", "trailing words"]
hint = _page_hint_text(pages, "3")
assert hint is not None
assert hint["kindle_page"] == "3"
assert "target page words" in hint["text"]
assert _page_hint_text(pages, None) is None
assert _page_hint_text(pages, "99") is None
def test_citekey_mentions_become_zotero_links():
from kindle_zotero_importer.import_plan import build_import_plan
clippings_payload = {
"format": "x",
"clippings": [
{
"id": "c1",
"title": "Book A",
"kind": "highlight",
"text": "As argued by @bookb2020, this follows.",
"page": None,
"location": "10",
}
],
}
zotero_index = {
"format": "x",
"items": [
{
"item_id": 1,
"key": "PARENT01",
"title": "Book A",
"fields": {"citationKey": "booka2020"},
"attachments": [
{
"item_id": 11,
"key": "ATTACH01",
"title": "pdf",
"content_type": "application/pdf",
"path": "x.pdf",
"resolved_path": "x.pdf",
}
],
},
{
"item_id": 2,
"key": "TARGET02",
"title": "Book B",
"fields": {"citationKey": "bookb2020"},
"attachments": [],
},
],
}
match_report = {
"format": "x",
"matches": [
{
"clipping_title": "Book A",
"status": "matched",
"candidates": [{"item_id": 1}],
}
],
}
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
comment = plan["items"][0]["annotation"]["comment"] or ""
assert "@bookb2020" in comment
assert "zotero://select/library/items/TARGET02" in comment
assert plan["items"][0]["zotero"]["related_items"] == [
{
"item_id": 2,
"key": "TARGET02",
"citation_key": "bookb2020",
"link": "zotero://select/library/items/TARGET02",
}
]
# self-links are skipped
clippings_payload["clippings"][0]["text"] = "See @booka2020 here."
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
assert "zotero://select" not in (plan["items"][0]["annotation"]["comment"] or "")
assert plan["items"][0]["zotero"]["related_items"] == []
def test_historical_annotations_gain_citekey_relations():
from kindle_zotero_importer.cli import _refresh_history_relations
history = [
{
"clipping_id": "old",
"parent_item_id": 1,
"annotation": {
"text": "An old highlight",
"comment": "body without organs @deleuze1983",
},
}
]
zotero_index = {
"items": [
{
"item_id": 1,
"key": "SOURCE01",
"fields": {"citationKey": "simondon2017a"},
},
{
"item_id": 2,
"key": "TARGET02",
"fields": {"citationKey": "deleuze1983"},
},
]
}
_refresh_history_relations(history, zotero_index)
assert history[0]["related_items"] == [
{
"item_id": 2,
"key": "TARGET02",
"citation_key": "deleuze1983",
"link": "zotero://select/library/items/TARGET02",
}
]
def test_group_library_targets_use_group_select_links():
from kindle_zotero_importer.import_plan import _select_link
assert (
_select_link({"key": "ABC123", "group_id": 6644344})
== "zotero://select/groups/6644344/items/ABC123"
)
assert (
_select_link({"key": "ABC123", "group_id": None})
== "zotero://select/library/items/ABC123"
)
assert (
_select_link({"key": "ABC123"})
== "zotero://select/library/items/ABC123"
)
def test_epub_tolerates_extraction_artifacts():
from kindle_zotero_importer.epub_position import find_epub_cfi, TextNode
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="idolatry o f the machine", start=0, end=24)]
quote = "idolatry of the machine and which"
# quote is longer than node, so exact fails; use full haystack variant
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="nothing other than idolatry o f the machine and which", start=0, end=53)]
result = find_epub_cfi(nodes, quote)
assert result is not None
assert result["selector"]["value"].startswith("epubcfi(")

View file

@ -6,7 +6,11 @@ from kindle_zotero_importer import cli
from kindle_zotero_importer.cli import ( from kindle_zotero_importer.cli import (
_add_previous_imported_comments, _add_previous_imported_comments,
_changed_note_clipping_ids, _changed_note_clipping_ids,
_changed_override_titles,
_clipping_ids_for_titles,
_deferred_clipping_ids,
_deletion_ids, _deletion_ids,
_discarded_clipping_ids,
_merge_annotation_history, _merge_annotation_history,
_merge_positioned_history, _merge_positioned_history,
_unresolved_clipping_ids, _unresolved_clipping_ids,
@ -78,6 +82,114 @@ def test_deletions_include_removed_and_previous_failures():
) == {"removed", "retry"} ) == {"removed", "retry"}
def test_deferred_clippings_are_reprocessed_while_they_remain_in_source():
previous = {"deferred_clipping_ids": ["retry", "removed"]}
assert _deferred_clipping_ids(previous, {"retry", "current"}) == {"retry"}
def test_discarded_clips_never_requeue_and_skips_are_not_unresolved():
previous = {"discarded_clipping_ids": ["gone", "removed"]}
assert _discarded_clipping_ids(previous, {"gone", "current"}) == {"gone"}
plan = {
"items": [
{"clipping": {"id": "note"}, "status": "pdf-position-skipped-non-highlight"},
{"clipping": {"id": "fail"}, "status": "pdf-text-not-found"},
{"clipping": {"id": "done"}, "status": "positioned"},
]
}
assert _unresolved_clipping_ids(plan, {"note", "fail", "done"}) == {"fail"}
def test_alternate_attachment_retry_prefers_pdf_when_epub_misses():
from kindle_zotero_importer.cli import _reattempt_failed_with_alternate_attachments
from kindle_zotero_importer.epub_position import add_epub_positions
from kindle_zotero_importer.import_plan import build_import_plan
from kindle_zotero_importer.pdf_position import add_pdf_positions
fixtures = Path(__file__).parent / "fixtures" / "windows-smoke"
epub_path = str(fixtures / "kanzi-dummy.epub")
pdf_path = str(fixtures / "kanzi-dummy.pdf")
quote = "This is a test highlight long enough to import for PDF."
clippings_payload = {
"format": "kindle-zotero-importer.clippings.v1",
"clippings": [
{
"id": "q1",
"title": "Dummy Book",
"kind": "highlight",
"text": quote,
"page": None,
"location": "100",
}
],
}
zotero_index = {
"format": "index",
"items": [
{
"item_id": 1,
"key": "ABC12345",
"title": "Dummy Book",
"fields": {},
"attachments": [
{
"item_id": 11,
"key": "EPUBKEY1",
"title": "epub",
"content_type": "application/epub+zip",
"path": epub_path,
"resolved_path": epub_path,
},
{
"item_id": 12,
"key": "PDFKEY123",
"title": "pdf",
"content_type": "application/pdf",
"path": pdf_path,
"resolved_path": pdf_path,
},
],
}
],
}
match_report = {
"format": "matches",
"matches": [
{
"clipping_title": "Dummy Book",
"status": "matched",
"candidates": [{"item_id": 1}],
}
],
}
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
assert plan["items"][0]["zotero"]["attachment"]["content_type"] == "application/epub+zip"
positioned = add_pdf_positions(add_epub_positions(plan))
assert positioned["items"][0]["status"] == "positioned"
assert "partial-text-match" in positioned["items"][0]["problems"]
rescued = _reattempt_failed_with_alternate_attachments(positioned)
assert rescued["items"][0]["status"] == "positioned"
assert "positioned-via-alternate-attachment" in rescued["items"][0]["problems"]
assert "partial-text-match" not in rescued["items"][0]["problems"]
def test_changed_override_requeues_every_clipping_for_exact_title():
previous = {"Shared title": {"citation_key": "old"}}
current = {"Shared title": {"citation_key": "new"}}
clippings = [
{"id": "first", "title": "Shared title"},
{"id": "second", "title": "Shared title"},
{"id": "third", "title": "Different title"},
]
changed_titles = _changed_override_titles(previous, current)
assert changed_titles == {"Shared title"}
assert _clipping_ids_for_titles(clippings, changed_titles) == {"first", "second"}
def test_previous_imported_comment_is_attached_to_reprocessed_entry(): def test_previous_imported_comment_is_attached_to_reprocessed_entry():
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}] current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
previous = { previous = {
@ -240,6 +352,6 @@ def test_noop_incremental_writes_complete_artifacts():
assert result == 0 assert result == 0
final_plan = json.loads((workdir / "import-plan.final.json").read_text()) final_plan = json.loads((workdir / "import-plan.final.json").read_text())
assert final_plan["annotations"] == [] assert final_plan["annotations"] == []
assert final_plan["all_annotations"] == [annotation] assert final_plan["all_annotations"] == [{**annotation, "related_items": []}]
assert final_plan["delta_annotation_count"] == 0 assert final_plan["delta_annotation_count"] == 0
assert (workdir / "import-plan.epub.json").exists() assert (workdir / "import-plan.epub.json").exists()

View file

@ -4,7 +4,7 @@ const vm = require('vm');
const bootstrapContext = {}; const bootstrapContext = {};
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext); vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
const plugin = bootstrapContext.KindleZoteroImporter; const plugin = bootstrapContext.Kanzi;
const maintainImportedAnnotations = plugin.maintainImportedAnnotations; const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
assert.equal( assert.equal(
@ -96,6 +96,47 @@ const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
const managerContext = { TextEncoder }; const managerContext = { TextEncoder };
vm.runInNewContext(scripts[0][1], managerContext); vm.runInNewContext(scripts[0][1], managerContext);
const manager = managerContext.KindleManager; const manager = managerContext.KindleManager;
assert.equal(plugin.countClippingsForTitle({ clippings: [
{ title: 'Shared title' },
{ title: 'Shared title' },
{ title: 'Different title' },
] }, 'Shared title'), 2);
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
{ id: 'n1', title: 'Shared title', kind: 'note', text: 'note' },
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
] },
matches: { matches: [{
clipping_title: 'Shared title',
clipping_count: 3,
status: 'matched',
candidates: [],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [{
clipping: { id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
status: 'pdf-text-not-found',
problems: ['not found'],
}] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const conflictRows = manager.conflictRows();
assert.equal(conflictRows.length, 1);
assert.equal(conflictRows[0].count, 3);
assert.equal(conflictRows[0].clippings.length, 3);
assert.equal(conflictRows[0].highlights, 2);
assert.equal(conflictRows[0].notes, 1);
manager.sortState.conflicts = { column: 'count', direction: 'desc' };
const sortedRows = manager.sortRows(
[{ title: 'small', count: 2 }, { title: 'large', count: 20 }, { title: 'middle', count: 10 }],
'conflicts',
{ count: row => row.count }
);
assert.deepEqual(Array.from(sortedRows, row => row.title), ['large', 'middle', 'small']);
manager.integratedPage = 1; manager.integratedPage = 1;
manager.integratedPageSize = 50; manager.integratedPageSize = 50;
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize'); const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
@ -143,6 +184,34 @@ async function testRuntimeExtraction() {
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py'))); assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
} }
async function testPythonPathRecovery() {
bootstrapContext.Services = { appinfo: { OS: 'Darwin' } };
bootstrapContext.IOUtils = {
exists: async path => path === '/opt/homebrew/opt/python@3.13/bin/python3.13',
};
assert.equal(
await plugin.resolvePythonPath('/opt/homebrew/bin/python3'),
'/opt/homebrew/opt/python@3.13/bin/python3.13'
);
bootstrapContext.IOUtils.exists = async path => path === '/custom/python3';
assert.equal(await plugin.resolvePythonPath('/custom/python3'), '/custom/python3');
bootstrapContext.Services.appinfo.OS = 'WINNT';
const windowsEnvironment = {
LOCALAPPDATA: 'C:\\Users\\test\\AppData\\Local',
ProgramFiles: 'C:\\Program Files',
'ProgramFiles(x86)': 'C:\\Program Files (x86)',
USERPROFILE: 'C:\\Users\\test',
SystemDrive: 'C:',
PATH: 'C:\\Tools\\Python;C:\\Windows\\System32',
};
plugin.getEnvironmentVariable = name => windowsEnvironment[name] || '';
const expected = 'C:\\Users\\test\\AppData\\Local\\Programs\\Python\\Python313\\python.exe';
bootstrapContext.IOUtils.exists = async path => path === expected;
assert.equal(await plugin.resolvePythonPath('python'), expected);
assert.ok(plugin.getWindowsPythonCandidates('python').includes('C:\\Tools\\Python\\python.exe'));
}
async function testMaintenanceSweep() { async function testMaintenanceSweep() {
const annotation = (id, text, end, comment = '') => ({ const annotation = (id, text, end, comment = '') => ({
id, id,
@ -315,7 +384,317 @@ async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
assert.equal(results.failed[0].clipping_id, 'replacement'); assert.equal(results.failed[0].clipping_id, 'replacement');
} }
async function testCitekeyRelationsAreBidirectionalAndAdditive() {
const originalZotero = bootstrapContext.Zotero;
const originalCollect = plugin.collectImportedAnnotations;
const originalMaintain = plugin.maintainImportedAnnotations;
const item = (id, key) => ({
id,
key,
related: new Set(),
saved: 0,
addRelatedItem(other) {
if (this.related.has(other.key)) return false;
this.related.add(other.key);
return true;
},
async save(options) {
assert.equal(options.skipDateModifiedUpdate, true);
this.saved += 1;
},
});
const source = item(1, 'SOURCE01');
const target = item(2, 'TARGET02');
bootstrapContext.Zotero = {
Items: { get: id => id === 1 ? source : id === 2 ? target : id === 11 ? attachment : null },
DB: { executeTransaction: async callback => callback() },
logError: () => {},
};
const entries = [
{
clipping_id: 'history-1',
parent_item_id: 1,
attachment_item_id: 11,
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
},
{
parent_item_id: 1,
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
},
];
const existingAnnotation = {
annotationComment: 'an old note',
annotationTags: [{ name: 'kindle-id:history-1' }],
saved: 0,
saveTx: async function () { this.saved += 1; },
};
const attachment = { getAnnotations: () => [existingAnnotation] };
const results = {
relationsAdded: 0,
relationsSkipped: 0,
relationsPlanned: 0,
relationFailures: [],
};
try {
plugin.collectImportedAnnotations = async () => [];
plugin.maintainImportedAnnotations = async () => {};
const writerResults = await plugin.writeAnnotations({
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
annotations: [],
all_annotations: entries,
deletions: [],
}, false);
assert.deepEqual(Array.from(source.related), ['TARGET02']);
assert.deepEqual(Array.from(target.related), ['SOURCE01']);
assert.equal(source.saved, 1);
assert.equal(target.saved, 1);
assert.equal(writerResults.relationsAdded, 1);
assert.equal(writerResults.relationFailures.length, 0);
assert.equal(writerResults.commentLinksAdded, 1);
assert.ok(existingAnnotation.annotationComment.includes(
'Related: @target2020 zotero://select/library/items/TARGET02'
));
assert.equal(existingAnnotation.saved, 1);
await plugin.addRelatedItemRelations(entries, false, results);
assert.equal(source.saved, 1);
assert.equal(target.saved, 1);
assert.equal(results.relationsSkipped, 1);
} finally {
bootstrapContext.Zotero = originalZotero;
plugin.collectImportedAnnotations = originalCollect;
plugin.maintainImportedAnnotations = originalMaintain;
}
}
function testPendingReviewGrouping() {
const groups = manager.groupPendingAnnotations([
{ clipping_id: 'a1', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'spice', pageLabel: '12', color: '#ffd400' } },
{ clipping_id: 'a2', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'worm', pageLabel: '', color: '#ff6666' } },
{ clipping_id: 'b1', clipping_title: 'Unknown Book', citation_key: '', annotation: { text: 'mystery', pageLabel: '', color: '#ffd400' } },
]);
assert.equal(groups.length, 2);
assert.equal(groups[0].items.length, 2);
assert.equal(groups[0].citationKey, 'herbert2020');
assert.equal(groups[1].items.length, 1);
assert.deepEqual(manager.groupPendingAnnotations([]), []);
}
function testFilterPlanForSelection() {
const plan = {
annotations: [{ clipping_id: 'keep' }, { clipping_id: 'skip' }],
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'keep' }, { clipping_id: 'skip' }],
};
const previous = { deferred_clipping_ids: ['older'] };
const filtered = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep'], previous);
assert.deepEqual(Array.from(filtered.annotations, a => a.clipping_id), ['keep']);
assert.deepEqual(Array.from(filtered.all_annotations, a => a.clipping_id), ['old', 'keep']);
assert.deepEqual(Array.from(filtered.deferred_clipping_ids), ['older', 'skip']);
// Original plan is not mutated.
assert.equal(plan.annotations.length, 2);
const all = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep', 'skip'], previous);
assert.equal(all.annotations.length, 2);
assert.equal(all.all_annotations.length, 3);
assert.deepEqual(Array.from(all.deferred_clipping_ids), ['older']);
const priorAnnotation = { clipping_id: 'skip', annotation: { text: 'existing' } };
const preserved = plugin.filterPlanForSelection(
plan,
['keep', 'skip'],
['keep'],
{ all_annotations: [priorAnnotation] }
);
assert.equal(preserved.all_annotations.find(a => a.clipping_id === 'skip'), priorAnnotation);
const deferred = plugin.deferPendingPlan(
{ all_annotations: [{ clipping_id: 'old' }], deferred_clipping_ids: ['earlier'] },
plan,
['keep', 'skip']
);
assert.deepEqual(Array.from(deferred.deferred_clipping_ids), ['earlier', 'keep', 'skip']);
assert.deepEqual(Array.from(deferred.all_annotations, a => a.clipping_id), ['old']);
}
function testTitleCountsAreAuthoritative() {
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
{ id: 'h3', title: 'Shared title', kind: 'highlight', text: 'third' },
] },
matches: { matches: [{
clipping_title: 'Shared title',
clipping_count: 99,
status: 'ambiguous',
candidates: [{ citation_key: 'shared2020', key: '', item_id: 1, title: 'Shared', score: 0.9 }],
}] },
generatedOverrides: { overrides: [{
clipping_title: 'Shared title',
review: { status: 'review', clipping_count: 1 },
candidates: [],
}] },
positionedPlan: { items: [{
clipping: { id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
status: 'pdf-text-not-found',
problems: ['not found'],
}] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const rows = manager.conflictRows();
assert.equal(rows.length, 1);
// Source has 3 clippings; misleading match (99) and override (1) counts must not win.
assert.equal(rows[0].count, 3);
assert.equal(rows[0].clippings.length, 3);
// Only 1 of 3 has position problems.
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 3 });
// Without clippings.json, fall back to conflict counts.
manager.data = {
clippings: null,
matches: { matches: [{
clipping_title: 'Lonely title',
clipping_count: 5,
status: 'ambiguous',
candidates: [],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [] },
finalPlan: { all_annotations: [] },
persistentOverrides: { overrides: [] },
};
const fallback = manager.conflictRows();
assert.equal(fallback.length, 1);
assert.equal(fallback[0].count, 5);
}
function testMappedTargetAndDiscardedClips() {
manager.data = {
clippings: { clippings: [
{ id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' },
{ id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' },
] },
matches: { matches: [{
clipping_title: 'Mapped title',
clipping_count: 2,
status: 'matched',
candidates: [{ citation_key: 'mapped2020', key: '', item_id: 7, title: 'Mapped', score: 1 }],
}] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [
{ clipping: { id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' }, status: 'pdf-text-not-found', problems: ['pdf-text-not-found'] },
{ clipping: { id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' }, status: 'pdf-position-skipped-non-highlight', problems: ['pdf-position-skipped-non-highlight'] },
] },
finalPlan: { all_annotations: [], discarded_clipping_ids: [] },
persistentOverrides: { overrides: [{ clipping_title: 'Mapped title', resolution: { citation_key: 'mapped2020' } }] },
};
let rows = manager.conflictRows();
assert.equal(rows.length, 1);
assert.equal(rows[0].mappedTarget, 'mapped2020');
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 2, total: 2 });
manager.data.finalPlan.discarded_clipping_ids = ['n1'];
rows = manager.conflictRows();
assert.equal(rows.length, 1);
assert.equal(rows[0].count, 1);
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 1 });
}
async function testDiscardClipPersists() {
const originalZotero = bootstrapContext.Zotero;
let written = null;
const plan = { discarded_clipping_ids: [], deferred_clipping_ids: ['x1'] };
bootstrapContext.Zotero = {
File: {
getContentsAsync: async () => JSON.stringify(plan),
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
},
};
const originalArtifactPath = plugin.artifactPath;
plugin.artifactPath = rel => `/proj/${rel}`;
try {
await plugin.discardClip('x1');
assert.deepEqual(written.discarded_clipping_ids, ['x1']);
assert.deepEqual(written.deferred_clipping_ids, []);
} finally {
plugin.artifactPath = originalArtifactPath;
bootstrapContext.Zotero = originalZotero;
}
}
function testTitleOverviewAndSorting() {
manager.data = {
clippings: { clippings: [
{ id: 'a1', title: 'Alpha', kind: 'highlight', text: 'one' },
{ id: 'a2', title: 'Alpha', kind: 'highlight', text: 'two' },
{ id: 'b1', title: 'Beta', kind: 'highlight', text: 'three' },
] },
matches: { matches: [
{ clipping_title: 'Alpha', clipping_count: 2, status: 'matched', candidates: [{ citation_key: 'alpha2020' }] },
{ clipping_title: 'Beta', clipping_count: 1, status: 'unmatched', candidates: [] },
] },
generatedOverrides: { overrides: [] },
positionedPlan: { items: [] },
finalPlan: {
all_annotations: [{ clipping_id: 'a1', clipping_title: 'Alpha' }],
deferred_clipping_ids: ['b1'],
},
persistentOverrides: { overrides: [] },
};
const overview = manager.titleOverviewRows();
assert.equal(overview.length, 2);
const alpha = overview.find(r => r.title === 'Alpha');
const beta = overview.find(r => r.title === 'Beta');
assert.equal(alpha.integrated, 1);
assert.equal(beta.pending, 1);
assert.equal(beta.status, 'pending review');
manager.sortState.titles = { column: 'title', direction: 'desc' };
const sorted = manager.sortRows(overview, 'titles', { title: row => row.title });
assert.deepEqual(Array.from(sorted, row => row.title), ['Beta', 'Alpha']);
}
async function testSaveOverrideUsesSourceCount() {
const originalRead = plugin.readJsonArtifact;
const originalArtifactPath = plugin.artifactPath;
const originalZotero = bootstrapContext.Zotero;
let written = null;
plugin.artifactPath = rel => `/proj/${rel}`;
plugin.readJsonArtifact = async rel => rel === 'clippings.json'
? { clippings: [{ title: 'Shared title' }, { title: 'Shared title' }, { title: 'Shared title' }] }
: null;
bootstrapContext.Zotero = {
File: {
getContentsAsync: async () => JSON.stringify({ format: 'kindle-zotero-importer.match-overrides.v1', overrides: [] }),
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
},
};
try {
const entry = await plugin.saveOverride({
clipping_title: 'Shared title',
clipping_count: 1,
resolution: { citation_key: 'shared2020' },
review: { status: 'pdf-text-not-found', clipping_count: 1 },
});
assert.equal(entry.review.clipping_count, 3);
assert.equal(written.overrides[0].review.clipping_count, 3);
} finally {
plugin.readJsonArtifact = originalRead;
plugin.artifactPath = originalArtifactPath;
bootstrapContext.Zotero = originalZotero;
}
}
testTitleCountsAreAuthoritative();
testTitleOverviewAndSorting();
testMappedTargetAndDiscardedClips();
testPendingReviewGrouping();
testFilterPlanForSelection();
testRuntimeExtraction() testRuntimeExtraction()
.then(testPythonPathRecovery)
.then(testSaveOverrideUsesSourceCount)
.then(testDiscardClipPersists)
.then(testMaintenanceSweep) .then(testMaintenanceSweep)
.then(testWriterReconciliation) .then(testWriterReconciliation)
.then(testDeletionFailureIsRetried) .then(testDeletionFailureIsRetried)
@ -323,6 +702,7 @@ testRuntimeExtraction()
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor) .then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
.then(testEnumerationFailurePersistsAllDeletions) .then(testEnumerationFailurePersistsAllDeletions)
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry) .then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
.then(testCitekeyRelationsAreBidirectionalAndAdditive)
.then(() => { .then(() => {
console.log('manager and annotation logic ok'); console.log('manager and annotation logic ok');
}).catch(error => { }).catch(error => {