Compare commits
6 commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aa3fc5ebc5 | ||
|
|
495f50989d | ||
|
|
ff0d0c572c | ||
|
|
608098d378 | ||
|
|
afac75153e | ||
|
|
936971b7f8 |
19 changed files with 2824 additions and 160 deletions
41
README.md
41
README.md
|
|
@ -1,7 +1,7 @@
|
|||
> [!CAUTION]
|
||||
> **This project has been heavily vibecoded.**
|
||||
|
||||
# Kanzi — Kindle annotations to Zotero
|
||||
# Kanzi — `K`indle `an`notations to `Z`otero `I`mporter
|
||||
|
||||
Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, directly inside Zotero. Once the required host tools are installed, imports run from the manager without CLI commands. Formerly *Kindle Zotero Importer*.
|
||||
|
||||
|
|
@ -17,16 +17,39 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
|
|||
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
|
||||
3. `Tools` → `Kanzi…` to open the manager.
|
||||
|
||||
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. macOS and Linux are tested. Windows requires setting the full Python executable path in `Settings` and installing the PDF tools separately.
|
||||
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. Kanzi detects standard Python installations automatically on macOS, Linux and Windows.
|
||||
|
||||
## Windows
|
||||
|
||||
Open PowerShell and install the host tools:
|
||||
|
||||
```powershell
|
||||
winget install --id Python.Python.3.13 --exact
|
||||
winget install --id oschwartz10612.Poppler --exact
|
||||
winget install --id QPDF.QPDF --exact # Optional: handles copy-protected PDFs
|
||||
```
|
||||
|
||||
Fully quit and reopen Zotero after installation so it receives the updated `PATH`, then install `kanzi.xpi` and open `Tools` → `Kanzi…`. Kanzi checks standard per-user and system Python locations as well as `PATH`; you should not need to enter an executable path manually. EPUB-only imports require Python but not Poppler.
|
||||
|
||||
To verify the installation in a new PowerShell window:
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
pdftotext -v
|
||||
pdftohtml -v
|
||||
pdfinfo -v
|
||||
qpdf --version # Optional
|
||||
```
|
||||
|
||||
# Use
|
||||
|
||||
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
|
||||
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
||||
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
|
||||
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
||||
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
||||
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
||||
2. **Review before writing.** New highlights are grouped by title and selected by default. Untick individual highlights or whole titles, then press `Integrate selected`. Skipped highlights return on the next run. `Cancel` and closing the manager write nothing to Zotero and do not mark pending highlights as integrated.
|
||||
3. **Check Integrated.** After integration, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
||||
4. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows one expandable row for that title with up to three suggestions `citekey · title (score%)`. Expand it to inspect individual highlights and positioning problems. Click `Use` on the right candidate, enter a citation key, Zotero key or item ID under `Use Custom`, or choose `Ignore Title`. Tables can be sorted and resized.
|
||||
5. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new, changed, deferred or removed highlights are processed. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
||||
6. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
||||
7. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
||||
|
||||
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
|
||||
|
||||
|
|
@ -62,11 +85,11 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
|
|||
|
||||
`Tools` → `Kanzi…` opens the manager.
|
||||
|
||||
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
|
||||
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. After positioning, the pending review lets you select exactly what is written to Zotero.
|
||||
|
||||
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
|
||||
|
||||
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
|
||||
**Conflicts.** One row per unresolved Kindle title, combining title-match and attachment/position statuses. Expand a row to inspect its highlights and positioning problems. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. Columns are sortable and resizable.
|
||||
|
||||
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
|
||||
|
||||
|
|
|
|||
506
plugin/bootstrap.js
vendored
506
plugin/bootstrap.js
vendored
|
|
@ -1,4 +1,4 @@
|
|||
var KindleZoteroImporter = {
|
||||
var Kanzi = {
|
||||
id: "kindle-zotero-importer@utkubilen.de",
|
||||
menuItems: [],
|
||||
runtimeProjectDir: "",
|
||||
|
|
@ -16,7 +16,7 @@ var KindleZoteroImporter = {
|
|||
);
|
||||
this.chromeHandle = addonManagerStartup.registerChrome(
|
||||
Services.io.newURI(this.rootURI + "manifest.json"),
|
||||
[["content", "kindle-zotero-importer", ""]]
|
||||
[["content", "kanzi", ""]]
|
||||
);
|
||||
await Zotero.initializationPromise;
|
||||
await this.ensureRuntime();
|
||||
|
|
@ -81,7 +81,7 @@ var KindleZoteroImporter = {
|
|||
}
|
||||
|
||||
const doc = win.document;
|
||||
if (doc.getElementById("kindle-zotero-importer-manager")) {
|
||||
if (doc.getElementById("kanzi-manager")) {
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
@ -92,7 +92,7 @@ var KindleZoteroImporter = {
|
|||
}
|
||||
|
||||
const managerItem = doc.createXULElement("menuitem");
|
||||
managerItem.setAttribute("id", "kindle-zotero-importer-manager");
|
||||
managerItem.setAttribute("id", "kanzi-manager");
|
||||
managerItem.setAttribute("label", "Kanzi...");
|
||||
managerItem.addEventListener("command", () => this.openManager());
|
||||
popup.appendChild(managerItem);
|
||||
|
|
@ -136,7 +136,14 @@ var KindleZoteroImporter = {
|
|||
},
|
||||
|
||||
getPythonPath() {
|
||||
return this.getPref("pythonPath", "python3");
|
||||
const fallback = this.getDefaultPython();
|
||||
const value = this.getPref("pythonPath", fallback);
|
||||
// Existing Windows installs may have "python3" stored from the old
|
||||
// Unix-only default; that binary name does not exist on Windows.
|
||||
if (this.isWindows() && value === "python3") {
|
||||
return "python";
|
||||
}
|
||||
return value || fallback;
|
||||
},
|
||||
|
||||
getZoteroDbPath() {
|
||||
|
|
@ -149,12 +156,115 @@ var KindleZoteroImporter = {
|
|||
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
|
||||
},
|
||||
|
||||
isWindows() {
|
||||
try {
|
||||
return Services.appinfo.OS === "WINNT";
|
||||
} catch (_error) {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
|
||||
getDefaultPython() {
|
||||
return this.isWindows() ? "python" : "python3";
|
||||
},
|
||||
|
||||
getEnvironmentVariable(name) {
|
||||
try {
|
||||
return Cc["@mozilla.org/process/environment;1"]
|
||||
.getService(Ci.nsIEnvironment)
|
||||
.get(name) || "";
|
||||
} catch (_error) {
|
||||
return "";
|
||||
}
|
||||
},
|
||||
|
||||
joinWindowsPath(base, ...parts) {
|
||||
if (!base) {
|
||||
return "";
|
||||
}
|
||||
return [String(base).replace(/[\\/]+$/, ""), ...parts]
|
||||
.map((part, index) => index ? String(part).replace(/^[\\/]+|[\\/]+$/g, "") : part)
|
||||
.join("\\");
|
||||
},
|
||||
|
||||
getWindowsPythonCandidates(configuredPath) {
|
||||
const localAppData = this.getEnvironmentVariable("LOCALAPPDATA");
|
||||
const programFiles = this.getEnvironmentVariable("ProgramFiles");
|
||||
const programFilesX86 = this.getEnvironmentVariable("ProgramFiles(x86)");
|
||||
const userProfile = this.getEnvironmentVariable("USERPROFILE");
|
||||
const systemDrive = this.getEnvironmentVariable("SystemDrive") || "C:";
|
||||
const versions = ["314", "313", "312", "311"];
|
||||
const candidates = [configuredPath];
|
||||
for (const version of versions) {
|
||||
candidates.push(
|
||||
this.joinWindowsPath(localAppData, "Programs", "Python", `Python${version}`, "python.exe"),
|
||||
this.joinWindowsPath(programFiles, `Python${version}`, "python.exe"),
|
||||
this.joinWindowsPath(programFilesX86, `Python${version}-32`, "python.exe"),
|
||||
this.joinWindowsPath(systemDrive, `Python${version}`, "python.exe")
|
||||
);
|
||||
}
|
||||
candidates.push(
|
||||
this.joinWindowsPath(localAppData, "Python", "bin", "python.exe"),
|
||||
this.joinWindowsPath(userProfile, "scoop", "apps", "python", "current", "python.exe"),
|
||||
this.joinWindowsPath(userProfile, "miniconda3", "python.exe"),
|
||||
this.joinWindowsPath(userProfile, "anaconda3", "python.exe")
|
||||
);
|
||||
for (const directory of this.getEnvironmentVariable("PATH").split(";")) {
|
||||
const cleanDirectory = directory.trim().replace(/^"|"$/g, "");
|
||||
if (cleanDirectory) {
|
||||
candidates.push(
|
||||
this.joinWindowsPath(cleanDirectory, "python.exe"),
|
||||
this.joinWindowsPath(cleanDirectory, "python3.exe")
|
||||
);
|
||||
}
|
||||
}
|
||||
return [...new Set(candidates.filter(Boolean))];
|
||||
},
|
||||
|
||||
async resolvePythonPath(configuredPath) {
|
||||
let candidates;
|
||||
if (this.isWindows()) {
|
||||
candidates = this.getWindowsPythonCandidates(configuredPath);
|
||||
} else if (Services.appinfo.OS === "Darwin") {
|
||||
candidates = [configuredPath];
|
||||
candidates.push(
|
||||
"/opt/homebrew/bin/python3",
|
||||
"/opt/homebrew/opt/python@3.14/bin/python3.14",
|
||||
"/opt/homebrew/opt/python@3.13/bin/python3.13",
|
||||
"/opt/homebrew/opt/python@3.12/bin/python3.12",
|
||||
"/opt/homebrew/opt/python@3.11/bin/python3.11",
|
||||
"/usr/local/bin/python3",
|
||||
"/usr/local/opt/python@3.14/bin/python3.14",
|
||||
"/usr/local/opt/python@3.13/bin/python3.13",
|
||||
"/usr/local/opt/python@3.12/bin/python3.12",
|
||||
"/usr/local/opt/python@3.11/bin/python3.11",
|
||||
"/usr/bin/python3"
|
||||
);
|
||||
} else {
|
||||
candidates = [configuredPath, "/usr/bin/python3", "/usr/local/bin/python3"];
|
||||
}
|
||||
for (const candidate of candidates) {
|
||||
if (!candidate || (!candidate.includes("/") && !candidate.includes("\\"))) continue;
|
||||
try {
|
||||
if (await IOUtils.exists(candidate)) return candidate;
|
||||
} catch (_error) {}
|
||||
}
|
||||
return configuredPath || this.getDefaultPython();
|
||||
},
|
||||
|
||||
joinPath(...parts) {
|
||||
if (typeof PathUtils !== "undefined" && PathUtils.join) {
|
||||
return PathUtils.join(...parts);
|
||||
}
|
||||
return parts.join("/").replaceAll("//", "/");
|
||||
},
|
||||
|
||||
getConfigPath() {
|
||||
return this.getProjectDir() + "/plugin-config.json";
|
||||
return this.joinPath(this.getProjectDir(), "plugin-config.json");
|
||||
},
|
||||
|
||||
artifactPath(relativePath) {
|
||||
return this.getProjectDir() + "/" + relativePath;
|
||||
return this.joinPath(this.getProjectDir(), ...String(relativePath).split("/"));
|
||||
},
|
||||
|
||||
async openArtifact(relativePath) {
|
||||
|
|
@ -177,17 +287,6 @@ var KindleZoteroImporter = {
|
|||
zoteroDbPath: this.getZoteroDbPath(),
|
||||
zoteroStorageRoot: this.getZoteroStorageRoot(),
|
||||
};
|
||||
if (settings.pythonPath === "python3") {
|
||||
const candidates = Services.appinfo.OS === "Darwin"
|
||||
? ["/opt/homebrew/bin/python3", "/usr/local/bin/python3", "/usr/bin/python3"]
|
||||
: ["/usr/bin/python3", "/usr/local/bin/python3"];
|
||||
for (const candidate of candidates) {
|
||||
if (await IOUtils.exists(candidate)) {
|
||||
settings.pythonPath = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
try {
|
||||
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
|
||||
const parsed = JSON.parse(text);
|
||||
|
|
@ -200,12 +299,13 @@ var KindleZoteroImporter = {
|
|||
if (parsed.colorMap) settings.colourMap = parsed.colorMap;
|
||||
// also check colour-map.json for Python pipeline
|
||||
try {
|
||||
const cmapText = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json");
|
||||
const cmapText = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
|
||||
settings.colourMap = JSON.parse(cmapText);
|
||||
} catch (_e2) {}
|
||||
} catch (_error) {
|
||||
// The config file is optional; defaults/preferences are enough for first run.
|
||||
}
|
||||
settings.pythonPath = await this.resolvePythonPath(settings.pythonPath);
|
||||
return settings;
|
||||
},
|
||||
|
||||
|
|
@ -267,8 +367,8 @@ var KindleZoteroImporter = {
|
|||
const data = await this.loadManagerData();
|
||||
const io = { plugin: this, settings, data };
|
||||
this.managerWindow = this.getMainWindow().openDialog(
|
||||
"chrome://kindle-zotero-importer/content/manager.html",
|
||||
"kindle-zotero-importer-manager-window",
|
||||
"chrome://kanzi/content/manager.html",
|
||||
"kanzi-manager-window",
|
||||
"chrome,centerscreen,resizable",
|
||||
io
|
||||
);
|
||||
|
|
@ -276,6 +376,7 @@ var KindleZoteroImporter = {
|
|||
|
||||
async loadManagerData() {
|
||||
return {
|
||||
clippings: await this.readJsonArtifact("clippings.json"),
|
||||
matches: await this.readJsonArtifact("matches.json"),
|
||||
generatedOverrides: await this.readJsonArtifact("match-overrides.generated.json"),
|
||||
positionedPlan: await this.readJsonArtifact("import-plan.positioned.json"),
|
||||
|
|
@ -305,11 +406,17 @@ var KindleZoteroImporter = {
|
|||
const index = payload.overrides.findIndex(
|
||||
(entry) => entry.clipping_title === resolution.clipping_title
|
||||
);
|
||||
const clippings = await this.readJsonArtifact("clippings.json");
|
||||
const sourceCount = this.countClippingsForTitle(clippings, resolution.clipping_title);
|
||||
const clippingCount = sourceCount || resolution.clipping_count || resolution.review?.clipping_count || 0;
|
||||
const nowIso = new Date().toISOString();
|
||||
const entry = {
|
||||
clipping_title: resolution.clipping_title,
|
||||
resolution: resolution.resolution,
|
||||
review: resolution.review || { status: "manager", clipping_count: resolution.clipping_count || 0 },
|
||||
review: {
|
||||
...(resolution.review || { status: "manager" }),
|
||||
clipping_count: clippingCount,
|
||||
},
|
||||
created_at: nowIso,
|
||||
updated_at: nowIso,
|
||||
};
|
||||
|
|
@ -329,6 +436,13 @@ var KindleZoteroImporter = {
|
|||
return entry;
|
||||
},
|
||||
|
||||
countClippingsForTitle(payload, clippingTitle) {
|
||||
return (payload?.clippings || []).reduce(
|
||||
(count, clipping) => count + (clipping.title === clippingTitle ? 1 : 0),
|
||||
0
|
||||
);
|
||||
},
|
||||
|
||||
async deleteOverride(clippingTitle) {
|
||||
if (!clippingTitle) {
|
||||
throw new Error("Missing clipping title for delete");
|
||||
|
|
@ -346,6 +460,23 @@ var KindleZoteroImporter = {
|
|||
return true;
|
||||
},
|
||||
|
||||
async discardClip(clippingId) {
|
||||
if (!clippingId) {
|
||||
throw new Error("Missing clipping id for discard");
|
||||
}
|
||||
const path = this.artifactPath("import-plan.final.json");
|
||||
const plan = JSON.parse(await Zotero.File.getContentsAsync(path));
|
||||
const discarded = new Set(plan.discarded_clipping_ids || []);
|
||||
discarded.add(clippingId);
|
||||
plan.discarded_clipping_ids = [...discarded].sort();
|
||||
// Also drop from deferred so it is never re-queued.
|
||||
if (Array.isArray(plan.deferred_clipping_ids)) {
|
||||
plan.deferred_clipping_ids = plan.deferred_clipping_ids.filter((id) => id !== clippingId);
|
||||
}
|
||||
await Zotero.File.putContentsAsync(path, JSON.stringify(plan, null, 2) + "\n");
|
||||
return true;
|
||||
},
|
||||
|
||||
async saveSettingsFromManager(newSettings) {
|
||||
const toSave = {
|
||||
projectDir: newSettings.projectDir || this.getProjectDir(),
|
||||
|
|
@ -358,7 +489,7 @@ var KindleZoteroImporter = {
|
|||
// also write colour-map.json for Python pipeline (mirrors plugin-config)
|
||||
try {
|
||||
await Zotero.File.putContentsAsync(
|
||||
this.getProjectDir() + "/colour-map.json",
|
||||
this.joinPath(this.getProjectDir(), "colour-map.json"),
|
||||
JSON.stringify(newSettings.colourMap, null, 2) + "\n"
|
||||
);
|
||||
} catch (_e) {}
|
||||
|
|
@ -376,7 +507,7 @@ var KindleZoteroImporter = {
|
|||
|
||||
async loadColourMap() {
|
||||
try {
|
||||
const text = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json");
|
||||
const text = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
|
||||
return JSON.parse(text);
|
||||
} catch (_e) {}
|
||||
try {
|
||||
|
|
@ -416,7 +547,7 @@ var KindleZoteroImporter = {
|
|||
if (result !== fp.returnOK) {
|
||||
return null;
|
||||
}
|
||||
return fp.file;
|
||||
return this.filePickerResultToPath(fp.file);
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
return this.promptForPath(
|
||||
|
|
@ -427,12 +558,32 @@ var KindleZoteroImporter = {
|
|||
}
|
||||
},
|
||||
|
||||
filePickerResultToPath(file) {
|
||||
if (!file) {
|
||||
return null;
|
||||
}
|
||||
if (typeof file === "string") {
|
||||
return file;
|
||||
}
|
||||
// Mozilla FilePicker returns an nsIFile; Python needs a native string path.
|
||||
if (typeof file.path === "string" && file.path) {
|
||||
return file.path;
|
||||
}
|
||||
try {
|
||||
const str = String(file);
|
||||
if (str && str !== "[object Object]") {
|
||||
return str;
|
||||
}
|
||||
} catch (_error) {}
|
||||
return null;
|
||||
},
|
||||
|
||||
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
|
||||
const python = settings.pythonPath;
|
||||
const python = settings.pythonPath || this.getDefaultPython();
|
||||
const projectDir = settings.projectDir;
|
||||
const runner = projectDir + "/plugin_runner.py";
|
||||
const summaryOutput = projectDir + "/plugin-summary.json";
|
||||
const progressOutput = projectDir + "/plugin-progress.json";
|
||||
const runner = this.joinPath(projectDir, "plugin_runner.py");
|
||||
const summaryOutput = this.joinPath(projectDir, "plugin-summary.json");
|
||||
const progressOutput = this.joinPath(projectDir, "plugin-progress.json");
|
||||
const args = [
|
||||
runner,
|
||||
"run",
|
||||
|
|
@ -458,12 +609,34 @@ var KindleZoteroImporter = {
|
|||
args.push("--retry-unresolved");
|
||||
}
|
||||
|
||||
await this.exec(python, args);
|
||||
let execResult = null;
|
||||
try {
|
||||
execResult = await this.exec(python, args);
|
||||
} catch (error) {
|
||||
const installHint = this.isWindows()
|
||||
? `Install Python 3.11 or newer, fully restart Zotero, and let Kanzi detect python.exe. `
|
||||
: `Install Python 3.11 or newer and check the Python path in Kanzi Settings. `;
|
||||
throw new Error(
|
||||
`Python could not run the Kanzi pipeline (${python}): ${error && error.message ? error.message : error}. ` +
|
||||
installHint
|
||||
);
|
||||
}
|
||||
try {
|
||||
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
|
||||
} catch (error) {
|
||||
const detail = execResult ? ` Python output: ${JSON.stringify(execResult).slice(0, 1000)}.` : "";
|
||||
throw new Error(
|
||||
`Python pipeline did not write ${summaryOutput}.${detail} ` +
|
||||
`Python: ${python}. Runner: ${runner}. ` +
|
||||
`Make sure Python 3 is installed and reachable, then check the project directory is writable.`
|
||||
);
|
||||
}
|
||||
},
|
||||
|
||||
async runManagedImport(manager) {
|
||||
const clippingsPath = await this.pickFile("Select Kindle My Clippings.txt", "*.txt");
|
||||
// NOTE: writing now happens only after the user reviews the pending
|
||||
// highlights and presses Integrate (see integratePending).
|
||||
if (!clippingsPath) {
|
||||
return;
|
||||
}
|
||||
|
|
@ -474,17 +647,23 @@ var KindleZoteroImporter = {
|
|||
|
||||
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
|
||||
try {
|
||||
clippingsPath = this.filePickerResultToPath(clippingsPath) || clippingsPath;
|
||||
if (!clippingsPath) {
|
||||
throw new Error("No clippings file selected");
|
||||
}
|
||||
const settings = await this.getRuntimeSettings();
|
||||
const progressPath = settings.projectDir + "/plugin-progress.json";
|
||||
const progressPath = this.joinPath(settings.projectDir, "plugin-progress.json");
|
||||
// Determine full vs incremental from explicit arg or manager checkbox
|
||||
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
|
||||
const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
|
||||
? retryUnresolvedOverride
|
||||
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
|
||||
manager.beginRun(clippingsPath);
|
||||
const finalPlanPath = this.joinPath(settings.projectDir, "import-plan.final.json");
|
||||
let previousFinalPlan = null;
|
||||
try {
|
||||
previousFinalPlan = JSON.parse(await Zotero.File.getContentsAsync(finalPlanPath));
|
||||
} catch (_backupError) {}
|
||||
try {
|
||||
await Zotero.File.removeIfExists(progressPath);
|
||||
} catch (_error) {}
|
||||
|
|
@ -511,20 +690,142 @@ var KindleZoteroImporter = {
|
|||
const finalPlan = JSON.parse(
|
||||
await Zotero.File.getContentsAsync(summary.outputs.final_plan)
|
||||
);
|
||||
manager.updateProgress(96, "Writing annotations", "Saving annotations in Zotero");
|
||||
const results = await this.writeAnnotations(finalPlan, false, (completed, total) => {
|
||||
const pendingCount = (finalPlan.annotations || []).length;
|
||||
const pendingDeletions = (finalPlan.deletions || []).length;
|
||||
if (!pendingCount && !pendingDeletions) {
|
||||
// Nothing new — run the maintenance sweep and finish as before.
|
||||
manager.updateProgress(96, "Writing annotations", "Checking existing annotations");
|
||||
const emptyResults = await this.writeAnnotations(finalPlan, false, (completed, total) => {
|
||||
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
||||
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
||||
});
|
||||
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, results);
|
||||
const data = await this.loadManagerData();
|
||||
await manager.completeRun(summary, results, data);
|
||||
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, emptyResults);
|
||||
const emptyData = await this.loadManagerData();
|
||||
await manager.completeRun(summary, emptyResults, emptyData);
|
||||
return;
|
||||
}
|
||||
// Pause before writing: let the user review and select what to integrate.
|
||||
const fullDeltaIds = (finalPlan.annotations || []).map((a) => a.clipping_id);
|
||||
// The pipeline has already written its generated history. Replace it with
|
||||
// a safe ledger until Integrate is pressed, including if the window closes.
|
||||
const deferredPlan = this.deferPendingPlan(previousFinalPlan, finalPlan, fullDeltaIds);
|
||||
await Zotero.File.putContentsAsync(
|
||||
summary.outputs.final_plan,
|
||||
JSON.stringify(deferredPlan, null, 2) + "\n"
|
||||
);
|
||||
this._pending = {
|
||||
summary,
|
||||
finalPlan,
|
||||
finalPlanPath: summary.outputs.final_plan,
|
||||
fullDeltaIds,
|
||||
previousFinalPlan,
|
||||
};
|
||||
manager.updateProgress(
|
||||
90,
|
||||
"Review pending",
|
||||
`${pendingCount} highlight(s) ready — select and press Integrate`
|
||||
);
|
||||
manager.showPendingReview(summary, finalPlan);
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
manager.failRun(String(error && error.stack ? error.stack : error));
|
||||
}
|
||||
},
|
||||
|
||||
deferPendingPlan(previousPlan, generatedPlan, deferredIds) {
|
||||
const plan = previousPlan
|
||||
? { ...previousPlan }
|
||||
: {
|
||||
...generatedPlan,
|
||||
annotations: [],
|
||||
all_annotations: [],
|
||||
deletions: [],
|
||||
annotation_count: 0,
|
||||
delta_annotation_count: 0,
|
||||
};
|
||||
const deferred = new Set(plan.deferred_clipping_ids || []);
|
||||
for (const id of deferredIds || []) deferred.add(id);
|
||||
plan.deferred_clipping_ids = [...deferred].sort();
|
||||
return plan;
|
||||
},
|
||||
|
||||
// Pure helper (tested): narrow a pipeline plan to the user's selection.
|
||||
filterPlanForSelection(finalPlan, fullDeltaIds, selectedIds, previousPlan = null) {
|
||||
const selected = new Set(selectedIds || []);
|
||||
const unselected = new Set(
|
||||
(fullDeltaIds || []).filter((id) => !selected.has(id))
|
||||
);
|
||||
const plan = { ...finalPlan };
|
||||
plan.annotations = (finalPlan.annotations || []).filter((a) =>
|
||||
selected.has(a.clipping_id)
|
||||
);
|
||||
if (unselected.size && Array.isArray(finalPlan.all_annotations)) {
|
||||
plan.all_annotations = finalPlan.all_annotations.filter(
|
||||
(a) => !unselected.has(a.clipping_id)
|
||||
);
|
||||
const present = new Set(plan.all_annotations.map((a) => a.clipping_id));
|
||||
for (const annotation of previousPlan?.all_annotations || previousPlan?.annotations || []) {
|
||||
if (unselected.has(annotation.clipping_id) && !present.has(annotation.clipping_id)) {
|
||||
plan.all_annotations.push(annotation);
|
||||
present.add(annotation.clipping_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
const deferred = new Set(previousPlan?.deferred_clipping_ids || []);
|
||||
for (const id of selected) deferred.delete(id);
|
||||
for (const id of unselected) deferred.add(id);
|
||||
plan.deferred_clipping_ids = [...deferred].sort();
|
||||
plan.delta_annotation_count = plan.annotations.length;
|
||||
if (Array.isArray(plan.all_annotations)) {
|
||||
plan.annotation_count = plan.all_annotations.length;
|
||||
}
|
||||
return plan;
|
||||
},
|
||||
|
||||
async integratePending(manager, selectedIds) {
|
||||
const pending = this._pending;
|
||||
if (!pending) {
|
||||
throw new Error("No pending review to integrate");
|
||||
}
|
||||
try {
|
||||
const plan = this.filterPlanForSelection(
|
||||
pending.finalPlan,
|
||||
pending.fullDeltaIds,
|
||||
selectedIds,
|
||||
pending.previousFinalPlan
|
||||
);
|
||||
manager.updateProgress(96, "Writing annotations", "Saving selected annotations in Zotero");
|
||||
const results = await this.writeAnnotations(plan, false, (completed, total) => {
|
||||
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
||||
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
||||
});
|
||||
await this.reconcileWriterState(pending.finalPlanPath, plan, results);
|
||||
this._pending = null;
|
||||
const data = await this.loadManagerData();
|
||||
await manager.completeRun(pending.summary, results, data);
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
manager.failRun(String(error && error.stack ? error.stack : error));
|
||||
}
|
||||
},
|
||||
|
||||
async cancelPendingReview(manager) {
|
||||
// The deferred ledger was written before the review appeared, so closing
|
||||
// or cancelling never records pending highlights as integrated.
|
||||
this._pending = null;
|
||||
manager.hidePendingReview();
|
||||
manager.cancelRun("Review cancelled — nothing was written. Run again when ready.");
|
||||
try {
|
||||
const data = await this.loadManagerData();
|
||||
manager.data = data;
|
||||
manager.rows = manager.conflictRows();
|
||||
manager.refreshRows(true);
|
||||
manager.refreshTitleOverview();
|
||||
manager.refreshIntegrated(true);
|
||||
manager.refreshMappings(true);
|
||||
} catch (_reloadError) {}
|
||||
},
|
||||
|
||||
async exec(command, args) {
|
||||
if (!Zotero.Utilities || !Zotero.Utilities.Internal || !Zotero.Utilities.Internal.exec) {
|
||||
throw new Error("Zotero.Utilities.Internal.exec is unavailable in this Zotero build");
|
||||
|
|
@ -814,6 +1115,117 @@ var KindleZoteroImporter = {
|
|||
throw new Error("Existing annotation cannot be saved by this Zotero build");
|
||||
},
|
||||
|
||||
async saveRelatedItem(item) {
|
||||
const options = { skipDateModifiedUpdate: true };
|
||||
if (item.save) {
|
||||
await item.save(options);
|
||||
return;
|
||||
}
|
||||
if (item.saveTx) {
|
||||
await item.saveTx(options);
|
||||
return;
|
||||
}
|
||||
throw new Error("Related item cannot be saved by this Zotero build");
|
||||
},
|
||||
|
||||
async addRelatedItemRelations(entries, dryRun, results) {
|
||||
const pairs = new Map();
|
||||
for (const entry of entries || []) {
|
||||
for (const target of entry.related_items || []) {
|
||||
const sourceID = Number(entry.parent_item_id);
|
||||
const targetID = Number(target.item_id);
|
||||
if (!sourceID || !targetID || sourceID === targetID) continue;
|
||||
const pairKey = [sourceID, targetID].sort((a, b) => a - b).join(":");
|
||||
if (!pairs.has(pairKey)) {
|
||||
pairs.set(pairKey, { sourceID, targetID, citationKey: target.citation_key || "" });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const pair of pairs.values()) {
|
||||
try {
|
||||
const source = Zotero.Items.get(pair.sourceID);
|
||||
const target = Zotero.Items.get(pair.targetID);
|
||||
if (!source || !target) {
|
||||
throw new Error(`Related item pair not found: ${pair.sourceID} -> ${pair.targetID}`);
|
||||
}
|
||||
if (dryRun) {
|
||||
results.relationsPlanned += 1;
|
||||
continue;
|
||||
}
|
||||
if (!source.addRelatedItem || !target.addRelatedItem) {
|
||||
throw new Error("This Zotero build does not support addRelatedItem()");
|
||||
}
|
||||
|
||||
let changed = false;
|
||||
const addBothDirections = async () => {
|
||||
const sourceChanged = source.addRelatedItem(target);
|
||||
const targetChanged = target.addRelatedItem(source);
|
||||
if (sourceChanged) await this.saveRelatedItem(source);
|
||||
if (targetChanged) await this.saveRelatedItem(target);
|
||||
changed = sourceChanged || targetChanged;
|
||||
};
|
||||
if (Zotero.DB?.executeTransaction) {
|
||||
await Zotero.DB.executeTransaction(addBothDirections);
|
||||
} else {
|
||||
await addBothDirections();
|
||||
}
|
||||
if (changed) results.relationsAdded += 1;
|
||||
else results.relationsSkipped += 1;
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
results.relationFailures.push({
|
||||
parent_item_id: pair.sourceID,
|
||||
related_item_id: pair.targetID,
|
||||
citation_key: pair.citationKey,
|
||||
message: String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
async backfillRelationLinks(entries, dryRun, results) {
|
||||
// Appends missing `Related: @key <link>` blocks to pre-existing
|
||||
// annotations. Cross-library targets can never become Related items,
|
||||
// so the comment link is the only visible connection for them.
|
||||
for (const entry of entries || []) {
|
||||
const targets = entry.related_items || [];
|
||||
if (!targets.length || !entry.clipping_id || !entry.attachment_item_id) continue;
|
||||
const block = "Related: " + targets.map(
|
||||
(target) => `@${target.citation_key} ${target.link || ""}`.trim()
|
||||
).join(" · ");
|
||||
try {
|
||||
const attachment = Zotero.Items.get(entry.attachment_item_id);
|
||||
if (!attachment) {
|
||||
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
|
||||
}
|
||||
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
|
||||
const tag = `kindle-id:${entry.clipping_id}`;
|
||||
const existing = (annotations || []).find((annotation) =>
|
||||
this.existingTags(annotation).includes(tag)
|
||||
);
|
||||
if (!existing) continue;
|
||||
const comment = String(existing.annotationComment || "");
|
||||
if (comment.includes(block)) continue;
|
||||
if (dryRun) {
|
||||
results.commentLinksPlanned += 1;
|
||||
continue;
|
||||
}
|
||||
existing.annotationComment = comment ? `${comment}\n\n${block}` : block;
|
||||
await this.saveExistingAnnotation(existing);
|
||||
results.commentLinksAdded += 1;
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
results.relationFailures.push({
|
||||
parent_item_id: entry.parent_item_id,
|
||||
related_item_id: (targets[0] || {}).item_id,
|
||||
citation_key: (targets[0] || {}).citation_key,
|
||||
message: `comment link: ${error}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
async reconcileWriterState(path, plan, results) {
|
||||
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
|
||||
const failedIds = new Set(
|
||||
|
|
@ -861,6 +1273,12 @@ var KindleZoteroImporter = {
|
|||
blockedRecreateMissingTag: 0,
|
||||
recreateDeclined: 0,
|
||||
removedDuplicates: 0,
|
||||
relationsAdded: 0,
|
||||
relationsSkipped: 0,
|
||||
relationsPlanned: 0,
|
||||
commentLinksAdded: 0,
|
||||
commentLinksPlanned: 0,
|
||||
relationFailures: [],
|
||||
deletedForIncremental: 0,
|
||||
failedDeletionIds: [],
|
||||
maintenanceDeletedIds: [],
|
||||
|
|
@ -1045,6 +1463,14 @@ var KindleZoteroImporter = {
|
|||
}
|
||||
}
|
||||
|
||||
const currentIDs = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
|
||||
const relationEntries = (plan.all_annotations || []).filter(
|
||||
(entry) => !currentIDs.has(entry.clipping_id)
|
||||
);
|
||||
relationEntries.push(...successfulEntries.values());
|
||||
await this.addRelatedItemRelations(relationEntries, dryRun, results);
|
||||
await this.backfillRelationLinks(relationEntries, dryRun, results);
|
||||
|
||||
if (!dryRun && plan.deletions && plan.deletions.length) {
|
||||
const deletionIds = new Set(plan.deletions);
|
||||
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
|
||||
|
|
@ -1146,7 +1572,7 @@ function uninstall(_data, _reason) {}
|
|||
|
||||
async function startup(data, _reason) {
|
||||
try {
|
||||
await KindleZoteroImporter.startup(data);
|
||||
await Kanzi.startup(data);
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
}
|
||||
|
|
@ -1154,7 +1580,7 @@ async function startup(data, _reason) {
|
|||
|
||||
function shutdown(_data, _reason) {
|
||||
try {
|
||||
KindleZoteroImporter.shutdown();
|
||||
Kanzi.shutdown();
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
}
|
||||
|
|
@ -1162,7 +1588,7 @@ function shutdown(_data, _reason) {
|
|||
|
||||
function onMainWindowLoad({ window }) {
|
||||
try {
|
||||
KindleZoteroImporter.onMainWindowLoad(window);
|
||||
Kanzi.onMainWindowLoad(window);
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
}
|
||||
|
|
@ -1170,7 +1596,7 @@ function onMainWindowLoad({ window }) {
|
|||
|
||||
function onMainWindowUnload({ window }) {
|
||||
try {
|
||||
KindleZoteroImporter.onMainWindowUnload(window);
|
||||
Kanzi.onMainWindowUnload(window);
|
||||
} catch (error) {
|
||||
Zotero.logError(error);
|
||||
}
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"manifest_version": 2,
|
||||
"name": "Kanzi",
|
||||
"version": "0.7.0",
|
||||
"version": "0.7.1",
|
||||
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
|
||||
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
|
||||
"author": "Utku Bilen Demir",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[project]
|
||||
name = "kindle-zotero-importer"
|
||||
version = "0.7.0"
|
||||
version = "0.7.1"
|
||||
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
|
@ -8,7 +9,12 @@ from pathlib import Path
|
|||
from .clippings import clippings_to_jsonable, load_clippings
|
||||
from .epub_position import add_epub_positions
|
||||
from .final_plan import FINAL_FORMAT, build_final_writer_plan
|
||||
from .import_plan import DEFAULT_COLOUR_MAP, _attach_notes_to_highlights, build_import_plan
|
||||
from .import_plan import (
|
||||
DEFAULT_COLOUR_MAP,
|
||||
_attach_notes_to_highlights,
|
||||
_related_items_for_citekeys,
|
||||
build_import_plan,
|
||||
)
|
||||
from .matcher import build_match_report, load_json
|
||||
from .mismatch_review import build_mismatch_review
|
||||
from .overrides import generate_override_skeleton, load_overrides
|
||||
|
|
@ -65,6 +71,20 @@ def _merge_annotation_history(
|
|||
)
|
||||
|
||||
|
||||
def _refresh_history_relations(
|
||||
annotations: list[dict], zotero_index: dict
|
||||
) -> None:
|
||||
"""Resolve citekeys for old annotations without re-positioning them."""
|
||||
items_by_id = {item["item_id"]: item for item in zotero_index.get("items", [])}
|
||||
for entry in annotations:
|
||||
annotation = entry.get("annotation") or {}
|
||||
entry["related_items"] = _related_items_for_citekeys(
|
||||
(annotation.get("text") or "", annotation.get("comment") or ""),
|
||||
entry.get("parent_item_id"),
|
||||
items_by_id,
|
||||
)
|
||||
|
||||
|
||||
def _changed_note_clipping_ids(
|
||||
current_clippings: list[dict], previous_annotations: list[dict]
|
||||
) -> set[str]:
|
||||
|
|
@ -112,6 +132,137 @@ def _deletion_ids(
|
|||
return deleted
|
||||
|
||||
|
||||
def _deferred_clipping_ids(
|
||||
previous_plan: dict | None, current_ids: set[str]
|
||||
) -> set[str]:
|
||||
return {
|
||||
clipping_id
|
||||
for clipping_id in (previous_plan or {}).get("deferred_clipping_ids", [])
|
||||
if clipping_id in current_ids
|
||||
}
|
||||
|
||||
|
||||
def _discarded_clipping_ids(
|
||||
previous_plan: dict | None, current_ids: set[str]
|
||||
) -> set[str]:
|
||||
"""Clips the user permanently discarded in Conflicts (e.g. standalone notes)."""
|
||||
return {
|
||||
clipping_id
|
||||
for clipping_id in (previous_plan or {}).get("discarded_clipping_ids", [])
|
||||
if clipping_id in current_ids
|
||||
}
|
||||
|
||||
|
||||
def _changed_override_titles(
|
||||
previous_snapshot: dict | None, current_overrides: dict
|
||||
) -> set[str]:
|
||||
if previous_snapshot is None:
|
||||
return set(current_overrides)
|
||||
return {
|
||||
title
|
||||
for title in set(previous_snapshot) | set(current_overrides)
|
||||
if previous_snapshot.get(title) != current_overrides.get(title)
|
||||
}
|
||||
|
||||
|
||||
def _clipping_ids_for_titles(
|
||||
clippings: list[dict], titles: set[str]
|
||||
) -> set[str]:
|
||||
return {clipping["id"] for clipping in clippings if clipping["title"] in titles}
|
||||
|
||||
|
||||
def _reattempt_failed_with_alternate_attachments(positioned_plan: dict) -> dict:
|
||||
"""Retry text/rect failures against another attachment of the same item.
|
||||
|
||||
`_choose_attachment` commits to one file (e.g. the sole EPUB) without
|
||||
comparing text. When that file lacks the quote but a sibling PDF/EPUB
|
||||
has it (e.g. Dostoyevsky EPUB vs PDF), re-anchor the clone there and
|
||||
keep the winner, tagged `positioned-via-alternate-attachment`.
|
||||
Partial shingle anchors are also upgraded when a sibling holds the
|
||||
full quote.
|
||||
"""
|
||||
retryable = {"epub-text-not-found", "pdf-text-not-found", "pdf-rects-not-found"}
|
||||
retries: list[dict] = []
|
||||
upgrade_ids: set[str] = set()
|
||||
for item in positioned_plan.get("items", []):
|
||||
if item.get("status") in retryable:
|
||||
pass
|
||||
elif item.get("status") == "positioned" and "partial-text-match" in (
|
||||
item.get("problems") or []
|
||||
):
|
||||
clipping_id = (item.get("clipping") or {}).get("id")
|
||||
if clipping_id:
|
||||
upgrade_ids.add(clipping_id)
|
||||
else:
|
||||
continue
|
||||
zotero = item.get("zotero") or {}
|
||||
current = zotero.get("attachment") or {}
|
||||
choices = zotero.get("attachment_choices") or []
|
||||
alternates = [
|
||||
choice
|
||||
for choice in choices
|
||||
if choice.get("item_id") != current.get("item_id")
|
||||
]
|
||||
if not alternates:
|
||||
continue
|
||||
alternates.sort(
|
||||
key=lambda choice: 0
|
||||
if choice.get("content_type") != current.get("content_type")
|
||||
else 1
|
||||
)
|
||||
clone = copy.deepcopy(item)
|
||||
clone_zotero = dict(clone.get("zotero") or {})
|
||||
clone_zotero["attachment"] = alternates[0]
|
||||
clone["zotero"] = clone_zotero
|
||||
clone["status"] = "ready-for-positioning"
|
||||
clone["problems"] = []
|
||||
retries.append(clone)
|
||||
if not retries:
|
||||
return positioned_plan
|
||||
mini = {"format": positioned_plan.get("format"), "items": retries}
|
||||
mini = add_epub_positions(mini)
|
||||
mini = add_pdf_positions(mini)
|
||||
winners = {}
|
||||
for attempt in mini.get("items", []):
|
||||
clipping_id = (attempt.get("clipping") or {}).get("id")
|
||||
if clipping_id and attempt.get("status") == "positioned":
|
||||
winners[clipping_id] = attempt
|
||||
if not winners:
|
||||
return positioned_plan
|
||||
merged = []
|
||||
for item in positioned_plan.get("items", []):
|
||||
clipping_id = (item.get("clipping") or {}).get("id")
|
||||
if clipping_id in winners:
|
||||
won = winners[clipping_id]
|
||||
if clipping_id in upgrade_ids and "partial-text-match" in (
|
||||
won.get("problems") or []
|
||||
):
|
||||
merged.append(item)
|
||||
continue
|
||||
won = dict(won)
|
||||
carried = [
|
||||
problem
|
||||
for problem in item.get("problems", [])
|
||||
if problem != "partial-text-match"
|
||||
or "partial-text-match" in (won.get("problems") or [])
|
||||
]
|
||||
won["problems"] = [
|
||||
*carried,
|
||||
"positioned-via-alternate-attachment",
|
||||
]
|
||||
merged.append(won)
|
||||
else:
|
||||
merged.append(item)
|
||||
updated = dict(positioned_plan)
|
||||
updated["items"] = merged
|
||||
status_counts: dict[str, int] = {}
|
||||
for entry in merged:
|
||||
status = entry.get("status", "unknown")
|
||||
status_counts[status] = status_counts.get(status, 0) + 1
|
||||
updated["status_counts"] = status_counts
|
||||
return updated
|
||||
|
||||
|
||||
def _add_previous_imported_comments(
|
||||
annotations: list[dict], previous_plan: dict | None
|
||||
) -> None:
|
||||
|
|
@ -134,7 +285,12 @@ def _unresolved_clipping_ids(
|
|||
return {
|
||||
item.get("clipping", {}).get("id")
|
||||
for item in (previous_plan or {}).get("items", [])
|
||||
if item.get("status") not in {"positioned", "ignored-title"}
|
||||
if item.get("status") not in {
|
||||
"positioned",
|
||||
"ignored-title",
|
||||
"pdf-position-skipped-non-highlight",
|
||||
"epub-position-skipped-non-highlight",
|
||||
}
|
||||
and item.get("clipping", {}).get("id") in source_ids
|
||||
}
|
||||
|
||||
|
|
@ -535,18 +691,12 @@ def main(argv: list[str] | None = None) -> int:
|
|||
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
|
||||
except Exception:
|
||||
_ov_for_inc = {}
|
||||
previous_override_snapshot = (prev_final_plan or {}).get("overrides_snapshot")
|
||||
if previous_override_snapshot is None:
|
||||
changed_override_titles = set(_ov_for_inc)
|
||||
else:
|
||||
changed_override_titles = {
|
||||
title
|
||||
for title in set(previous_override_snapshot) | set(_ov_for_inc)
|
||||
if previous_override_snapshot.get(title) != _ov_for_inc.get(title)
|
||||
}
|
||||
for c in effective_clippings:
|
||||
if c["title"] in changed_override_titles:
|
||||
to_process_ids.add(c["id"])
|
||||
changed_override_titles = _changed_override_titles(
|
||||
(prev_final_plan or {}).get("overrides_snapshot"), _ov_for_inc
|
||||
)
|
||||
to_process_ids.update(
|
||||
_clipping_ids_for_titles(effective_clippings, changed_override_titles)
|
||||
)
|
||||
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
|
||||
for annotation in previous_annotations:
|
||||
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
|
||||
|
|
@ -563,10 +713,15 @@ def main(argv: list[str] | None = None) -> int:
|
|||
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
|
||||
if clipping_id in new_ids
|
||||
)
|
||||
to_process_ids.update(_deferred_clipping_ids(prev_final_plan, new_ids))
|
||||
if args.retry_unresolved and prev_positioned_plan:
|
||||
to_process_ids.update(
|
||||
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
|
||||
)
|
||||
# Permanently discarded clips (user action in Conflicts) never re-queue.
|
||||
to_process_ids.difference_update(
|
||||
_discarded_clipping_ids(prev_final_plan, new_ids)
|
||||
)
|
||||
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
|
||||
try:
|
||||
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
|
||||
|
|
@ -632,6 +787,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
epub_plan = add_epub_positions(plan)
|
||||
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
|
||||
positioned_plan = add_pdf_positions(epub_plan)
|
||||
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
|
||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||
final_plan = build_final_writer_plan(positioned_plan)
|
||||
# Attach deletions for writer
|
||||
|
|
@ -665,6 +821,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
epub_plan = add_epub_positions(plan)
|
||||
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
|
||||
positioned_plan = add_pdf_positions(epub_plan)
|
||||
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
|
||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||
final_plan = build_final_writer_plan(positioned_plan)
|
||||
final_plan["deletions"] = sorted(
|
||||
|
|
@ -700,10 +857,14 @@ def main(argv: list[str] | None = None) -> int:
|
|||
final_plan.get("annotations", []),
|
||||
final_plan.get("deletions", []),
|
||||
)
|
||||
_refresh_history_relations(final_plan["all_annotations"], zotero_index)
|
||||
final_plan["annotation_count"] = len(final_plan["all_annotations"])
|
||||
final_plan["delta_annotation_count"] = delta_annotation_count
|
||||
final_plan["processed_clipping_ids"] = sorted(new_ids)
|
||||
final_plan["overrides_snapshot"] = overrides or {}
|
||||
final_plan["discarded_clipping_ids"] = sorted(
|
||||
_discarded_clipping_ids(prev_final_plan, new_ids)
|
||||
)
|
||||
|
||||
indent = 2 if args.pretty else None
|
||||
outputs = {
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ from dataclasses import dataclass
|
|||
from html import unescape
|
||||
from html.parser import HTMLParser
|
||||
import posixpath
|
||||
import unicodedata
|
||||
import zipfile
|
||||
import xml.etree.ElementTree as ET
|
||||
from typing import Any
|
||||
|
|
@ -68,6 +69,8 @@ def _position_item(
|
|||
annotation["sortIndex"] = position["sortIndex"]
|
||||
updated["annotation"] = annotation
|
||||
updated["status"] = "positioned"
|
||||
if position.get("partial"):
|
||||
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
|
||||
return updated
|
||||
|
||||
|
||||
|
|
@ -104,11 +107,42 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
|
|||
return None
|
||||
|
||||
match_start = normalized_haystack.find(normalized_quote)
|
||||
if match_start < 0:
|
||||
return None
|
||||
raw_start: int | None = None
|
||||
raw_end: int | None = None
|
||||
partial = False
|
||||
if match_start >= 0:
|
||||
match_end = match_start + len(normalized_quote)
|
||||
raw_start = haystack_map[match_start]
|
||||
raw_end = haystack_map[match_end - 1] + 1
|
||||
else:
|
||||
stripped_quote, _ = _strip_with_map(normalized_quote)
|
||||
if stripped_quote and len(stripped_quote) >= 20:
|
||||
stripped_haystack, stripped_map = _strip_with_map(normalized_haystack)
|
||||
strip_start = stripped_haystack.find(stripped_quote)
|
||||
strip_len = len(stripped_quote)
|
||||
if strip_start < 0:
|
||||
hit = _partial_stripped_range(stripped_haystack, stripped_quote)
|
||||
if hit is not None:
|
||||
strip_start, strip_end = hit
|
||||
strip_len = strip_end - strip_start
|
||||
partial = True
|
||||
if strip_start >= 0:
|
||||
# stripped_map points into normalized_haystack;
|
||||
# convert via haystack_map back to raw haystack offsets.
|
||||
norm_start = stripped_map[strip_start]
|
||||
norm_end = stripped_map[strip_start + strip_len - 1]
|
||||
raw_start = haystack_map[norm_start]
|
||||
raw_end = haystack_map[norm_end] + 1
|
||||
else:
|
||||
span = _shingle_span(normalized_haystack, _quote_words(normalized_quote))
|
||||
if span is not None:
|
||||
raw_start = haystack_map[span[0]]
|
||||
raw_end = haystack_map[span[1] - 1] + 1
|
||||
partial = True
|
||||
else:
|
||||
partial = False
|
||||
if raw_start is None or raw_end is None:
|
||||
return None
|
||||
|
||||
# Account for the spaces inserted between text nodes in haystack construction.
|
||||
adjusted_nodes = _nodes_for_joined_text(text_nodes)
|
||||
|
|
@ -139,6 +173,7 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
|
|||
"value": value,
|
||||
},
|
||||
"sortIndex": f"{spine_step:05d}|{raw_start:08d}",
|
||||
"partial": partial,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -213,6 +248,11 @@ def _clean_text(text: str) -> str:
|
|||
|
||||
|
||||
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||
text = unicodedata.normalize("NFKC", text)
|
||||
text = (
|
||||
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
|
||||
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
|
||||
)
|
||||
normalized = []
|
||||
mapping = []
|
||||
last_was_space = False
|
||||
|
|
@ -232,6 +272,58 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
|||
return "".join(normalized), mapping
|
||||
|
||||
|
||||
def _strip_with_map(text: str) -> tuple[str, list[int]]:
|
||||
text = unicodedata.normalize("NFKC", text).casefold()
|
||||
stripped: list[str] = []
|
||||
mapping: list[int] = []
|
||||
for index, char in enumerate(text):
|
||||
if char.isalnum():
|
||||
stripped.append(char)
|
||||
mapping.append(index)
|
||||
return "".join(stripped), mapping
|
||||
|
||||
|
||||
def _partial_stripped_range(
|
||||
haystack: str, quote: str, min_len: int = 30
|
||||
) -> tuple[int, int] | None:
|
||||
if len(quote) < min_len * 2:
|
||||
return None
|
||||
mid = len(quote) // 2
|
||||
for part in (quote[:mid], quote[mid:]):
|
||||
if len(part) < min_len:
|
||||
continue
|
||||
start = haystack.find(part)
|
||||
if start >= 0:
|
||||
return start, start + len(part)
|
||||
return None
|
||||
|
||||
|
||||
def _quote_words(normalized: str) -> list[str]:
|
||||
words: list[str] = []
|
||||
for token in normalized.split():
|
||||
cleaned = "".join(char for char in token if char.isalnum())
|
||||
if cleaned:
|
||||
words.append(cleaned)
|
||||
return words
|
||||
|
||||
|
||||
def _shingle_span(
|
||||
normalized_text: str, quote_words: list[str], width: int = 6
|
||||
) -> tuple[int, int] | None:
|
||||
first: int | None = None
|
||||
last: int | None = None
|
||||
for start in range(len(quote_words) - width + 1):
|
||||
needle = " ".join(quote_words[start : start + width])
|
||||
at = normalized_text.find(needle)
|
||||
if at >= 0:
|
||||
end = at + len(needle)
|
||||
first = at if first is None else min(first, at)
|
||||
last = end if last is None else max(last, end)
|
||||
if first is None or last is None:
|
||||
return None
|
||||
return first, last
|
||||
|
||||
|
||||
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
|
||||
nodes = []
|
||||
cursor = 0
|
||||
|
|
|
|||
|
|
@ -71,6 +71,7 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
|
|||
"attachment_key": attachment["key"],
|
||||
"parent_item_id": item["zotero"]["parent_item_id"],
|
||||
"parent_key": item["zotero"]["parent_key"],
|
||||
"related_items": item["zotero"].get("related_items", []),
|
||||
"citation_key": item["zotero"].get("citation_key"),
|
||||
"doi": item["zotero"].get("doi"),
|
||||
"isbn": item["zotero"].get("isbn"),
|
||||
|
|
|
|||
|
|
@ -124,6 +124,12 @@ def _plan_clipping(
|
|||
zotero_item, overrides_by_title.get(clipping["title"]) or {}
|
||||
)
|
||||
status = "ready-for-positioning" if attachment else attachment_status
|
||||
stub = _annotation_stub(clipping, colour_map) if attachment else None
|
||||
related_items = (
|
||||
_append_citekey_links(stub, clipping, zotero_item, items_by_id)
|
||||
if stub
|
||||
else []
|
||||
)
|
||||
return {
|
||||
"clipping": clipping,
|
||||
"status": status,
|
||||
|
|
@ -138,17 +144,96 @@ def _plan_clipping(
|
|||
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
|
||||
"url": zotero_item["fields"].get("url"),
|
||||
"publication_title": zotero_item["fields"].get("publicationTitle"),
|
||||
"related_items": related_items,
|
||||
"attachment": attachment,
|
||||
"attachment_choices": zotero_item.get("attachments", []),
|
||||
"expected_attachment_type": _expected_attachment_type(
|
||||
clipping, zotero_item.get("attachments", [])
|
||||
),
|
||||
},
|
||||
"annotation": _annotation_stub(clipping, colour_map) if attachment else None,
|
||||
"annotation": stub,
|
||||
"problems": problems,
|
||||
}
|
||||
|
||||
|
||||
def _append_citekey_links(
|
||||
stub: dict[str, Any],
|
||||
clipping: dict[str, Any],
|
||||
parent_item: dict[str, Any],
|
||||
items_by_id: dict[int, dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Turn `@citekey` mentions into clickable Zotero item links.
|
||||
|
||||
Kindle-side `@key` tokens (Obsidian-style) resolve against the library
|
||||
index; matches to *other* items are appended to the annotation comment
|
||||
as `zotero://select` links, which Zotero renders clickable. Self-links
|
||||
to the parent item are skipped.
|
||||
"""
|
||||
related_items = _related_items_for_citekeys(
|
||||
(clipping.get("text") or "", clipping.get("comment") or ""),
|
||||
parent_item.get("item_id"),
|
||||
items_by_id,
|
||||
)
|
||||
links = [
|
||||
f"@{target['citation_key']} {target['link']}" for target in related_items
|
||||
]
|
||||
if not links:
|
||||
return []
|
||||
existing = stub.get("comment") or ""
|
||||
block = "Related: " + " · ".join(links)
|
||||
if block not in existing:
|
||||
stub["comment"] = f"{existing}\n\n{block}" if existing else block
|
||||
return related_items
|
||||
|
||||
|
||||
def _select_link(item: dict[str, Any]) -> str:
|
||||
"""Library-aware `zotero://select` link for an indexed item.
|
||||
|
||||
Group-library targets need the `/groups/{groupID}/items/{key}` form;
|
||||
personal-library (or unknown) targets use `/library/items/{key}`.
|
||||
"""
|
||||
group_id = item.get("group_id")
|
||||
if group_id:
|
||||
return f"zotero://select/groups/{group_id}/items/{item['key']}"
|
||||
return f"zotero://select/library/items/{item['key']}"
|
||||
|
||||
|
||||
def _related_items_for_citekeys(
|
||||
sources: tuple[str, ...],
|
||||
parent_item_id: int | None,
|
||||
items_by_id: dict[int, dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
tokens = {
|
||||
token.casefold()
|
||||
for source in sources
|
||||
for token in re.findall(r"@([A-Za-z0-9_][\w:.-]*)", source or "")
|
||||
}
|
||||
if not tokens:
|
||||
return []
|
||||
by_citekey = {
|
||||
str(item.get("fields", {}).get("citationKey", "")).casefold(): item
|
||||
for item in items_by_id.values()
|
||||
if item.get("fields", {}).get("citationKey")
|
||||
}
|
||||
related = []
|
||||
seen_item_ids = set()
|
||||
for token in sorted(tokens):
|
||||
target = by_citekey.get(token)
|
||||
target_id = target.get("item_id") if target else None
|
||||
if not target or target_id == parent_item_id or target_id in seen_item_ids:
|
||||
continue
|
||||
seen_item_ids.add(target_id)
|
||||
related.append(
|
||||
{
|
||||
"item_id": target_id,
|
||||
"key": target["key"],
|
||||
"citation_key": target["fields"]["citationKey"],
|
||||
"link": _select_link(target),
|
||||
}
|
||||
)
|
||||
return related
|
||||
|
||||
|
||||
def _choose_attachment(
|
||||
zotero_item: dict[str, Any],
|
||||
override: dict[str, Any] | None = None,
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import re
|
|||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unicodedata
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
|
@ -126,7 +127,12 @@ def _position_item(
|
|||
text_pages, item["clipping"]["text"], item["clipping"].get("page")
|
||||
)
|
||||
if page_index is None:
|
||||
return _with_problem(item, "pdf-text-not-found")
|
||||
failed = _with_problem(item, "pdf-text-not-found")
|
||||
hint = _page_hint_text(text_pages, item["clipping"].get("page"))
|
||||
if hint:
|
||||
failed = dict(failed)
|
||||
failed["page_hint"] = hint
|
||||
return failed
|
||||
page_xml = xml_cache.setdefault(
|
||||
(extraction_path, page_index),
|
||||
extract_pdf_page_xml(extraction_path, page_index),
|
||||
|
|
@ -152,6 +158,8 @@ def _position_item(
|
|||
annotation["sortIndex"] = position.get("sortIndex")
|
||||
updated["annotation"] = annotation
|
||||
updated["status"] = "positioned"
|
||||
if position.get("partial"):
|
||||
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
|
||||
return updated
|
||||
|
||||
|
||||
|
|
@ -227,6 +235,37 @@ def find_pdf_text_page(
|
|||
normalized_page, _ = _normalize_with_map(pages[index])
|
||||
if normalized_quote in normalized_page:
|
||||
return index
|
||||
# Fallback: alphanumeric-only search tolerates PDF extraction artifacts
|
||||
# such as "o f" for "of", hyphenation, and punctuation differences.
|
||||
stripped_quote, _ = _strip_with_map(quote)
|
||||
if stripped_quote and len(stripped_quote) >= 20:
|
||||
for index in page_order:
|
||||
stripped_page, _ = _strip_with_map(pages[index])
|
||||
if stripped_quote in stripped_page:
|
||||
return index
|
||||
# Partial fallback: long highlights may span pages or contain a
|
||||
# divergent middle (edition differences). Anchor to either half so
|
||||
# Zotero-style word search behaviour is approximated.
|
||||
for index in page_order:
|
||||
stripped_page, _ = _strip_with_map(pages[index])
|
||||
if _partial_stripped_hit(stripped_page, stripped_quote):
|
||||
return index
|
||||
# Shingle fallback: scattered OCR substitutions (e.g. Kindle
|
||||
# "yadsınaraz" vs PDF "yadsınamaz") break every long substring but
|
||||
# leave most 6-word runs intact. Rank pages by shingle hits.
|
||||
quote_words = _quote_words(normalized_quote)
|
||||
if len(quote_words) >= 6:
|
||||
required = 2 if len(quote_words) >= 12 else 1
|
||||
best_index: int | None = None
|
||||
best_hits = 0
|
||||
for index in page_order:
|
||||
normalized_page, _ = _normalize_with_map(pages[index])
|
||||
hits = _shingle_hits(normalized_page, quote_words)
|
||||
if hits > best_hits:
|
||||
best_hits = hits
|
||||
best_index = index
|
||||
if best_index is not None and best_hits >= required:
|
||||
return best_index
|
||||
return None
|
||||
|
||||
|
||||
|
|
@ -290,25 +329,52 @@ def find_pdf_rects(
|
|||
text = " ".join(word.text for word in page.words)
|
||||
normalized_text, text_map = _normalize_with_map(text)
|
||||
start = normalized_text.find(normalized_quote)
|
||||
if start < 0:
|
||||
return None
|
||||
raw_start: int | None = None
|
||||
raw_end: int | None = None
|
||||
partial = False
|
||||
if start >= 0:
|
||||
end = start + len(normalized_quote)
|
||||
raw_start = text_map[start]
|
||||
raw_end = text_map[end - 1] + 1
|
||||
else:
|
||||
stripped_quote, _ = _strip_with_map(quote)
|
||||
if stripped_quote and len(stripped_quote) >= 20:
|
||||
stripped_text, stripped_map = _strip_with_map(text)
|
||||
strip_start = stripped_text.find(stripped_quote)
|
||||
if strip_start >= 0:
|
||||
strip_end = strip_start + len(stripped_quote)
|
||||
raw_start = stripped_map[strip_start]
|
||||
raw_end = stripped_map[strip_end - 1] + 1
|
||||
else:
|
||||
hit = _partial_stripped_range(stripped_text, stripped_quote)
|
||||
if hit is not None:
|
||||
strip_start, strip_end = hit
|
||||
raw_start = stripped_map[strip_start]
|
||||
raw_end = stripped_map[strip_end - 1] + 1
|
||||
partial = True
|
||||
else:
|
||||
span = _shingle_span(normalized_text, _quote_words(normalized_quote))
|
||||
if span is not None:
|
||||
raw_start = text_map[span[0]]
|
||||
raw_end = text_map[span[1] - 1] + 1
|
||||
partial = True
|
||||
if raw_start is None or raw_end is None:
|
||||
return None
|
||||
matched_words = [
|
||||
word for word in page.words if word.end >= raw_start and word.start <= raw_end
|
||||
]
|
||||
if not matched_words:
|
||||
return None
|
||||
pdf_width, pdf_height = size_cache.setdefault(
|
||||
(path, page.index),
|
||||
_pdf_page_size(path, page.index + 1, page.width, page.height),
|
||||
)
|
||||
size_key = (path, page.index)
|
||||
if size_key not in size_cache:
|
||||
size_cache[size_key] = _pdf_page_size(path, page.index + 1, page.width, page.height)
|
||||
pdf_width, pdf_height = size_cache[size_key]
|
||||
rects = _word_rects(matched_words, page, pdf_width, pdf_height)
|
||||
return {
|
||||
"pageIndex": page.index,
|
||||
"rects": rects,
|
||||
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
|
||||
"partial": partial,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -361,6 +427,11 @@ def _pdf_page_size(
|
|||
|
||||
|
||||
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||
text = unicodedata.normalize("NFKC", text)
|
||||
text = (
|
||||
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
|
||||
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
|
||||
)
|
||||
normalized = []
|
||||
mapping = []
|
||||
last_was_space = False
|
||||
|
|
@ -380,6 +451,112 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
|||
return "".join(normalized), mapping
|
||||
|
||||
|
||||
def _strip_with_map(text: str) -> tuple[str, list[int]]:
|
||||
"""Alphanumeric-only folding with index mapping.
|
||||
|
||||
Removes spaces and punctuation so PDF artifacts like "o f" for "of",
|
||||
hyphenated line breaks, and curly-quote/dash variants still match.
|
||||
Mapping points back into the input string for rect recovery.
|
||||
"""
|
||||
text = unicodedata.normalize("NFKC", text).casefold()
|
||||
stripped: list[str] = []
|
||||
mapping: list[int] = []
|
||||
for index, char in enumerate(text):
|
||||
if char.isalnum():
|
||||
stripped.append(char)
|
||||
mapping.append(index)
|
||||
return "".join(stripped), mapping
|
||||
|
||||
|
||||
def _partial_stripped_range(
|
||||
haystack: str, quote: str, min_len: int = 30
|
||||
) -> tuple[int, int] | None:
|
||||
"""Best half-quote hit for long highlights spanning pages/editions."""
|
||||
if len(quote) < min_len * 2:
|
||||
return None
|
||||
mid = len(quote) // 2
|
||||
for part in (quote[:mid], quote[mid:]):
|
||||
if len(part) < min_len:
|
||||
continue
|
||||
start = haystack.find(part)
|
||||
if start >= 0:
|
||||
return start, start + len(part)
|
||||
return None
|
||||
|
||||
|
||||
def _partial_stripped_hit(haystack: str, quote: str, min_len: int = 30) -> bool:
|
||||
return _partial_stripped_range(haystack, quote, min_len) is not None
|
||||
|
||||
|
||||
def _quote_words(normalized: str, width: int = 6) -> list[str]:
|
||||
del width
|
||||
words: list[str] = []
|
||||
for token in normalized.split():
|
||||
cleaned = "".join(char for char in token if char.isalnum())
|
||||
if cleaned:
|
||||
words.append(cleaned)
|
||||
return words
|
||||
|
||||
|
||||
def _shingle_hits(normalized_page: str, quote_words: list[str], width: int = 6) -> int:
|
||||
hits = 0
|
||||
seen: set[tuple[str, ...]] = set()
|
||||
for start in range(len(quote_words) - width + 1):
|
||||
shingle = tuple(quote_words[start : start + width])
|
||||
if shingle in seen:
|
||||
continue
|
||||
seen.add(shingle)
|
||||
if " ".join(shingle) in normalized_page:
|
||||
hits += 1
|
||||
return hits
|
||||
|
||||
|
||||
def _shingle_span(
|
||||
normalized_text: str, quote_words: list[str], width: int = 6
|
||||
) -> tuple[int, int] | None:
|
||||
first: int | None = None
|
||||
last: int | None = None
|
||||
for start in range(len(quote_words) - width + 1):
|
||||
needle = " ".join(quote_words[start : start + width])
|
||||
at = normalized_text.find(needle)
|
||||
if at >= 0:
|
||||
end = at + len(needle)
|
||||
first = at if first is None else min(first, at)
|
||||
last = end if last is None else max(last, end)
|
||||
if first is None or last is None:
|
||||
return None
|
||||
return first, last
|
||||
|
||||
|
||||
def _page_hint_text(
|
||||
pages: list[str], kindle_page: str | None, radius: int = 1, limit: int = 1200
|
||||
) -> dict[str, Any] | None:
|
||||
"""Snippet of the PDF pages around the Kindle page hint.
|
||||
|
||||
Lets the user eyeball which part a `pdf-text-not-found` highlight
|
||||
belongs to and place it manually. Truncated to keep artifacts small.
|
||||
"""
|
||||
if not kindle_page or not kindle_page.isdigit():
|
||||
return None
|
||||
center = int(kindle_page) - 1
|
||||
if center < 0 or center >= len(pages):
|
||||
return None
|
||||
parts = []
|
||||
for index in range(max(0, center - radius), min(len(pages), center + radius + 1)):
|
||||
snippet = " ".join(pages[index].split())
|
||||
if snippet:
|
||||
parts.append(f"[PDF p.{index + 1}] {snippet[:limit]}")
|
||||
if not parts:
|
||||
return None
|
||||
return {
|
||||
"kindle_page": kindle_page,
|
||||
"pdf_pages": list(
|
||||
range(max(1, center - radius + 1), min(len(pages), center + radius + 1) + 1)
|
||||
),
|
||||
"text": "\n\n".join(parts)[: limit * 2 + 64],
|
||||
}
|
||||
|
||||
|
||||
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
|
||||
updated = dict(item)
|
||||
updated["status"] = problem
|
||||
|
|
|
|||
|
|
@ -34,6 +34,8 @@ class ZoteroItem:
|
|||
key: str
|
||||
item_type: str
|
||||
title: str | None
|
||||
library_id: int | None = None
|
||||
group_id: int | None = None
|
||||
creators: list[str] = field(default_factory=list)
|
||||
fields: dict[str, str] = field(default_factory=dict)
|
||||
attachments: list[ZoteroAttachment] = field(default_factory=list)
|
||||
|
|
@ -47,15 +49,19 @@ def build_zotero_index(db_path: str, storage_root: str | None = None) -> dict[st
|
|||
creators = _load_creators(connection)
|
||||
attachments = _load_attachments(connection, storage_root)
|
||||
|
||||
groups = _load_groups(connection)
|
||||
indexed_items: list[ZoteroItem] = []
|
||||
for row in items:
|
||||
fields = _metadata_for_item(row)
|
||||
library_id = row["libraryID"]
|
||||
indexed_items.append(
|
||||
ZoteroItem(
|
||||
item_id=row["itemID"],
|
||||
key=row["key"],
|
||||
item_type=row["typeName"],
|
||||
title=fields.get("title"),
|
||||
library_id=library_id,
|
||||
group_id=groups.get(library_id),
|
||||
creators=creators.get(row["itemID"], []),
|
||||
fields=fields,
|
||||
attachments=attachments.get(row["itemID"], []),
|
||||
|
|
@ -80,6 +86,7 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
|
|||
SELECT
|
||||
i.itemID,
|
||||
i.key,
|
||||
i.libraryID,
|
||||
it.typeName,
|
||||
{field_columns}
|
||||
FROM items i
|
||||
|
|
@ -95,6 +102,14 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
|
|||
return list(connection.execute(query))
|
||||
|
||||
|
||||
def _load_groups(connection: sqlite3.Connection) -> dict[int, int]:
|
||||
try:
|
||||
rows = connection.execute("SELECT groupID, libraryID FROM groups")
|
||||
except Exception:
|
||||
return {}
|
||||
return {row["libraryID"]: row["groupID"] for row in rows}
|
||||
|
||||
|
||||
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
|
||||
rows = connection.execute(
|
||||
"""
|
||||
|
|
|
|||
15
tests/fixtures/windows-smoke/My Clippings.txt
vendored
Normal file
15
tests/fixtures/windows-smoke/My Clippings.txt
vendored
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
Kanzi Dummy EPUB
|
||||
- Your Highlight on page 1 | location 100-120 | Added on Tuesday, 18 May 2021 14:18:13
|
||||
|
||||
This is a test highlight long enough to import for EPUB.
|
||||
==========
|
||||
Kanzi Dummy PDF
|
||||
- Your Highlight on page 2 | location 200-220 | Added on Tuesday, 18 May 2021 14:20:00
|
||||
|
||||
This is a test highlight long enough to import for PDF.
|
||||
==========
|
||||
Unknown Book Title XYZ
|
||||
- Your Highlight on page 1 | location 10-11 | Added on Tuesday, 18 May 2021 14:18:13
|
||||
|
||||
This should land in Conflicts for manual mapping.
|
||||
==========
|
||||
36
tests/fixtures/windows-smoke/checklist.txt
vendored
Normal file
36
tests/fixtures/windows-smoke/checklist.txt
vendored
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
Kanzi Windows smoke test (Parallels VM)
|
||||
|
||||
COPY THESE 3 FILES INTO THE VM:
|
||||
- My Clippings.txt
|
||||
- kanzi-dummy.epub
|
||||
- kanzi-dummy.pdf
|
||||
- dist/kanzi.xpi (from repo root, same cross-platform build)
|
||||
|
||||
1) PowerShell:
|
||||
winget install --id Python.Python.3.13 --exact
|
||||
winget install --id oschwartz10612.Poppler --exact
|
||||
winget install --id QPDF.QPDF --exact
|
||||
python --version
|
||||
pdftotext -v
|
||||
pdfinfo -v
|
||||
qpdf --version
|
||||
|
||||
2) Fully quit + reopen Zotero (picks up PATH).
|
||||
|
||||
3) Fresh Zotero, sync off. Create:
|
||||
- Book "Kanzi Dummy EPUB" + attach kanzi-dummy.epub
|
||||
- Book "Kanzi Dummy PDF" + attach kanzi-dummy.pdf
|
||||
|
||||
4) Tools -> Kanzi -> Choose My Clippings.txt (the copied one).
|
||||
Leave Python path empty to test auto-detect.
|
||||
|
||||
EXPECT:
|
||||
- 2 highlights position and await review, 1 title in Conflicts
|
||||
("Unknown Book Title XYZ").
|
||||
- Summary shows new Status by Kindle Title table with source counts.
|
||||
- Save a mapping for the unknown title -> Re-import processes it.
|
||||
- Table headers sort and persist across pages.
|
||||
|
||||
IF PYTHON NOT FOUND:
|
||||
Settings shows detected path hint; paste full python.exe path,
|
||||
Save Settings, retry.
|
||||
BIN
tests/fixtures/windows-smoke/kanzi-dummy.epub
vendored
Normal file
BIN
tests/fixtures/windows-smoke/kanzi-dummy.epub
vendored
Normal file
Binary file not shown.
33
tests/fixtures/windows-smoke/kanzi-dummy.pdf
vendored
Normal file
33
tests/fixtures/windows-smoke/kanzi-dummy.pdf
vendored
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<< /Type /Catalog /Pages 2 0 R >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Length 134 >>
|
||||
stream
|
||||
BT /F1 12 Tf 72 720 Td (Kanzi Dummy PDF) Tj ET
|
||||
BT /F1 12 Tf 72 700 Td (This is a test highlight long enough to import for PDF.) Tj ET
|
||||
endstream
|
||||
endobj
|
||||
xref
|
||||
0 6
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000058 00000 n
|
||||
0000000115 00000 n
|
||||
0000000241 00000 n
|
||||
0000000311 00000 n
|
||||
trailer
|
||||
<< /Size 6 /Root 1 0 R >>
|
||||
startxref
|
||||
495
|
||||
%%EOF
|
||||
93
tests/fixtures/windows-smoke/make_fixtures.py
vendored
Normal file
93
tests/fixtures/windows-smoke/make_fixtures.py
vendored
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
"""Generate minimal EPUB + PDF fixtures for the Windows smoke test.
|
||||
|
||||
Run: python3 tests/fixtures/windows-smoke/make_fixtures.py
|
||||
Outputs kanzi-dummy.epub and kanzi-dummy.pdf next to this script.
|
||||
Both contain the highlight sentences from 'My Clippings.txt' so
|
||||
EPUB CFI and PDF text positioning can succeed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import zipfile
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
EPUB_TEXT = "This is a test highlight long enough to import for EPUB."
|
||||
PDF_TEXT = "This is a test highlight long enough to import for PDF."
|
||||
|
||||
|
||||
def make_epub(path: Path) -> None:
|
||||
chapter = f"""<?xml version="1.0" encoding="utf-8"?>
|
||||
<html xmlns="http://www.w3.org/1999/xhtml">
|
||||
<head><title>Kanzi Dummy</title></head>
|
||||
<body>
|
||||
<h1>Kanzi Dummy EPUB</h1>
|
||||
<p>Some intro text before the highlight.</p>
|
||||
<p>{EPUB_TEXT}</p>
|
||||
<p>Some trailing text after the highlight.</p>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
container = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
|
||||
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
|
||||
</container>
|
||||
"""
|
||||
opf = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<package version="3.0" unique-identifier="uid" xmlns="http://www.idpf.org/2007/opf">
|
||||
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Kanzi Dummy EPUB</dc:title><dc:identifier id="uid">kanzi-dummy-epub</dc:identifier><dc:language>en</dc:language></metadata>
|
||||
<manifest><item id="ch1" href="chapter1.xhtml" media-type="application/xhtml+xml"/></manifest>
|
||||
<spine><itemref idref="ch1"/></spine>
|
||||
</package>
|
||||
"""
|
||||
with zipfile.ZipFile(path, "w") as z:
|
||||
z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
|
||||
z.writestr("META-INF/container.xml", container, compress_type=zipfile.ZIP_DEFLATED)
|
||||
z.writestr("OEBPS/content.opf", opf, compress_type=zipfile.ZIP_DEFLATED)
|
||||
z.writestr("OEBPS/chapter1.xhtml", chapter, compress_type=zipfile.ZIP_DEFLATED)
|
||||
|
||||
|
||||
def _pdf_escape(text: str) -> str:
|
||||
return text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
|
||||
|
||||
|
||||
def make_pdf(path: Path) -> None:
|
||||
# Minimal one-page PDF with Helvetica text. pdftotext extracts it fine.
|
||||
content = (
|
||||
f"BT /F1 12 Tf 72 720 Td ({_pdf_escape('Kanzi Dummy PDF')}) Tj ET\n"
|
||||
f"BT /F1 12 Tf 72 700 Td ({_pdf_escape(PDF_TEXT)}) Tj ET\n"
|
||||
)
|
||||
objects: list[bytes] = []
|
||||
objects.append(b"<< /Type /Catalog /Pages 2 0 R >>")
|
||||
objects.append(b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
|
||||
objects.append(
|
||||
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
|
||||
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"
|
||||
)
|
||||
objects.append(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
|
||||
objects.append(f"<< /Length {len(content.encode('latin-1'))} >>\nstream\n{content}endstream".encode("latin-1"))
|
||||
|
||||
out = bytearray(b"%PDF-1.4\n")
|
||||
offsets = [0]
|
||||
for i, body in enumerate(objects, start=1):
|
||||
offsets.append(len(out))
|
||||
out += f"{i} 0 obj\n".encode("ascii") + body + b"\nendobj\n"
|
||||
xref_pos = len(out)
|
||||
out += f"xref\n0 {len(objects) + 1}\n".encode("ascii")
|
||||
out += b"0000000000 65535 f \n"
|
||||
for off in offsets[1:]:
|
||||
out += f"{off:010d} 00000 n \n".encode("ascii")
|
||||
out += (
|
||||
f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
|
||||
f"startxref\n{xref_pos}\n%%EOF\n"
|
||||
).encode("ascii")
|
||||
path.write_bytes(bytes(out))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
make_epub(HERE / "kanzi-dummy.epub")
|
||||
make_pdf(HERE / "kanzi-dummy.pdf")
|
||||
print("wrote kanzi-dummy.epub + kanzi-dummy.pdf")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -28,10 +28,206 @@ def test_pdf_fallback_tag():
|
|||
"format":"x",
|
||||
"items":[
|
||||
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
|
||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
|
||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1","related_items":[{"item_id":20,"key":"P2","citation_key":"c2"}]},
|
||||
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
||||
]
|
||||
}
|
||||
plan = build_final_writer_plan(positioned)
|
||||
tags = plan["annotations"][0]["annotation"]["tags"]
|
||||
assert any(t["name"].startswith("kindle-id:") for t in tags)
|
||||
assert plan["annotations"][0]["related_items"][0]["item_id"] == 20
|
||||
|
||||
|
||||
def test_pdf_tolerates_of_split_artifact():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page, find_pdf_rects, PDFPageXML, PDFWord
|
||||
quote = "nothing other than idolatry of the machine and which"
|
||||
page_text = "nothing other than idolatry o f the machine and which"
|
||||
assert find_pdf_text_page([page_text], quote) == 0
|
||||
words = []
|
||||
cursor = 0
|
||||
for w in page_text.split():
|
||||
words.append(PDFWord(text=w, left=0.0, top=0.0, width=10.0, height=10.0, start=cursor, end=cursor + len(w)))
|
||||
cursor += len(w) + 1
|
||||
page = PDFPageXML(index=0, width=100.0, height=100.0, words=words)
|
||||
rects = find_pdf_rects("dummy.pdf", page, quote, {("dummy.pdf", 0): (100.0, 100.0)})
|
||||
assert rects is not None
|
||||
assert rects["pageIndex"] == 0
|
||||
assert rects["rects"]
|
||||
|
||||
|
||||
def test_pdf_tolerates_punctuation_and_quotes():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||
quote = "This, then, gives rise to “intemperate technicism”."
|
||||
page = "this then gives rise to intemperate technicism"
|
||||
assert find_pdf_text_page([page], quote) == 0
|
||||
|
||||
|
||||
def test_pdf_partial_half_anchors_spanning_text():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||
first = "A" * 70
|
||||
second = "Z" * 70
|
||||
quote = first + second
|
||||
page = "xx " + first.lower() + " yy"
|
||||
assert find_pdf_text_page([page], quote) == 0
|
||||
|
||||
|
||||
def test_pdf_shingle_fallback_scattered_ocr_errors():
|
||||
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||
quote = (
|
||||
"Gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
|
||||
"buna inanmamasım sağlayacak güç ve yeteneği bulur mucize yadsınaraz "
|
||||
"dururda bile olsa boyun eğmez"
|
||||
)
|
||||
page = (
|
||||
"gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
|
||||
"buna inanmaması sağlayacak güç ve yeteneği bulur mucize yadsınamaz "
|
||||
"dururda bile olsa boyun eğmez"
|
||||
)
|
||||
assert find_pdf_text_page([page], quote) == 0
|
||||
|
||||
|
||||
def test_page_hint_snippet_around_kindle_page():
|
||||
from kindle_zotero_importer.pdf_position import _page_hint_text
|
||||
pages = ["", "intro words here", "target page words here", "trailing words"]
|
||||
hint = _page_hint_text(pages, "3")
|
||||
assert hint is not None
|
||||
assert hint["kindle_page"] == "3"
|
||||
assert "target page words" in hint["text"]
|
||||
assert _page_hint_text(pages, None) is None
|
||||
assert _page_hint_text(pages, "99") is None
|
||||
|
||||
|
||||
def test_citekey_mentions_become_zotero_links():
|
||||
from kindle_zotero_importer.import_plan import build_import_plan
|
||||
clippings_payload = {
|
||||
"format": "x",
|
||||
"clippings": [
|
||||
{
|
||||
"id": "c1",
|
||||
"title": "Book A",
|
||||
"kind": "highlight",
|
||||
"text": "As argued by @bookb2020, this follows.",
|
||||
"page": None,
|
||||
"location": "10",
|
||||
}
|
||||
],
|
||||
}
|
||||
zotero_index = {
|
||||
"format": "x",
|
||||
"items": [
|
||||
{
|
||||
"item_id": 1,
|
||||
"key": "PARENT01",
|
||||
"title": "Book A",
|
||||
"fields": {"citationKey": "booka2020"},
|
||||
"attachments": [
|
||||
{
|
||||
"item_id": 11,
|
||||
"key": "ATTACH01",
|
||||
"title": "pdf",
|
||||
"content_type": "application/pdf",
|
||||
"path": "x.pdf",
|
||||
"resolved_path": "x.pdf",
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"title": "Book B",
|
||||
"fields": {"citationKey": "bookb2020"},
|
||||
"attachments": [],
|
||||
},
|
||||
],
|
||||
}
|
||||
match_report = {
|
||||
"format": "x",
|
||||
"matches": [
|
||||
{
|
||||
"clipping_title": "Book A",
|
||||
"status": "matched",
|
||||
"candidates": [{"item_id": 1}],
|
||||
}
|
||||
],
|
||||
}
|
||||
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||
comment = plan["items"][0]["annotation"]["comment"] or ""
|
||||
assert "@bookb2020" in comment
|
||||
assert "zotero://select/library/items/TARGET02" in comment
|
||||
assert plan["items"][0]["zotero"]["related_items"] == [
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"citation_key": "bookb2020",
|
||||
"link": "zotero://select/library/items/TARGET02",
|
||||
}
|
||||
]
|
||||
# self-links are skipped
|
||||
clippings_payload["clippings"][0]["text"] = "See @booka2020 here."
|
||||
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||
assert "zotero://select" not in (plan["items"][0]["annotation"]["comment"] or "")
|
||||
assert plan["items"][0]["zotero"]["related_items"] == []
|
||||
|
||||
|
||||
def test_historical_annotations_gain_citekey_relations():
|
||||
from kindle_zotero_importer.cli import _refresh_history_relations
|
||||
history = [
|
||||
{
|
||||
"clipping_id": "old",
|
||||
"parent_item_id": 1,
|
||||
"annotation": {
|
||||
"text": "An old highlight",
|
||||
"comment": "body without organs @deleuze1983",
|
||||
},
|
||||
}
|
||||
]
|
||||
zotero_index = {
|
||||
"items": [
|
||||
{
|
||||
"item_id": 1,
|
||||
"key": "SOURCE01",
|
||||
"fields": {"citationKey": "simondon2017a"},
|
||||
},
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"fields": {"citationKey": "deleuze1983"},
|
||||
},
|
||||
]
|
||||
}
|
||||
_refresh_history_relations(history, zotero_index)
|
||||
assert history[0]["related_items"] == [
|
||||
{
|
||||
"item_id": 2,
|
||||
"key": "TARGET02",
|
||||
"citation_key": "deleuze1983",
|
||||
"link": "zotero://select/library/items/TARGET02",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_group_library_targets_use_group_select_links():
|
||||
from kindle_zotero_importer.import_plan import _select_link
|
||||
assert (
|
||||
_select_link({"key": "ABC123", "group_id": 6644344})
|
||||
== "zotero://select/groups/6644344/items/ABC123"
|
||||
)
|
||||
assert (
|
||||
_select_link({"key": "ABC123", "group_id": None})
|
||||
== "zotero://select/library/items/ABC123"
|
||||
)
|
||||
assert (
|
||||
_select_link({"key": "ABC123"})
|
||||
== "zotero://select/library/items/ABC123"
|
||||
)
|
||||
|
||||
|
||||
def test_epub_tolerates_extraction_artifacts():
|
||||
from kindle_zotero_importer.epub_position import find_epub_cfi, TextNode
|
||||
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="idolatry o f the machine", start=0, end=24)]
|
||||
quote = "idolatry of the machine and which"
|
||||
# quote is longer than node, so exact fails; use full haystack variant
|
||||
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="nothing other than idolatry o f the machine and which", start=0, end=53)]
|
||||
result = find_epub_cfi(nodes, quote)
|
||||
assert result is not None
|
||||
assert result["selector"]["value"].startswith("epubcfi(")
|
||||
|
|
|
|||
|
|
@ -6,7 +6,11 @@ from kindle_zotero_importer import cli
|
|||
from kindle_zotero_importer.cli import (
|
||||
_add_previous_imported_comments,
|
||||
_changed_note_clipping_ids,
|
||||
_changed_override_titles,
|
||||
_clipping_ids_for_titles,
|
||||
_deferred_clipping_ids,
|
||||
_deletion_ids,
|
||||
_discarded_clipping_ids,
|
||||
_merge_annotation_history,
|
||||
_merge_positioned_history,
|
||||
_unresolved_clipping_ids,
|
||||
|
|
@ -78,6 +82,114 @@ def test_deletions_include_removed_and_previous_failures():
|
|||
) == {"removed", "retry"}
|
||||
|
||||
|
||||
def test_deferred_clippings_are_reprocessed_while_they_remain_in_source():
|
||||
previous = {"deferred_clipping_ids": ["retry", "removed"]}
|
||||
|
||||
assert _deferred_clipping_ids(previous, {"retry", "current"}) == {"retry"}
|
||||
|
||||
|
||||
def test_discarded_clips_never_requeue_and_skips_are_not_unresolved():
|
||||
previous = {"discarded_clipping_ids": ["gone", "removed"]}
|
||||
assert _discarded_clipping_ids(previous, {"gone", "current"}) == {"gone"}
|
||||
|
||||
plan = {
|
||||
"items": [
|
||||
{"clipping": {"id": "note"}, "status": "pdf-position-skipped-non-highlight"},
|
||||
{"clipping": {"id": "fail"}, "status": "pdf-text-not-found"},
|
||||
{"clipping": {"id": "done"}, "status": "positioned"},
|
||||
]
|
||||
}
|
||||
assert _unresolved_clipping_ids(plan, {"note", "fail", "done"}) == {"fail"}
|
||||
|
||||
|
||||
def test_alternate_attachment_retry_prefers_pdf_when_epub_misses():
|
||||
from kindle_zotero_importer.cli import _reattempt_failed_with_alternate_attachments
|
||||
from kindle_zotero_importer.epub_position import add_epub_positions
|
||||
from kindle_zotero_importer.import_plan import build_import_plan
|
||||
from kindle_zotero_importer.pdf_position import add_pdf_positions
|
||||
|
||||
fixtures = Path(__file__).parent / "fixtures" / "windows-smoke"
|
||||
epub_path = str(fixtures / "kanzi-dummy.epub")
|
||||
pdf_path = str(fixtures / "kanzi-dummy.pdf")
|
||||
quote = "This is a test highlight long enough to import for PDF."
|
||||
clippings_payload = {
|
||||
"format": "kindle-zotero-importer.clippings.v1",
|
||||
"clippings": [
|
||||
{
|
||||
"id": "q1",
|
||||
"title": "Dummy Book",
|
||||
"kind": "highlight",
|
||||
"text": quote,
|
||||
"page": None,
|
||||
"location": "100",
|
||||
}
|
||||
],
|
||||
}
|
||||
zotero_index = {
|
||||
"format": "index",
|
||||
"items": [
|
||||
{
|
||||
"item_id": 1,
|
||||
"key": "ABC12345",
|
||||
"title": "Dummy Book",
|
||||
"fields": {},
|
||||
"attachments": [
|
||||
{
|
||||
"item_id": 11,
|
||||
"key": "EPUBKEY1",
|
||||
"title": "epub",
|
||||
"content_type": "application/epub+zip",
|
||||
"path": epub_path,
|
||||
"resolved_path": epub_path,
|
||||
},
|
||||
{
|
||||
"item_id": 12,
|
||||
"key": "PDFKEY123",
|
||||
"title": "pdf",
|
||||
"content_type": "application/pdf",
|
||||
"path": pdf_path,
|
||||
"resolved_path": pdf_path,
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
match_report = {
|
||||
"format": "matches",
|
||||
"matches": [
|
||||
{
|
||||
"clipping_title": "Dummy Book",
|
||||
"status": "matched",
|
||||
"candidates": [{"item_id": 1}],
|
||||
}
|
||||
],
|
||||
}
|
||||
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||
assert plan["items"][0]["zotero"]["attachment"]["content_type"] == "application/epub+zip"
|
||||
positioned = add_pdf_positions(add_epub_positions(plan))
|
||||
assert positioned["items"][0]["status"] == "positioned"
|
||||
assert "partial-text-match" in positioned["items"][0]["problems"]
|
||||
rescued = _reattempt_failed_with_alternate_attachments(positioned)
|
||||
assert rescued["items"][0]["status"] == "positioned"
|
||||
assert "positioned-via-alternate-attachment" in rescued["items"][0]["problems"]
|
||||
assert "partial-text-match" not in rescued["items"][0]["problems"]
|
||||
|
||||
|
||||
def test_changed_override_requeues_every_clipping_for_exact_title():
|
||||
previous = {"Shared title": {"citation_key": "old"}}
|
||||
current = {"Shared title": {"citation_key": "new"}}
|
||||
clippings = [
|
||||
{"id": "first", "title": "Shared title"},
|
||||
{"id": "second", "title": "Shared title"},
|
||||
{"id": "third", "title": "Different title"},
|
||||
]
|
||||
|
||||
changed_titles = _changed_override_titles(previous, current)
|
||||
|
||||
assert changed_titles == {"Shared title"}
|
||||
assert _clipping_ids_for_titles(clippings, changed_titles) == {"first", "second"}
|
||||
|
||||
|
||||
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
|
||||
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
|
||||
previous = {
|
||||
|
|
@ -240,6 +352,6 @@ def test_noop_incremental_writes_complete_artifacts():
|
|||
assert result == 0
|
||||
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
|
||||
assert final_plan["annotations"] == []
|
||||
assert final_plan["all_annotations"] == [annotation]
|
||||
assert final_plan["all_annotations"] == [{**annotation, "related_items": []}]
|
||||
assert final_plan["delta_annotation_count"] == 0
|
||||
assert (workdir / "import-plan.epub.json").exists()
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ const vm = require('vm');
|
|||
|
||||
const bootstrapContext = {};
|
||||
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
|
||||
const plugin = bootstrapContext.KindleZoteroImporter;
|
||||
const plugin = bootstrapContext.Kanzi;
|
||||
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
|
||||
|
||||
assert.equal(
|
||||
|
|
@ -96,6 +96,47 @@ const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
|
|||
const managerContext = { TextEncoder };
|
||||
vm.runInNewContext(scripts[0][1], managerContext);
|
||||
const manager = managerContext.KindleManager;
|
||||
assert.equal(plugin.countClippingsForTitle({ clippings: [
|
||||
{ title: 'Shared title' },
|
||||
{ title: 'Shared title' },
|
||||
{ title: 'Different title' },
|
||||
] }, 'Shared title'), 2);
|
||||
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||
{ id: 'n1', title: 'Shared title', kind: 'note', text: 'note' },
|
||||
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||
] },
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Shared title',
|
||||
clipping_count: 3,
|
||||
status: 'matched',
|
||||
candidates: [],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [{
|
||||
clipping: { id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||
status: 'pdf-text-not-found',
|
||||
problems: ['not found'],
|
||||
}] },
|
||||
finalPlan: { all_annotations: [] },
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const conflictRows = manager.conflictRows();
|
||||
assert.equal(conflictRows.length, 1);
|
||||
assert.equal(conflictRows[0].count, 3);
|
||||
assert.equal(conflictRows[0].clippings.length, 3);
|
||||
assert.equal(conflictRows[0].highlights, 2);
|
||||
assert.equal(conflictRows[0].notes, 1);
|
||||
|
||||
manager.sortState.conflicts = { column: 'count', direction: 'desc' };
|
||||
const sortedRows = manager.sortRows(
|
||||
[{ title: 'small', count: 2 }, { title: 'large', count: 20 }, { title: 'middle', count: 10 }],
|
||||
'conflicts',
|
||||
{ count: row => row.count }
|
||||
);
|
||||
assert.deepEqual(Array.from(sortedRows, row => row.title), ['large', 'middle', 'small']);
|
||||
manager.integratedPage = 1;
|
||||
manager.integratedPageSize = 50;
|
||||
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
|
||||
|
|
@ -143,6 +184,34 @@ async function testRuntimeExtraction() {
|
|||
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
|
||||
}
|
||||
|
||||
async function testPythonPathRecovery() {
|
||||
bootstrapContext.Services = { appinfo: { OS: 'Darwin' } };
|
||||
bootstrapContext.IOUtils = {
|
||||
exists: async path => path === '/opt/homebrew/opt/python@3.13/bin/python3.13',
|
||||
};
|
||||
assert.equal(
|
||||
await plugin.resolvePythonPath('/opt/homebrew/bin/python3'),
|
||||
'/opt/homebrew/opt/python@3.13/bin/python3.13'
|
||||
);
|
||||
bootstrapContext.IOUtils.exists = async path => path === '/custom/python3';
|
||||
assert.equal(await plugin.resolvePythonPath('/custom/python3'), '/custom/python3');
|
||||
|
||||
bootstrapContext.Services.appinfo.OS = 'WINNT';
|
||||
const windowsEnvironment = {
|
||||
LOCALAPPDATA: 'C:\\Users\\test\\AppData\\Local',
|
||||
ProgramFiles: 'C:\\Program Files',
|
||||
'ProgramFiles(x86)': 'C:\\Program Files (x86)',
|
||||
USERPROFILE: 'C:\\Users\\test',
|
||||
SystemDrive: 'C:',
|
||||
PATH: 'C:\\Tools\\Python;C:\\Windows\\System32',
|
||||
};
|
||||
plugin.getEnvironmentVariable = name => windowsEnvironment[name] || '';
|
||||
const expected = 'C:\\Users\\test\\AppData\\Local\\Programs\\Python\\Python313\\python.exe';
|
||||
bootstrapContext.IOUtils.exists = async path => path === expected;
|
||||
assert.equal(await plugin.resolvePythonPath('python'), expected);
|
||||
assert.ok(plugin.getWindowsPythonCandidates('python').includes('C:\\Tools\\Python\\python.exe'));
|
||||
}
|
||||
|
||||
async function testMaintenanceSweep() {
|
||||
const annotation = (id, text, end, comment = '') => ({
|
||||
id,
|
||||
|
|
@ -315,7 +384,317 @@ async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
|
|||
assert.equal(results.failed[0].clipping_id, 'replacement');
|
||||
}
|
||||
|
||||
async function testCitekeyRelationsAreBidirectionalAndAdditive() {
|
||||
const originalZotero = bootstrapContext.Zotero;
|
||||
const originalCollect = plugin.collectImportedAnnotations;
|
||||
const originalMaintain = plugin.maintainImportedAnnotations;
|
||||
const item = (id, key) => ({
|
||||
id,
|
||||
key,
|
||||
related: new Set(),
|
||||
saved: 0,
|
||||
addRelatedItem(other) {
|
||||
if (this.related.has(other.key)) return false;
|
||||
this.related.add(other.key);
|
||||
return true;
|
||||
},
|
||||
async save(options) {
|
||||
assert.equal(options.skipDateModifiedUpdate, true);
|
||||
this.saved += 1;
|
||||
},
|
||||
});
|
||||
const source = item(1, 'SOURCE01');
|
||||
const target = item(2, 'TARGET02');
|
||||
bootstrapContext.Zotero = {
|
||||
Items: { get: id => id === 1 ? source : id === 2 ? target : id === 11 ? attachment : null },
|
||||
DB: { executeTransaction: async callback => callback() },
|
||||
logError: () => {},
|
||||
};
|
||||
const entries = [
|
||||
{
|
||||
clipping_id: 'history-1',
|
||||
parent_item_id: 1,
|
||||
attachment_item_id: 11,
|
||||
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
|
||||
},
|
||||
{
|
||||
parent_item_id: 1,
|
||||
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
|
||||
},
|
||||
];
|
||||
const existingAnnotation = {
|
||||
annotationComment: 'an old note',
|
||||
annotationTags: [{ name: 'kindle-id:history-1' }],
|
||||
saved: 0,
|
||||
saveTx: async function () { this.saved += 1; },
|
||||
};
|
||||
const attachment = { getAnnotations: () => [existingAnnotation] };
|
||||
const results = {
|
||||
relationsAdded: 0,
|
||||
relationsSkipped: 0,
|
||||
relationsPlanned: 0,
|
||||
relationFailures: [],
|
||||
};
|
||||
try {
|
||||
plugin.collectImportedAnnotations = async () => [];
|
||||
plugin.maintainImportedAnnotations = async () => {};
|
||||
const writerResults = await plugin.writeAnnotations({
|
||||
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||
annotations: [],
|
||||
all_annotations: entries,
|
||||
deletions: [],
|
||||
}, false);
|
||||
assert.deepEqual(Array.from(source.related), ['TARGET02']);
|
||||
assert.deepEqual(Array.from(target.related), ['SOURCE01']);
|
||||
assert.equal(source.saved, 1);
|
||||
assert.equal(target.saved, 1);
|
||||
assert.equal(writerResults.relationsAdded, 1);
|
||||
assert.equal(writerResults.relationFailures.length, 0);
|
||||
assert.equal(writerResults.commentLinksAdded, 1);
|
||||
assert.ok(existingAnnotation.annotationComment.includes(
|
||||
'Related: @target2020 zotero://select/library/items/TARGET02'
|
||||
));
|
||||
assert.equal(existingAnnotation.saved, 1);
|
||||
|
||||
await plugin.addRelatedItemRelations(entries, false, results);
|
||||
assert.equal(source.saved, 1);
|
||||
assert.equal(target.saved, 1);
|
||||
assert.equal(results.relationsSkipped, 1);
|
||||
} finally {
|
||||
bootstrapContext.Zotero = originalZotero;
|
||||
plugin.collectImportedAnnotations = originalCollect;
|
||||
plugin.maintainImportedAnnotations = originalMaintain;
|
||||
}
|
||||
}
|
||||
|
||||
function testPendingReviewGrouping() {
|
||||
const groups = manager.groupPendingAnnotations([
|
||||
{ clipping_id: 'a1', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'spice', pageLabel: '12', color: '#ffd400' } },
|
||||
{ clipping_id: 'a2', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'worm', pageLabel: '', color: '#ff6666' } },
|
||||
{ clipping_id: 'b1', clipping_title: 'Unknown Book', citation_key: '', annotation: { text: 'mystery', pageLabel: '', color: '#ffd400' } },
|
||||
]);
|
||||
assert.equal(groups.length, 2);
|
||||
assert.equal(groups[0].items.length, 2);
|
||||
assert.equal(groups[0].citationKey, 'herbert2020');
|
||||
assert.equal(groups[1].items.length, 1);
|
||||
assert.deepEqual(manager.groupPendingAnnotations([]), []);
|
||||
}
|
||||
|
||||
function testFilterPlanForSelection() {
|
||||
const plan = {
|
||||
annotations: [{ clipping_id: 'keep' }, { clipping_id: 'skip' }],
|
||||
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'keep' }, { clipping_id: 'skip' }],
|
||||
};
|
||||
const previous = { deferred_clipping_ids: ['older'] };
|
||||
const filtered = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep'], previous);
|
||||
assert.deepEqual(Array.from(filtered.annotations, a => a.clipping_id), ['keep']);
|
||||
assert.deepEqual(Array.from(filtered.all_annotations, a => a.clipping_id), ['old', 'keep']);
|
||||
assert.deepEqual(Array.from(filtered.deferred_clipping_ids), ['older', 'skip']);
|
||||
// Original plan is not mutated.
|
||||
assert.equal(plan.annotations.length, 2);
|
||||
const all = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep', 'skip'], previous);
|
||||
assert.equal(all.annotations.length, 2);
|
||||
assert.equal(all.all_annotations.length, 3);
|
||||
assert.deepEqual(Array.from(all.deferred_clipping_ids), ['older']);
|
||||
|
||||
const priorAnnotation = { clipping_id: 'skip', annotation: { text: 'existing' } };
|
||||
const preserved = plugin.filterPlanForSelection(
|
||||
plan,
|
||||
['keep', 'skip'],
|
||||
['keep'],
|
||||
{ all_annotations: [priorAnnotation] }
|
||||
);
|
||||
assert.equal(preserved.all_annotations.find(a => a.clipping_id === 'skip'), priorAnnotation);
|
||||
|
||||
const deferred = plugin.deferPendingPlan(
|
||||
{ all_annotations: [{ clipping_id: 'old' }], deferred_clipping_ids: ['earlier'] },
|
||||
plan,
|
||||
['keep', 'skip']
|
||||
);
|
||||
assert.deepEqual(Array.from(deferred.deferred_clipping_ids), ['earlier', 'keep', 'skip']);
|
||||
assert.deepEqual(Array.from(deferred.all_annotations, a => a.clipping_id), ['old']);
|
||||
}
|
||||
|
||||
function testTitleCountsAreAuthoritative() {
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||
{ id: 'h3', title: 'Shared title', kind: 'highlight', text: 'third' },
|
||||
] },
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Shared title',
|
||||
clipping_count: 99,
|
||||
status: 'ambiguous',
|
||||
candidates: [{ citation_key: 'shared2020', key: '', item_id: 1, title: 'Shared', score: 0.9 }],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [{
|
||||
clipping_title: 'Shared title',
|
||||
review: { status: 'review', clipping_count: 1 },
|
||||
candidates: [],
|
||||
}] },
|
||||
positionedPlan: { items: [{
|
||||
clipping: { id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||
status: 'pdf-text-not-found',
|
||||
problems: ['not found'],
|
||||
}] },
|
||||
finalPlan: { all_annotations: [] },
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const rows = manager.conflictRows();
|
||||
assert.equal(rows.length, 1);
|
||||
// Source has 3 clippings; misleading match (99) and override (1) counts must not win.
|
||||
assert.equal(rows[0].count, 3);
|
||||
assert.equal(rows[0].clippings.length, 3);
|
||||
// Only 1 of 3 has position problems.
|
||||
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 3 });
|
||||
|
||||
// Without clippings.json, fall back to conflict counts.
|
||||
manager.data = {
|
||||
clippings: null,
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Lonely title',
|
||||
clipping_count: 5,
|
||||
status: 'ambiguous',
|
||||
candidates: [],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [] },
|
||||
finalPlan: { all_annotations: [] },
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const fallback = manager.conflictRows();
|
||||
assert.equal(fallback.length, 1);
|
||||
assert.equal(fallback[0].count, 5);
|
||||
}
|
||||
|
||||
function testMappedTargetAndDiscardedClips() {
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' },
|
||||
{ id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' },
|
||||
] },
|
||||
matches: { matches: [{
|
||||
clipping_title: 'Mapped title',
|
||||
clipping_count: 2,
|
||||
status: 'matched',
|
||||
candidates: [{ citation_key: 'mapped2020', key: '', item_id: 7, title: 'Mapped', score: 1 }],
|
||||
}] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [
|
||||
{ clipping: { id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' }, status: 'pdf-text-not-found', problems: ['pdf-text-not-found'] },
|
||||
{ clipping: { id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' }, status: 'pdf-position-skipped-non-highlight', problems: ['pdf-position-skipped-non-highlight'] },
|
||||
] },
|
||||
finalPlan: { all_annotations: [], discarded_clipping_ids: [] },
|
||||
persistentOverrides: { overrides: [{ clipping_title: 'Mapped title', resolution: { citation_key: 'mapped2020' } }] },
|
||||
};
|
||||
let rows = manager.conflictRows();
|
||||
assert.equal(rows.length, 1);
|
||||
assert.equal(rows[0].mappedTarget, 'mapped2020');
|
||||
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 2, total: 2 });
|
||||
|
||||
manager.data.finalPlan.discarded_clipping_ids = ['n1'];
|
||||
rows = manager.conflictRows();
|
||||
assert.equal(rows.length, 1);
|
||||
assert.equal(rows[0].count, 1);
|
||||
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 1 });
|
||||
}
|
||||
|
||||
async function testDiscardClipPersists() {
|
||||
const originalZotero = bootstrapContext.Zotero;
|
||||
let written = null;
|
||||
const plan = { discarded_clipping_ids: [], deferred_clipping_ids: ['x1'] };
|
||||
bootstrapContext.Zotero = {
|
||||
File: {
|
||||
getContentsAsync: async () => JSON.stringify(plan),
|
||||
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
|
||||
},
|
||||
};
|
||||
const originalArtifactPath = plugin.artifactPath;
|
||||
plugin.artifactPath = rel => `/proj/${rel}`;
|
||||
try {
|
||||
await plugin.discardClip('x1');
|
||||
assert.deepEqual(written.discarded_clipping_ids, ['x1']);
|
||||
assert.deepEqual(written.deferred_clipping_ids, []);
|
||||
} finally {
|
||||
plugin.artifactPath = originalArtifactPath;
|
||||
bootstrapContext.Zotero = originalZotero;
|
||||
}
|
||||
}
|
||||
|
||||
function testTitleOverviewAndSorting() {
|
||||
manager.data = {
|
||||
clippings: { clippings: [
|
||||
{ id: 'a1', title: 'Alpha', kind: 'highlight', text: 'one' },
|
||||
{ id: 'a2', title: 'Alpha', kind: 'highlight', text: 'two' },
|
||||
{ id: 'b1', title: 'Beta', kind: 'highlight', text: 'three' },
|
||||
] },
|
||||
matches: { matches: [
|
||||
{ clipping_title: 'Alpha', clipping_count: 2, status: 'matched', candidates: [{ citation_key: 'alpha2020' }] },
|
||||
{ clipping_title: 'Beta', clipping_count: 1, status: 'unmatched', candidates: [] },
|
||||
] },
|
||||
generatedOverrides: { overrides: [] },
|
||||
positionedPlan: { items: [] },
|
||||
finalPlan: {
|
||||
all_annotations: [{ clipping_id: 'a1', clipping_title: 'Alpha' }],
|
||||
deferred_clipping_ids: ['b1'],
|
||||
},
|
||||
persistentOverrides: { overrides: [] },
|
||||
};
|
||||
const overview = manager.titleOverviewRows();
|
||||
assert.equal(overview.length, 2);
|
||||
const alpha = overview.find(r => r.title === 'Alpha');
|
||||
const beta = overview.find(r => r.title === 'Beta');
|
||||
assert.equal(alpha.integrated, 1);
|
||||
assert.equal(beta.pending, 1);
|
||||
assert.equal(beta.status, 'pending review');
|
||||
|
||||
manager.sortState.titles = { column: 'title', direction: 'desc' };
|
||||
const sorted = manager.sortRows(overview, 'titles', { title: row => row.title });
|
||||
assert.deepEqual(Array.from(sorted, row => row.title), ['Beta', 'Alpha']);
|
||||
}
|
||||
|
||||
async function testSaveOverrideUsesSourceCount() {
|
||||
const originalRead = plugin.readJsonArtifact;
|
||||
const originalArtifactPath = plugin.artifactPath;
|
||||
const originalZotero = bootstrapContext.Zotero;
|
||||
let written = null;
|
||||
plugin.artifactPath = rel => `/proj/${rel}`;
|
||||
plugin.readJsonArtifact = async rel => rel === 'clippings.json'
|
||||
? { clippings: [{ title: 'Shared title' }, { title: 'Shared title' }, { title: 'Shared title' }] }
|
||||
: null;
|
||||
bootstrapContext.Zotero = {
|
||||
File: {
|
||||
getContentsAsync: async () => JSON.stringify({ format: 'kindle-zotero-importer.match-overrides.v1', overrides: [] }),
|
||||
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
|
||||
},
|
||||
};
|
||||
try {
|
||||
const entry = await plugin.saveOverride({
|
||||
clipping_title: 'Shared title',
|
||||
clipping_count: 1,
|
||||
resolution: { citation_key: 'shared2020' },
|
||||
review: { status: 'pdf-text-not-found', clipping_count: 1 },
|
||||
});
|
||||
assert.equal(entry.review.clipping_count, 3);
|
||||
assert.equal(written.overrides[0].review.clipping_count, 3);
|
||||
} finally {
|
||||
plugin.readJsonArtifact = originalRead;
|
||||
plugin.artifactPath = originalArtifactPath;
|
||||
bootstrapContext.Zotero = originalZotero;
|
||||
}
|
||||
}
|
||||
|
||||
testTitleCountsAreAuthoritative();
|
||||
testTitleOverviewAndSorting();
|
||||
testMappedTargetAndDiscardedClips();
|
||||
testPendingReviewGrouping();
|
||||
testFilterPlanForSelection();
|
||||
|
||||
testRuntimeExtraction()
|
||||
.then(testPythonPathRecovery)
|
||||
.then(testSaveOverrideUsesSourceCount)
|
||||
.then(testDiscardClipPersists)
|
||||
.then(testMaintenanceSweep)
|
||||
.then(testWriterReconciliation)
|
||||
.then(testDeletionFailureIsRetried)
|
||||
|
|
@ -323,6 +702,7 @@ testRuntimeExtraction()
|
|||
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
|
||||
.then(testEnumerationFailurePersistsAllDeletions)
|
||||
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
|
||||
.then(testCitekeyRelationsAreBidirectionalAndAdditive)
|
||||
.then(() => {
|
||||
console.log('manager and annotation logic ok');
|
||||
}).catch(error => {
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue