Kanzi 0.7.1: safer imports and resilient positioning
This commit is contained in:
parent
495f50989d
commit
aa3fc5ebc5
19 changed files with 2823 additions and 159 deletions
39
README.md
39
README.md
|
|
@ -17,16 +17,39 @@ Import Kindle `My Clippings.txt` highlights into Zotero as native annotations, d
|
||||||
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
|
2. In Zotero: `Tools` → `Plugins` → `gear` → `Install Plugin From File…` → pick the `.xpi` → restart Zotero.
|
||||||
3. `Tools` → `Kanzi…` to open the manager.
|
3. `Tools` → `Kanzi…` to open the manager.
|
||||||
|
|
||||||
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. macOS and Linux are tested. Windows requires setting the full Python executable path in `Settings` and installing the PDF tools separately.
|
Works with Zotero 7 to 10. Kanzi packages its importer source, but requires Python 3.11 or newer on the host. EPUB imports need no external positioning tools. PDF imports additionally require Poppler (`pdftotext`, `pdftohtml`, `pdfinfo`); `qpdf` is recommended for encrypted PDFs. Kanzi detects standard Python installations automatically on macOS, Linux and Windows.
|
||||||
|
|
||||||
|
## Windows
|
||||||
|
|
||||||
|
Open PowerShell and install the host tools:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
winget install --id Python.Python.3.13 --exact
|
||||||
|
winget install --id oschwartz10612.Poppler --exact
|
||||||
|
winget install --id QPDF.QPDF --exact # Optional: handles copy-protected PDFs
|
||||||
|
```
|
||||||
|
|
||||||
|
Fully quit and reopen Zotero after installation so it receives the updated `PATH`, then install `kanzi.xpi` and open `Tools` → `Kanzi…`. Kanzi checks standard per-user and system Python locations as well as `PATH`; you should not need to enter an executable path manually. EPUB-only imports require Python but not Poppler.
|
||||||
|
|
||||||
|
To verify the installation in a new PowerShell window:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python --version
|
||||||
|
pdftotext -v
|
||||||
|
pdftohtml -v
|
||||||
|
pdfinfo -v
|
||||||
|
qpdf --version # Optional
|
||||||
|
```
|
||||||
|
|
||||||
# Use
|
# Use
|
||||||
|
|
||||||
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
|
1. **Choose your file.** Click `Choose My Clippings.txt` and pick your *cumulative* `My Clippings.txt` from Kindle (`Documents/My Clippings.txt`). Keep the window open. You will see live progress.
|
||||||
2. **Check Integrated.** After the run, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
2. **Review before writing.** New highlights are grouped by title and selected by default. Untick individual highlights or whole titles, then press `Integrate selected`. Skipped highlights return on the next run. `Cancel` and closing the manager write nothing to Zotero and do not mark pending highlights as integrated.
|
||||||
3. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows it with up to three suggestions `citekey · title (score%)`. Click `Use` on the right one. Or type any citation key, Zotero key or item ID under `Use Custom`. Or `Ignore Title` to skip it forever. If you map `chabot2013` for one `Simondon` variant, it will offer to apply the same mapping to the other variants with that candidate.
|
3. **Check Integrated.** After integration, `Integrated` shows what was imported. Columns are `Highlight Text`, `Citekey`, `Added On`, `Integrated` and `Page`. Use the filter to find anything.
|
||||||
4. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new or changed highlights are repositioned. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
4. **Fix Conflicts.** If a Kindle title did not match a Zotero item, `Conflicts` shows one expandable row for that title with up to three suggestions `citekey · title (score%)`. Expand it to inspect individual highlights and positioning problems. Click `Use` on the right candidate, enter a citation key, Zotero key or item ID under `Use Custom`, or choose `Ignore Title`. Tables can be sorted and resized.
|
||||||
5. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
5. **Re-import.** After you have fixed one or more titles, click `Re-import with saved overrides`. It reuses the last file, or asks for it. Only new, changed, deferred or removed highlights are processed. Already integrated ones are skipped, so the second run is fast. Use `Retry unresolved attachments and positions` for prior failures without rebuilding successful annotations. Tick `Full re-import from scratch` only if you want to rebuild everything.
|
||||||
6. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
6. **Mappings.** See all titles you have approved or ignored, newest first, with date. Use `Delete` to send any entry back to `Conflicts` for rematching.
|
||||||
|
7. **Settings and Artifacts.** Change `Python` or `Zotero DB` paths and `Save Settings`. Or use `Open` or `Reveal` for any generated file, such as `mismatch-review.md` or `match-overrides.json`.
|
||||||
|
|
||||||
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
|
Your choices are saved in `match-overrides.json` in Kanzi's private runtime folder inside your Zotero profile. Back it up or delete an entry to undo a mapping; it is excluded from Git because it contains your document titles.
|
||||||
|
|
||||||
|
|
@ -62,11 +85,11 @@ Both `[o]` and `[orange]` work, case is not sensitive, and space after the brack
|
||||||
|
|
||||||
`Tools` → `Kanzi…` opens the manager.
|
`Tools` → `Kanzi…` opens the manager.
|
||||||
|
|
||||||
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox.
|
**Summary.** Shows `clippings`, `unique titles`, `final annotations` and `conflicts`. Also `Match Statuses` and `Plan Statuses`. The run panel shows `Choose My Clippings.txt`, progress, stage and detail, and the `Full re-import` checkbox. After positioning, the pending review lets you select exactly what is written to Zotero.
|
||||||
|
|
||||||
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
|
**Integrated.** All annotations remembered in `import-plan.final.json`. Columns are `Kindle Title`, `Citekey`, `Highlight Text`, `Added On`, `Integrated` and `Page` with colour swatch. Filter at the top right; results are resizable and paginated.
|
||||||
|
|
||||||
**Conflicts.** Unresolved titles. Type is `Title match`, `Override suggestion` or `Attachment/position`. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. If you use `chabot2013` for one Simondon variant, it offers to apply to other variants with the same candidate.
|
**Conflicts.** One row per unresolved Kindle title, combining title-match and attachment/position statuses. Expand a row to inspect its highlights and positioning problems. `Candidates` shows `N suggestion(s) — top: citekey (score%)`. Each candidate has a `Use` button. Below that is `Use Custom` and `Ignore Title`. Saving shows the `Re-import` bar. Columns are sortable and resizable.
|
||||||
|
|
||||||
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
|
**Mappings.** Persistent `match-overrides.json`. Columns are `Kindle Title`, `Resolution`, `Status`, `Count` and `Updated`. Use `Delete` to send it back to `Conflicts`. Filterable.
|
||||||
|
|
||||||
|
|
|
||||||
514
plugin/bootstrap.js
vendored
514
plugin/bootstrap.js
vendored
|
|
@ -1,4 +1,4 @@
|
||||||
var KindleZoteroImporter = {
|
var Kanzi = {
|
||||||
id: "kindle-zotero-importer@utkubilen.de",
|
id: "kindle-zotero-importer@utkubilen.de",
|
||||||
menuItems: [],
|
menuItems: [],
|
||||||
runtimeProjectDir: "",
|
runtimeProjectDir: "",
|
||||||
|
|
@ -16,7 +16,7 @@ var KindleZoteroImporter = {
|
||||||
);
|
);
|
||||||
this.chromeHandle = addonManagerStartup.registerChrome(
|
this.chromeHandle = addonManagerStartup.registerChrome(
|
||||||
Services.io.newURI(this.rootURI + "manifest.json"),
|
Services.io.newURI(this.rootURI + "manifest.json"),
|
||||||
[["content", "kindle-zotero-importer", ""]]
|
[["content", "kanzi", ""]]
|
||||||
);
|
);
|
||||||
await Zotero.initializationPromise;
|
await Zotero.initializationPromise;
|
||||||
await this.ensureRuntime();
|
await this.ensureRuntime();
|
||||||
|
|
@ -81,7 +81,7 @@ var KindleZoteroImporter = {
|
||||||
}
|
}
|
||||||
|
|
||||||
const doc = win.document;
|
const doc = win.document;
|
||||||
if (doc.getElementById("kindle-zotero-importer-manager")) {
|
if (doc.getElementById("kanzi-manager")) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -92,7 +92,7 @@ var KindleZoteroImporter = {
|
||||||
}
|
}
|
||||||
|
|
||||||
const managerItem = doc.createXULElement("menuitem");
|
const managerItem = doc.createXULElement("menuitem");
|
||||||
managerItem.setAttribute("id", "kindle-zotero-importer-manager");
|
managerItem.setAttribute("id", "kanzi-manager");
|
||||||
managerItem.setAttribute("label", "Kanzi...");
|
managerItem.setAttribute("label", "Kanzi...");
|
||||||
managerItem.addEventListener("command", () => this.openManager());
|
managerItem.addEventListener("command", () => this.openManager());
|
||||||
popup.appendChild(managerItem);
|
popup.appendChild(managerItem);
|
||||||
|
|
@ -136,7 +136,14 @@ var KindleZoteroImporter = {
|
||||||
},
|
},
|
||||||
|
|
||||||
getPythonPath() {
|
getPythonPath() {
|
||||||
return this.getPref("pythonPath", "python3");
|
const fallback = this.getDefaultPython();
|
||||||
|
const value = this.getPref("pythonPath", fallback);
|
||||||
|
// Existing Windows installs may have "python3" stored from the old
|
||||||
|
// Unix-only default; that binary name does not exist on Windows.
|
||||||
|
if (this.isWindows() && value === "python3") {
|
||||||
|
return "python";
|
||||||
|
}
|
||||||
|
return value || fallback;
|
||||||
},
|
},
|
||||||
|
|
||||||
getZoteroDbPath() {
|
getZoteroDbPath() {
|
||||||
|
|
@ -149,12 +156,115 @@ var KindleZoteroImporter = {
|
||||||
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
|
return this.getPref("zoteroStorageRoot", dataDir ? PathUtils.join(dataDir, "storage") : "");
|
||||||
},
|
},
|
||||||
|
|
||||||
|
isWindows() {
|
||||||
|
try {
|
||||||
|
return Services.appinfo.OS === "WINNT";
|
||||||
|
} catch (_error) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
getDefaultPython() {
|
||||||
|
return this.isWindows() ? "python" : "python3";
|
||||||
|
},
|
||||||
|
|
||||||
|
getEnvironmentVariable(name) {
|
||||||
|
try {
|
||||||
|
return Cc["@mozilla.org/process/environment;1"]
|
||||||
|
.getService(Ci.nsIEnvironment)
|
||||||
|
.get(name) || "";
|
||||||
|
} catch (_error) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
joinWindowsPath(base, ...parts) {
|
||||||
|
if (!base) {
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
return [String(base).replace(/[\\/]+$/, ""), ...parts]
|
||||||
|
.map((part, index) => index ? String(part).replace(/^[\\/]+|[\\/]+$/g, "") : part)
|
||||||
|
.join("\\");
|
||||||
|
},
|
||||||
|
|
||||||
|
getWindowsPythonCandidates(configuredPath) {
|
||||||
|
const localAppData = this.getEnvironmentVariable("LOCALAPPDATA");
|
||||||
|
const programFiles = this.getEnvironmentVariable("ProgramFiles");
|
||||||
|
const programFilesX86 = this.getEnvironmentVariable("ProgramFiles(x86)");
|
||||||
|
const userProfile = this.getEnvironmentVariable("USERPROFILE");
|
||||||
|
const systemDrive = this.getEnvironmentVariable("SystemDrive") || "C:";
|
||||||
|
const versions = ["314", "313", "312", "311"];
|
||||||
|
const candidates = [configuredPath];
|
||||||
|
for (const version of versions) {
|
||||||
|
candidates.push(
|
||||||
|
this.joinWindowsPath(localAppData, "Programs", "Python", `Python${version}`, "python.exe"),
|
||||||
|
this.joinWindowsPath(programFiles, `Python${version}`, "python.exe"),
|
||||||
|
this.joinWindowsPath(programFilesX86, `Python${version}-32`, "python.exe"),
|
||||||
|
this.joinWindowsPath(systemDrive, `Python${version}`, "python.exe")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
candidates.push(
|
||||||
|
this.joinWindowsPath(localAppData, "Python", "bin", "python.exe"),
|
||||||
|
this.joinWindowsPath(userProfile, "scoop", "apps", "python", "current", "python.exe"),
|
||||||
|
this.joinWindowsPath(userProfile, "miniconda3", "python.exe"),
|
||||||
|
this.joinWindowsPath(userProfile, "anaconda3", "python.exe")
|
||||||
|
);
|
||||||
|
for (const directory of this.getEnvironmentVariable("PATH").split(";")) {
|
||||||
|
const cleanDirectory = directory.trim().replace(/^"|"$/g, "");
|
||||||
|
if (cleanDirectory) {
|
||||||
|
candidates.push(
|
||||||
|
this.joinWindowsPath(cleanDirectory, "python.exe"),
|
||||||
|
this.joinWindowsPath(cleanDirectory, "python3.exe")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return [...new Set(candidates.filter(Boolean))];
|
||||||
|
},
|
||||||
|
|
||||||
|
async resolvePythonPath(configuredPath) {
|
||||||
|
let candidates;
|
||||||
|
if (this.isWindows()) {
|
||||||
|
candidates = this.getWindowsPythonCandidates(configuredPath);
|
||||||
|
} else if (Services.appinfo.OS === "Darwin") {
|
||||||
|
candidates = [configuredPath];
|
||||||
|
candidates.push(
|
||||||
|
"/opt/homebrew/bin/python3",
|
||||||
|
"/opt/homebrew/opt/python@3.14/bin/python3.14",
|
||||||
|
"/opt/homebrew/opt/python@3.13/bin/python3.13",
|
||||||
|
"/opt/homebrew/opt/python@3.12/bin/python3.12",
|
||||||
|
"/opt/homebrew/opt/python@3.11/bin/python3.11",
|
||||||
|
"/usr/local/bin/python3",
|
||||||
|
"/usr/local/opt/python@3.14/bin/python3.14",
|
||||||
|
"/usr/local/opt/python@3.13/bin/python3.13",
|
||||||
|
"/usr/local/opt/python@3.12/bin/python3.12",
|
||||||
|
"/usr/local/opt/python@3.11/bin/python3.11",
|
||||||
|
"/usr/bin/python3"
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
candidates = [configuredPath, "/usr/bin/python3", "/usr/local/bin/python3"];
|
||||||
|
}
|
||||||
|
for (const candidate of candidates) {
|
||||||
|
if (!candidate || (!candidate.includes("/") && !candidate.includes("\\"))) continue;
|
||||||
|
try {
|
||||||
|
if (await IOUtils.exists(candidate)) return candidate;
|
||||||
|
} catch (_error) {}
|
||||||
|
}
|
||||||
|
return configuredPath || this.getDefaultPython();
|
||||||
|
},
|
||||||
|
|
||||||
|
joinPath(...parts) {
|
||||||
|
if (typeof PathUtils !== "undefined" && PathUtils.join) {
|
||||||
|
return PathUtils.join(...parts);
|
||||||
|
}
|
||||||
|
return parts.join("/").replaceAll("//", "/");
|
||||||
|
},
|
||||||
|
|
||||||
getConfigPath() {
|
getConfigPath() {
|
||||||
return this.getProjectDir() + "/plugin-config.json";
|
return this.joinPath(this.getProjectDir(), "plugin-config.json");
|
||||||
},
|
},
|
||||||
|
|
||||||
artifactPath(relativePath) {
|
artifactPath(relativePath) {
|
||||||
return this.getProjectDir() + "/" + relativePath;
|
return this.joinPath(this.getProjectDir(), ...String(relativePath).split("/"));
|
||||||
},
|
},
|
||||||
|
|
||||||
async openArtifact(relativePath) {
|
async openArtifact(relativePath) {
|
||||||
|
|
@ -177,17 +287,6 @@ var KindleZoteroImporter = {
|
||||||
zoteroDbPath: this.getZoteroDbPath(),
|
zoteroDbPath: this.getZoteroDbPath(),
|
||||||
zoteroStorageRoot: this.getZoteroStorageRoot(),
|
zoteroStorageRoot: this.getZoteroStorageRoot(),
|
||||||
};
|
};
|
||||||
if (settings.pythonPath === "python3") {
|
|
||||||
const candidates = Services.appinfo.OS === "Darwin"
|
|
||||||
? ["/opt/homebrew/bin/python3", "/usr/local/bin/python3", "/usr/bin/python3"]
|
|
||||||
: ["/usr/bin/python3", "/usr/local/bin/python3"];
|
|
||||||
for (const candidate of candidates) {
|
|
||||||
if (await IOUtils.exists(candidate)) {
|
|
||||||
settings.pythonPath = candidate;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
try {
|
try {
|
||||||
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
|
const text = await Zotero.File.getContentsAsync(this.getConfigPath());
|
||||||
const parsed = JSON.parse(text);
|
const parsed = JSON.parse(text);
|
||||||
|
|
@ -200,12 +299,13 @@ var KindleZoteroImporter = {
|
||||||
if (parsed.colorMap) settings.colourMap = parsed.colorMap;
|
if (parsed.colorMap) settings.colourMap = parsed.colorMap;
|
||||||
// also check colour-map.json for Python pipeline
|
// also check colour-map.json for Python pipeline
|
||||||
try {
|
try {
|
||||||
const cmapText = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json");
|
const cmapText = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
|
||||||
settings.colourMap = JSON.parse(cmapText);
|
settings.colourMap = JSON.parse(cmapText);
|
||||||
} catch (_e2) {}
|
} catch (_e2) {}
|
||||||
} catch (_error) {
|
} catch (_error) {
|
||||||
// The config file is optional; defaults/preferences are enough for first run.
|
// The config file is optional; defaults/preferences are enough for first run.
|
||||||
}
|
}
|
||||||
|
settings.pythonPath = await this.resolvePythonPath(settings.pythonPath);
|
||||||
return settings;
|
return settings;
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|
@ -267,8 +367,8 @@ var KindleZoteroImporter = {
|
||||||
const data = await this.loadManagerData();
|
const data = await this.loadManagerData();
|
||||||
const io = { plugin: this, settings, data };
|
const io = { plugin: this, settings, data };
|
||||||
this.managerWindow = this.getMainWindow().openDialog(
|
this.managerWindow = this.getMainWindow().openDialog(
|
||||||
"chrome://kindle-zotero-importer/content/manager.html",
|
"chrome://kanzi/content/manager.html",
|
||||||
"kindle-zotero-importer-manager-window",
|
"kanzi-manager-window",
|
||||||
"chrome,centerscreen,resizable",
|
"chrome,centerscreen,resizable",
|
||||||
io
|
io
|
||||||
);
|
);
|
||||||
|
|
@ -276,6 +376,7 @@ var KindleZoteroImporter = {
|
||||||
|
|
||||||
async loadManagerData() {
|
async loadManagerData() {
|
||||||
return {
|
return {
|
||||||
|
clippings: await this.readJsonArtifact("clippings.json"),
|
||||||
matches: await this.readJsonArtifact("matches.json"),
|
matches: await this.readJsonArtifact("matches.json"),
|
||||||
generatedOverrides: await this.readJsonArtifact("match-overrides.generated.json"),
|
generatedOverrides: await this.readJsonArtifact("match-overrides.generated.json"),
|
||||||
positionedPlan: await this.readJsonArtifact("import-plan.positioned.json"),
|
positionedPlan: await this.readJsonArtifact("import-plan.positioned.json"),
|
||||||
|
|
@ -305,11 +406,17 @@ var KindleZoteroImporter = {
|
||||||
const index = payload.overrides.findIndex(
|
const index = payload.overrides.findIndex(
|
||||||
(entry) => entry.clipping_title === resolution.clipping_title
|
(entry) => entry.clipping_title === resolution.clipping_title
|
||||||
);
|
);
|
||||||
|
const clippings = await this.readJsonArtifact("clippings.json");
|
||||||
|
const sourceCount = this.countClippingsForTitle(clippings, resolution.clipping_title);
|
||||||
|
const clippingCount = sourceCount || resolution.clipping_count || resolution.review?.clipping_count || 0;
|
||||||
const nowIso = new Date().toISOString();
|
const nowIso = new Date().toISOString();
|
||||||
const entry = {
|
const entry = {
|
||||||
clipping_title: resolution.clipping_title,
|
clipping_title: resolution.clipping_title,
|
||||||
resolution: resolution.resolution,
|
resolution: resolution.resolution,
|
||||||
review: resolution.review || { status: "manager", clipping_count: resolution.clipping_count || 0 },
|
review: {
|
||||||
|
...(resolution.review || { status: "manager" }),
|
||||||
|
clipping_count: clippingCount,
|
||||||
|
},
|
||||||
created_at: nowIso,
|
created_at: nowIso,
|
||||||
updated_at: nowIso,
|
updated_at: nowIso,
|
||||||
};
|
};
|
||||||
|
|
@ -329,6 +436,13 @@ var KindleZoteroImporter = {
|
||||||
return entry;
|
return entry;
|
||||||
},
|
},
|
||||||
|
|
||||||
|
countClippingsForTitle(payload, clippingTitle) {
|
||||||
|
return (payload?.clippings || []).reduce(
|
||||||
|
(count, clipping) => count + (clipping.title === clippingTitle ? 1 : 0),
|
||||||
|
0
|
||||||
|
);
|
||||||
|
},
|
||||||
|
|
||||||
async deleteOverride(clippingTitle) {
|
async deleteOverride(clippingTitle) {
|
||||||
if (!clippingTitle) {
|
if (!clippingTitle) {
|
||||||
throw new Error("Missing clipping title for delete");
|
throw new Error("Missing clipping title for delete");
|
||||||
|
|
@ -346,6 +460,23 @@ var KindleZoteroImporter = {
|
||||||
return true;
|
return true;
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async discardClip(clippingId) {
|
||||||
|
if (!clippingId) {
|
||||||
|
throw new Error("Missing clipping id for discard");
|
||||||
|
}
|
||||||
|
const path = this.artifactPath("import-plan.final.json");
|
||||||
|
const plan = JSON.parse(await Zotero.File.getContentsAsync(path));
|
||||||
|
const discarded = new Set(plan.discarded_clipping_ids || []);
|
||||||
|
discarded.add(clippingId);
|
||||||
|
plan.discarded_clipping_ids = [...discarded].sort();
|
||||||
|
// Also drop from deferred so it is never re-queued.
|
||||||
|
if (Array.isArray(plan.deferred_clipping_ids)) {
|
||||||
|
plan.deferred_clipping_ids = plan.deferred_clipping_ids.filter((id) => id !== clippingId);
|
||||||
|
}
|
||||||
|
await Zotero.File.putContentsAsync(path, JSON.stringify(plan, null, 2) + "\n");
|
||||||
|
return true;
|
||||||
|
},
|
||||||
|
|
||||||
async saveSettingsFromManager(newSettings) {
|
async saveSettingsFromManager(newSettings) {
|
||||||
const toSave = {
|
const toSave = {
|
||||||
projectDir: newSettings.projectDir || this.getProjectDir(),
|
projectDir: newSettings.projectDir || this.getProjectDir(),
|
||||||
|
|
@ -358,7 +489,7 @@ var KindleZoteroImporter = {
|
||||||
// also write colour-map.json for Python pipeline (mirrors plugin-config)
|
// also write colour-map.json for Python pipeline (mirrors plugin-config)
|
||||||
try {
|
try {
|
||||||
await Zotero.File.putContentsAsync(
|
await Zotero.File.putContentsAsync(
|
||||||
this.getProjectDir() + "/colour-map.json",
|
this.joinPath(this.getProjectDir(), "colour-map.json"),
|
||||||
JSON.stringify(newSettings.colourMap, null, 2) + "\n"
|
JSON.stringify(newSettings.colourMap, null, 2) + "\n"
|
||||||
);
|
);
|
||||||
} catch (_e) {}
|
} catch (_e) {}
|
||||||
|
|
@ -376,7 +507,7 @@ var KindleZoteroImporter = {
|
||||||
|
|
||||||
async loadColourMap() {
|
async loadColourMap() {
|
||||||
try {
|
try {
|
||||||
const text = await Zotero.File.getContentsAsync(this.getProjectDir() + "/colour-map.json");
|
const text = await Zotero.File.getContentsAsync(this.joinPath(this.getProjectDir(), "colour-map.json"));
|
||||||
return JSON.parse(text);
|
return JSON.parse(text);
|
||||||
} catch (_e) {}
|
} catch (_e) {}
|
||||||
try {
|
try {
|
||||||
|
|
@ -416,7 +547,7 @@ var KindleZoteroImporter = {
|
||||||
if (result !== fp.returnOK) {
|
if (result !== fp.returnOK) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
return fp.file;
|
return this.filePickerResultToPath(fp.file);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
return this.promptForPath(
|
return this.promptForPath(
|
||||||
|
|
@ -427,12 +558,32 @@ var KindleZoteroImporter = {
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
|
filePickerResultToPath(file) {
|
||||||
|
if (!file) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
if (typeof file === "string") {
|
||||||
|
return file;
|
||||||
|
}
|
||||||
|
// Mozilla FilePicker returns an nsIFile; Python needs a native string path.
|
||||||
|
if (typeof file.path === "string" && file.path) {
|
||||||
|
return file.path;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const str = String(file);
|
||||||
|
if (str && str !== "[object Object]") {
|
||||||
|
return str;
|
||||||
|
}
|
||||||
|
} catch (_error) {}
|
||||||
|
return null;
|
||||||
|
},
|
||||||
|
|
||||||
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
|
async runPipeline(clippingsPath, settings, isFull = false, retryUnresolved = false) {
|
||||||
const python = settings.pythonPath;
|
const python = settings.pythonPath || this.getDefaultPython();
|
||||||
const projectDir = settings.projectDir;
|
const projectDir = settings.projectDir;
|
||||||
const runner = projectDir + "/plugin_runner.py";
|
const runner = this.joinPath(projectDir, "plugin_runner.py");
|
||||||
const summaryOutput = projectDir + "/plugin-summary.json";
|
const summaryOutput = this.joinPath(projectDir, "plugin-summary.json");
|
||||||
const progressOutput = projectDir + "/plugin-progress.json";
|
const progressOutput = this.joinPath(projectDir, "plugin-progress.json");
|
||||||
const args = [
|
const args = [
|
||||||
runner,
|
runner,
|
||||||
"run",
|
"run",
|
||||||
|
|
@ -458,12 +609,34 @@ var KindleZoteroImporter = {
|
||||||
args.push("--retry-unresolved");
|
args.push("--retry-unresolved");
|
||||||
}
|
}
|
||||||
|
|
||||||
await this.exec(python, args);
|
let execResult = null;
|
||||||
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
|
try {
|
||||||
|
execResult = await this.exec(python, args);
|
||||||
|
} catch (error) {
|
||||||
|
const installHint = this.isWindows()
|
||||||
|
? `Install Python 3.11 or newer, fully restart Zotero, and let Kanzi detect python.exe. `
|
||||||
|
: `Install Python 3.11 or newer and check the Python path in Kanzi Settings. `;
|
||||||
|
throw new Error(
|
||||||
|
`Python could not run the Kanzi pipeline (${python}): ${error && error.message ? error.message : error}. ` +
|
||||||
|
installHint
|
||||||
|
);
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
return JSON.parse(await Zotero.File.getContentsAsync(summaryOutput));
|
||||||
|
} catch (error) {
|
||||||
|
const detail = execResult ? ` Python output: ${JSON.stringify(execResult).slice(0, 1000)}.` : "";
|
||||||
|
throw new Error(
|
||||||
|
`Python pipeline did not write ${summaryOutput}.${detail} ` +
|
||||||
|
`Python: ${python}. Runner: ${runner}. ` +
|
||||||
|
`Make sure Python 3 is installed and reachable, then check the project directory is writable.`
|
||||||
|
);
|
||||||
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
async runManagedImport(manager) {
|
async runManagedImport(manager) {
|
||||||
const clippingsPath = await this.pickFile("Select Kindle My Clippings.txt", "*.txt");
|
const clippingsPath = await this.pickFile("Select Kindle My Clippings.txt", "*.txt");
|
||||||
|
// NOTE: writing now happens only after the user reviews the pending
|
||||||
|
// highlights and presses Integrate (see integratePending).
|
||||||
if (!clippingsPath) {
|
if (!clippingsPath) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
@ -474,17 +647,23 @@ var KindleZoteroImporter = {
|
||||||
|
|
||||||
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
|
async runManagedImportWithPath(manager, clippingsPath, isFullOverride, retryUnresolvedOverride) {
|
||||||
try {
|
try {
|
||||||
|
clippingsPath = this.filePickerResultToPath(clippingsPath) || clippingsPath;
|
||||||
if (!clippingsPath) {
|
if (!clippingsPath) {
|
||||||
throw new Error("No clippings file selected");
|
throw new Error("No clippings file selected");
|
||||||
}
|
}
|
||||||
const settings = await this.getRuntimeSettings();
|
const settings = await this.getRuntimeSettings();
|
||||||
const progressPath = settings.projectDir + "/plugin-progress.json";
|
const progressPath = this.joinPath(settings.projectDir, "plugin-progress.json");
|
||||||
// Determine full vs incremental from explicit arg or manager checkbox
|
// Determine full vs incremental from explicit arg or manager checkbox
|
||||||
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
|
const isFull = typeof isFullOverride === "boolean" ? isFullOverride : (manager.getFullReimport ? manager.getFullReimport() : false);
|
||||||
const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
|
const retryUnresolved = typeof retryUnresolvedOverride === "boolean"
|
||||||
? retryUnresolvedOverride
|
? retryUnresolvedOverride
|
||||||
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
|
: (manager.getRetryUnresolved ? manager.getRetryUnresolved() : false);
|
||||||
manager.beginRun(clippingsPath);
|
manager.beginRun(clippingsPath);
|
||||||
|
const finalPlanPath = this.joinPath(settings.projectDir, "import-plan.final.json");
|
||||||
|
let previousFinalPlan = null;
|
||||||
|
try {
|
||||||
|
previousFinalPlan = JSON.parse(await Zotero.File.getContentsAsync(finalPlanPath));
|
||||||
|
} catch (_backupError) {}
|
||||||
try {
|
try {
|
||||||
await Zotero.File.removeIfExists(progressPath);
|
await Zotero.File.removeIfExists(progressPath);
|
||||||
} catch (_error) {}
|
} catch (_error) {}
|
||||||
|
|
@ -511,20 +690,142 @@ var KindleZoteroImporter = {
|
||||||
const finalPlan = JSON.parse(
|
const finalPlan = JSON.parse(
|
||||||
await Zotero.File.getContentsAsync(summary.outputs.final_plan)
|
await Zotero.File.getContentsAsync(summary.outputs.final_plan)
|
||||||
);
|
);
|
||||||
manager.updateProgress(96, "Writing annotations", "Saving annotations in Zotero");
|
const pendingCount = (finalPlan.annotations || []).length;
|
||||||
const results = await this.writeAnnotations(finalPlan, false, (completed, total) => {
|
const pendingDeletions = (finalPlan.deletions || []).length;
|
||||||
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
if (!pendingCount && !pendingDeletions) {
|
||||||
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
// Nothing new — run the maintenance sweep and finish as before.
|
||||||
});
|
manager.updateProgress(96, "Writing annotations", "Checking existing annotations");
|
||||||
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, results);
|
const emptyResults = await this.writeAnnotations(finalPlan, false, (completed, total) => {
|
||||||
const data = await this.loadManagerData();
|
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
||||||
await manager.completeRun(summary, results, data);
|
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
||||||
|
});
|
||||||
|
await this.reconcileWriterState(summary.outputs.final_plan, finalPlan, emptyResults);
|
||||||
|
const emptyData = await this.loadManagerData();
|
||||||
|
await manager.completeRun(summary, emptyResults, emptyData);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Pause before writing: let the user review and select what to integrate.
|
||||||
|
const fullDeltaIds = (finalPlan.annotations || []).map((a) => a.clipping_id);
|
||||||
|
// The pipeline has already written its generated history. Replace it with
|
||||||
|
// a safe ledger until Integrate is pressed, including if the window closes.
|
||||||
|
const deferredPlan = this.deferPendingPlan(previousFinalPlan, finalPlan, fullDeltaIds);
|
||||||
|
await Zotero.File.putContentsAsync(
|
||||||
|
summary.outputs.final_plan,
|
||||||
|
JSON.stringify(deferredPlan, null, 2) + "\n"
|
||||||
|
);
|
||||||
|
this._pending = {
|
||||||
|
summary,
|
||||||
|
finalPlan,
|
||||||
|
finalPlanPath: summary.outputs.final_plan,
|
||||||
|
fullDeltaIds,
|
||||||
|
previousFinalPlan,
|
||||||
|
};
|
||||||
|
manager.updateProgress(
|
||||||
|
90,
|
||||||
|
"Review pending",
|
||||||
|
`${pendingCount} highlight(s) ready — select and press Integrate`
|
||||||
|
);
|
||||||
|
manager.showPendingReview(summary, finalPlan);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
manager.failRun(String(error && error.stack ? error.stack : error));
|
manager.failRun(String(error && error.stack ? error.stack : error));
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
||||||
|
deferPendingPlan(previousPlan, generatedPlan, deferredIds) {
|
||||||
|
const plan = previousPlan
|
||||||
|
? { ...previousPlan }
|
||||||
|
: {
|
||||||
|
...generatedPlan,
|
||||||
|
annotations: [],
|
||||||
|
all_annotations: [],
|
||||||
|
deletions: [],
|
||||||
|
annotation_count: 0,
|
||||||
|
delta_annotation_count: 0,
|
||||||
|
};
|
||||||
|
const deferred = new Set(plan.deferred_clipping_ids || []);
|
||||||
|
for (const id of deferredIds || []) deferred.add(id);
|
||||||
|
plan.deferred_clipping_ids = [...deferred].sort();
|
||||||
|
return plan;
|
||||||
|
},
|
||||||
|
|
||||||
|
// Pure helper (tested): narrow a pipeline plan to the user's selection.
|
||||||
|
filterPlanForSelection(finalPlan, fullDeltaIds, selectedIds, previousPlan = null) {
|
||||||
|
const selected = new Set(selectedIds || []);
|
||||||
|
const unselected = new Set(
|
||||||
|
(fullDeltaIds || []).filter((id) => !selected.has(id))
|
||||||
|
);
|
||||||
|
const plan = { ...finalPlan };
|
||||||
|
plan.annotations = (finalPlan.annotations || []).filter((a) =>
|
||||||
|
selected.has(a.clipping_id)
|
||||||
|
);
|
||||||
|
if (unselected.size && Array.isArray(finalPlan.all_annotations)) {
|
||||||
|
plan.all_annotations = finalPlan.all_annotations.filter(
|
||||||
|
(a) => !unselected.has(a.clipping_id)
|
||||||
|
);
|
||||||
|
const present = new Set(plan.all_annotations.map((a) => a.clipping_id));
|
||||||
|
for (const annotation of previousPlan?.all_annotations || previousPlan?.annotations || []) {
|
||||||
|
if (unselected.has(annotation.clipping_id) && !present.has(annotation.clipping_id)) {
|
||||||
|
plan.all_annotations.push(annotation);
|
||||||
|
present.add(annotation.clipping_id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const deferred = new Set(previousPlan?.deferred_clipping_ids || []);
|
||||||
|
for (const id of selected) deferred.delete(id);
|
||||||
|
for (const id of unselected) deferred.add(id);
|
||||||
|
plan.deferred_clipping_ids = [...deferred].sort();
|
||||||
|
plan.delta_annotation_count = plan.annotations.length;
|
||||||
|
if (Array.isArray(plan.all_annotations)) {
|
||||||
|
plan.annotation_count = plan.all_annotations.length;
|
||||||
|
}
|
||||||
|
return plan;
|
||||||
|
},
|
||||||
|
|
||||||
|
async integratePending(manager, selectedIds) {
|
||||||
|
const pending = this._pending;
|
||||||
|
if (!pending) {
|
||||||
|
throw new Error("No pending review to integrate");
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const plan = this.filterPlanForSelection(
|
||||||
|
pending.finalPlan,
|
||||||
|
pending.fullDeltaIds,
|
||||||
|
selectedIds,
|
||||||
|
pending.previousFinalPlan
|
||||||
|
);
|
||||||
|
manager.updateProgress(96, "Writing annotations", "Saving selected annotations in Zotero");
|
||||||
|
const results = await this.writeAnnotations(plan, false, (completed, total) => {
|
||||||
|
const percent = total ? 96 + Math.floor((completed / total) * 4) : 100;
|
||||||
|
manager.updateProgress(percent, "Writing annotations", `${completed} of ${total} checked`);
|
||||||
|
});
|
||||||
|
await this.reconcileWriterState(pending.finalPlanPath, plan, results);
|
||||||
|
this._pending = null;
|
||||||
|
const data = await this.loadManagerData();
|
||||||
|
await manager.completeRun(pending.summary, results, data);
|
||||||
|
} catch (error) {
|
||||||
|
Zotero.logError(error);
|
||||||
|
manager.failRun(String(error && error.stack ? error.stack : error));
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
async cancelPendingReview(manager) {
|
||||||
|
// The deferred ledger was written before the review appeared, so closing
|
||||||
|
// or cancelling never records pending highlights as integrated.
|
||||||
|
this._pending = null;
|
||||||
|
manager.hidePendingReview();
|
||||||
|
manager.cancelRun("Review cancelled — nothing was written. Run again when ready.");
|
||||||
|
try {
|
||||||
|
const data = await this.loadManagerData();
|
||||||
|
manager.data = data;
|
||||||
|
manager.rows = manager.conflictRows();
|
||||||
|
manager.refreshRows(true);
|
||||||
|
manager.refreshTitleOverview();
|
||||||
|
manager.refreshIntegrated(true);
|
||||||
|
manager.refreshMappings(true);
|
||||||
|
} catch (_reloadError) {}
|
||||||
|
},
|
||||||
|
|
||||||
async exec(command, args) {
|
async exec(command, args) {
|
||||||
if (!Zotero.Utilities || !Zotero.Utilities.Internal || !Zotero.Utilities.Internal.exec) {
|
if (!Zotero.Utilities || !Zotero.Utilities.Internal || !Zotero.Utilities.Internal.exec) {
|
||||||
throw new Error("Zotero.Utilities.Internal.exec is unavailable in this Zotero build");
|
throw new Error("Zotero.Utilities.Internal.exec is unavailable in this Zotero build");
|
||||||
|
|
@ -814,6 +1115,117 @@ var KindleZoteroImporter = {
|
||||||
throw new Error("Existing annotation cannot be saved by this Zotero build");
|
throw new Error("Existing annotation cannot be saved by this Zotero build");
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async saveRelatedItem(item) {
|
||||||
|
const options = { skipDateModifiedUpdate: true };
|
||||||
|
if (item.save) {
|
||||||
|
await item.save(options);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (item.saveTx) {
|
||||||
|
await item.saveTx(options);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
throw new Error("Related item cannot be saved by this Zotero build");
|
||||||
|
},
|
||||||
|
|
||||||
|
async addRelatedItemRelations(entries, dryRun, results) {
|
||||||
|
const pairs = new Map();
|
||||||
|
for (const entry of entries || []) {
|
||||||
|
for (const target of entry.related_items || []) {
|
||||||
|
const sourceID = Number(entry.parent_item_id);
|
||||||
|
const targetID = Number(target.item_id);
|
||||||
|
if (!sourceID || !targetID || sourceID === targetID) continue;
|
||||||
|
const pairKey = [sourceID, targetID].sort((a, b) => a - b).join(":");
|
||||||
|
if (!pairs.has(pairKey)) {
|
||||||
|
pairs.set(pairKey, { sourceID, targetID, citationKey: target.citation_key || "" });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const pair of pairs.values()) {
|
||||||
|
try {
|
||||||
|
const source = Zotero.Items.get(pair.sourceID);
|
||||||
|
const target = Zotero.Items.get(pair.targetID);
|
||||||
|
if (!source || !target) {
|
||||||
|
throw new Error(`Related item pair not found: ${pair.sourceID} -> ${pair.targetID}`);
|
||||||
|
}
|
||||||
|
if (dryRun) {
|
||||||
|
results.relationsPlanned += 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!source.addRelatedItem || !target.addRelatedItem) {
|
||||||
|
throw new Error("This Zotero build does not support addRelatedItem()");
|
||||||
|
}
|
||||||
|
|
||||||
|
let changed = false;
|
||||||
|
const addBothDirections = async () => {
|
||||||
|
const sourceChanged = source.addRelatedItem(target);
|
||||||
|
const targetChanged = target.addRelatedItem(source);
|
||||||
|
if (sourceChanged) await this.saveRelatedItem(source);
|
||||||
|
if (targetChanged) await this.saveRelatedItem(target);
|
||||||
|
changed = sourceChanged || targetChanged;
|
||||||
|
};
|
||||||
|
if (Zotero.DB?.executeTransaction) {
|
||||||
|
await Zotero.DB.executeTransaction(addBothDirections);
|
||||||
|
} else {
|
||||||
|
await addBothDirections();
|
||||||
|
}
|
||||||
|
if (changed) results.relationsAdded += 1;
|
||||||
|
else results.relationsSkipped += 1;
|
||||||
|
} catch (error) {
|
||||||
|
Zotero.logError(error);
|
||||||
|
results.relationFailures.push({
|
||||||
|
parent_item_id: pair.sourceID,
|
||||||
|
related_item_id: pair.targetID,
|
||||||
|
citation_key: pair.citationKey,
|
||||||
|
message: String(error),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
async backfillRelationLinks(entries, dryRun, results) {
|
||||||
|
// Appends missing `Related: @key <link>` blocks to pre-existing
|
||||||
|
// annotations. Cross-library targets can never become Related items,
|
||||||
|
// so the comment link is the only visible connection for them.
|
||||||
|
for (const entry of entries || []) {
|
||||||
|
const targets = entry.related_items || [];
|
||||||
|
if (!targets.length || !entry.clipping_id || !entry.attachment_item_id) continue;
|
||||||
|
const block = "Related: " + targets.map(
|
||||||
|
(target) => `@${target.citation_key} ${target.link || ""}`.trim()
|
||||||
|
).join(" · ");
|
||||||
|
try {
|
||||||
|
const attachment = Zotero.Items.get(entry.attachment_item_id);
|
||||||
|
if (!attachment) {
|
||||||
|
throw new Error(`Attachment not found: ${entry.attachment_item_id}`);
|
||||||
|
}
|
||||||
|
const annotations = attachment.getAnnotations ? attachment.getAnnotations() : [];
|
||||||
|
const tag = `kindle-id:${entry.clipping_id}`;
|
||||||
|
const existing = (annotations || []).find((annotation) =>
|
||||||
|
this.existingTags(annotation).includes(tag)
|
||||||
|
);
|
||||||
|
if (!existing) continue;
|
||||||
|
const comment = String(existing.annotationComment || "");
|
||||||
|
if (comment.includes(block)) continue;
|
||||||
|
if (dryRun) {
|
||||||
|
results.commentLinksPlanned += 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
existing.annotationComment = comment ? `${comment}\n\n${block}` : block;
|
||||||
|
await this.saveExistingAnnotation(existing);
|
||||||
|
results.commentLinksAdded += 1;
|
||||||
|
} catch (error) {
|
||||||
|
Zotero.logError(error);
|
||||||
|
results.relationFailures.push({
|
||||||
|
parent_item_id: entry.parent_item_id,
|
||||||
|
related_item_id: (targets[0] || {}).item_id,
|
||||||
|
citation_key: (targets[0] || {}).citation_key,
|
||||||
|
message: `comment link: ${error}`,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
async reconcileWriterState(path, plan, results) {
|
async reconcileWriterState(path, plan, results) {
|
||||||
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
|
const plannedIds = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
|
||||||
const failedIds = new Set(
|
const failedIds = new Set(
|
||||||
|
|
@ -861,6 +1273,12 @@ var KindleZoteroImporter = {
|
||||||
blockedRecreateMissingTag: 0,
|
blockedRecreateMissingTag: 0,
|
||||||
recreateDeclined: 0,
|
recreateDeclined: 0,
|
||||||
removedDuplicates: 0,
|
removedDuplicates: 0,
|
||||||
|
relationsAdded: 0,
|
||||||
|
relationsSkipped: 0,
|
||||||
|
relationsPlanned: 0,
|
||||||
|
commentLinksAdded: 0,
|
||||||
|
commentLinksPlanned: 0,
|
||||||
|
relationFailures: [],
|
||||||
deletedForIncremental: 0,
|
deletedForIncremental: 0,
|
||||||
failedDeletionIds: [],
|
failedDeletionIds: [],
|
||||||
maintenanceDeletedIds: [],
|
maintenanceDeletedIds: [],
|
||||||
|
|
@ -1045,6 +1463,14 @@ var KindleZoteroImporter = {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const currentIDs = new Set((plan.annotations || []).map((entry) => entry.clipping_id));
|
||||||
|
const relationEntries = (plan.all_annotations || []).filter(
|
||||||
|
(entry) => !currentIDs.has(entry.clipping_id)
|
||||||
|
);
|
||||||
|
relationEntries.push(...successfulEntries.values());
|
||||||
|
await this.addRelatedItemRelations(relationEntries, dryRun, results);
|
||||||
|
await this.backfillRelationLinks(relationEntries, dryRun, results);
|
||||||
|
|
||||||
if (!dryRun && plan.deletions && plan.deletions.length) {
|
if (!dryRun && plan.deletions && plan.deletions.length) {
|
||||||
const deletionIds = new Set(plan.deletions);
|
const deletionIds = new Set(plan.deletions);
|
||||||
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
|
const alreadyDeletedIds = new Set(results.maintenanceDeletedIds);
|
||||||
|
|
@ -1146,7 +1572,7 @@ function uninstall(_data, _reason) {}
|
||||||
|
|
||||||
async function startup(data, _reason) {
|
async function startup(data, _reason) {
|
||||||
try {
|
try {
|
||||||
await KindleZoteroImporter.startup(data);
|
await Kanzi.startup(data);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
}
|
}
|
||||||
|
|
@ -1154,7 +1580,7 @@ async function startup(data, _reason) {
|
||||||
|
|
||||||
function shutdown(_data, _reason) {
|
function shutdown(_data, _reason) {
|
||||||
try {
|
try {
|
||||||
KindleZoteroImporter.shutdown();
|
Kanzi.shutdown();
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
}
|
}
|
||||||
|
|
@ -1162,7 +1588,7 @@ function shutdown(_data, _reason) {
|
||||||
|
|
||||||
function onMainWindowLoad({ window }) {
|
function onMainWindowLoad({ window }) {
|
||||||
try {
|
try {
|
||||||
KindleZoteroImporter.onMainWindowLoad(window);
|
Kanzi.onMainWindowLoad(window);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
}
|
}
|
||||||
|
|
@ -1170,7 +1596,7 @@ function onMainWindowLoad({ window }) {
|
||||||
|
|
||||||
function onMainWindowUnload({ window }) {
|
function onMainWindowUnload({ window }) {
|
||||||
try {
|
try {
|
||||||
KindleZoteroImporter.onMainWindowUnload(window);
|
Kanzi.onMainWindowUnload(window);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
Zotero.logError(error);
|
Zotero.logError(error);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,7 +1,7 @@
|
||||||
{
|
{
|
||||||
"manifest_version": 2,
|
"manifest_version": 2,
|
||||||
"name": "Kanzi",
|
"name": "Kanzi",
|
||||||
"version": "0.7.0",
|
"version": "0.7.1",
|
||||||
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
|
"description": "Kindle annotations to Zotero — My Clippings.txt → native annotations.",
|
||||||
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
|
"homepage_url": "https://github.com/UtkuBilenDemir/kanzi",
|
||||||
"author": "Utku Bilen Demir",
|
"author": "Utku Bilen Demir",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
[project]
|
[project]
|
||||||
name = "kindle-zotero-importer"
|
name = "kindle-zotero-importer"
|
||||||
version = "0.7.0"
|
version = "0.7.1"
|
||||||
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
|
description = "Generate Zotero annotation import plans from Kindle My Clippings.txt files."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import copy
|
||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
@ -8,7 +9,12 @@ from pathlib import Path
|
||||||
from .clippings import clippings_to_jsonable, load_clippings
|
from .clippings import clippings_to_jsonable, load_clippings
|
||||||
from .epub_position import add_epub_positions
|
from .epub_position import add_epub_positions
|
||||||
from .final_plan import FINAL_FORMAT, build_final_writer_plan
|
from .final_plan import FINAL_FORMAT, build_final_writer_plan
|
||||||
from .import_plan import DEFAULT_COLOUR_MAP, _attach_notes_to_highlights, build_import_plan
|
from .import_plan import (
|
||||||
|
DEFAULT_COLOUR_MAP,
|
||||||
|
_attach_notes_to_highlights,
|
||||||
|
_related_items_for_citekeys,
|
||||||
|
build_import_plan,
|
||||||
|
)
|
||||||
from .matcher import build_match_report, load_json
|
from .matcher import build_match_report, load_json
|
||||||
from .mismatch_review import build_mismatch_review
|
from .mismatch_review import build_mismatch_review
|
||||||
from .overrides import generate_override_skeleton, load_overrides
|
from .overrides import generate_override_skeleton, load_overrides
|
||||||
|
|
@ -65,6 +71,20 @@ def _merge_annotation_history(
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _refresh_history_relations(
|
||||||
|
annotations: list[dict], zotero_index: dict
|
||||||
|
) -> None:
|
||||||
|
"""Resolve citekeys for old annotations without re-positioning them."""
|
||||||
|
items_by_id = {item["item_id"]: item for item in zotero_index.get("items", [])}
|
||||||
|
for entry in annotations:
|
||||||
|
annotation = entry.get("annotation") or {}
|
||||||
|
entry["related_items"] = _related_items_for_citekeys(
|
||||||
|
(annotation.get("text") or "", annotation.get("comment") or ""),
|
||||||
|
entry.get("parent_item_id"),
|
||||||
|
items_by_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _changed_note_clipping_ids(
|
def _changed_note_clipping_ids(
|
||||||
current_clippings: list[dict], previous_annotations: list[dict]
|
current_clippings: list[dict], previous_annotations: list[dict]
|
||||||
) -> set[str]:
|
) -> set[str]:
|
||||||
|
|
@ -112,6 +132,137 @@ def _deletion_ids(
|
||||||
return deleted
|
return deleted
|
||||||
|
|
||||||
|
|
||||||
|
def _deferred_clipping_ids(
|
||||||
|
previous_plan: dict | None, current_ids: set[str]
|
||||||
|
) -> set[str]:
|
||||||
|
return {
|
||||||
|
clipping_id
|
||||||
|
for clipping_id in (previous_plan or {}).get("deferred_clipping_ids", [])
|
||||||
|
if clipping_id in current_ids
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _discarded_clipping_ids(
|
||||||
|
previous_plan: dict | None, current_ids: set[str]
|
||||||
|
) -> set[str]:
|
||||||
|
"""Clips the user permanently discarded in Conflicts (e.g. standalone notes)."""
|
||||||
|
return {
|
||||||
|
clipping_id
|
||||||
|
for clipping_id in (previous_plan or {}).get("discarded_clipping_ids", [])
|
||||||
|
if clipping_id in current_ids
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _changed_override_titles(
|
||||||
|
previous_snapshot: dict | None, current_overrides: dict
|
||||||
|
) -> set[str]:
|
||||||
|
if previous_snapshot is None:
|
||||||
|
return set(current_overrides)
|
||||||
|
return {
|
||||||
|
title
|
||||||
|
for title in set(previous_snapshot) | set(current_overrides)
|
||||||
|
if previous_snapshot.get(title) != current_overrides.get(title)
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _clipping_ids_for_titles(
|
||||||
|
clippings: list[dict], titles: set[str]
|
||||||
|
) -> set[str]:
|
||||||
|
return {clipping["id"] for clipping in clippings if clipping["title"] in titles}
|
||||||
|
|
||||||
|
|
||||||
|
def _reattempt_failed_with_alternate_attachments(positioned_plan: dict) -> dict:
|
||||||
|
"""Retry text/rect failures against another attachment of the same item.
|
||||||
|
|
||||||
|
`_choose_attachment` commits to one file (e.g. the sole EPUB) without
|
||||||
|
comparing text. When that file lacks the quote but a sibling PDF/EPUB
|
||||||
|
has it (e.g. Dostoyevsky EPUB vs PDF), re-anchor the clone there and
|
||||||
|
keep the winner, tagged `positioned-via-alternate-attachment`.
|
||||||
|
Partial shingle anchors are also upgraded when a sibling holds the
|
||||||
|
full quote.
|
||||||
|
"""
|
||||||
|
retryable = {"epub-text-not-found", "pdf-text-not-found", "pdf-rects-not-found"}
|
||||||
|
retries: list[dict] = []
|
||||||
|
upgrade_ids: set[str] = set()
|
||||||
|
for item in positioned_plan.get("items", []):
|
||||||
|
if item.get("status") in retryable:
|
||||||
|
pass
|
||||||
|
elif item.get("status") == "positioned" and "partial-text-match" in (
|
||||||
|
item.get("problems") or []
|
||||||
|
):
|
||||||
|
clipping_id = (item.get("clipping") or {}).get("id")
|
||||||
|
if clipping_id:
|
||||||
|
upgrade_ids.add(clipping_id)
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
zotero = item.get("zotero") or {}
|
||||||
|
current = zotero.get("attachment") or {}
|
||||||
|
choices = zotero.get("attachment_choices") or []
|
||||||
|
alternates = [
|
||||||
|
choice
|
||||||
|
for choice in choices
|
||||||
|
if choice.get("item_id") != current.get("item_id")
|
||||||
|
]
|
||||||
|
if not alternates:
|
||||||
|
continue
|
||||||
|
alternates.sort(
|
||||||
|
key=lambda choice: 0
|
||||||
|
if choice.get("content_type") != current.get("content_type")
|
||||||
|
else 1
|
||||||
|
)
|
||||||
|
clone = copy.deepcopy(item)
|
||||||
|
clone_zotero = dict(clone.get("zotero") or {})
|
||||||
|
clone_zotero["attachment"] = alternates[0]
|
||||||
|
clone["zotero"] = clone_zotero
|
||||||
|
clone["status"] = "ready-for-positioning"
|
||||||
|
clone["problems"] = []
|
||||||
|
retries.append(clone)
|
||||||
|
if not retries:
|
||||||
|
return positioned_plan
|
||||||
|
mini = {"format": positioned_plan.get("format"), "items": retries}
|
||||||
|
mini = add_epub_positions(mini)
|
||||||
|
mini = add_pdf_positions(mini)
|
||||||
|
winners = {}
|
||||||
|
for attempt in mini.get("items", []):
|
||||||
|
clipping_id = (attempt.get("clipping") or {}).get("id")
|
||||||
|
if clipping_id and attempt.get("status") == "positioned":
|
||||||
|
winners[clipping_id] = attempt
|
||||||
|
if not winners:
|
||||||
|
return positioned_plan
|
||||||
|
merged = []
|
||||||
|
for item in positioned_plan.get("items", []):
|
||||||
|
clipping_id = (item.get("clipping") or {}).get("id")
|
||||||
|
if clipping_id in winners:
|
||||||
|
won = winners[clipping_id]
|
||||||
|
if clipping_id in upgrade_ids and "partial-text-match" in (
|
||||||
|
won.get("problems") or []
|
||||||
|
):
|
||||||
|
merged.append(item)
|
||||||
|
continue
|
||||||
|
won = dict(won)
|
||||||
|
carried = [
|
||||||
|
problem
|
||||||
|
for problem in item.get("problems", [])
|
||||||
|
if problem != "partial-text-match"
|
||||||
|
or "partial-text-match" in (won.get("problems") or [])
|
||||||
|
]
|
||||||
|
won["problems"] = [
|
||||||
|
*carried,
|
||||||
|
"positioned-via-alternate-attachment",
|
||||||
|
]
|
||||||
|
merged.append(won)
|
||||||
|
else:
|
||||||
|
merged.append(item)
|
||||||
|
updated = dict(positioned_plan)
|
||||||
|
updated["items"] = merged
|
||||||
|
status_counts: dict[str, int] = {}
|
||||||
|
for entry in merged:
|
||||||
|
status = entry.get("status", "unknown")
|
||||||
|
status_counts[status] = status_counts.get(status, 0) + 1
|
||||||
|
updated["status_counts"] = status_counts
|
||||||
|
return updated
|
||||||
|
|
||||||
|
|
||||||
def _add_previous_imported_comments(
|
def _add_previous_imported_comments(
|
||||||
annotations: list[dict], previous_plan: dict | None
|
annotations: list[dict], previous_plan: dict | None
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|
@ -134,7 +285,12 @@ def _unresolved_clipping_ids(
|
||||||
return {
|
return {
|
||||||
item.get("clipping", {}).get("id")
|
item.get("clipping", {}).get("id")
|
||||||
for item in (previous_plan or {}).get("items", [])
|
for item in (previous_plan or {}).get("items", [])
|
||||||
if item.get("status") not in {"positioned", "ignored-title"}
|
if item.get("status") not in {
|
||||||
|
"positioned",
|
||||||
|
"ignored-title",
|
||||||
|
"pdf-position-skipped-non-highlight",
|
||||||
|
"epub-position-skipped-non-highlight",
|
||||||
|
}
|
||||||
and item.get("clipping", {}).get("id") in source_ids
|
and item.get("clipping", {}).get("id") in source_ids
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -535,18 +691,12 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
|
_ov_for_inc = load_overrides(load_json(str(overrides_path))) if overrides_path.exists() else {}
|
||||||
except Exception:
|
except Exception:
|
||||||
_ov_for_inc = {}
|
_ov_for_inc = {}
|
||||||
previous_override_snapshot = (prev_final_plan or {}).get("overrides_snapshot")
|
changed_override_titles = _changed_override_titles(
|
||||||
if previous_override_snapshot is None:
|
(prev_final_plan or {}).get("overrides_snapshot"), _ov_for_inc
|
||||||
changed_override_titles = set(_ov_for_inc)
|
)
|
||||||
else:
|
to_process_ids.update(
|
||||||
changed_override_titles = {
|
_clipping_ids_for_titles(effective_clippings, changed_override_titles)
|
||||||
title
|
)
|
||||||
for title in set(previous_override_snapshot) | set(_ov_for_inc)
|
|
||||||
if previous_override_snapshot.get(title) != _ov_for_inc.get(title)
|
|
||||||
}
|
|
||||||
for c in effective_clippings:
|
|
||||||
if c["title"] in changed_override_titles:
|
|
||||||
to_process_ids.add(c["id"])
|
|
||||||
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
|
previous_annotations = (prev_final_plan or {}).get("all_annotations") or (prev_final_plan or {}).get("annotations", [])
|
||||||
for annotation in previous_annotations:
|
for annotation in previous_annotations:
|
||||||
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
|
if annotation.get("clipping_title") in changed_override_titles and annotation.get("clipping_id"):
|
||||||
|
|
@ -563,10 +713,15 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
|
for clipping_id in (prev_final_plan or {}).get("write_failed_ids", [])
|
||||||
if clipping_id in new_ids
|
if clipping_id in new_ids
|
||||||
)
|
)
|
||||||
|
to_process_ids.update(_deferred_clipping_ids(prev_final_plan, new_ids))
|
||||||
if args.retry_unresolved and prev_positioned_plan:
|
if args.retry_unresolved and prev_positioned_plan:
|
||||||
to_process_ids.update(
|
to_process_ids.update(
|
||||||
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
|
_unresolved_clipping_ids(prev_positioned_plan, new_ids)
|
||||||
)
|
)
|
||||||
|
# Permanently discarded clips (user action in Conflicts) never re-queue.
|
||||||
|
to_process_ids.difference_update(
|
||||||
|
_discarded_clipping_ids(prev_final_plan, new_ids)
|
||||||
|
)
|
||||||
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
|
# If colourMap itself changed since last run, re-queue all prev integrated (so Settings change recolours)
|
||||||
try:
|
try:
|
||||||
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
|
prev_map = (prev_final_plan or {}).get("colourMap") or (prev_final_plan or {}).get("colorMap")
|
||||||
|
|
@ -632,6 +787,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
epub_plan = add_epub_positions(plan)
|
epub_plan = add_epub_positions(plan)
|
||||||
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
|
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files (incremental)")
|
||||||
positioned_plan = add_pdf_positions(epub_plan)
|
positioned_plan = add_pdf_positions(epub_plan)
|
||||||
|
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
|
||||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||||
final_plan = build_final_writer_plan(positioned_plan)
|
final_plan = build_final_writer_plan(positioned_plan)
|
||||||
# Attach deletions for writer
|
# Attach deletions for writer
|
||||||
|
|
@ -665,6 +821,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
epub_plan = add_epub_positions(plan)
|
epub_plan = add_epub_positions(plan)
|
||||||
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
|
report_progress(55, "Positioning PDF highlights", "Locating highlights in PDF files; this is usually the longest stage")
|
||||||
positioned_plan = add_pdf_positions(epub_plan)
|
positioned_plan = add_pdf_positions(epub_plan)
|
||||||
|
positioned_plan = _reattempt_failed_with_alternate_attachments(positioned_plan)
|
||||||
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
report_progress(90, "Finalizing", "Preparing Zotero annotations and mismatch report")
|
||||||
final_plan = build_final_writer_plan(positioned_plan)
|
final_plan = build_final_writer_plan(positioned_plan)
|
||||||
final_plan["deletions"] = sorted(
|
final_plan["deletions"] = sorted(
|
||||||
|
|
@ -700,10 +857,14 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
final_plan.get("annotations", []),
|
final_plan.get("annotations", []),
|
||||||
final_plan.get("deletions", []),
|
final_plan.get("deletions", []),
|
||||||
)
|
)
|
||||||
|
_refresh_history_relations(final_plan["all_annotations"], zotero_index)
|
||||||
final_plan["annotation_count"] = len(final_plan["all_annotations"])
|
final_plan["annotation_count"] = len(final_plan["all_annotations"])
|
||||||
final_plan["delta_annotation_count"] = delta_annotation_count
|
final_plan["delta_annotation_count"] = delta_annotation_count
|
||||||
final_plan["processed_clipping_ids"] = sorted(new_ids)
|
final_plan["processed_clipping_ids"] = sorted(new_ids)
|
||||||
final_plan["overrides_snapshot"] = overrides or {}
|
final_plan["overrides_snapshot"] = overrides or {}
|
||||||
|
final_plan["discarded_clipping_ids"] = sorted(
|
||||||
|
_discarded_clipping_ids(prev_final_plan, new_ids)
|
||||||
|
)
|
||||||
|
|
||||||
indent = 2 if args.pretty else None
|
indent = 2 if args.pretty else None
|
||||||
outputs = {
|
outputs = {
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,7 @@ from dataclasses import dataclass
|
||||||
from html import unescape
|
from html import unescape
|
||||||
from html.parser import HTMLParser
|
from html.parser import HTMLParser
|
||||||
import posixpath
|
import posixpath
|
||||||
|
import unicodedata
|
||||||
import zipfile
|
import zipfile
|
||||||
import xml.etree.ElementTree as ET
|
import xml.etree.ElementTree as ET
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
@ -68,6 +69,8 @@ def _position_item(
|
||||||
annotation["sortIndex"] = position["sortIndex"]
|
annotation["sortIndex"] = position["sortIndex"]
|
||||||
updated["annotation"] = annotation
|
updated["annotation"] = annotation
|
||||||
updated["status"] = "positioned"
|
updated["status"] = "positioned"
|
||||||
|
if position.get("partial"):
|
||||||
|
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
|
||||||
return updated
|
return updated
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -104,11 +107,42 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
|
||||||
return None
|
return None
|
||||||
|
|
||||||
match_start = normalized_haystack.find(normalized_quote)
|
match_start = normalized_haystack.find(normalized_quote)
|
||||||
if match_start < 0:
|
raw_start: int | None = None
|
||||||
|
raw_end: int | None = None
|
||||||
|
partial = False
|
||||||
|
if match_start >= 0:
|
||||||
|
match_end = match_start + len(normalized_quote)
|
||||||
|
raw_start = haystack_map[match_start]
|
||||||
|
raw_end = haystack_map[match_end - 1] + 1
|
||||||
|
else:
|
||||||
|
stripped_quote, _ = _strip_with_map(normalized_quote)
|
||||||
|
if stripped_quote and len(stripped_quote) >= 20:
|
||||||
|
stripped_haystack, stripped_map = _strip_with_map(normalized_haystack)
|
||||||
|
strip_start = stripped_haystack.find(stripped_quote)
|
||||||
|
strip_len = len(stripped_quote)
|
||||||
|
if strip_start < 0:
|
||||||
|
hit = _partial_stripped_range(stripped_haystack, stripped_quote)
|
||||||
|
if hit is not None:
|
||||||
|
strip_start, strip_end = hit
|
||||||
|
strip_len = strip_end - strip_start
|
||||||
|
partial = True
|
||||||
|
if strip_start >= 0:
|
||||||
|
# stripped_map points into normalized_haystack;
|
||||||
|
# convert via haystack_map back to raw haystack offsets.
|
||||||
|
norm_start = stripped_map[strip_start]
|
||||||
|
norm_end = stripped_map[strip_start + strip_len - 1]
|
||||||
|
raw_start = haystack_map[norm_start]
|
||||||
|
raw_end = haystack_map[norm_end] + 1
|
||||||
|
else:
|
||||||
|
span = _shingle_span(normalized_haystack, _quote_words(normalized_quote))
|
||||||
|
if span is not None:
|
||||||
|
raw_start = haystack_map[span[0]]
|
||||||
|
raw_end = haystack_map[span[1] - 1] + 1
|
||||||
|
partial = True
|
||||||
|
else:
|
||||||
|
partial = False
|
||||||
|
if raw_start is None or raw_end is None:
|
||||||
return None
|
return None
|
||||||
match_end = match_start + len(normalized_quote)
|
|
||||||
raw_start = haystack_map[match_start]
|
|
||||||
raw_end = haystack_map[match_end - 1] + 1
|
|
||||||
|
|
||||||
# Account for the spaces inserted between text nodes in haystack construction.
|
# Account for the spaces inserted between text nodes in haystack construction.
|
||||||
adjusted_nodes = _nodes_for_joined_text(text_nodes)
|
adjusted_nodes = _nodes_for_joined_text(text_nodes)
|
||||||
|
|
@ -139,6 +173,7 @@ def find_epub_cfi(text_nodes: list[TextNode], quote: str) -> dict[str, Any] | No
|
||||||
"value": value,
|
"value": value,
|
||||||
},
|
},
|
||||||
"sortIndex": f"{spine_step:05d}|{raw_start:08d}",
|
"sortIndex": f"{spine_step:05d}|{raw_start:08d}",
|
||||||
|
"partial": partial,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -213,6 +248,11 @@ def _clean_text(text: str) -> str:
|
||||||
|
|
||||||
|
|
||||||
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||||
|
text = unicodedata.normalize("NFKC", text)
|
||||||
|
text = (
|
||||||
|
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
|
||||||
|
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
|
||||||
|
)
|
||||||
normalized = []
|
normalized = []
|
||||||
mapping = []
|
mapping = []
|
||||||
last_was_space = False
|
last_was_space = False
|
||||||
|
|
@ -232,6 +272,58 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||||
return "".join(normalized), mapping
|
return "".join(normalized), mapping
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_with_map(text: str) -> tuple[str, list[int]]:
|
||||||
|
text = unicodedata.normalize("NFKC", text).casefold()
|
||||||
|
stripped: list[str] = []
|
||||||
|
mapping: list[int] = []
|
||||||
|
for index, char in enumerate(text):
|
||||||
|
if char.isalnum():
|
||||||
|
stripped.append(char)
|
||||||
|
mapping.append(index)
|
||||||
|
return "".join(stripped), mapping
|
||||||
|
|
||||||
|
|
||||||
|
def _partial_stripped_range(
|
||||||
|
haystack: str, quote: str, min_len: int = 30
|
||||||
|
) -> tuple[int, int] | None:
|
||||||
|
if len(quote) < min_len * 2:
|
||||||
|
return None
|
||||||
|
mid = len(quote) // 2
|
||||||
|
for part in (quote[:mid], quote[mid:]):
|
||||||
|
if len(part) < min_len:
|
||||||
|
continue
|
||||||
|
start = haystack.find(part)
|
||||||
|
if start >= 0:
|
||||||
|
return start, start + len(part)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _quote_words(normalized: str) -> list[str]:
|
||||||
|
words: list[str] = []
|
||||||
|
for token in normalized.split():
|
||||||
|
cleaned = "".join(char for char in token if char.isalnum())
|
||||||
|
if cleaned:
|
||||||
|
words.append(cleaned)
|
||||||
|
return words
|
||||||
|
|
||||||
|
|
||||||
|
def _shingle_span(
|
||||||
|
normalized_text: str, quote_words: list[str], width: int = 6
|
||||||
|
) -> tuple[int, int] | None:
|
||||||
|
first: int | None = None
|
||||||
|
last: int | None = None
|
||||||
|
for start in range(len(quote_words) - width + 1):
|
||||||
|
needle = " ".join(quote_words[start : start + width])
|
||||||
|
at = normalized_text.find(needle)
|
||||||
|
if at >= 0:
|
||||||
|
end = at + len(needle)
|
||||||
|
first = at if first is None else min(first, at)
|
||||||
|
last = end if last is None else max(last, end)
|
||||||
|
if first is None or last is None:
|
||||||
|
return None
|
||||||
|
return first, last
|
||||||
|
|
||||||
|
|
||||||
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
|
def _nodes_for_joined_text(text_nodes: list[TextNode]) -> list[TextNode]:
|
||||||
nodes = []
|
nodes = []
|
||||||
cursor = 0
|
cursor = 0
|
||||||
|
|
|
||||||
|
|
@ -71,6 +71,7 @@ def build_final_writer_plan(positioned_plan: dict[str, Any]) -> dict[str, Any]:
|
||||||
"attachment_key": attachment["key"],
|
"attachment_key": attachment["key"],
|
||||||
"parent_item_id": item["zotero"]["parent_item_id"],
|
"parent_item_id": item["zotero"]["parent_item_id"],
|
||||||
"parent_key": item["zotero"]["parent_key"],
|
"parent_key": item["zotero"]["parent_key"],
|
||||||
|
"related_items": item["zotero"].get("related_items", []),
|
||||||
"citation_key": item["zotero"].get("citation_key"),
|
"citation_key": item["zotero"].get("citation_key"),
|
||||||
"doi": item["zotero"].get("doi"),
|
"doi": item["zotero"].get("doi"),
|
||||||
"isbn": item["zotero"].get("isbn"),
|
"isbn": item["zotero"].get("isbn"),
|
||||||
|
|
|
||||||
|
|
@ -124,6 +124,12 @@ def _plan_clipping(
|
||||||
zotero_item, overrides_by_title.get(clipping["title"]) or {}
|
zotero_item, overrides_by_title.get(clipping["title"]) or {}
|
||||||
)
|
)
|
||||||
status = "ready-for-positioning" if attachment else attachment_status
|
status = "ready-for-positioning" if attachment else attachment_status
|
||||||
|
stub = _annotation_stub(clipping, colour_map) if attachment else None
|
||||||
|
related_items = (
|
||||||
|
_append_citekey_links(stub, clipping, zotero_item, items_by_id)
|
||||||
|
if stub
|
||||||
|
else []
|
||||||
|
)
|
||||||
return {
|
return {
|
||||||
"clipping": clipping,
|
"clipping": clipping,
|
||||||
"status": status,
|
"status": status,
|
||||||
|
|
@ -138,17 +144,96 @@ def _plan_clipping(
|
||||||
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
|
"issn": zotero_item["fields"].get("ISSN") if "ISSN" in zotero_item["fields"] else None,
|
||||||
"url": zotero_item["fields"].get("url"),
|
"url": zotero_item["fields"].get("url"),
|
||||||
"publication_title": zotero_item["fields"].get("publicationTitle"),
|
"publication_title": zotero_item["fields"].get("publicationTitle"),
|
||||||
|
"related_items": related_items,
|
||||||
"attachment": attachment,
|
"attachment": attachment,
|
||||||
"attachment_choices": zotero_item.get("attachments", []),
|
"attachment_choices": zotero_item.get("attachments", []),
|
||||||
"expected_attachment_type": _expected_attachment_type(
|
"expected_attachment_type": _expected_attachment_type(
|
||||||
clipping, zotero_item.get("attachments", [])
|
clipping, zotero_item.get("attachments", [])
|
||||||
),
|
),
|
||||||
},
|
},
|
||||||
"annotation": _annotation_stub(clipping, colour_map) if attachment else None,
|
"annotation": stub,
|
||||||
"problems": problems,
|
"problems": problems,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _append_citekey_links(
|
||||||
|
stub: dict[str, Any],
|
||||||
|
clipping: dict[str, Any],
|
||||||
|
parent_item: dict[str, Any],
|
||||||
|
items_by_id: dict[int, dict[str, Any]],
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""Turn `@citekey` mentions into clickable Zotero item links.
|
||||||
|
|
||||||
|
Kindle-side `@key` tokens (Obsidian-style) resolve against the library
|
||||||
|
index; matches to *other* items are appended to the annotation comment
|
||||||
|
as `zotero://select` links, which Zotero renders clickable. Self-links
|
||||||
|
to the parent item are skipped.
|
||||||
|
"""
|
||||||
|
related_items = _related_items_for_citekeys(
|
||||||
|
(clipping.get("text") or "", clipping.get("comment") or ""),
|
||||||
|
parent_item.get("item_id"),
|
||||||
|
items_by_id,
|
||||||
|
)
|
||||||
|
links = [
|
||||||
|
f"@{target['citation_key']} {target['link']}" for target in related_items
|
||||||
|
]
|
||||||
|
if not links:
|
||||||
|
return []
|
||||||
|
existing = stub.get("comment") or ""
|
||||||
|
block = "Related: " + " · ".join(links)
|
||||||
|
if block not in existing:
|
||||||
|
stub["comment"] = f"{existing}\n\n{block}" if existing else block
|
||||||
|
return related_items
|
||||||
|
|
||||||
|
|
||||||
|
def _select_link(item: dict[str, Any]) -> str:
|
||||||
|
"""Library-aware `zotero://select` link for an indexed item.
|
||||||
|
|
||||||
|
Group-library targets need the `/groups/{groupID}/items/{key}` form;
|
||||||
|
personal-library (or unknown) targets use `/library/items/{key}`.
|
||||||
|
"""
|
||||||
|
group_id = item.get("group_id")
|
||||||
|
if group_id:
|
||||||
|
return f"zotero://select/groups/{group_id}/items/{item['key']}"
|
||||||
|
return f"zotero://select/library/items/{item['key']}"
|
||||||
|
|
||||||
|
|
||||||
|
def _related_items_for_citekeys(
|
||||||
|
sources: tuple[str, ...],
|
||||||
|
parent_item_id: int | None,
|
||||||
|
items_by_id: dict[int, dict[str, Any]],
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
tokens = {
|
||||||
|
token.casefold()
|
||||||
|
for source in sources
|
||||||
|
for token in re.findall(r"@([A-Za-z0-9_][\w:.-]*)", source or "")
|
||||||
|
}
|
||||||
|
if not tokens:
|
||||||
|
return []
|
||||||
|
by_citekey = {
|
||||||
|
str(item.get("fields", {}).get("citationKey", "")).casefold(): item
|
||||||
|
for item in items_by_id.values()
|
||||||
|
if item.get("fields", {}).get("citationKey")
|
||||||
|
}
|
||||||
|
related = []
|
||||||
|
seen_item_ids = set()
|
||||||
|
for token in sorted(tokens):
|
||||||
|
target = by_citekey.get(token)
|
||||||
|
target_id = target.get("item_id") if target else None
|
||||||
|
if not target or target_id == parent_item_id or target_id in seen_item_ids:
|
||||||
|
continue
|
||||||
|
seen_item_ids.add(target_id)
|
||||||
|
related.append(
|
||||||
|
{
|
||||||
|
"item_id": target_id,
|
||||||
|
"key": target["key"],
|
||||||
|
"citation_key": target["fields"]["citationKey"],
|
||||||
|
"link": _select_link(target),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return related
|
||||||
|
|
||||||
|
|
||||||
def _choose_attachment(
|
def _choose_attachment(
|
||||||
zotero_item: dict[str, Any],
|
zotero_item: dict[str, Any],
|
||||||
override: dict[str, Any] | None = None,
|
override: dict[str, Any] | None = None,
|
||||||
|
|
|
||||||
|
|
@ -7,6 +7,7 @@ import re
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import tempfile
|
import tempfile
|
||||||
|
import unicodedata
|
||||||
import xml.etree.ElementTree as ET
|
import xml.etree.ElementTree as ET
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
@ -126,7 +127,12 @@ def _position_item(
|
||||||
text_pages, item["clipping"]["text"], item["clipping"].get("page")
|
text_pages, item["clipping"]["text"], item["clipping"].get("page")
|
||||||
)
|
)
|
||||||
if page_index is None:
|
if page_index is None:
|
||||||
return _with_problem(item, "pdf-text-not-found")
|
failed = _with_problem(item, "pdf-text-not-found")
|
||||||
|
hint = _page_hint_text(text_pages, item["clipping"].get("page"))
|
||||||
|
if hint:
|
||||||
|
failed = dict(failed)
|
||||||
|
failed["page_hint"] = hint
|
||||||
|
return failed
|
||||||
page_xml = xml_cache.setdefault(
|
page_xml = xml_cache.setdefault(
|
||||||
(extraction_path, page_index),
|
(extraction_path, page_index),
|
||||||
extract_pdf_page_xml(extraction_path, page_index),
|
extract_pdf_page_xml(extraction_path, page_index),
|
||||||
|
|
@ -152,6 +158,8 @@ def _position_item(
|
||||||
annotation["sortIndex"] = position.get("sortIndex")
|
annotation["sortIndex"] = position.get("sortIndex")
|
||||||
updated["annotation"] = annotation
|
updated["annotation"] = annotation
|
||||||
updated["status"] = "positioned"
|
updated["status"] = "positioned"
|
||||||
|
if position.get("partial"):
|
||||||
|
updated["problems"] = [*item.get("problems", []), "partial-text-match"]
|
||||||
return updated
|
return updated
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -227,6 +235,37 @@ def find_pdf_text_page(
|
||||||
normalized_page, _ = _normalize_with_map(pages[index])
|
normalized_page, _ = _normalize_with_map(pages[index])
|
||||||
if normalized_quote in normalized_page:
|
if normalized_quote in normalized_page:
|
||||||
return index
|
return index
|
||||||
|
# Fallback: alphanumeric-only search tolerates PDF extraction artifacts
|
||||||
|
# such as "o f" for "of", hyphenation, and punctuation differences.
|
||||||
|
stripped_quote, _ = _strip_with_map(quote)
|
||||||
|
if stripped_quote and len(stripped_quote) >= 20:
|
||||||
|
for index in page_order:
|
||||||
|
stripped_page, _ = _strip_with_map(pages[index])
|
||||||
|
if stripped_quote in stripped_page:
|
||||||
|
return index
|
||||||
|
# Partial fallback: long highlights may span pages or contain a
|
||||||
|
# divergent middle (edition differences). Anchor to either half so
|
||||||
|
# Zotero-style word search behaviour is approximated.
|
||||||
|
for index in page_order:
|
||||||
|
stripped_page, _ = _strip_with_map(pages[index])
|
||||||
|
if _partial_stripped_hit(stripped_page, stripped_quote):
|
||||||
|
return index
|
||||||
|
# Shingle fallback: scattered OCR substitutions (e.g. Kindle
|
||||||
|
# "yadsınaraz" vs PDF "yadsınamaz") break every long substring but
|
||||||
|
# leave most 6-word runs intact. Rank pages by shingle hits.
|
||||||
|
quote_words = _quote_words(normalized_quote)
|
||||||
|
if len(quote_words) >= 6:
|
||||||
|
required = 2 if len(quote_words) >= 12 else 1
|
||||||
|
best_index: int | None = None
|
||||||
|
best_hits = 0
|
||||||
|
for index in page_order:
|
||||||
|
normalized_page, _ = _normalize_with_map(pages[index])
|
||||||
|
hits = _shingle_hits(normalized_page, quote_words)
|
||||||
|
if hits > best_hits:
|
||||||
|
best_hits = hits
|
||||||
|
best_index = index
|
||||||
|
if best_index is not None and best_hits >= required:
|
||||||
|
return best_index
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -290,25 +329,52 @@ def find_pdf_rects(
|
||||||
text = " ".join(word.text for word in page.words)
|
text = " ".join(word.text for word in page.words)
|
||||||
normalized_text, text_map = _normalize_with_map(text)
|
normalized_text, text_map = _normalize_with_map(text)
|
||||||
start = normalized_text.find(normalized_quote)
|
start = normalized_text.find(normalized_quote)
|
||||||
if start < 0:
|
raw_start: int | None = None
|
||||||
|
raw_end: int | None = None
|
||||||
|
partial = False
|
||||||
|
if start >= 0:
|
||||||
|
end = start + len(normalized_quote)
|
||||||
|
raw_start = text_map[start]
|
||||||
|
raw_end = text_map[end - 1] + 1
|
||||||
|
else:
|
||||||
|
stripped_quote, _ = _strip_with_map(quote)
|
||||||
|
if stripped_quote and len(stripped_quote) >= 20:
|
||||||
|
stripped_text, stripped_map = _strip_with_map(text)
|
||||||
|
strip_start = stripped_text.find(stripped_quote)
|
||||||
|
if strip_start >= 0:
|
||||||
|
strip_end = strip_start + len(stripped_quote)
|
||||||
|
raw_start = stripped_map[strip_start]
|
||||||
|
raw_end = stripped_map[strip_end - 1] + 1
|
||||||
|
else:
|
||||||
|
hit = _partial_stripped_range(stripped_text, stripped_quote)
|
||||||
|
if hit is not None:
|
||||||
|
strip_start, strip_end = hit
|
||||||
|
raw_start = stripped_map[strip_start]
|
||||||
|
raw_end = stripped_map[strip_end - 1] + 1
|
||||||
|
partial = True
|
||||||
|
else:
|
||||||
|
span = _shingle_span(normalized_text, _quote_words(normalized_quote))
|
||||||
|
if span is not None:
|
||||||
|
raw_start = text_map[span[0]]
|
||||||
|
raw_end = text_map[span[1] - 1] + 1
|
||||||
|
partial = True
|
||||||
|
if raw_start is None or raw_end is None:
|
||||||
return None
|
return None
|
||||||
end = start + len(normalized_quote)
|
|
||||||
raw_start = text_map[start]
|
|
||||||
raw_end = text_map[end - 1] + 1
|
|
||||||
matched_words = [
|
matched_words = [
|
||||||
word for word in page.words if word.end >= raw_start and word.start <= raw_end
|
word for word in page.words if word.end >= raw_start and word.start <= raw_end
|
||||||
]
|
]
|
||||||
if not matched_words:
|
if not matched_words:
|
||||||
return None
|
return None
|
||||||
pdf_width, pdf_height = size_cache.setdefault(
|
size_key = (path, page.index)
|
||||||
(path, page.index),
|
if size_key not in size_cache:
|
||||||
_pdf_page_size(path, page.index + 1, page.width, page.height),
|
size_cache[size_key] = _pdf_page_size(path, page.index + 1, page.width, page.height)
|
||||||
)
|
pdf_width, pdf_height = size_cache[size_key]
|
||||||
rects = _word_rects(matched_words, page, pdf_width, pdf_height)
|
rects = _word_rects(matched_words, page, pdf_width, pdf_height)
|
||||||
return {
|
return {
|
||||||
"pageIndex": page.index,
|
"pageIndex": page.index,
|
||||||
"rects": rects,
|
"rects": rects,
|
||||||
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
|
"sortIndex": f"{page.index:05d}|{int(rects[0][1]):06d}|{int(rects[0][0]):05d}",
|
||||||
|
"partial": partial,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -361,6 +427,11 @@ def _pdf_page_size(
|
||||||
|
|
||||||
|
|
||||||
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||||
|
text = unicodedata.normalize("NFKC", text)
|
||||||
|
text = (
|
||||||
|
text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"')
|
||||||
|
.replace("–", "-").replace("—", "-").replace("\xa0", " ").replace("\xad", "")
|
||||||
|
)
|
||||||
normalized = []
|
normalized = []
|
||||||
mapping = []
|
mapping = []
|
||||||
last_was_space = False
|
last_was_space = False
|
||||||
|
|
@ -380,6 +451,112 @@ def _normalize_with_map(text: str) -> tuple[str, list[int]]:
|
||||||
return "".join(normalized), mapping
|
return "".join(normalized), mapping
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_with_map(text: str) -> tuple[str, list[int]]:
|
||||||
|
"""Alphanumeric-only folding with index mapping.
|
||||||
|
|
||||||
|
Removes spaces and punctuation so PDF artifacts like "o f" for "of",
|
||||||
|
hyphenated line breaks, and curly-quote/dash variants still match.
|
||||||
|
Mapping points back into the input string for rect recovery.
|
||||||
|
"""
|
||||||
|
text = unicodedata.normalize("NFKC", text).casefold()
|
||||||
|
stripped: list[str] = []
|
||||||
|
mapping: list[int] = []
|
||||||
|
for index, char in enumerate(text):
|
||||||
|
if char.isalnum():
|
||||||
|
stripped.append(char)
|
||||||
|
mapping.append(index)
|
||||||
|
return "".join(stripped), mapping
|
||||||
|
|
||||||
|
|
||||||
|
def _partial_stripped_range(
|
||||||
|
haystack: str, quote: str, min_len: int = 30
|
||||||
|
) -> tuple[int, int] | None:
|
||||||
|
"""Best half-quote hit for long highlights spanning pages/editions."""
|
||||||
|
if len(quote) < min_len * 2:
|
||||||
|
return None
|
||||||
|
mid = len(quote) // 2
|
||||||
|
for part in (quote[:mid], quote[mid:]):
|
||||||
|
if len(part) < min_len:
|
||||||
|
continue
|
||||||
|
start = haystack.find(part)
|
||||||
|
if start >= 0:
|
||||||
|
return start, start + len(part)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _partial_stripped_hit(haystack: str, quote: str, min_len: int = 30) -> bool:
|
||||||
|
return _partial_stripped_range(haystack, quote, min_len) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _quote_words(normalized: str, width: int = 6) -> list[str]:
|
||||||
|
del width
|
||||||
|
words: list[str] = []
|
||||||
|
for token in normalized.split():
|
||||||
|
cleaned = "".join(char for char in token if char.isalnum())
|
||||||
|
if cleaned:
|
||||||
|
words.append(cleaned)
|
||||||
|
return words
|
||||||
|
|
||||||
|
|
||||||
|
def _shingle_hits(normalized_page: str, quote_words: list[str], width: int = 6) -> int:
|
||||||
|
hits = 0
|
||||||
|
seen: set[tuple[str, ...]] = set()
|
||||||
|
for start in range(len(quote_words) - width + 1):
|
||||||
|
shingle = tuple(quote_words[start : start + width])
|
||||||
|
if shingle in seen:
|
||||||
|
continue
|
||||||
|
seen.add(shingle)
|
||||||
|
if " ".join(shingle) in normalized_page:
|
||||||
|
hits += 1
|
||||||
|
return hits
|
||||||
|
|
||||||
|
|
||||||
|
def _shingle_span(
|
||||||
|
normalized_text: str, quote_words: list[str], width: int = 6
|
||||||
|
) -> tuple[int, int] | None:
|
||||||
|
first: int | None = None
|
||||||
|
last: int | None = None
|
||||||
|
for start in range(len(quote_words) - width + 1):
|
||||||
|
needle = " ".join(quote_words[start : start + width])
|
||||||
|
at = normalized_text.find(needle)
|
||||||
|
if at >= 0:
|
||||||
|
end = at + len(needle)
|
||||||
|
first = at if first is None else min(first, at)
|
||||||
|
last = end if last is None else max(last, end)
|
||||||
|
if first is None or last is None:
|
||||||
|
return None
|
||||||
|
return first, last
|
||||||
|
|
||||||
|
|
||||||
|
def _page_hint_text(
|
||||||
|
pages: list[str], kindle_page: str | None, radius: int = 1, limit: int = 1200
|
||||||
|
) -> dict[str, Any] | None:
|
||||||
|
"""Snippet of the PDF pages around the Kindle page hint.
|
||||||
|
|
||||||
|
Lets the user eyeball which part a `pdf-text-not-found` highlight
|
||||||
|
belongs to and place it manually. Truncated to keep artifacts small.
|
||||||
|
"""
|
||||||
|
if not kindle_page or not kindle_page.isdigit():
|
||||||
|
return None
|
||||||
|
center = int(kindle_page) - 1
|
||||||
|
if center < 0 or center >= len(pages):
|
||||||
|
return None
|
||||||
|
parts = []
|
||||||
|
for index in range(max(0, center - radius), min(len(pages), center + radius + 1)):
|
||||||
|
snippet = " ".join(pages[index].split())
|
||||||
|
if snippet:
|
||||||
|
parts.append(f"[PDF p.{index + 1}] {snippet[:limit]}")
|
||||||
|
if not parts:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"kindle_page": kindle_page,
|
||||||
|
"pdf_pages": list(
|
||||||
|
range(max(1, center - radius + 1), min(len(pages), center + radius + 1) + 1)
|
||||||
|
),
|
||||||
|
"text": "\n\n".join(parts)[: limit * 2 + 64],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
|
def _with_problem(item: dict[str, Any], problem: str) -> dict[str, Any]:
|
||||||
updated = dict(item)
|
updated = dict(item)
|
||||||
updated["status"] = problem
|
updated["status"] = problem
|
||||||
|
|
|
||||||
|
|
@ -34,6 +34,8 @@ class ZoteroItem:
|
||||||
key: str
|
key: str
|
||||||
item_type: str
|
item_type: str
|
||||||
title: str | None
|
title: str | None
|
||||||
|
library_id: int | None = None
|
||||||
|
group_id: int | None = None
|
||||||
creators: list[str] = field(default_factory=list)
|
creators: list[str] = field(default_factory=list)
|
||||||
fields: dict[str, str] = field(default_factory=dict)
|
fields: dict[str, str] = field(default_factory=dict)
|
||||||
attachments: list[ZoteroAttachment] = field(default_factory=list)
|
attachments: list[ZoteroAttachment] = field(default_factory=list)
|
||||||
|
|
@ -47,15 +49,19 @@ def build_zotero_index(db_path: str, storage_root: str | None = None) -> dict[st
|
||||||
creators = _load_creators(connection)
|
creators = _load_creators(connection)
|
||||||
attachments = _load_attachments(connection, storage_root)
|
attachments = _load_attachments(connection, storage_root)
|
||||||
|
|
||||||
|
groups = _load_groups(connection)
|
||||||
indexed_items: list[ZoteroItem] = []
|
indexed_items: list[ZoteroItem] = []
|
||||||
for row in items:
|
for row in items:
|
||||||
fields = _metadata_for_item(row)
|
fields = _metadata_for_item(row)
|
||||||
|
library_id = row["libraryID"]
|
||||||
indexed_items.append(
|
indexed_items.append(
|
||||||
ZoteroItem(
|
ZoteroItem(
|
||||||
item_id=row["itemID"],
|
item_id=row["itemID"],
|
||||||
key=row["key"],
|
key=row["key"],
|
||||||
item_type=row["typeName"],
|
item_type=row["typeName"],
|
||||||
title=fields.get("title"),
|
title=fields.get("title"),
|
||||||
|
library_id=library_id,
|
||||||
|
group_id=groups.get(library_id),
|
||||||
creators=creators.get(row["itemID"], []),
|
creators=creators.get(row["itemID"], []),
|
||||||
fields=fields,
|
fields=fields,
|
||||||
attachments=attachments.get(row["itemID"], []),
|
attachments=attachments.get(row["itemID"], []),
|
||||||
|
|
@ -80,6 +86,7 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
|
||||||
SELECT
|
SELECT
|
||||||
i.itemID,
|
i.itemID,
|
||||||
i.key,
|
i.key,
|
||||||
|
i.libraryID,
|
||||||
it.typeName,
|
it.typeName,
|
||||||
{field_columns}
|
{field_columns}
|
||||||
FROM items i
|
FROM items i
|
||||||
|
|
@ -95,6 +102,14 @@ def _load_parent_items(connection: sqlite3.Connection) -> list[sqlite3.Row]:
|
||||||
return list(connection.execute(query))
|
return list(connection.execute(query))
|
||||||
|
|
||||||
|
|
||||||
|
def _load_groups(connection: sqlite3.Connection) -> dict[int, int]:
|
||||||
|
try:
|
||||||
|
rows = connection.execute("SELECT groupID, libraryID FROM groups")
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
return {row["libraryID"]: row["groupID"] for row in rows}
|
||||||
|
|
||||||
|
|
||||||
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
|
def _load_creators(connection: sqlite3.Connection) -> dict[int, list[str]]:
|
||||||
rows = connection.execute(
|
rows = connection.execute(
|
||||||
"""
|
"""
|
||||||
|
|
|
||||||
15
tests/fixtures/windows-smoke/My Clippings.txt
vendored
Normal file
15
tests/fixtures/windows-smoke/My Clippings.txt
vendored
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
Kanzi Dummy EPUB
|
||||||
|
- Your Highlight on page 1 | location 100-120 | Added on Tuesday, 18 May 2021 14:18:13
|
||||||
|
|
||||||
|
This is a test highlight long enough to import for EPUB.
|
||||||
|
==========
|
||||||
|
Kanzi Dummy PDF
|
||||||
|
- Your Highlight on page 2 | location 200-220 | Added on Tuesday, 18 May 2021 14:20:00
|
||||||
|
|
||||||
|
This is a test highlight long enough to import for PDF.
|
||||||
|
==========
|
||||||
|
Unknown Book Title XYZ
|
||||||
|
- Your Highlight on page 1 | location 10-11 | Added on Tuesday, 18 May 2021 14:18:13
|
||||||
|
|
||||||
|
This should land in Conflicts for manual mapping.
|
||||||
|
==========
|
||||||
36
tests/fixtures/windows-smoke/checklist.txt
vendored
Normal file
36
tests/fixtures/windows-smoke/checklist.txt
vendored
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
Kanzi Windows smoke test (Parallels VM)
|
||||||
|
|
||||||
|
COPY THESE 3 FILES INTO THE VM:
|
||||||
|
- My Clippings.txt
|
||||||
|
- kanzi-dummy.epub
|
||||||
|
- kanzi-dummy.pdf
|
||||||
|
- dist/kanzi.xpi (from repo root, same cross-platform build)
|
||||||
|
|
||||||
|
1) PowerShell:
|
||||||
|
winget install --id Python.Python.3.13 --exact
|
||||||
|
winget install --id oschwartz10612.Poppler --exact
|
||||||
|
winget install --id QPDF.QPDF --exact
|
||||||
|
python --version
|
||||||
|
pdftotext -v
|
||||||
|
pdfinfo -v
|
||||||
|
qpdf --version
|
||||||
|
|
||||||
|
2) Fully quit + reopen Zotero (picks up PATH).
|
||||||
|
|
||||||
|
3) Fresh Zotero, sync off. Create:
|
||||||
|
- Book "Kanzi Dummy EPUB" + attach kanzi-dummy.epub
|
||||||
|
- Book "Kanzi Dummy PDF" + attach kanzi-dummy.pdf
|
||||||
|
|
||||||
|
4) Tools -> Kanzi -> Choose My Clippings.txt (the copied one).
|
||||||
|
Leave Python path empty to test auto-detect.
|
||||||
|
|
||||||
|
EXPECT:
|
||||||
|
- 2 highlights position and await review, 1 title in Conflicts
|
||||||
|
("Unknown Book Title XYZ").
|
||||||
|
- Summary shows new Status by Kindle Title table with source counts.
|
||||||
|
- Save a mapping for the unknown title -> Re-import processes it.
|
||||||
|
- Table headers sort and persist across pages.
|
||||||
|
|
||||||
|
IF PYTHON NOT FOUND:
|
||||||
|
Settings shows detected path hint; paste full python.exe path,
|
||||||
|
Save Settings, retry.
|
||||||
BIN
tests/fixtures/windows-smoke/kanzi-dummy.epub
vendored
Normal file
BIN
tests/fixtures/windows-smoke/kanzi-dummy.epub
vendored
Normal file
Binary file not shown.
33
tests/fixtures/windows-smoke/kanzi-dummy.pdf
vendored
Normal file
33
tests/fixtures/windows-smoke/kanzi-dummy.pdf
vendored
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
%PDF-1.4
|
||||||
|
1 0 obj
|
||||||
|
<< /Type /Catalog /Pages 2 0 R >>
|
||||||
|
endobj
|
||||||
|
2 0 obj
|
||||||
|
<< /Type /Pages /Kids [3 0 R] /Count 1 >>
|
||||||
|
endobj
|
||||||
|
3 0 obj
|
||||||
|
<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>
|
||||||
|
endobj
|
||||||
|
4 0 obj
|
||||||
|
<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>
|
||||||
|
endobj
|
||||||
|
5 0 obj
|
||||||
|
<< /Length 134 >>
|
||||||
|
stream
|
||||||
|
BT /F1 12 Tf 72 720 Td (Kanzi Dummy PDF) Tj ET
|
||||||
|
BT /F1 12 Tf 72 700 Td (This is a test highlight long enough to import for PDF.) Tj ET
|
||||||
|
endstream
|
||||||
|
endobj
|
||||||
|
xref
|
||||||
|
0 6
|
||||||
|
0000000000 65535 f
|
||||||
|
0000000009 00000 n
|
||||||
|
0000000058 00000 n
|
||||||
|
0000000115 00000 n
|
||||||
|
0000000241 00000 n
|
||||||
|
0000000311 00000 n
|
||||||
|
trailer
|
||||||
|
<< /Size 6 /Root 1 0 R >>
|
||||||
|
startxref
|
||||||
|
495
|
||||||
|
%%EOF
|
||||||
93
tests/fixtures/windows-smoke/make_fixtures.py
vendored
Normal file
93
tests/fixtures/windows-smoke/make_fixtures.py
vendored
Normal file
|
|
@ -0,0 +1,93 @@
|
||||||
|
"""Generate minimal EPUB + PDF fixtures for the Windows smoke test.
|
||||||
|
|
||||||
|
Run: python3 tests/fixtures/windows-smoke/make_fixtures.py
|
||||||
|
Outputs kanzi-dummy.epub and kanzi-dummy.pdf next to this script.
|
||||||
|
Both contain the highlight sentences from 'My Clippings.txt' so
|
||||||
|
EPUB CFI and PDF text positioning can succeed.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
import zipfile
|
||||||
|
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
EPUB_TEXT = "This is a test highlight long enough to import for EPUB."
|
||||||
|
PDF_TEXT = "This is a test highlight long enough to import for PDF."
|
||||||
|
|
||||||
|
|
||||||
|
def make_epub(path: Path) -> None:
|
||||||
|
chapter = f"""<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
||||||
|
<head><title>Kanzi Dummy</title></head>
|
||||||
|
<body>
|
||||||
|
<h1>Kanzi Dummy EPUB</h1>
|
||||||
|
<p>Some intro text before the highlight.</p>
|
||||||
|
<p>{EPUB_TEXT}</p>
|
||||||
|
<p>Some trailing text after the highlight.</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
"""
|
||||||
|
container = """<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
|
||||||
|
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
|
||||||
|
</container>
|
||||||
|
"""
|
||||||
|
opf = """<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<package version="3.0" unique-identifier="uid" xmlns="http://www.idpf.org/2007/opf">
|
||||||
|
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:title>Kanzi Dummy EPUB</dc:title><dc:identifier id="uid">kanzi-dummy-epub</dc:identifier><dc:language>en</dc:language></metadata>
|
||||||
|
<manifest><item id="ch1" href="chapter1.xhtml" media-type="application/xhtml+xml"/></manifest>
|
||||||
|
<spine><itemref idref="ch1"/></spine>
|
||||||
|
</package>
|
||||||
|
"""
|
||||||
|
with zipfile.ZipFile(path, "w") as z:
|
||||||
|
z.writestr("mimetype", "application/epub+zip", compress_type=zipfile.ZIP_STORED)
|
||||||
|
z.writestr("META-INF/container.xml", container, compress_type=zipfile.ZIP_DEFLATED)
|
||||||
|
z.writestr("OEBPS/content.opf", opf, compress_type=zipfile.ZIP_DEFLATED)
|
||||||
|
z.writestr("OEBPS/chapter1.xhtml", chapter, compress_type=zipfile.ZIP_DEFLATED)
|
||||||
|
|
||||||
|
|
||||||
|
def _pdf_escape(text: str) -> str:
|
||||||
|
return text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
|
||||||
|
|
||||||
|
|
||||||
|
def make_pdf(path: Path) -> None:
|
||||||
|
# Minimal one-page PDF with Helvetica text. pdftotext extracts it fine.
|
||||||
|
content = (
|
||||||
|
f"BT /F1 12 Tf 72 720 Td ({_pdf_escape('Kanzi Dummy PDF')}) Tj ET\n"
|
||||||
|
f"BT /F1 12 Tf 72 700 Td ({_pdf_escape(PDF_TEXT)}) Tj ET\n"
|
||||||
|
)
|
||||||
|
objects: list[bytes] = []
|
||||||
|
objects.append(b"<< /Type /Catalog /Pages 2 0 R >>")
|
||||||
|
objects.append(b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>")
|
||||||
|
objects.append(
|
||||||
|
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] "
|
||||||
|
b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>"
|
||||||
|
)
|
||||||
|
objects.append(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
|
||||||
|
objects.append(f"<< /Length {len(content.encode('latin-1'))} >>\nstream\n{content}endstream".encode("latin-1"))
|
||||||
|
|
||||||
|
out = bytearray(b"%PDF-1.4\n")
|
||||||
|
offsets = [0]
|
||||||
|
for i, body in enumerate(objects, start=1):
|
||||||
|
offsets.append(len(out))
|
||||||
|
out += f"{i} 0 obj\n".encode("ascii") + body + b"\nendobj\n"
|
||||||
|
xref_pos = len(out)
|
||||||
|
out += f"xref\n0 {len(objects) + 1}\n".encode("ascii")
|
||||||
|
out += b"0000000000 65535 f \n"
|
||||||
|
for off in offsets[1:]:
|
||||||
|
out += f"{off:010d} 00000 n \n".encode("ascii")
|
||||||
|
out += (
|
||||||
|
f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
|
||||||
|
f"startxref\n{xref_pos}\n%%EOF\n"
|
||||||
|
).encode("ascii")
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
make_epub(HERE / "kanzi-dummy.epub")
|
||||||
|
make_pdf(HERE / "kanzi-dummy.pdf")
|
||||||
|
print("wrote kanzi-dummy.epub + kanzi-dummy.pdf")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -28,10 +28,206 @@ def test_pdf_fallback_tag():
|
||||||
"format":"x",
|
"format":"x",
|
||||||
"items":[
|
"items":[
|
||||||
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
|
{"status":"positioned","clipping":{"id":"id1","title":"T","added_on":None,"added_on_iso":None},
|
||||||
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1"},
|
"zotero":{"attachment":{"item_id":1,"key":"K1"},"parent_item_id":10,"parent_key":"P1","citation_key":"c1","related_items":[{"item_id":20,"key":"P2","citation_key":"c2"}]},
|
||||||
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
"annotation":{"type":"highlight","text":"hi","position":{"type":"FragmentSelector","value":"cfi"}}}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
plan = build_final_writer_plan(positioned)
|
plan = build_final_writer_plan(positioned)
|
||||||
tags = plan["annotations"][0]["annotation"]["tags"]
|
tags = plan["annotations"][0]["annotation"]["tags"]
|
||||||
assert any(t["name"].startswith("kindle-id:") for t in tags)
|
assert any(t["name"].startswith("kindle-id:") for t in tags)
|
||||||
|
assert plan["annotations"][0]["related_items"][0]["item_id"] == 20
|
||||||
|
|
||||||
|
|
||||||
|
def test_pdf_tolerates_of_split_artifact():
|
||||||
|
from kindle_zotero_importer.pdf_position import find_pdf_text_page, find_pdf_rects, PDFPageXML, PDFWord
|
||||||
|
quote = "nothing other than idolatry of the machine and which"
|
||||||
|
page_text = "nothing other than idolatry o f the machine and which"
|
||||||
|
assert find_pdf_text_page([page_text], quote) == 0
|
||||||
|
words = []
|
||||||
|
cursor = 0
|
||||||
|
for w in page_text.split():
|
||||||
|
words.append(PDFWord(text=w, left=0.0, top=0.0, width=10.0, height=10.0, start=cursor, end=cursor + len(w)))
|
||||||
|
cursor += len(w) + 1
|
||||||
|
page = PDFPageXML(index=0, width=100.0, height=100.0, words=words)
|
||||||
|
rects = find_pdf_rects("dummy.pdf", page, quote, {("dummy.pdf", 0): (100.0, 100.0)})
|
||||||
|
assert rects is not None
|
||||||
|
assert rects["pageIndex"] == 0
|
||||||
|
assert rects["rects"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_pdf_tolerates_punctuation_and_quotes():
|
||||||
|
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||||
|
quote = "This, then, gives rise to “intemperate technicism”."
|
||||||
|
page = "this then gives rise to intemperate technicism"
|
||||||
|
assert find_pdf_text_page([page], quote) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_pdf_partial_half_anchors_spanning_text():
|
||||||
|
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||||
|
first = "A" * 70
|
||||||
|
second = "Z" * 70
|
||||||
|
quote = first + second
|
||||||
|
page = "xx " + first.lower() + " yy"
|
||||||
|
assert find_pdf_text_page([page], quote) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_pdf_shingle_fallback_scattered_ocr_errors():
|
||||||
|
from kindle_zotero_importer.pdf_position import find_pdf_text_page
|
||||||
|
quote = (
|
||||||
|
"Gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
|
||||||
|
"buna inanmamasım sağlayacak güç ve yeteneği bulur mucize yadsınaraz "
|
||||||
|
"dururda bile olsa boyun eğmez"
|
||||||
|
)
|
||||||
|
page = (
|
||||||
|
"gerçekçi zındıklık yolunu tutuşsa bir mucize görse bile kendinde "
|
||||||
|
"buna inanmaması sağlayacak güç ve yeteneği bulur mucize yadsınamaz "
|
||||||
|
"dururda bile olsa boyun eğmez"
|
||||||
|
)
|
||||||
|
assert find_pdf_text_page([page], quote) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_page_hint_snippet_around_kindle_page():
|
||||||
|
from kindle_zotero_importer.pdf_position import _page_hint_text
|
||||||
|
pages = ["", "intro words here", "target page words here", "trailing words"]
|
||||||
|
hint = _page_hint_text(pages, "3")
|
||||||
|
assert hint is not None
|
||||||
|
assert hint["kindle_page"] == "3"
|
||||||
|
assert "target page words" in hint["text"]
|
||||||
|
assert _page_hint_text(pages, None) is None
|
||||||
|
assert _page_hint_text(pages, "99") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_citekey_mentions_become_zotero_links():
|
||||||
|
from kindle_zotero_importer.import_plan import build_import_plan
|
||||||
|
clippings_payload = {
|
||||||
|
"format": "x",
|
||||||
|
"clippings": [
|
||||||
|
{
|
||||||
|
"id": "c1",
|
||||||
|
"title": "Book A",
|
||||||
|
"kind": "highlight",
|
||||||
|
"text": "As argued by @bookb2020, this follows.",
|
||||||
|
"page": None,
|
||||||
|
"location": "10",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
zotero_index = {
|
||||||
|
"format": "x",
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"item_id": 1,
|
||||||
|
"key": "PARENT01",
|
||||||
|
"title": "Book A",
|
||||||
|
"fields": {"citationKey": "booka2020"},
|
||||||
|
"attachments": [
|
||||||
|
{
|
||||||
|
"item_id": 11,
|
||||||
|
"key": "ATTACH01",
|
||||||
|
"title": "pdf",
|
||||||
|
"content_type": "application/pdf",
|
||||||
|
"path": "x.pdf",
|
||||||
|
"resolved_path": "x.pdf",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"item_id": 2,
|
||||||
|
"key": "TARGET02",
|
||||||
|
"title": "Book B",
|
||||||
|
"fields": {"citationKey": "bookb2020"},
|
||||||
|
"attachments": [],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
match_report = {
|
||||||
|
"format": "x",
|
||||||
|
"matches": [
|
||||||
|
{
|
||||||
|
"clipping_title": "Book A",
|
||||||
|
"status": "matched",
|
||||||
|
"candidates": [{"item_id": 1}],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||||
|
comment = plan["items"][0]["annotation"]["comment"] or ""
|
||||||
|
assert "@bookb2020" in comment
|
||||||
|
assert "zotero://select/library/items/TARGET02" in comment
|
||||||
|
assert plan["items"][0]["zotero"]["related_items"] == [
|
||||||
|
{
|
||||||
|
"item_id": 2,
|
||||||
|
"key": "TARGET02",
|
||||||
|
"citation_key": "bookb2020",
|
||||||
|
"link": "zotero://select/library/items/TARGET02",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
# self-links are skipped
|
||||||
|
clippings_payload["clippings"][0]["text"] = "See @booka2020 here."
|
||||||
|
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||||
|
assert "zotero://select" not in (plan["items"][0]["annotation"]["comment"] or "")
|
||||||
|
assert plan["items"][0]["zotero"]["related_items"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_historical_annotations_gain_citekey_relations():
|
||||||
|
from kindle_zotero_importer.cli import _refresh_history_relations
|
||||||
|
history = [
|
||||||
|
{
|
||||||
|
"clipping_id": "old",
|
||||||
|
"parent_item_id": 1,
|
||||||
|
"annotation": {
|
||||||
|
"text": "An old highlight",
|
||||||
|
"comment": "body without organs @deleuze1983",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
zotero_index = {
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"item_id": 1,
|
||||||
|
"key": "SOURCE01",
|
||||||
|
"fields": {"citationKey": "simondon2017a"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"item_id": 2,
|
||||||
|
"key": "TARGET02",
|
||||||
|
"fields": {"citationKey": "deleuze1983"},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
_refresh_history_relations(history, zotero_index)
|
||||||
|
assert history[0]["related_items"] == [
|
||||||
|
{
|
||||||
|
"item_id": 2,
|
||||||
|
"key": "TARGET02",
|
||||||
|
"citation_key": "deleuze1983",
|
||||||
|
"link": "zotero://select/library/items/TARGET02",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_group_library_targets_use_group_select_links():
|
||||||
|
from kindle_zotero_importer.import_plan import _select_link
|
||||||
|
assert (
|
||||||
|
_select_link({"key": "ABC123", "group_id": 6644344})
|
||||||
|
== "zotero://select/groups/6644344/items/ABC123"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
_select_link({"key": "ABC123", "group_id": None})
|
||||||
|
== "zotero://select/library/items/ABC123"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
_select_link({"key": "ABC123"})
|
||||||
|
== "zotero://select/library/items/ABC123"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_epub_tolerates_extraction_artifacts():
|
||||||
|
from kindle_zotero_importer.epub_position import find_epub_cfi, TextNode
|
||||||
|
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="idolatry o f the machine", start=0, end=24)]
|
||||||
|
quote = "idolatry of the machine and which"
|
||||||
|
# quote is longer than node, so exact fails; use full haystack variant
|
||||||
|
nodes = [TextNode(spine_index=0, cfi_parent_path="/4/2", text_step="1", text="nothing other than idolatry o f the machine and which", start=0, end=53)]
|
||||||
|
result = find_epub_cfi(nodes, quote)
|
||||||
|
assert result is not None
|
||||||
|
assert result["selector"]["value"].startswith("epubcfi(")
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,11 @@ from kindle_zotero_importer import cli
|
||||||
from kindle_zotero_importer.cli import (
|
from kindle_zotero_importer.cli import (
|
||||||
_add_previous_imported_comments,
|
_add_previous_imported_comments,
|
||||||
_changed_note_clipping_ids,
|
_changed_note_clipping_ids,
|
||||||
|
_changed_override_titles,
|
||||||
|
_clipping_ids_for_titles,
|
||||||
|
_deferred_clipping_ids,
|
||||||
_deletion_ids,
|
_deletion_ids,
|
||||||
|
_discarded_clipping_ids,
|
||||||
_merge_annotation_history,
|
_merge_annotation_history,
|
||||||
_merge_positioned_history,
|
_merge_positioned_history,
|
||||||
_unresolved_clipping_ids,
|
_unresolved_clipping_ids,
|
||||||
|
|
@ -78,6 +82,114 @@ def test_deletions_include_removed_and_previous_failures():
|
||||||
) == {"removed", "retry"}
|
) == {"removed", "retry"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_deferred_clippings_are_reprocessed_while_they_remain_in_source():
|
||||||
|
previous = {"deferred_clipping_ids": ["retry", "removed"]}
|
||||||
|
|
||||||
|
assert _deferred_clipping_ids(previous, {"retry", "current"}) == {"retry"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_discarded_clips_never_requeue_and_skips_are_not_unresolved():
|
||||||
|
previous = {"discarded_clipping_ids": ["gone", "removed"]}
|
||||||
|
assert _discarded_clipping_ids(previous, {"gone", "current"}) == {"gone"}
|
||||||
|
|
||||||
|
plan = {
|
||||||
|
"items": [
|
||||||
|
{"clipping": {"id": "note"}, "status": "pdf-position-skipped-non-highlight"},
|
||||||
|
{"clipping": {"id": "fail"}, "status": "pdf-text-not-found"},
|
||||||
|
{"clipping": {"id": "done"}, "status": "positioned"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
assert _unresolved_clipping_ids(plan, {"note", "fail", "done"}) == {"fail"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_alternate_attachment_retry_prefers_pdf_when_epub_misses():
|
||||||
|
from kindle_zotero_importer.cli import _reattempt_failed_with_alternate_attachments
|
||||||
|
from kindle_zotero_importer.epub_position import add_epub_positions
|
||||||
|
from kindle_zotero_importer.import_plan import build_import_plan
|
||||||
|
from kindle_zotero_importer.pdf_position import add_pdf_positions
|
||||||
|
|
||||||
|
fixtures = Path(__file__).parent / "fixtures" / "windows-smoke"
|
||||||
|
epub_path = str(fixtures / "kanzi-dummy.epub")
|
||||||
|
pdf_path = str(fixtures / "kanzi-dummy.pdf")
|
||||||
|
quote = "This is a test highlight long enough to import for PDF."
|
||||||
|
clippings_payload = {
|
||||||
|
"format": "kindle-zotero-importer.clippings.v1",
|
||||||
|
"clippings": [
|
||||||
|
{
|
||||||
|
"id": "q1",
|
||||||
|
"title": "Dummy Book",
|
||||||
|
"kind": "highlight",
|
||||||
|
"text": quote,
|
||||||
|
"page": None,
|
||||||
|
"location": "100",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
zotero_index = {
|
||||||
|
"format": "index",
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"item_id": 1,
|
||||||
|
"key": "ABC12345",
|
||||||
|
"title": "Dummy Book",
|
||||||
|
"fields": {},
|
||||||
|
"attachments": [
|
||||||
|
{
|
||||||
|
"item_id": 11,
|
||||||
|
"key": "EPUBKEY1",
|
||||||
|
"title": "epub",
|
||||||
|
"content_type": "application/epub+zip",
|
||||||
|
"path": epub_path,
|
||||||
|
"resolved_path": epub_path,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"item_id": 12,
|
||||||
|
"key": "PDFKEY123",
|
||||||
|
"title": "pdf",
|
||||||
|
"content_type": "application/pdf",
|
||||||
|
"path": pdf_path,
|
||||||
|
"resolved_path": pdf_path,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
match_report = {
|
||||||
|
"format": "matches",
|
||||||
|
"matches": [
|
||||||
|
{
|
||||||
|
"clipping_title": "Dummy Book",
|
||||||
|
"status": "matched",
|
||||||
|
"candidates": [{"item_id": 1}],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
}
|
||||||
|
plan = build_import_plan(clippings_payload, zotero_index, match_report, None)
|
||||||
|
assert plan["items"][0]["zotero"]["attachment"]["content_type"] == "application/epub+zip"
|
||||||
|
positioned = add_pdf_positions(add_epub_positions(plan))
|
||||||
|
assert positioned["items"][0]["status"] == "positioned"
|
||||||
|
assert "partial-text-match" in positioned["items"][0]["problems"]
|
||||||
|
rescued = _reattempt_failed_with_alternate_attachments(positioned)
|
||||||
|
assert rescued["items"][0]["status"] == "positioned"
|
||||||
|
assert "positioned-via-alternate-attachment" in rescued["items"][0]["problems"]
|
||||||
|
assert "partial-text-match" not in rescued["items"][0]["problems"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_changed_override_requeues_every_clipping_for_exact_title():
|
||||||
|
previous = {"Shared title": {"citation_key": "old"}}
|
||||||
|
current = {"Shared title": {"citation_key": "new"}}
|
||||||
|
clippings = [
|
||||||
|
{"id": "first", "title": "Shared title"},
|
||||||
|
{"id": "second", "title": "Shared title"},
|
||||||
|
{"id": "third", "title": "Different title"},
|
||||||
|
]
|
||||||
|
|
||||||
|
changed_titles = _changed_override_titles(previous, current)
|
||||||
|
|
||||||
|
assert changed_titles == {"Shared title"}
|
||||||
|
assert _clipping_ids_for_titles(clippings, changed_titles) == {"first", "second"}
|
||||||
|
|
||||||
|
|
||||||
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
|
def test_previous_imported_comment_is_attached_to_reprocessed_entry():
|
||||||
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
|
current = [{"clipping_id": "a", "annotation": {"comment": "new"}}]
|
||||||
previous = {
|
previous = {
|
||||||
|
|
@ -240,6 +352,6 @@ def test_noop_incremental_writes_complete_artifacts():
|
||||||
assert result == 0
|
assert result == 0
|
||||||
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
|
final_plan = json.loads((workdir / "import-plan.final.json").read_text())
|
||||||
assert final_plan["annotations"] == []
|
assert final_plan["annotations"] == []
|
||||||
assert final_plan["all_annotations"] == [annotation]
|
assert final_plan["all_annotations"] == [{**annotation, "related_items": []}]
|
||||||
assert final_plan["delta_annotation_count"] == 0
|
assert final_plan["delta_annotation_count"] == 0
|
||||||
assert (workdir / "import-plan.epub.json").exists()
|
assert (workdir / "import-plan.epub.json").exists()
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ const vm = require('vm');
|
||||||
|
|
||||||
const bootstrapContext = {};
|
const bootstrapContext = {};
|
||||||
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
|
vm.runInNewContext(fs.readFileSync('plugin/bootstrap.js', 'utf8'), bootstrapContext);
|
||||||
const plugin = bootstrapContext.KindleZoteroImporter;
|
const plugin = bootstrapContext.Kanzi;
|
||||||
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
|
const maintainImportedAnnotations = plugin.maintainImportedAnnotations;
|
||||||
|
|
||||||
assert.equal(
|
assert.equal(
|
||||||
|
|
@ -96,6 +96,47 @@ const scripts = [...managerHtml.matchAll(/<script>([\s\S]*?)<\/script>/g)];
|
||||||
const managerContext = { TextEncoder };
|
const managerContext = { TextEncoder };
|
||||||
vm.runInNewContext(scripts[0][1], managerContext);
|
vm.runInNewContext(scripts[0][1], managerContext);
|
||||||
const manager = managerContext.KindleManager;
|
const manager = managerContext.KindleManager;
|
||||||
|
assert.equal(plugin.countClippingsForTitle({ clippings: [
|
||||||
|
{ title: 'Shared title' },
|
||||||
|
{ title: 'Shared title' },
|
||||||
|
{ title: 'Different title' },
|
||||||
|
] }, 'Shared title'), 2);
|
||||||
|
|
||||||
|
manager.data = {
|
||||||
|
clippings: { clippings: [
|
||||||
|
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||||
|
{ id: 'n1', title: 'Shared title', kind: 'note', text: 'note' },
|
||||||
|
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||||
|
] },
|
||||||
|
matches: { matches: [{
|
||||||
|
clipping_title: 'Shared title',
|
||||||
|
clipping_count: 3,
|
||||||
|
status: 'matched',
|
||||||
|
candidates: [],
|
||||||
|
}] },
|
||||||
|
generatedOverrides: { overrides: [] },
|
||||||
|
positionedPlan: { items: [{
|
||||||
|
clipping: { id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||||
|
status: 'pdf-text-not-found',
|
||||||
|
problems: ['not found'],
|
||||||
|
}] },
|
||||||
|
finalPlan: { all_annotations: [] },
|
||||||
|
persistentOverrides: { overrides: [] },
|
||||||
|
};
|
||||||
|
const conflictRows = manager.conflictRows();
|
||||||
|
assert.equal(conflictRows.length, 1);
|
||||||
|
assert.equal(conflictRows[0].count, 3);
|
||||||
|
assert.equal(conflictRows[0].clippings.length, 3);
|
||||||
|
assert.equal(conflictRows[0].highlights, 2);
|
||||||
|
assert.equal(conflictRows[0].notes, 1);
|
||||||
|
|
||||||
|
manager.sortState.conflicts = { column: 'count', direction: 'desc' };
|
||||||
|
const sortedRows = manager.sortRows(
|
||||||
|
[{ title: 'small', count: 2 }, { title: 'large', count: 20 }, { title: 'middle', count: 10 }],
|
||||||
|
'conflicts',
|
||||||
|
{ count: row => row.count }
|
||||||
|
);
|
||||||
|
assert.deepEqual(Array.from(sortedRows, row => row.title), ['large', 'middle', 'small']);
|
||||||
manager.integratedPage = 1;
|
manager.integratedPage = 1;
|
||||||
manager.integratedPageSize = 50;
|
manager.integratedPageSize = 50;
|
||||||
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
|
const page = manager.paginate(Array.from({ length: 121 }, (_, index) => index), 'integratedPage', 'integratedPageSize');
|
||||||
|
|
@ -143,6 +184,34 @@ async function testRuntimeExtraction() {
|
||||||
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
|
assert.ok(writes.some(([path]) => path.endsWith('/src/kindle_zotero_importer/cli.py')));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function testPythonPathRecovery() {
|
||||||
|
bootstrapContext.Services = { appinfo: { OS: 'Darwin' } };
|
||||||
|
bootstrapContext.IOUtils = {
|
||||||
|
exists: async path => path === '/opt/homebrew/opt/python@3.13/bin/python3.13',
|
||||||
|
};
|
||||||
|
assert.equal(
|
||||||
|
await plugin.resolvePythonPath('/opt/homebrew/bin/python3'),
|
||||||
|
'/opt/homebrew/opt/python@3.13/bin/python3.13'
|
||||||
|
);
|
||||||
|
bootstrapContext.IOUtils.exists = async path => path === '/custom/python3';
|
||||||
|
assert.equal(await plugin.resolvePythonPath('/custom/python3'), '/custom/python3');
|
||||||
|
|
||||||
|
bootstrapContext.Services.appinfo.OS = 'WINNT';
|
||||||
|
const windowsEnvironment = {
|
||||||
|
LOCALAPPDATA: 'C:\\Users\\test\\AppData\\Local',
|
||||||
|
ProgramFiles: 'C:\\Program Files',
|
||||||
|
'ProgramFiles(x86)': 'C:\\Program Files (x86)',
|
||||||
|
USERPROFILE: 'C:\\Users\\test',
|
||||||
|
SystemDrive: 'C:',
|
||||||
|
PATH: 'C:\\Tools\\Python;C:\\Windows\\System32',
|
||||||
|
};
|
||||||
|
plugin.getEnvironmentVariable = name => windowsEnvironment[name] || '';
|
||||||
|
const expected = 'C:\\Users\\test\\AppData\\Local\\Programs\\Python\\Python313\\python.exe';
|
||||||
|
bootstrapContext.IOUtils.exists = async path => path === expected;
|
||||||
|
assert.equal(await plugin.resolvePythonPath('python'), expected);
|
||||||
|
assert.ok(plugin.getWindowsPythonCandidates('python').includes('C:\\Tools\\Python\\python.exe'));
|
||||||
|
}
|
||||||
|
|
||||||
async function testMaintenanceSweep() {
|
async function testMaintenanceSweep() {
|
||||||
const annotation = (id, text, end, comment = '') => ({
|
const annotation = (id, text, end, comment = '') => ({
|
||||||
id,
|
id,
|
||||||
|
|
@ -315,7 +384,317 @@ async function testReplacementFailurePreservesOldAnnotationAndDeletionRetry() {
|
||||||
assert.equal(results.failed[0].clipping_id, 'replacement');
|
assert.equal(results.failed[0].clipping_id, 'replacement');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function testCitekeyRelationsAreBidirectionalAndAdditive() {
|
||||||
|
const originalZotero = bootstrapContext.Zotero;
|
||||||
|
const originalCollect = plugin.collectImportedAnnotations;
|
||||||
|
const originalMaintain = plugin.maintainImportedAnnotations;
|
||||||
|
const item = (id, key) => ({
|
||||||
|
id,
|
||||||
|
key,
|
||||||
|
related: new Set(),
|
||||||
|
saved: 0,
|
||||||
|
addRelatedItem(other) {
|
||||||
|
if (this.related.has(other.key)) return false;
|
||||||
|
this.related.add(other.key);
|
||||||
|
return true;
|
||||||
|
},
|
||||||
|
async save(options) {
|
||||||
|
assert.equal(options.skipDateModifiedUpdate, true);
|
||||||
|
this.saved += 1;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const source = item(1, 'SOURCE01');
|
||||||
|
const target = item(2, 'TARGET02');
|
||||||
|
bootstrapContext.Zotero = {
|
||||||
|
Items: { get: id => id === 1 ? source : id === 2 ? target : id === 11 ? attachment : null },
|
||||||
|
DB: { executeTransaction: async callback => callback() },
|
||||||
|
logError: () => {},
|
||||||
|
};
|
||||||
|
const entries = [
|
||||||
|
{
|
||||||
|
clipping_id: 'history-1',
|
||||||
|
parent_item_id: 1,
|
||||||
|
attachment_item_id: 11,
|
||||||
|
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
parent_item_id: 1,
|
||||||
|
related_items: [{ item_id: 2, key: 'TARGET02', citation_key: 'target2020', link: 'zotero://select/library/items/TARGET02' }],
|
||||||
|
},
|
||||||
|
];
|
||||||
|
const existingAnnotation = {
|
||||||
|
annotationComment: 'an old note',
|
||||||
|
annotationTags: [{ name: 'kindle-id:history-1' }],
|
||||||
|
saved: 0,
|
||||||
|
saveTx: async function () { this.saved += 1; },
|
||||||
|
};
|
||||||
|
const attachment = { getAnnotations: () => [existingAnnotation] };
|
||||||
|
const results = {
|
||||||
|
relationsAdded: 0,
|
||||||
|
relationsSkipped: 0,
|
||||||
|
relationsPlanned: 0,
|
||||||
|
relationFailures: [],
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
plugin.collectImportedAnnotations = async () => [];
|
||||||
|
plugin.maintainImportedAnnotations = async () => {};
|
||||||
|
const writerResults = await plugin.writeAnnotations({
|
||||||
|
format: 'kindle-zotero-importer.zotero-writer-plan.v1',
|
||||||
|
annotations: [],
|
||||||
|
all_annotations: entries,
|
||||||
|
deletions: [],
|
||||||
|
}, false);
|
||||||
|
assert.deepEqual(Array.from(source.related), ['TARGET02']);
|
||||||
|
assert.deepEqual(Array.from(target.related), ['SOURCE01']);
|
||||||
|
assert.equal(source.saved, 1);
|
||||||
|
assert.equal(target.saved, 1);
|
||||||
|
assert.equal(writerResults.relationsAdded, 1);
|
||||||
|
assert.equal(writerResults.relationFailures.length, 0);
|
||||||
|
assert.equal(writerResults.commentLinksAdded, 1);
|
||||||
|
assert.ok(existingAnnotation.annotationComment.includes(
|
||||||
|
'Related: @target2020 zotero://select/library/items/TARGET02'
|
||||||
|
));
|
||||||
|
assert.equal(existingAnnotation.saved, 1);
|
||||||
|
|
||||||
|
await plugin.addRelatedItemRelations(entries, false, results);
|
||||||
|
assert.equal(source.saved, 1);
|
||||||
|
assert.equal(target.saved, 1);
|
||||||
|
assert.equal(results.relationsSkipped, 1);
|
||||||
|
} finally {
|
||||||
|
bootstrapContext.Zotero = originalZotero;
|
||||||
|
plugin.collectImportedAnnotations = originalCollect;
|
||||||
|
plugin.maintainImportedAnnotations = originalMaintain;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function testPendingReviewGrouping() {
|
||||||
|
const groups = manager.groupPendingAnnotations([
|
||||||
|
{ clipping_id: 'a1', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'spice', pageLabel: '12', color: '#ffd400' } },
|
||||||
|
{ clipping_id: 'a2', clipping_title: 'Dune (Frank Herbert)', citation_key: 'herbert2020', annotation: { text: 'worm', pageLabel: '', color: '#ff6666' } },
|
||||||
|
{ clipping_id: 'b1', clipping_title: 'Unknown Book', citation_key: '', annotation: { text: 'mystery', pageLabel: '', color: '#ffd400' } },
|
||||||
|
]);
|
||||||
|
assert.equal(groups.length, 2);
|
||||||
|
assert.equal(groups[0].items.length, 2);
|
||||||
|
assert.equal(groups[0].citationKey, 'herbert2020');
|
||||||
|
assert.equal(groups[1].items.length, 1);
|
||||||
|
assert.deepEqual(manager.groupPendingAnnotations([]), []);
|
||||||
|
}
|
||||||
|
|
||||||
|
function testFilterPlanForSelection() {
|
||||||
|
const plan = {
|
||||||
|
annotations: [{ clipping_id: 'keep' }, { clipping_id: 'skip' }],
|
||||||
|
all_annotations: [{ clipping_id: 'old' }, { clipping_id: 'keep' }, { clipping_id: 'skip' }],
|
||||||
|
};
|
||||||
|
const previous = { deferred_clipping_ids: ['older'] };
|
||||||
|
const filtered = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep'], previous);
|
||||||
|
assert.deepEqual(Array.from(filtered.annotations, a => a.clipping_id), ['keep']);
|
||||||
|
assert.deepEqual(Array.from(filtered.all_annotations, a => a.clipping_id), ['old', 'keep']);
|
||||||
|
assert.deepEqual(Array.from(filtered.deferred_clipping_ids), ['older', 'skip']);
|
||||||
|
// Original plan is not mutated.
|
||||||
|
assert.equal(plan.annotations.length, 2);
|
||||||
|
const all = plugin.filterPlanForSelection(plan, ['keep', 'skip'], ['keep', 'skip'], previous);
|
||||||
|
assert.equal(all.annotations.length, 2);
|
||||||
|
assert.equal(all.all_annotations.length, 3);
|
||||||
|
assert.deepEqual(Array.from(all.deferred_clipping_ids), ['older']);
|
||||||
|
|
||||||
|
const priorAnnotation = { clipping_id: 'skip', annotation: { text: 'existing' } };
|
||||||
|
const preserved = plugin.filterPlanForSelection(
|
||||||
|
plan,
|
||||||
|
['keep', 'skip'],
|
||||||
|
['keep'],
|
||||||
|
{ all_annotations: [priorAnnotation] }
|
||||||
|
);
|
||||||
|
assert.equal(preserved.all_annotations.find(a => a.clipping_id === 'skip'), priorAnnotation);
|
||||||
|
|
||||||
|
const deferred = plugin.deferPendingPlan(
|
||||||
|
{ all_annotations: [{ clipping_id: 'old' }], deferred_clipping_ids: ['earlier'] },
|
||||||
|
plan,
|
||||||
|
['keep', 'skip']
|
||||||
|
);
|
||||||
|
assert.deepEqual(Array.from(deferred.deferred_clipping_ids), ['earlier', 'keep', 'skip']);
|
||||||
|
assert.deepEqual(Array.from(deferred.all_annotations, a => a.clipping_id), ['old']);
|
||||||
|
}
|
||||||
|
|
||||||
|
function testTitleCountsAreAuthoritative() {
|
||||||
|
manager.data = {
|
||||||
|
clippings: { clippings: [
|
||||||
|
{ id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||||
|
{ id: 'h2', title: 'Shared title', kind: 'highlight', text: 'second' },
|
||||||
|
{ id: 'h3', title: 'Shared title', kind: 'highlight', text: 'third' },
|
||||||
|
] },
|
||||||
|
matches: { matches: [{
|
||||||
|
clipping_title: 'Shared title',
|
||||||
|
clipping_count: 99,
|
||||||
|
status: 'ambiguous',
|
||||||
|
candidates: [{ citation_key: 'shared2020', key: '', item_id: 1, title: 'Shared', score: 0.9 }],
|
||||||
|
}] },
|
||||||
|
generatedOverrides: { overrides: [{
|
||||||
|
clipping_title: 'Shared title',
|
||||||
|
review: { status: 'review', clipping_count: 1 },
|
||||||
|
candidates: [],
|
||||||
|
}] },
|
||||||
|
positionedPlan: { items: [{
|
||||||
|
clipping: { id: 'h1', title: 'Shared title', kind: 'highlight', text: 'first' },
|
||||||
|
status: 'pdf-text-not-found',
|
||||||
|
problems: ['not found'],
|
||||||
|
}] },
|
||||||
|
finalPlan: { all_annotations: [] },
|
||||||
|
persistentOverrides: { overrides: [] },
|
||||||
|
};
|
||||||
|
const rows = manager.conflictRows();
|
||||||
|
assert.equal(rows.length, 1);
|
||||||
|
// Source has 3 clippings; misleading match (99) and override (1) counts must not win.
|
||||||
|
assert.equal(rows[0].count, 3);
|
||||||
|
assert.equal(rows[0].clippings.length, 3);
|
||||||
|
// Only 1 of 3 has position problems.
|
||||||
|
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 3 });
|
||||||
|
|
||||||
|
// Without clippings.json, fall back to conflict counts.
|
||||||
|
manager.data = {
|
||||||
|
clippings: null,
|
||||||
|
matches: { matches: [{
|
||||||
|
clipping_title: 'Lonely title',
|
||||||
|
clipping_count: 5,
|
||||||
|
status: 'ambiguous',
|
||||||
|
candidates: [],
|
||||||
|
}] },
|
||||||
|
generatedOverrides: { overrides: [] },
|
||||||
|
positionedPlan: { items: [] },
|
||||||
|
finalPlan: { all_annotations: [] },
|
||||||
|
persistentOverrides: { overrides: [] },
|
||||||
|
};
|
||||||
|
const fallback = manager.conflictRows();
|
||||||
|
assert.equal(fallback.length, 1);
|
||||||
|
assert.equal(fallback[0].count, 5);
|
||||||
|
}
|
||||||
|
|
||||||
|
function testMappedTargetAndDiscardedClips() {
|
||||||
|
manager.data = {
|
||||||
|
clippings: { clippings: [
|
||||||
|
{ id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' },
|
||||||
|
{ id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' },
|
||||||
|
] },
|
||||||
|
matches: { matches: [{
|
||||||
|
clipping_title: 'Mapped title',
|
||||||
|
clipping_count: 2,
|
||||||
|
status: 'matched',
|
||||||
|
candidates: [{ citation_key: 'mapped2020', key: '', item_id: 7, title: 'Mapped', score: 1 }],
|
||||||
|
}] },
|
||||||
|
generatedOverrides: { overrides: [] },
|
||||||
|
positionedPlan: { items: [
|
||||||
|
{ clipping: { id: 'h1', title: 'Mapped title', kind: 'highlight', text: 'first' }, status: 'pdf-text-not-found', problems: ['pdf-text-not-found'] },
|
||||||
|
{ clipping: { id: 'n1', title: 'Mapped title', kind: 'note', text: 'standalone note' }, status: 'pdf-position-skipped-non-highlight', problems: ['pdf-position-skipped-non-highlight'] },
|
||||||
|
] },
|
||||||
|
finalPlan: { all_annotations: [], discarded_clipping_ids: [] },
|
||||||
|
persistentOverrides: { overrides: [{ clipping_title: 'Mapped title', resolution: { citation_key: 'mapped2020' } }] },
|
||||||
|
};
|
||||||
|
let rows = manager.conflictRows();
|
||||||
|
assert.equal(rows.length, 1);
|
||||||
|
assert.equal(rows[0].mappedTarget, 'mapped2020');
|
||||||
|
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 2, total: 2 });
|
||||||
|
|
||||||
|
manager.data.finalPlan.discarded_clipping_ids = ['n1'];
|
||||||
|
rows = manager.conflictRows();
|
||||||
|
assert.equal(rows.length, 1);
|
||||||
|
assert.equal(rows[0].count, 1);
|
||||||
|
assert.deepEqual(manager.conflictAffected(rows[0]), { failed: 1, total: 1 });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testDiscardClipPersists() {
|
||||||
|
const originalZotero = bootstrapContext.Zotero;
|
||||||
|
let written = null;
|
||||||
|
const plan = { discarded_clipping_ids: [], deferred_clipping_ids: ['x1'] };
|
||||||
|
bootstrapContext.Zotero = {
|
||||||
|
File: {
|
||||||
|
getContentsAsync: async () => JSON.stringify(plan),
|
||||||
|
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const originalArtifactPath = plugin.artifactPath;
|
||||||
|
plugin.artifactPath = rel => `/proj/${rel}`;
|
||||||
|
try {
|
||||||
|
await plugin.discardClip('x1');
|
||||||
|
assert.deepEqual(written.discarded_clipping_ids, ['x1']);
|
||||||
|
assert.deepEqual(written.deferred_clipping_ids, []);
|
||||||
|
} finally {
|
||||||
|
plugin.artifactPath = originalArtifactPath;
|
||||||
|
bootstrapContext.Zotero = originalZotero;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function testTitleOverviewAndSorting() {
|
||||||
|
manager.data = {
|
||||||
|
clippings: { clippings: [
|
||||||
|
{ id: 'a1', title: 'Alpha', kind: 'highlight', text: 'one' },
|
||||||
|
{ id: 'a2', title: 'Alpha', kind: 'highlight', text: 'two' },
|
||||||
|
{ id: 'b1', title: 'Beta', kind: 'highlight', text: 'three' },
|
||||||
|
] },
|
||||||
|
matches: { matches: [
|
||||||
|
{ clipping_title: 'Alpha', clipping_count: 2, status: 'matched', candidates: [{ citation_key: 'alpha2020' }] },
|
||||||
|
{ clipping_title: 'Beta', clipping_count: 1, status: 'unmatched', candidates: [] },
|
||||||
|
] },
|
||||||
|
generatedOverrides: { overrides: [] },
|
||||||
|
positionedPlan: { items: [] },
|
||||||
|
finalPlan: {
|
||||||
|
all_annotations: [{ clipping_id: 'a1', clipping_title: 'Alpha' }],
|
||||||
|
deferred_clipping_ids: ['b1'],
|
||||||
|
},
|
||||||
|
persistentOverrides: { overrides: [] },
|
||||||
|
};
|
||||||
|
const overview = manager.titleOverviewRows();
|
||||||
|
assert.equal(overview.length, 2);
|
||||||
|
const alpha = overview.find(r => r.title === 'Alpha');
|
||||||
|
const beta = overview.find(r => r.title === 'Beta');
|
||||||
|
assert.equal(alpha.integrated, 1);
|
||||||
|
assert.equal(beta.pending, 1);
|
||||||
|
assert.equal(beta.status, 'pending review');
|
||||||
|
|
||||||
|
manager.sortState.titles = { column: 'title', direction: 'desc' };
|
||||||
|
const sorted = manager.sortRows(overview, 'titles', { title: row => row.title });
|
||||||
|
assert.deepEqual(Array.from(sorted, row => row.title), ['Beta', 'Alpha']);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function testSaveOverrideUsesSourceCount() {
|
||||||
|
const originalRead = plugin.readJsonArtifact;
|
||||||
|
const originalArtifactPath = plugin.artifactPath;
|
||||||
|
const originalZotero = bootstrapContext.Zotero;
|
||||||
|
let written = null;
|
||||||
|
plugin.artifactPath = rel => `/proj/${rel}`;
|
||||||
|
plugin.readJsonArtifact = async rel => rel === 'clippings.json'
|
||||||
|
? { clippings: [{ title: 'Shared title' }, { title: 'Shared title' }, { title: 'Shared title' }] }
|
||||||
|
: null;
|
||||||
|
bootstrapContext.Zotero = {
|
||||||
|
File: {
|
||||||
|
getContentsAsync: async () => JSON.stringify({ format: 'kindle-zotero-importer.match-overrides.v1', overrides: [] }),
|
||||||
|
putContentsAsync: async (_path, text) => { written = JSON.parse(text); },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
const entry = await plugin.saveOverride({
|
||||||
|
clipping_title: 'Shared title',
|
||||||
|
clipping_count: 1,
|
||||||
|
resolution: { citation_key: 'shared2020' },
|
||||||
|
review: { status: 'pdf-text-not-found', clipping_count: 1 },
|
||||||
|
});
|
||||||
|
assert.equal(entry.review.clipping_count, 3);
|
||||||
|
assert.equal(written.overrides[0].review.clipping_count, 3);
|
||||||
|
} finally {
|
||||||
|
plugin.readJsonArtifact = originalRead;
|
||||||
|
plugin.artifactPath = originalArtifactPath;
|
||||||
|
bootstrapContext.Zotero = originalZotero;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
testTitleCountsAreAuthoritative();
|
||||||
|
testTitleOverviewAndSorting();
|
||||||
|
testMappedTargetAndDiscardedClips();
|
||||||
|
testPendingReviewGrouping();
|
||||||
|
testFilterPlanForSelection();
|
||||||
|
|
||||||
testRuntimeExtraction()
|
testRuntimeExtraction()
|
||||||
|
.then(testPythonPathRecovery)
|
||||||
|
.then(testSaveOverrideUsesSourceCount)
|
||||||
|
.then(testDiscardClipPersists)
|
||||||
.then(testMaintenanceSweep)
|
.then(testMaintenanceSweep)
|
||||||
.then(testWriterReconciliation)
|
.then(testWriterReconciliation)
|
||||||
.then(testDeletionFailureIsRetried)
|
.then(testDeletionFailureIsRetried)
|
||||||
|
|
@ -323,6 +702,7 @@ testRuntimeExtraction()
|
||||||
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
|
.then(testSameClippingDuplicateKeepsLogicalHistoryAndDeletesSurvivor)
|
||||||
.then(testEnumerationFailurePersistsAllDeletions)
|
.then(testEnumerationFailurePersistsAllDeletions)
|
||||||
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
|
.then(testReplacementFailurePreservesOldAnnotationAndDeletionRetry)
|
||||||
|
.then(testCitekeyRelationsAreBidirectionalAndAdditive)
|
||||||
.then(() => {
|
.then(() => {
|
||||||
console.log('manager and annotation logic ok');
|
console.log('manager and annotation logic ok');
|
||||||
}).catch(error => {
|
}).catch(error => {
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue