Zotero: group-wide export, CLI export, and per-collection routing

.specify/specs/zotero-export-and-routing.md

@@ -0,0 +1,305 @@
+# Zotero: group-wide export, CLI export, and per-collection routing
+
+**Issue**: [con/docflow#2](https://github.com/con/docflow/issues/2)
+
+**Date**: 2026-07-14
+**Status**: In progress (branch `enh-zotero`)
+**Scope**: parts (1)–(3) fully; parts (4)–(5) implemented at a level that
+covers the concrete use-case in the issue, with room for follow-ups.
+
+## Summary
+
+Close the "one command from Zotero to `refs.bib`" loop for grant-proposal
+and paper workflows. Three gaps in the current `ZoteroClient` /
+`docflow zotero` CLI:
+
+1. `export_bibtex()` exists in Python but has no CLI subcommand.
+2. `export_bibtex()` requires a `collection_key`, so exporting the whole
+   library (`/groups/{id}/items`) is not supported.
+3. `format=bibtex` in Zotero caps responses at 100 items per request; a
+   library >100 refs is silently truncated.
+
+Additionally: `docflow zotero push` targets a single collection, and the
+generated Makefile has no target for regenerating `refs.bib`.
+
+## Design
+
+### 1. `ZoteroClient.export_bibtex()` — group-wide + paginated + formats
+
+Signature changes:
+
+```python
+def export_bibtex(
+    self,
+    collection_key: str | None = None,
+    output_path: Path | None = None,
+    format: str = "bibtex",         # bibtex | csljson | ris
+) -> str:
+    ...
+```
+
+Behaviour:
+
+- `collection_key=None` → export the whole library (`self.zot.items(...)`).
+- `collection_key=KEY` → export that collection (existing behaviour, but
+  now using the same paginated helper).
+- `format` is passed straight through as the Zotero `format=` parameter.
+- Result is the concatenation of all pages.
+- `output_path` still writes the result to disk.
+
+**Pagination.** Zotero caps `format=bibtex` / `format=ris` responses at
+100 items per request; a library >100 items is silently truncated
+without explicit pagination.  We bypass `pyzotero` entirely for the
+export path and talk to `https://api.zotero.org` directly via
+`urllib.request` (`_zotero_api_get` low-level helper, `_paginated_export`
+generator).  Two reasons:
+
+1. `pyzotero` returns a **parsed** `BibDatabase` object (from
+   `bibtexparser`) for `format=bibtex`, not a string.  Concatenating
+   these across pages requires re-serialising via `bibtexparser.dumps`,
+   which introduces formatting drift.
+2. `pyzotero` does not surface the `Total-Results` response header,
+   which we use to decide when to stop paginating.
+
+The loop:
+
+```python
+def _paginated_export(self, endpoint: str, format: str) -> Iterator[str]:
+    start = 0
+    total: int | None = None
+    while True:
+        params = {"format": format, "limit": "100", "start": str(start)}
+        url = f"{ZOTERO_API_BASE}{endpoint}?{urlencode(params)}"
+        req = urllib.request.Request(url)
+        req.add_header("Zotero-API-Version", "3")
+        req.add_header("Authorization", f"Bearer {self.api_key}")
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            yield resp.read().decode("utf-8")
+            if total is None:
+                total = int(resp.headers.get("Total-Results", "0"))
+        start += 100
+        if total <= 0 or start >= total:
+            return
+```
+
+Result assembly: `bibtex` and `ris` are plain-text pages joined with a
+blank-line separator; `csljson` pages are parsed and merged into a
+single JSON array which is then pretty-printed.
+
+### 2. `docflow zotero export` (CLI)
+
+```
+docflow zotero export [--collection KEY] [--format bibtex|csljson|ris]
+                      [--output PATH]
+```
+
+- Defaults: `--format=bibtex`, output → stdout, no collection (whole
+  group).
+- Zotero credentials come from `.docflow/config.yaml` when present, or
+  from env vars when not (see "Config-less operation" below).
+- If a config is present and `zotero.collection_key` is set, that value
+  is used when `--collection` is omitted.  Pass `--collection ""` to
+  force whole-group export even when a config default exists.  Without
+  a config, the default is whole-library export.
+
+### 3. Per-collection routing on `docflow zotero push`
+
+Two routing paths, both optional and composable:
+
+**(a) Per-citation `collection` field.** Any citation loaded from an
+input JSON that carries a `collection` key routes to that collection
+regardless of the CLI `--collection` flag. Extend `Citation` to carry an
+optional `collection` field (defaults to `None`), and thread it through
+`push_items` / `check_items`. The CLI accepts either the historical
+schema (`{"citations": [{"type": ..., "value": ...}, ...]}`) or the new
+one with an added `"collection"` field per entry.
+
+**(b) `--rules RULES.yaml`.** A tiny YAML:
+
+```yaml
+default_collection: ABCD1234
+routes:
+  - match: {tag: biosketch}
+    collection: EFGH5678
+  - match: {doi_prefix: "10.1016/j.cell"}
+    collection: EFGH5678
+```
+
+Rules are evaluated top-to-bottom; the first matching rule wins. Match
+predicates supported in v1:
+
+- `tag`: the citation has this tag (see below).
+- `doi_prefix`: the citation is a DOI and starts with this prefix.
+
+Citations don't currently carry tags. We piggy-back on the extraction
+side: comment blocks can annotate citations with `#tag` tokens in
+context text, and the citation-JSON writer emits them as a
+`tags: []` list per citation. In v1 the pathway is: **users author
+`{doi, tag}` (or `{doi, collection}`) JSON directly**; comment-derived
+tagging is a follow-up.
+
+Fallback order per citation:
+1. `citation.collection` if set on the input.
+2. First matching rule from `--rules` (if given).
+3. `default_collection` from `--rules` (if given).
+4. `--collection` CLI flag.
+5. None → item lands in library root (existing behaviour).
+
+`push_items` gains a per-citation resolution step (`_resolve_collection`).
+The push loop iterates **one citation at a time** and calls
+`zot.create_items([item])` per citation with the resolved collection
+attached to the item's `collections` field.  Batching multiple items
+into a single `create_items` call is a valid future optimisation but is
+not implemented in v1 — pyzotero handles small write batches
+transparently, and the deduplication path (`check_items` DOI cache)
+already avoids redundant reads.  The result dict reports at the
+citation level.
+
+### 4. `docflow zotero add-dois`
+
+Ergonomic loader for the common "here's a DOI list, put them in Zotero"
+flow, so users don't have to hand-craft citation-JSON:
+
+```
+docflow zotero add-dois [--collection KEY] [--dry-run] [FILE]
+```
+
+- Reads DOIs from FILE (or stdin if omitted).
+- One DOI per line; blank lines and `#`-prefixed comment lines skipped.
+- Each line becomes a `Citation` whose type is `BIORXIV` if the DOI
+  begins with `10.1101/` (so version-suffix normalisation `.v1`/`.v2`
+  applies), else `DOI`.
+- Delegates to `client.push_items(...)`.
+
+The `--rules` / per-item routing does not apply here (add-dois is
+deliberately dumb); users who need routing should compose their own
+citation-JSON and use `push`.
+
+### 5. Makefile template
+
+Add a `refs.bib` target and a `bib` phony alias.  The template is
+**always emitted** by `docflow init` (regardless of whether
+`--zotero-group-id` was passed); the target is gated at *runtime* by
+the same `ZOTERO_API_KEY` env-var check used by `zotero-check` /
+`zotero-push`, so it prints an actionable message and exits 1 when the
+key is missing rather than silently doing nothing.
+
+```makefile
+.PHONY: bib
+bib: refs.bib
+
+refs.bib:
+	@if [ -z "$$ZOTERO_API_KEY" ]; then \
+	  echo "Error: ZOTERO_API_KEY not set"; \
+	  exit 1; \
+	fi
+	docflow zotero export --output "$@"
+```
+
+`bib` is also included in the `all:` recipe:
+
+```makefile
+all: sync md tsv comments zotero-check bib
+```
+
+## Config-less operation
+
+None of the `docflow zotero` subcommands (`check`, `push`, `export`,
+`add-dois`) require a `.docflow/config.yaml`.  In env-only mode:
+
+- `ZOTERO_API_KEY` (or `DOCFLOW_ZOTERO_API_KEY`) — the only strictly
+  required variable.
+- `ZOTERO_GROUP_ID` (or `DOCFLOW_ZOTERO_GROUP_ID`) — optional.  If unset,
+  the library is **auto-discovered** by calling Zotero's
+  `/keys/current` endpoint:
+    - If the key grants access to exactly one library (user or group),
+      that library is used.
+    - If multiple groups are accessible **and** a `--collection` is
+      given on the CLI, each accessible group is probed for that
+      collection; the matching one is used.
+    - Otherwise `LibraryResolutionError` is raised, listing accessible
+      libraries so the caller can set `ZOTERO_GROUP_ID` explicitly.
+- `ZOTERO_MODE` — optional, defaults to `group`; set to `personal` for
+  user libraries (with `ZOTERO_GROUP_ID` = user ID).
+
+`export`'s "config default collection" behaviour (picking up
+`zotero.collection_key` when `--collection` is omitted) applies only
+when a config file is present; without one, the default is
+whole-library export.
+
+## Push mechanics: single-call item creation
+
+When routing a citation to a destination collection, `push_items` sets
+the item's `collections` field **at creation time** (part of the
+`create_items` payload).  It does **not** call `addto_collection`
+afterwards.  Reason: `addto_collection` requires the server-assigned
+`key` / `version` / `data.collections` fields of the *created* item —
+none of which exist on the local template dict.  Assigning the
+collection at creation time is atomic (no window during which the item
+exists outside any collection) and needs one API call instead of two.
+
+## Exit codes
+
+`docflow zotero push` and `docflow zotero add-dois` **must exit with a
+non-zero status code (1) whenever at least one citation failed to
+push**.  A partial success (some added, some failed) is not a success;
+downstream Makefiles and CI must be able to detect the failure without
+scanning stdout.  The failure list is still printed to stdout for the
+user.  Dry-run mode never fails on this basis (nothing was actually
+attempted).
+
+## Non-goals (this branch)
+
+- Comment-driven tagging → citation → `--rules {tag: ...}` end-to-end.
+  Left as a follow-up; v1 assumes the user provides JSON with either
+  explicit `collection` or a routable field.
+- Bulk BibTeX ↔ Zotero round-trip.
+- CSL-JSON output for `docflow zotero push`.
+- Retries / rate-limit backoff on Zotero API (relies on `pyzotero`
+  defaults).
+
+## Test plan
+
+Unit tests (mocked `pyzotero.Zotero`):
+
+- `export_bibtex(collection_key=None)` calls `zot.items()`, not
+  `zot.collection_items()`.
+- `export_bibtex` paginates: mock two consecutive pages of length 100
+  and a third of length <100; verify result is the concatenation and
+  the loop stopped.
+- `export_bibtex(format="csljson")` returns a JSON string that parses
+  to a list of the mocked dicts.
+- `export_bibtex(format="ris")` passes `format=ris` through.
+- Config default: `ZoteroConfig(collection_key=...)` is picked up when
+  CLI `--collection` is omitted; `--collection ""` overrides to
+  whole-group.
+- Per-citation `collection` field routes correctly and overrides the
+  CLI flag.
+- `--rules` matcher: `doi_prefix` and `tag` each route a citation to
+  the expected collection; `default_collection` catches unmatched.
+- `add-dois` reads DOIs from stdin, drops blank/`#` lines, and
+  produces a plausible push_items call.
+- `add-dois` classifies `10.1101/*` DOIs as `BIORXIV` (so version
+  suffixes are normalised on dedup).
+- `docflow init` writes a Makefile containing the `refs.bib` target
+  (regardless of whether `--zotero-group-id` was supplied) and lists
+  `bib` in the `all:` recipe.
+
+CLI tests:
+
+- `docflow zotero export --help` shows the new options.
+- `docflow zotero export --format bibtex` (no collection) prints to
+  stdout.
+- `docflow zotero export --output /tmp/out.bib` writes the file and
+  is silent on stdout.
+- `docflow zotero add-dois --help` shows options.
+
+All new tests marked `@pytest.mark.ai_generated`.
+
+## Rollout
+
+- No config-schema breaking changes; `ZoteroConfig.collection_key` was
+  already optional.
+- `Citation` gains an optional field with a `None` default — safe.
+- `export_bibtex(collection_key, ...)` remains callable positionally
+  with the same argument as before; only additional kwargs are new.