feat(endpoint-microsub): PR 1 - Core Microsub server with channels and timeline

[rmdes]

Summary

This PR adds the foundational Microsub endpoint with channel and timeline management.

What's included

Microsub API:

• GET/POST ?action=channels - list, create, update, delete, reorder channels
• GET/POST ?action=timeline - list items, mark read/unread, remove

Storage:

• MongoDB collections for channels and items
• Cursor-based pagination for timeline
• Per-user channel ordering and read state tracking

Features:

• Follows Microsub spec for channel and timeline actions
• Testable with existing Microsub clients (Monocle, Indigenous, etc.)
• Multi-user support via userId from session/token

How to test

1. Enable the plugin in your Indiekit config
2. Use a Microsub client like Monocle pointed at your /microsub endpoint
3. Or test via curl:

# List channels
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://your-site.com/microsub?action=channels"

# Create a channel
curl -X POST -H "Authorization: Bearer YOUR_TOKEN" \
  -d "action=channels&name=Tech News" \
  "https://your-site.com/microsub"

# Get timeline (will be empty until feed fetching is added in PR 3)
curl -H "Authorization: Bearer YOUR_TOKEN" \
  "https://your-site.com/microsub?action=timeline&channel=CHANNEL_UID"

PR breakdown

This is PR 1 of 6 for the Microsub implementation:

• PR 1 (this): Core + Channels + Timeline API ← functional Microsub server
• PR 2: Feed discovery and subscription
• PR 3: Feed fetching and parsing
• PR 4: Reader UI
• PR 5: Compose and Micropub integration
• PR 6: Settings and filtering

---

Co-Authored-By: Claude Opus 4.5 noreply@anthropic.com

[paulrobertlloyd]

Hi @rmdes. Firstly, sorry it’s taken me so long to get back to you on this; life got in the way. I should hopefully now have some time to review/collaborate on this feature.

I’ve pushed two extra commits:

1. Added the plug-in to the development config so that when running the application locally the plugin is available; once the plugin is completed, we can possibly make it a plugin that ships by default with Indiekit, and then not need to add it here.
2. Added an icon for the plug-in, using the icon for Microsub.

Before merging this into main, might we be able to do the following:

1. Add unit and integration tests (see other endpoints for examples)
2. Run and fix issues reported by npm run lint (I recently updated eslint’s rules so there are a few new errors)

Thanks again for your patience, looking forward to getting this contribution into Indiekit!

[rmdes]

Thanks @paulrobertlloyd, and no problem about the delay. I’ve pushed three commits covering both points, plus one fix that writing the tests uncovered.

Two things are worth calling out, as neither is obvious from the diff.

mongodb version mismatch (d6499142)

The plug-in declared mongodb: ^6.0.0 while @indiekit/indiekit declares ^7.4.0. npm hoisted 7.4.0 to the root and installed a nested 6.21.0 under packages/endpoint-microsub/node_modules, so lib/storage/items.js resolved its ObjectId import to bson 6 while the collections it operates on are served by the host’s bson 7 driver.

Any query carrying a plug-in-created ObjectId then failed to serialise:

BSONVersionError: Unsupported BSON version, bson types must be from bson 7.x.x

This affected markItemsRead, markItemsUnread and removeItems. All three build ObjectIds from the entry IDs in the request:

const objectIds = entryIds
  .map((id) => {
    try {
      return new ObjectId(id);
    } catch {
      return;
    }
  })
  .filter(Boolean);

It matters in practice because transformToJf2 exposes _id to clients as a string, so a Microsub client marking items as read sends those IDs straight back and lands on this path. Construction succeeds; the failure happens at serialisation when the query is sent.

The fix is just to declare the same major as the host, which removes the nested copy (package-lock.json shrinks accordingly). Happy to drop the dependency entirely instead if you’d rather the plug-in never imported mongodb directly — ObjectId is the only thing it uses.

init() now awaits index creation (3d5f66c6)

This one is a genuine behaviour change rather than a pure lint fix, so flagging it explicitly.

Index creation was deliberately fire-and-forget:

// Create indexes for optimal performance (runs in background)
if (indiekit.database) {
  createIndexes(indiekit).catch((error) => {
    console.warn("[Microsub] Index creation failed:", error.message);
  });
}

unicorn/prefer-await flags the .catch() chain, and I couldn’t satisfy it without awaiting, so init() is now async and initialisation waits for the indexes:

if (indiekit.database) {
  try {
    await createIndexes(indiekit);
  } catch (error) {
    console.warn("[Microsub] Index creation failed:", error.message);
  }
}

The plug-in loader already does await plugin.init(Indiekit) (packages/indiekit/lib/plugins.js:22), so this is within the existing contract, and the try/catch keeps the original “warn, don’t crash” behaviour if the database is unreachable. It also removes a race where timeline queries could run before the indexes existed. The trade-off is that start-up now blocks on index creation. If you’d prefer to keep it in the background, an inline disable with a comment would be the alternative — your call.

Tests

Unit tests cover lib/utils and lib/storage, in a directory structure mirroring lib/ (as packages/indiekit/test/unit/middleware does). Controllers are covered by the integration tests instead, matching endpoint-auth, endpoint-media and endpoint-micropub. 135 tests in total, and npm run lint is clean for the package.

One note on the failing check: it’s the Localazy step failing on a missing readKey, which can’t pass on a pull request from a fork since secrets aren’t exposed. I don’t think there’s anything I can do about that from here.

[paulrobertlloyd]

On the MongoDB version mismatch, @indiekit/util exports a getObjectId method for this very reason.

Right now the MongoDB version used is specified in @indiekit/util and @indiekit/indiekit. I wonder if it might make sense to move the getMongodbClient method currently used in @indiekit/indiekit to @indiekit/util. That would then mean Indiekit’s MongoDB version is specified in a single package/location.

Am happy to do that as a separate PR if you think that’d help?

[rmdes]

getObjectId is exactly what was needed,

thanks, I’d missed it. Pushed a commit that uses it in lib/storage/items.js and lib/utils/pagination.js, so the plug-in no longer imports or declares mongodb at all and @indiekit/util owns the version.

That supersedes the earlier commit that bumped the pin by hand. Matching the version manually would have worked until the next bump moved it again; not declaring the dependency means there’s nothing left to drift.

On moving getMongodbClient into @indiekit/util: I think that’s worth doing, though it wouldn’t have prevented this particular problem getObjectId already covered it, I just wasn’t using it. The case for it is that mongodb is currently declared in two places, and they have already diverged slightly:

packages/indiekit/package.json   ^7.5.0
packages/util/package.json       ^7.4.0

Both resolve to the same install today since they’re caret ranges on the same major, so nothing is broken.

But it’s the same shape of mismatch, and consolidating would leave one place to get it right. Happy either way, and happy to review if you do open that PR.

Both of your original points are now addressed:

• Tests - 101 unit tests covering lib/utils and lib/storage, in a directory structure mirroring lib/, plus 34 integration tests covering the endpoint, channel and timeline actions, invalid actions and unauthenticated access. Controllers are covered by the integration tests, as in the other endpoint plug-ins.
• Lint - npm run lint is clean for the package.

The failing check is the Localazy step, which needs a readKey secret and so can’t pass on a pull request from a fork.

[rmdes]

Rebased this locally onto main and it applies cleanly — only
package-lock.json conflicted, resolved by regenerating it.

Before I push it, a question: feat/microsub is now 32 commits behind main
and has no commits of its own, so should this retarget to main instead?
Happy either way — I'd just rather not hand you drift to untangle later.

Edited to correct the second half of this comment. I originally wrote that
I couldn't get a clean local test run. That was my mistake: I had not set
PASSWORD_SECRET, which build.yml sets alongside SECRET, and without it
the session-backed tests redirect and fail. With it, endpoint-microsub on the
rebased branch is 135 tests, 135 passing, and main is green across the
packages I checked.

So there is nothing outstanding here from my side. Apologies for the noise —
the failures I was seeing were entirely my own environment.

[paulrobertlloyd]

I have a few questions, none of which need to be resolved now.

Before I push it, a question: feat/microsub is now 32 commits behind main and has no commits of its own, so should this retarget to main instead?

Let stick to the plan, and have this set of PRs merge into feat/microsub for now. I have rebased that branch with main, but you might need to resolve a conflict before merging this PR.

[paulrobertlloyd]

Possibly one for a separate PR, but I wonder if this work could be moved into @indiekit/util and used for pagination on @indkitkit/endpoint-micropub and @indiekit/endpoint-media too. I’d expect all three endpoints to paginate in the same way, and I think there’s a bit of overlap here with the existing getCursor utility in @indiekit/util.

[rmdes]

Worth doing, though not a direct swap: getCursor keys on _id, and Microsub sorts by published time, so these cursors encode {timestamp, id} to break ties between items published in the same second. A shared version would need to handle both orderings.

Happy to take it as a separate PR.

[paulrobertlloyd]

Brill. No rush, so maybe we can create an issue for this for now, and revisit once this plugin is nearer being released?

[rmdes]

Also rebased onto feat/microsub now you've brought it up to date — applies cleanly, 135 tests passing.

[paulrobertlloyd]

The failing check is the Localazy step, which needs a readKey secret and so can’t pass on a pull request from a fork.

For the next PR, as you are now a member of the getindiekit organisation, you should be able to create branches directly on this repo. That will then mean the readKey secret will be available and tests no longer fail 🤞

[rmdes]

The failing check is the Localazy step, which needs a readKey secret and so can’t pass on a pull request from a fork.

For the next PR, as you are now a member of the getindiekit organisation, you should be able to create branches directly on this repo. That will then mean the readKey secret will be available and tests no longer fail 🤞

Thanks — though it looks like I don't have push access yet: I'm an active org member, but not a collaborator on the repo (permissions.push is false). Org membership on its own doesn't grant it, so a team would need write access on the repo, or me adding directly. Happy to test once you've had a chance.

• Org membership: active, role member ✓
• Repo collaborator: no
• permissions.push: false

[paulrobertlloyd]

@rmdes I have no understanding of the byzantine roles and permissions model used by GitHub, but I think you should now have write access.

[rmdes]

Confirmed, write access works — thanks. I'll branch directly here for the next one so CI can actually run the tests.