docs(SDK-899): add RFC for autogenerated API docs

[mariechatfield]

Summary

Add an RFC to autogenerate API docs based on inline comments. This is a complementary workstream to #1507, which sets up the infrastructure pipeline for a docs site based on markdown files.

This proposal outlines the strategy to improve our inline comment coverage, which can be used for tooltips and other intellisense features in IDEs and to generate high-quality markdown documentation.

Related

• Jira ticket: SDK-899

[jeffredodd]

This looks great! Thanks for putting this together 🙏

[jeffredodd]

We should maybe enforce the requirement of docs perhaps at some point so we don't forget to write the docs.

[mariechatfield]

Ah forgot to mention it in the top-level approach but that is where the beautiful eslint plugins come in! We can set them up to error and fail the build if you don't add documentation to an exported symbol. Unfortunately, it can't tell the difference between "I exported this from a /shared directory to use in another function" versus "I exported this from a barrel file so we can export it for public usage" so it will yell at every export without documentation

But that's what we can use the @internal tag for (to indicate this really is private implementation detail only) along with an allowlist / blocklist to let us roll out the lint rules incrementally

[serikjensen]

One thing worth thinking about with this: where are TSDoc comments enforced? Would it just be on any publicly exported component/function/hook/utility? or all regardless of export?

[mariechatfield]

This is the sample config I used in my spike branch. It does mean every export from a source file will need a comment to pass linting (it doesn't care about what gets exported to the consumer).

const jsdocRequireOptions = {
  // Only require JSDoc for exports
  publicOnly: true,
  require: {
    FunctionDeclaration: true,
    ClassDeclaration: true,
    ArrowFunctionExpression: true,
    FunctionExpression: true,
    MethodDefinition: true,
  },
  contexts: [
    // Require JSDoc for exported variables (often constants)
    'ExportNamedDeclaration > VariableDeclaration',
    // and for class properties (methods enabled via the `require` block)
    'PropertyDefinition',
    // and for types, interfaces, and enums
    'ExportNamedDeclaration > TSTypeAliasDeclaration',
    'ExportNamedDeclaration > TSInterfaceDeclaration',
    'ExportNamedDeclaration > TSEnumDeclaration',
  ],
  // If enabled, --fix will simply add an empty jsdoc comment
  enableFixer: false,
}

Here's a sample of errors that it would currently generate:

src/.../useEmployeeDetailsForm.tsx
   44:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export interface EmployeeDetailsSubmitCallbacks
   58:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export type UseEmployeeDetailsFormProps
   62:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export interface EmployeeDetailsFields
   72:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export interface UseEmployeeDetailsFormReady
  109:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export function useEmployeeDetailsForm
  309:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export type UseEmployeeDetailsFormResult
  310:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export type EmployeeDetailsFieldsMetadata
  311:8  warning  Missing JSDoc comment  jsdoc/require-jsdoc  →  export type EmployeeDetailsFormFields

✖ 8 problems (0 errors, 8 warnings)

This is why the incremental strategy is important (and also so that as I start adding JSDoc via the skills I'm planning, if we realize it's too much bloat, we can adjust the rules as we go)

[jeffredodd]

Love this idea too

[jeffredodd]

Interesting idea. I think we will only want to release docs on version bumps so this isn't a bad idea. I think it all depends on the time cost to run these scripts often. 🤔

[mariechatfield]

Theoretically if we use the @alpha tags when we're working on something that's fully in progress it shouldn't ever change the docs until you promote something to @beta or @public:
https://api-extractor.com/pages/tsdoc/doc_comment_syntax/#release-tags

If we do want to generate the actual docs much less frequently, I think it may be worth adding the API extractor back into the mix, in part because it does do a lot of additional heavy validation. The only diff that would be included in each PR at that point is the type interface and any validation errors (including "this wasn't documented at all.")

For example, if you have a @beta function use an @alpha type for props, this will log an error: https://api-extractor.com/pages/messages/ae-incompatible-release-tags/

[serikjensen]

This is looking amazing! thanks for the diligence here

Couple of questions

• Do we have a separate effort ticketed to autogenerate events? That may be part of an additional effort to type the onEvent calls correctly. But events are a big part of our public contract and really hard to maintain. That would definitely be more complex to document, but should be possible with tsMorph and the onEvent prop being passed consistently
• Do we have a separate effort ticketed to autogenerate translations? It would be nice to consider how we autogenerate all translations available for a given component and show usage examples. Another stretch here would be figuring out how to best show where the translations would map to the ui component so partners know what to override

Left a couple of comments on the RFC as well

[serikjensen]

One thing worth thinking about with this: where are TSDoc comments enforced? Would it just be on any publicly exported component/function/hook/utility? or all regardless of export?

[serikjensen]

Couple of thoughts here, i worry that having guides layer separate from the component docs might make it easier to forget about the corresponding guide file for any given block or workflow. I've had it in the past where we place a readme file alongside the component source which works as the guide. I also feel like the guide and the api docs should be located next to each other in the end documentation even if they live separately in the source. Ex. having one tab for the guide and usage and another for the props tables

[mariechatfield]

I completely agree about guides and API living alongside each other in the docs! I did an audit of a bunch of the docs sites we've said we like, and I think Chakra UI does it best:

Docs
├── Get Started
│   ├── Overview
│   ├── AI for Agents
│   ├── Frameworks
│   └── Environments
├── Components
│   ├── Concepts          ← guide prose: "how to use `as`/`asChild`", "how color
│   │                       mode works", "how to test" — no Props tables
│   ├── Layout            ← component ref: ExampleTabs + Props table per file
│   ├── Typography           (box.mdx, flex.mdx, grid.mdx…)
│   ├── Buttons
│   ├── Date and Time
│   ├── Forms
│   ├── Collections
│   ├── Overlays
│   ├── Disclosure
│   ├── Feedback
│   ├── Data Display
│   ├── Internationalization
│   └── Utilities
├── Charts
│   ├── Overview
│   └── Charts
├── Styling
│   ├── Concepts          ← guide prose: "how responsive design works",
│   │                       "how dark mode works", "what cascade layers are"
│   ├── Compositions
│   └── Style Props       ← reference tables: one page per CSS property group
└── Theming
    ├── Concepts          ← guide prose: "what tokens are", "how recipes work"
    ├── Design Tokens     ← reference: token values per category
    ├── Compositions
    └── Customization

In particular, having each "group" start with Concepts or Overview for embedded guide content, followed by the relevant API reference content for that group. Agree that if we can have the source for that doc live closer to the code it's documenting is even better

[serikjensen]

I think i'm following on this, the FormData and FormOutputs would definitely be good to generate correctly. fieldValidators is an implementation detail and if i'm reading this right wouldn't be documented (which would be preferable)

[mariechatfield]

If you look at our dist/ files today, fieldValidators is declared as a const because we need it to build the type for EmployeeDetailsFormData:

declare const fieldValidators: {
    firstName: z.ZodString;
    middleInitial: z.ZodString;
    lastName: z.ZodString;
    email: z.ZodEmail;
    dateOfBirth: z.ZodISODate;
    ssn: z.ZodString;
    selfOnboarding: z.ZodBoolean;
};
export type EmployeeDetailsFormData = {
    [K in keyof typeof fieldValidators]: z.infer<(typeof fieldValidators)[K]>;
};

The ts-morph proposal would be to go through the dist and replace it inline with the text version, which is the same thing that displays if you hover over that value in an IDE:

type EmployeeDetailsFormData = {
    firstName: string;
    middleInitial: string;
    lastName: string;
    email: string;
    dateOfBirth: string;
    ssn: string;
    selfOnboarding: boolean;
}

fieldValidators right now would end up being referenced in any generated documentation because it's referenced in the d.ts we generate

[serikjensen]

oh interesting. Are there zod types that wouldn't map cleanly to TS types?

[mariechatfield]

I don't think so, I think this is specifically a side effect of how we're mapping the data based on zod infer so we don't have to declare them twice?

[mariechatfield]

@serikjensen updated this to show a sample of the kind of TSDoc we'd generate based on markdown. If the prose or example content gets too big, we can also use the @include or @includeCode tags in typedoc to pull in markdown from sibling files (best of both worlds in terms of nice markdown formatting for guides, without adding too much text to source code files)

[mariechatfield]

More details in this section to cover:

• general type surface improvements to make it easier to document (not just the zod field validators, but more broadly)
• moving guide content closer to the source code it documents
• even more autogenerated documentation, including an audit of which of our current autogen docs could be migrated to a type-based solution (component adapter props, theming variables)