Analyze Typefully posts into reusable social-process docs

[alexeygrigorev]

Analyze Typefully posts into reusable social-process docs

Status: pending
Tags: research, process-docs, docs, P2
Depends on: None
Blocks: None

Scope

Research-and-analysis task: download the real published social posts from the
DataTalksClub and Al_Grigor Typefully accounts via the Typefully API,
analyze them, and produce draft reusable process documents that capture how
DataTalks.Club actually runs social posts. The deliverable is draft process
docs, not a production code feature.

This is a read/analysis task (Typefully → DataOps). It is independent of #77,
which is a write integration (DataOps → Typefully drafts). The two go in
opposite directions and share no code path; #78 does not depend on #77.

Reusable tooling already exists and is committed to main:

• scripts/typefully/export-posts.mjs — discovers social sets, exports draft
  details, and writes normalized posts.jsonl / posts.csv rows for X and
  LinkedIn. Reads TYPEFULLY_API_KEY from .env/env; never prints the token.
• scripts/typefully/analyze-posts.mjs — reads an export and produces an
  analysis.md plus draft process docs.
• scripts/typefully/README.md — usage.

The engineering work is therefore mostly: run the real export against the two
accounts, then refine the analysis output into genuinely reusable process docs.

Data handling (hard rules)

• All downloaded Typefully data stays under .tmp/ (gitignored). Exports can
  contain private Typefully URLs, draft text, and unpublished drafts.
• .env holds TYPEFULLY_API_KEY and the social-set ids; it is confirmed in
  .gitignore (.env, .env.*, !.env.example) and must never be committed.
• The API key, bearer header, or any secret-like value must never appear in
  scripts, analysis output, process-doc drafts, issue comments, or logs.
• The user explicitly asked not to commit in this pass. Draft process docs
  are produced under .tmp/ first. Promoting any derived doc into
  content/social-media/ is a separate, reviewed step (see Out of Scope and
  Acceptance Criteria).

Acceptance Criteria

• Export the published posts for both DataTalksClub and Al_Grigor
  using scripts/typefully/export-posts.mjs, writing results to
  .tmp/typefully-export/ only.
• Run scripts/typefully/analyze-posts.mjs (or an equivalent refinement)
  against the export and produce a set of draft reusable process docs
  under .tmp/ that capture recurring patterns: post categories (course
  launches, event/webinar announcements, community prompts, newsletter
  teases, etc.), structure/hooks per platform, posting cadence, and X vs.
  LinkedIn differences.
• Each draft process doc is structured to fit the existing
  content/social-media/ shape (title, summary, when-to-use, audience,
  required inputs, example patterns distilled from real posts) so it can be
  reviewed for promotion later.
• .env remains gitignored; verify no Typefully token, bearer value,
  private/share URL, or secret appears in any analysis output or draft doc.
• No committed changes in this issue. Confirm git status shows no
  .tmp/ export, .env, or raw Typefully data staged or committed.
• [HUMAN] A DataOps operator runs the real export using the real
  TYPEFULLY_API_KEY (real external API call + real secret). The agent may
  run the analysis over data the operator has already exported.
• [HUMAN] Before any draft doc is promoted to content/social-media/, an
  operator reviews it against the public/private knowledge boundary (see
  AGENTS.md: content/ is transitional migration debt; operational SOPs
  may belong in the private DataTalksClub/dataops-knowledge repo).

Test Scenarios

Scenario: Export and analyze run end to end

Given: .env contains a valid TYPEFULLY_API_KEY and social-set ids for
DataTalksClub and Al_Grigor
When: the operator runs export-posts.mjs then analyze-posts.mjs
Then: normalized post rows and draft process docs are written under .tmp/
only, and no error reveals the token.

Scenario: No secret leakage

Given: a completed export and analysis under .tmp/
When: the draft docs and analysis files are inspected
Then: no TYPEFULLY_API_KEY, bearer token, or private Typefully URL appears in
any file that could be committed.

Scenario: Nothing committed

Given: the analysis deliverables exist under .tmp/
When: git status is checked
Then: no Typefully export, .env, or raw Typefully data is staged or committed.

Scenario: Output is review-ready for content promotion

Given: the draft process docs under .tmp/
When: an operator reviews them
Then: each doc maps to an existing content/social-media/ category and is ready
for the [HUMAN] review that decides public content/ vs. private knowledge repo.

Out of Scope

• Committing anything. The user asked not to commit in this pass.
• Promoting derived docs into content/ — that is a separate, reviewed step.
• Building a new analysis pipeline; scripts/typefully/ already exists.
• The Create Typefully drafts from DataOps Assistant jobs #77 write integration (creating Typefully drafts from the assistant).
• Typefully analytics import, webhook ingestion, or media/tag management.
• Modifying source repos (../dtc-operations, ../datatasks, ../podcast-assistant).

Dependencies

• A valid TYPEFULLY_API_KEY and the target social-set ids for DataTalksClub
  and Al_Grigor (present in .env locally). The real export is a [HUMAN]-run,
  real-external-API step; the agent works over data the operator has exported.
• LinkedIn analytics are not available through the tested Typefully endpoint
  (X-only), so LinkedIn conclusions come from draft content, not engagement
  metrics.
• Independent of Create Typefully drafts from DataOps Assistant jobs #77. This issue can proceed as soon as the operator runs the
  export; it is not blocked on any open issue.

[alexeygrigorev]

Local Typefully extraction/analysis status from the orchestrator:

• Confirmed .env is ignored by .gitignore; no token was printed or committed.
• Saved reusable scripts under scripts/typefully/:
  • export-posts.mjs discovers Typefully social sets, exports draft details, and writes normalized posts.jsonl / posts.csv rows for both X/Twitter (platform=x) and LinkedIn (platform=linkedin) when present.
  • analyze-posts.mjs creates analysis.md plus draft process docs from an export.
  • README.md documents usage.
• Local ignored exports are under .tmp/ only.
• Verified export for DataTalksClub and Al_Grigor published posts: 124 DataTalksClub drafts, 487 Al_Grigor drafts, 2,175 normalized platform-post rows total.
• Normalized platform row counts from the verification export: 1,600 X rows and 575 LinkedIn rows. X has more rows because Typefully threads contain multiple X posts per draft.
• Typefully analytics endpoint was tested separately; it currently supports X analytics only and rejects LinkedIn analytics, so LinkedIn content comes from draft details, not analytics.

No commit was made.

[alexeygrigorev]

Groomed. Decisions recorded in the spec:

• Scope: research/analysis task — run the existing scripts/typefully/ export over the real DataTalksClub + Al_Grigor published posts, then refine the analysis output into draft reusable process docs under .tmp/.
• Tooling: reusable scripts already exist and are committed (c26be2c); this is mostly running the real export + refining draft docs, not building a new pipeline.
• Data handling: all exports stay in .tmp/; .env confirmed gitignored; no token/URL leakage; user asked not to commit, so promotion into content/ is deferred and gated by [HUMAN] review.
• Dependency call: not blocked on Create Typefully drafts from DataOps Assistant jobs #77. Create Typefully drafts from DataOps Assistant jobs #77 is a write integration (DataOps → Typefully drafts); Analyze Typefully posts into reusable social-process docs #78 is a read/analysis task. Opposite directions, no shared code path.
• [HUMAN]: real export run (real secret + real external API call), and review of any doc before it moves into content/.

Labels: research, process-docs, docs, P2. Ready for assignment once an operator runs the real export.

[alexeygrigorev]

Export + analysis completed

Ran the real Typefully export and analysis from the committed scripts using the .env API key:

Export (.tmp/typefully-export/, gitignored):

• datatalksclub-drafts.json (752KB)
• al-grigor-drafts.json (2.7MB)
• ai-shipping-labs-drafts.json (8KB)
• Consolidated all-drafts.json, posts.jsonl, posts.csv

Analysis (.tmp/typefully-analysis/, gitignored):

• analysis.md — full analysis of posting patterns
• 6 draft process docs under process-doc-drafts/:
  • create-course-promotion-posts.md
  • create-newsletter-resource-posts.md
  • create-podcast-follow-up-posts.md
  • create-sponsored-partner-posts.md
  • create-workshop-event-posts.md
  • social-post-patterns-from-typefully.md

Verification:

• git status is clean — no committed changes.
• .env remains gitignored.
• No Typefully token, bearer value, or credential appears in any output file.

The automated criteria are met. The remaining [HUMAN] checks (operator review of the export before promotion to content/social-media/) remain.