Skip to content
newsencenewsence Docs

Platform ingestion and rendering

Platform behavior has two owners: Core acquires and normalizes content; Web renders canonical resources. Shared source imports carry the ResourceKind and ResourcePlatform contracts, while each Worker owns its runtime behavior.

Core ingestion

Section titled “Core ingestion”
source cron or saved URL
  -> ingest/acquisition.ts
  -> ResourceProcessingV2Workflow
  -> ingest/platforms/*
  -> canonical resources/resource_translations
  -> AI Search sync

Relevant files:

  • workers/core-worker/src/index.ts: cron dispatch;
  • workers/core-worker/src/ingest/acquisition.ts: saved-URL detection and acquisition routing;
  • workers/core-worker/src/ingest/platforms/: platform-specific durable processing;
  • workers/core-worker/src/ingest/stored-resource.ts: stored-resource reads, row projection, and processability gates;
  • workers/core-worker/src/ingest/resource-persistence.ts: enrichment graph projection into the app-owned DOMAIN.persistResolvedResource transaction sink.

Keep the ingest axes independent:

  • monitored Source.platform: RSS, Twitter, or YouTube;
  • monitored Source.kind: blog or news feed/editorial classification;
  • Resource.kind: blog, forum, post, video, paper, image, or file;
  • Resource.resourcePlatform: Hacker News, Twitter, YouTube, or null;
  • representation: file_type MIME plus optional storage_key;
  • provenance: nullable source_id;
  • user lifecycle: user_resources, optional resource_placements, and minimal user_files raw-blob ownership.

The valid (kind, resourcePlatform) identities are blog / null, forum / hackernews, post / twitter, video / youtube, paper / null, image / null, and file / null. Source.kind remains separate: both blog and news sources render resources through the blog shell, and the Article content-kind filter groups both source kinds. The source kind remains editorial provenance rather than a second presentation identity.

PDF acquisition adds neither a platform nor a PDF-specific kind. Academic enrichment promotes an identified work to paper; an unresolved readable PDF falls back to file with file_type='application/pdf'. HN/Twitter carriers stay carrier Resources and link a separately resolved target through subject_resource_id.

To add a platform:

  1. implement durable processing in ingest/platforms/;
  2. add an acquisition branch when saved URLs should resolve to it;
  3. add only the required cron line when it has a monitored source;
  4. normalize provider facts into common creators, versioned kind metadata, exact identifiers, and an optional immediate subject;
  5. add its locator, valid identity pair, and narrowed kind adapter;
  6. keep profile saves/locations in user_resources / resource_placements, raw-blob ownership in user_files, and monitored provenance in resources.source_id.

There is no Core platform registry to update.

Web rendering

Section titled “Web rendering”

Web receives canonical resource cards from the shared read layer and switches exhaustively on persisted kind. Readers consume canonical creators, kind metadata, identifiers, file facts, and subject facts; they never interpret provider JSON or resolve target URLs.

  • web-tanstack/src/lib/resource/presentation.ts for labels and source policy;
  • web-tanstack/src/components/feed/ResourceCardBody.tsx and cards/ for kind-owned card rendering;
  • web-tanstack/src/components/resource/readers/ for kind-owned detail layout;
  • web-tanstack/src/components/resource/{hackerNewsDiscussion,postPresentation,videoPresentation}.ts for canonical kind metadata projection;
  • web-tanstack/src/lib/resource/file.ts for the few file labels shared by cards and readers.

Card and reader composition is exhaustive over every product kind; resourcePlatform supplies only platform presentation policy. PDF preview is layered on readable-document/paper presentation from the MIME representation. Source.kind remains in the source domain and is intentionally absent from resource presentation DTOs.

Dispatch product presentation only from canonical kind plus optional resourcePlatform.

Verification

Section titled “Verification”
  • Run Core lint/typecheck for every ingest change.
  • Use resyncResource to exercise acquisition → enrichment for an existing resource whose source changed.
  • Use wrangler dev --test-scheduled to exercise cron discovery and batch error isolation; resync does not cover cron.
  • Tail logs and compare enrichment_status='failed' counts before and after production smoke tests.