DigesterDigester

Category Digests

The Digests view is a category × time-window matrix. A colored cell is the newest saved result — click it to read the briefing, or check it to regenerate. Select missing cells, pick up to 20 combinations, and choose Generate selected.

How a digest is built

• LM Studio first condenses the matching posts in bounded batches, then writes one source-grounded briefing from those summaries.
• Each section links back to its supporting posts; relevant saved images and videos appear with the original source cards.
• The queue is created immediately and processed sequentially, keeping exactly one LM Studio request active at a time.
• Empty combinations are reported and skipped. A failure in one digest does not stop the rest of the queue.
• Completed digests are stored in SQLite and stay available across restarts.
• A chat model must be loaded in LM Studio. Semantic embeddings are not required for this feature.

Digests stay deliberately category-specific so each saved briefing has one stable category identity. The single-category generator remains available for quick one-off briefings, and the All posts toolbar has a Digest matrix shortcut.

Project Intelligence

This is the feature that makes a saved-post archive actionable: pick one of your local Codex projects and ask what the last week of saved posts actually means for that codebase.

Profiling a project

• Digester reads the project name and primary folder from Codex's local project registry — there are no manual metadata fields to fill in.
• LM Studio infers concise focus notes from the project's current evidence.
• A scan inventories readable code, documentation and recent Git work. It skips secrets, dependencies, caches, generated output, model weights and large or binary files.
• The resulting implementation profile cites the project-relative files that support each capability or limitation.
• Malformed local-model JSON is repaired when possible, retried once in a compact form, and then safely recovered from validated LM-generated component notes.

Impact digests

• Generate an impact digest for today, three days, a week, a month, six months or a year — optionally narrowed to one post category.
• The report only keeps developments supported by both saved posts and project evidence.
• It contains the current implementation, a capability comparison, pros, cons and unknowns, recommendations, source URLs and media, project file evidence, and prioritized next steps.
• Colored topic tags, expandable implementation cards and report sections, and an On this page index keep large profiles navigable. Open/closed sections stay stable while background progress arrives, and Expand all / Collapse all sit at the top.

Deep analysis

Click a topic tag, choose Deep analyze on an implementation or finding, or type a topic into Explore a topic. This rereads the project folder — not just the stored profile — searches within files for topic-specific source excerpts, and retrieves matching saved posts and comments through the existing RAG search. Compatible embeddings enable hybrid retrieval; keyword search works without an embedding model. This searches the whole saved library, independent of the impact digest's date and category filters.

A topic report includes implementation comparisons, pros and cons, concrete evaluation steps, original posts and media, and expandable source excerpts. Reports are saved in the project's digest history and are reachable through the read-only MCP tools.

Coverage and retrieval-fallback warnings appear in the saved report. Deep analysis does not run project code, execute benchmarks, or verify social-media claims.

Automation: Scheduled Crawling

The selected collections are re-read on a fixed interval — the default is X Bookmarks every 30 minutes — using the same saved browser session as a manual crawl. No developer API keys are involved, and nothing is posted, liked or followed.

• A cycle is skipped, with the reason recorded on the Automation page, when the platform is not signed in or another crawl, indexing, categorization or digest job is already running. A skipped cycle never leaves a failed entry in Activity.
• Runs appear in Activity as a parent automatic row plus one row per collection, so a scheduled crawl is never mistaken for a manual one.
• The interval, the collections and an optional per-run post limit are configurable. A blank limit uses the crawl limit from Settings.
• The next run time is shown on the page and the schedule survives restarts: a restart does not repeat a crawl that already ran inside the interval.

Automation: The GPU Idle Analyser

Digest generation competes with your own models for the same card, so Digester waits for a genuinely idle GPU instead of a fixed time of day. It reads nvidia-smi directly — it never loads a model or runs a benchmark to probe the cards.

Why it watches the right cards

• LM Studio runs inference in a separate llama-server backend process, and one model can be split across several cards. The analyser follows those inference processes to the cards they hold and watches that whole set.
• The LM Studio.exe desktop process is deliberately ignored: it is Electron and holds a graphics context on whichever card the window is drawn on — usually not an inference card at all. On the machine this was verified on, that distinction is the difference between watching the RTX 5090 and watching the RTX 4080 + RTX 5080 that actually run LM Studio.
• Nothing is guessed. If the LM Studio cards cannot be identified, the page says so and no digest is generated.
• A failed nvidia-smi reading is never counted as idle time, so a driver problem pauses digest generation instead of triggering a burst of it.
• While the window is still filling, the page reports how many samples have been collected out of the expected number, and no digest starts.

Selecting what an idle run may produce

Digests to create while idle takes either every category — including categories you add later — or a named subset, plus the time windows. Each selected category/window pair becomes one digest, created sequentially in the order shown, with at most one LM Studio request active at a time. Empty windows are skipped silently rather than reported as failures.

Both behaviours are independent and either can be paused without affecting the other. Both can also be triggered on demand with Crawl now and Generate now, which ignore the schedule and the GPU rule so a configuration can be verified immediately. Without a chat model configured, an automatic run is reported as skipped instead of failing repeatedly.

Storage & Exports

All mutable data is ignored by Git and stored below data/:

data/
  ├── local-lore.db
  ├── browser-profiles/
  │   ├── x/
  │   └── reddit/
  ├── session-cookies/
  │   ├── x.json
  │   └── reddit.json
  └── media/
      ├── x/
      └── reddit/

The database uses normalized items, item_collections, categories, item_categories, category_suggestion_jobs, digests, projects, project_digests, comments, media, chunks, item_workspace and saved_views tables. Embeddings live alongside chunks as JSON vectors, so the archive stays self-contained.

• Export current view writes one JSON object per line with stable fields for RAG pipelines.
• /api/export?format=markdown provides a human-readable alternative.
• The JSONL importer understands both Digester exports and compatible normalized records. Re-import restores notes, stars and read state.
• Selected posts can be exported as Markdown or JSONL from the bulk-action toolbar.
• JSONL is a record export, not a complete backup. It contains media references, not media files, settings, saved views or account cookies — keep a separate backup of the data directory if you need one.

Activity & Job Handling

• The Activity view keeps a local history of every operation, with status and text filters over the latest 30 entries.
• Clear failures removes all failed entries at once; × removes an individual completed or failed row.
• Active and queued operations cannot be removed. Queued collections start sequentially.
• If the service restarts during a job, that abandoned job is marked failed on the next startup instead of sitting under In progress forever.

Under the Hood

A small Express service with a dependency-free browser UI — no build step, no frontend framework, no bundler.

Built on Express 5, better-sqlite3, Playwright, cheerio, ipaddr.js, zod and the official MCP SDK. Checks and tests run with npm run check and npm test.

Source Code

Digester lives in a self-hosted GitLab repository. It is private and invite-only — the link will not open without an account that has been granted access.

https://gitlab.aikanova.com/nerdpraxis/Twitter_Reddit_verarbeiter