Updates July 13

PGrid

• worked with ai to investigate and reduce the thousands of ebay and amazon single-page crawls being queued each day
  • found scheduled stale crawls were the main source of most queue spikes, with marketplace search jobs also contributing
  • updated stale amazon and ebay single-page jobs to recheck snapshot freshness just before crawling and skip if another job had refreshed the item while it was queued
  • reduced repeat ebay work by crawling listings that had been out of stock for more than 3 days at most once a day
  • fixed ebay listings with the wrong currency being repeatedly queued
• went through the current specs search rules and consolidated the scattered docs into agent skills for writing and running specs searches
• added a cli command to allow agents to preview and add deals to the website. it can also rotate an existing deal for the same product
• continued work on laptops
  • iterated with fable on new designs for the laptop details page. still not satisfied with them.
  • ran laptop variant specs searches against prod and found reseller-upgraded laptops weren’t being handled correctly. this also exposed inconsistent ways for agent specs searches to finish without creating a catalog row, so unified the outcomes across laptop, monitor, gpu and ram workflows
  • added laptop listing crawls for msy, mwave and ple, and raised the laptop family search-term alert threshold from 100 to 150 as the catalog grew
• created a reusable finalization-agent pattern that runs after the research agent to independently verify and normalise its result, replacing the old baml verification and normalisation steps in agent mode. added it to monitor models, laptop variants, gpu variants, ram models and ram variants
• centralised specs search workflow config so the cli and workers use the same mode for each category and stage, then made agent mode the default everywhere it was supported
• did more specs search cli and runner cleanup: shared model selection across agents, clearer client and run reporting, and consistent saving of research links
• found an item with an empty title during classification. updated the generic item pipeline to stop saving new items without titles, added the same validation for replacement ebay listings and cleaned up the affected prod rows
• added single-page crawling and item enrichment support for:
  • scorptec
  • mwave
  • pc case gear
  • msy
  • ple
• moved ai cleanup of long listing descriptions into a database-backed queue with retries, so item enrichment and crawls no longer wait for the llm call to finish
• expanded browser debugging across all browser-backed worker tasks and updated the cli so agents can inspect local and prod captures, including events, timelines, frames and how old each frame was
• updated the gpu list table to group models by series, show visual bars for vram and listing counts and make out-of-stock rows easier to distinguish
• continued work on the reddit pc-building knowledge base. expanded the component and build-planning pages and updated the ingestion and synthesis prompts to make wiki edits more consistent
• misc infrastructure and cleanup
  • optimised clickhouse daily rebuilds, removed an unused aggregate field and fixed clickhouse refreshes after audit reclassification
  • added hosthatch3 worker capacity and ci reporting for backend and worker image sizes
  • deleted the old unused classification frontend