Updates July 27
PGrid
- continued work on laptops
- kept iterating on the laptop details and listing page designs, and added a dev-only listing page using real query data
- started work on a one-time agent to fill gaps in laptop data needed for the new pages, including battery-life research
- completed laptop catalog image selection and cutouts
- got agents to generate and visually review cutouts for every selected laptop image; the laptop cutout queue is now drained
- did a lot more work on specs search and classification
- simplified the ssd variant classifier prompt and built an eval harness to compare prompt changes against real past classification decisions
- changed cpu specs search to research the product first, instead of letting a nearby catalog model override its identity
- tightened monitor screen-size research after catching a finalizer using the size from a sibling model
- improved ram model identity handling for official rgb product lines and form factors
- fixed gpu model handling when the same chip exists with different vram capacities
- got codex agents to run specs search and classification on prod across laptops, monitors, ram and gpus
- when they hit blockers, investigated and fixed them with database cleanup or prompt changes before continuing
- this is still ongoing
- re-enabled scheduled classification after fixing a bug with openai token tracking always being used.
- continued work on deals
- added curated text highlights to recent deals
- started replacing noisy price alerts with an ai deal research agent that can decide whether an alert is useful
- fixed deal snapshots not refreshing after a lower price
- continued work on crawler proxies and cloudflare handling
- added surfshark as a new proxy provider alongside nordvpn and purevpn, with australia and us regions
- added source-based proxy routing and dashboard controls so individual crawlers can use the provider and region that works best
- improved cloudflare challenge detection, proxy timeout metrics and b&h photo crawl debugging
- improved crawler auditing and debugging
- unified the audit inputs across retailers and added more detailed debugging for newegg crawls
- added centrecom smartwatch crawling and fixed officeworks product-image handling
- did some operations cleanup
- added a backup/export workflow for the grafana dashboard
- changed make push to route through hmain, offloading deployment work from my laptop
- added helper commands for refreshing codex authentication on the long-running workers
- improved clickhouse refresh performance by batching product slug updates
- added a new python worker. it is currently used from typescript to cut out images so we can have transparent backgrounds
- updated cli to show this
