Updates October 6

PGrid

• created prompts and scripts to test amazon usa and amazon aus “change location using modal” functionality. done through using aus proxy for amazon usa and vice versa. used to fix minor issue with amazon usa change location functionality
• shipped a health check for each proxy service. they all ping ip.pgrid.com.au (which is a cf worker) to check if the proxy is working and kills if 3 failed requests (15 mins).
• enabled internal ssd scraping on amazon usa
• found issue with amazon usa scraping where the proxy times out. added a check for network issues page on chrome and refresh.
• added config and tests for amazon usa gpu scraping. started scraping.
• migrated aus amazon ssd and cpu scraping to use the new sharded amazon scraping.
• added makefile for grafana backups using grafanactl cli. put backups into git
• updated generic ebay scrapers to take screenshot configs to be controlled both by feature flags and by custom functions for ai e2e testing
  • updated ai e2e testing prompt and script for ebay to now use screenshot config as well as be more explicit on whats required. tested with codex as well
• added grafana charts to show snapshot information per source per category. useful during transition of amazon/ebay aus scraping to use sharded scraping. as well as for usa sites coming online
  • this helped see that amazon usa gpu data wasn’t being saved. after investigation issue was related to migration script between old gpu card table and generic item table. the script deleted orphaned generic item gpus where there wasn’t a corresponding gpucard anymore. disabled this feature
• wired up ebay sharded generic item search crawling. started with cpu and ssd for ebay usa. fixed some minor stuff while doing this and when it ran for the first time.
• fixed amazon single page scraping. added end to end prompts and test scripts for ai agents to run. made the location changing a bit more robust both for usa and aus. found title parsing was broken for single page scraping and fixed that.