Updates September 8

PGrid

• Did a bunch more SSD variant verification runs. Updated Claude review prompt and now it seems OK to review what the AI verification does. Claude also uses Prisma MCP tools to make sure values are in correct formats. Fixed some formatting issues there.
• Found there was a bug for Amazon GPU scraping where a timeout would cause the scraping run to end without saving any data. Fixed.
• Minor reliability fixes for some scrapers.
• Postgres database is getting quite large now. Briefly investigated extracting parts of database into DuckDB files but not much further work done here.
• Created a new prompts folder in the project which will have prompts for Claude Code to be able to do integration-style manual testing. The first of these is testing eBay scraping by running the scraper, getting it to take screenshots and comparing the screenshots with the scraper output. Seems to be working well and a good solution to make sure scrapers are functional.
  • Added Makefile to this folder to be able to run interactive Claude Code sessions but on specific prompts.
• Found JW data wasn’t being scraped. Added a JW Claude Code scraper test prompt but the scraper selectors were OK. Turns out JW was blocking my scrapers. Found that changing user agent got around the block.
  • After getting past the blocked issue, found my old Cloudflare bypass was no longer working. Tried to do some debugging for this but not successful yet.
• New feature flag to control screenshot uploading for each crawler.
• Changed browser to Brave. After some basic testing, it didn’t help with Amazon or Cloudflare issues, caused some unknown browser crashes and caused large spikes in CPU. So reverted to Chromium.
• Reinstalled the OS for HostHatch2 server as for some reason it had much higher job failure rates than all other servers and seems to just get stuck sometimes. The reinstall didn’t seem to help.
• More CLI commands.
• Lots of time spent trying to debug why Amazon proxy isn’t working that well but not solved yet.