Updates October 13

PGrid

• updated self hosted vpn based proxies to use dynamic ports and self register in the database. allows multiple proxies running on given hosts.
  • this was a large change in how the proxies worked and required all proxies to self register and connect to the db. i deployed this without testing and ran into issues with db connections. after fixing added new feature flag to allow disabling all crawl jobs.
  • updated proxy ip refreshing to be more scattered as had some issues when refreshing all proxy ips at the same time. suspect nordvpn erroring when trying to log in 7 times closely
• new e2e test for ebay single page scraping. found out from ozbargain that ebay single page coupon parsing was broken. fixed and then verified using the e2e test which the ai picked up correctly.
• updates to internal cli for displaying new proxy details as well as displaying job queue and allowing clearing of specific jobs in queue using the cli
• fixed cloudflare turnstile click coordinates. scrapers now pass it more often
• little play around with ai and threejs to generate an ascii animation for the logo for future redesign. unsure if it will be used.
• added region specific browser context folder. so when a worker uses a usa vpn, it will use a usa specific browser context and reuse it between requests.
• big rewrite of ssd classification again. starting with model stage classification, this time im working much closer with ai to try to keep tech debt low and the architecture configurable this took the rest of the week. ended up changing models to use gpt5 api for model classification reviews.