Updates June 2

Skipped 2 weeks due to european trip

PGrid

• Ebay intermittent errors causing more and more issues. There are times where queue size doesn’t decrease and all workers get stuck resulting in the next batch of ebay scraping to be queued. From debugging logs added to ebay scraper, suspect the issue is coming from initial bot check that potentially does a redirect and breaks scraper. Added a timeout for that section of the code
• Updated next js from 14 to 15. Resulted in some errors like mantine packages module resolution errors and nextjs not allowing imports from shared directory. Fixed and started using pnpm.
• Updated dockerfiles to use pnpm
• Started getting out of memory errors for some cpu listing queries on clickhouse when doing frontend builds. This would cause some cpu pages to not have any listings on them. Updated the base get listings query to be simpler.
• New gpus: AMD Radeon RX 9060 XT 16GB and 8GB. First time splitting a gpu into its 16 and 8gb versions, classification seems to work ok.
• Upgraded worker docker image to use pnpm, upgraded playwright versions
• Fixed scraper issue with MSY where it wouldn’t use pagination