Updates July 21
PGrid
- Saw there were very consistent failures of scraping eBay GPUs. From investigating, realised the Nord proxy services all had a memory leak. Restarted the proxy services but one kept erroring out. After investigating further it was potentially due to missing docker permissions on both nomad server config and deployment config files but that didnt fix it. More investigation led to finding NordVPN Linux client was updated to remove a specific
ipv6 command that was failing.
- Created a script to clear specific distributed tasks from the queue to help prevent the backlog becoming too much for current workers to handle.
- Have been observing a lot of Amazon GPUs being classified recently with up to 100 GPUs in one day being classified. From investigating this seems to be originating from AI that checks title changes for Amazon listings and can delete an existing classification verification step. Disabled this for now to debug further what kind of amazon title changes happen.
- Minor worker dependency update including BAML. Realised BAML wasn’t used in production so set up a test worker job to run it to make sure it’s ready to take over raw OpenRouter calls.
- Started implementing a basic version of subvariants that is only specific to SSDs. Keeping all existing functionality the same, the subvariants will only be used for classification. Updated Postgres to ClickHouse migration to handle SSD subvariants.
- Added a fix for Next.js frontend builds to fail if queries fail. PGrid website was empty tables for 30 minutes while ClickHouse was being upgraded to new schema.
- Minor classification frontend fixes to track items through a classification session the same way its done on the backend (frontend was missing tracking some steps).
- Found an issue with the recently optimised eBay generic item searching which wouldn’t index eBay listings which included variant dropdowns to select multiple items from a single page. Fixed this by going into individual pages of all eBay items the first time they are encountered.
- Some work refactoring the generic item create/process function. I wanted to make a unified function that handles creating generic items and saving snapshots but handling a lot of different cases made this a pretty complicated function.