Updates June 8

PGrid

• updates to internal cli
  • added codex command to see usage for multiple codex accounts and switch between them.
  • updated “stats” command to work for any category and deleted category specific stats commands
• added support for HDDs scraping. realised I wasn’t scraping data for this and started scraping it for sites that sold it and I had support for
• continue work on specs search for laptops
  • added shared util to auto classify when specs search resulted in finding a variant that already existed for a given item. this shared util was used for laptop and gpu variants
  • fixed bug where research agent can have trouble determining exact laptop variant and therefore returns null values for things like cpu, but then verifier just fills in that with a specific value instead of verifying
  • changed definition of laptop variant to make touch screen variant defining. this was after multiple searches resulted in mixing touch and non touch laptops into one variant. did not fix those
  • claude fable came out and codex agent based laptop variant specs search has been draining usage even with 2 basic accounts so asked fable to iterate on using google gemini comparing with ‘golden data’ already in database. it was able to iterate and test prompts increasing the accuracy from 80% to closer to 100%.
  • ran through variant specs search with 100+ new laptop variants added
  • switched to using vertex ai for gemini to make use of google cloud credits. but wasn’t able to find rate limits for this. updated specs search workflows to store information about llm usage for each call
• continue work on item enrichment
  • extracted out description, image and item sku columns to be a separate table to try reduce data in main generic items table which is getting quite big already
  • updating crawlers to support extracting information required like images, descriptions and skus. updated:
    • bestbuy api
    • newegg
    • amazon au and us
    • officeworks (initial was bad, updated after test runs)
    • jbhifi
  • updated cli to view item enrichment details
• added support for making use of job priority. stale recrawls are now lower priority
• attempt fix at ebay issue where featured deals keep flipping between active and not active. claude found potential issue where if coupon is retrieved from top of page a discount price doesn’t get saved. updated to reuse existing discount prices if same coupon and same original page.
• new item enrichment backlog command to see what needs to be done
• issue with workers on deployment, added lint check on docker build to prevent deployment failure condition
• started work on adding support for monitor category classification and specs search. initial version is using codex sdk agent to do specs search