Updates March 30

PGrid

• updated generic item schema to allow saving descriptions of items. started using this new column for laptops from Scorptec as the titles aren’t enough to classify.
• adding a local only Astro page that will allow AI agents to edit slideshows for social media which I will take screenshots of to create the slide.
• added a local only page to allow AI agents to create slideshows. created a new slideshow using this where I found images but the AI generated the slides.
  • iterated on this a bit to:
    • support multiple different sizes of social media posts
    • added some shared code so it’s easier to start new slides
    • added some docs for AI agents so they categorise images used into folders so they can be used later
    • added rules so Astro files created to create posts also get moved into a gitignored folder so they can be referenced later if needed
    • added scripts that use Playwright to create images from the Astro pages
• started scraping laptops from jb-hifi
• ran specs search for RAM models and created ~80 new RAM models and ~40 RAM variants
  • fixed issues with specs search after running RAM model specs search and it created a model with a duplicate brand in the model name. updated prompt
• started working on a Firefox extension that allows agents to take actions in my Firefox browser like running JS and taking screenshots. initially used this to try and monitor Reddit but haven’t found a good use case for it yet. Wanted it to exist anyway as an alternative method of scraping if it’s needed in the future
• found out that the snapshot dedupe task was broken since adding support for deals since deals can reference snapshots that were marked to be deleted causing errors for the entire job. fixed and made the snapshot dedupe task more efficient by doing most of the work using SQL queries
• found an issue where non australian ebay listings were showing up for pgrid au. found it was due to the newly added search based scraping. the single page scraping would have marked the item as failed due to currency differences. updated search based scraping to add a new filter to ignore items from international sellers based on a single attribute.
• started planning out how to handle classification and specs search for laptops
• investigating a networking issue with hosthatch2. randomly and at the same time, ClickHouse and proxies get timeout issues. same with SSH. AI unable to determine the cause. moved proxies away from hosthatch2. but ClickHouse connection issues still break multiple things.