Updates April 21

PGrid

• Last week I restarted a server with an old nomad config, resulted in nomad being open to the web and found some crypto miners deployed on some of my servers.
  • Updated all node configurations to include their tailscale ip address so restarting should keep the instance only on the tailscale network
  • Added cpu load tracking on grafana to get a better idea of node loads
• Updates to the classification tracking to persist tracking information into the database
• Updates to both frontend classification and backend ssd classification pipeline to both use the classification tracking which saves information about each step taken into the database
• Added a new ui to view the classification tracking for each session of ssd classification. This helped find that not all variants are available in the database and more variants will need to be searched for. Ideally classification can queue variant searching.
• There is more and more issues with ebay scraping resulting in a lot of workers timing out at 60 minutes and getting killed. spent a little time investigating and did some minor tweaking of error handling for screenshots and disabling media.
• Created new Interface and queries for viewing SSD data.
  • Required custom clickhouse queries that would return daily price history and current ssd variant listings based on filters set
  • Has detailed filters and sorting and is closer to what I wanted to exist for PGrid
  • Includes price history graph and link to listings for individual ssd models