Updates Dec 9

PGrid

• My entire nomad admin ui was visible to the internet which includes job configs containing secrets and ability to exec into containers. In my nomad config for all clients I had network_interface = "tailscale0" and then I checked one or two servers manually and the admin ui wasn’t available. So I assumed nomad web ui was running on the tailscale ip for each of the servers. This was incorrect and the servers i manually checked had default iptables blocking that ports. But given I have nomad now on 9 servers, some of them didn’t have any default blocking and port 4646 was available for the web to access. This has been the case for at least 1 month since I have migrated all servers to use nomad.
  • Setup systemd based nomad client service on each server that specifically binds to the tailscale ip.
  • While I didn’t notice any suspicious usage from any api keys, rotated keys for:
    • AWS (SES)
    • Github PAT (triggering github actions)
    • Self hosted docker registry password
    • Cloudflare keys (deploying to pages)
• Updated private docker registry to save contents onto disk. before if it restarted I lost all images heh
• Added a http server on top of the http proxy that uses nordvpn. This allows control to set the country using a http request and view current status of the vpn.
• Using proxy can connect to amazon.com using a US connection, issues connecting to amazon.de using a german connection.
• Updated amazon scraper to handle:
  • different sites: amazon.com, amazon.com.au, amazon.de
  • handle variants from single page
• Worked on performance of PGrid with 2m rows in gpu price snapshots table.
  • added a duplicate cleanup function for gpu price snapshots similar to generic item price snapshots
  • explored using clickhouse/duckdb but both not in use yet