I put up a vps with nginx and the logs show dodgy requests within minutes, how do you guys deal with these?

Edit: Thanks for the tips everyone!

  • Dr_Toofing@programming.devEnglish
    61·
    1 year ago

    These requests are probably made by search/indexing bots. My personal server gets a quite a lot of these, but they rarely use any bandwidth.
    The easiest choice (probably disliked by more savvy users) is to just enable cloudflare on your server. It won’t block the requests, but will stop anything malicious.
    With how advanced modern scraping techniques are there is so much you can do. I am not an expert, so take what I say with a grain of salt.

    • WasPentalive@lemmy.oneEnglish
      31·
      1 year ago

      The ligitimate web spiders (for example the crawler used by Google to map the web for search) should pay attention to robots.txt. I think though that that is only valid for web-based services.

    • Rusty@lemmy.worldEnglish
      31·
      1 year ago

      Fail2Ban is great and all, but Cloudflare provides such an amazing layer of protection with so little effort that it’s probably the best choice for most people.

      You press a few buttons and have a CDN, bot attack protection, DDOS protection, captcha for weird connections, email forwarding, static website hosting… It’s suspicious just how much stuff you get for free tbh.

            • ItsGhost@sh.itjust.worksEnglish
              3·
              1 year ago

              You can use a custom origin certificate, but that’s irrelevant when CloudFlare still re-encrypt everything to analyse the request in more detail. It does leave me torn when using it, I don’t use it on anything where sensitive plain text is flying around, especially authentication data (which is annoying when that’s the most valuable place to have the protection), but I do have it on my matrix homeserver as anything remotely important is E2EE anyway so there’s little they can gain, and with the amount of requests it gets some level of mitigation is desirable