• TheHarpyEagle@lemmy.world
    link
    fedilink
    arrow-up
    4
    ·
    1 year ago

    As someone who used to scrape government websites for a living (with permission from them cause they’d rather us break their single 100yr old server than give us a csv), I can confirm that maintaining scraping scripts is a huge pain in the ass.

    • Kuma@lemmy.world
      link
      fedilink
      arrow-up
      2
      ·
      1 year ago

      Ooof, i am glad you don’t have to do it anymore. I have a customer who is in the same situation. The company with the site were also ok with it (it was a running joke “this [bot] is our fastest user”) but it was very sketchy because you had to login as someone to run the bot. thankfully did they always tell us when they made changes so we never had to be surprised.