As an oversimplification, RSA works by taking two huge prime numbers (the private key) and multiplying them together (the public key). You could get the private key by factoring the public key but the best way to factor a prime number is to basically just try every combination with a few tricks sprinkled in until something works which could take millions of years with modern computers. You can hoard everyone’s public key until computers can crack it in a reasonable amount of time but everyone involved will be long gone by then.
The whole point of private and public keys is to have an operation that’s easy to do one way but would take so long to reverse that it’s virtually impossible. There’s nothing stopping you trying except the time and effort involved.
Gmail has had a problem for a while where people can send you spam calendar invites and they’ll automatically get added to your calendar.