I had a similar issue as you, and switching/reseating DP cables solved it for me. This is the switch I am using.

Also note my monitors can only do 4K@60Hz.

Not sure if you’ve considered software KVM for this, but it tends to make everything a lot easier, and even paid versions are typically far cheaper than a relatively basic hardware KVM. Hardware KVMs are essentially “dumb” devices, they have to process and retransmit every raw piece of information, regardless of whether it’s important or not, they have no choice. Which ends up requiring obscene amounts of bandwidth at the kind of resolutions you’re talking about and demands a lot from the hardware as well as all the connections.

Software KVM is able to be smarter, since it is running at a higher level in the stack, and can make better decisions about what it needs to transmit and how. If you’re concerned about the bandwidth, hooking the two computers together with a dedicated 10GB Ethernet card each is more than enough for even raw bitrate 4k/120Hz times 2. And realistically you don’t need that much, as I said, Software KVM has enough information about what’s going on on-screen that it can be far more selective and streamlined in what it is actually transmitting without any loss in quality, perceptible, or at all if you configure the settings high enough. Latency is going to be the main hurdle, and if you’ve already decided you can’t possibly live with even a hair more latency than physically possible, fine, but at least give software KVM some consideration before you throw it out completely, it may not be as bad as you presume.