So you want self hosted? And what you mean by changing the encryption? Changing the encryption protocol and encryption algorithm are two different thing. OMEMO is kind of de facto for XMPP. Last I checked it seems doesn’t let you choose the cipher suite like TLS 1.2 does. You can spin another if you have the capability to do so but not recommend.
And whatelse you want to tweek?
EDIT: typo
Key? Oh, welcome to the land of GPG/OTP where platform and protocol doesn’t matter. Good luck finding a friend willing to deal with this just to talk to you though. And loose some important security features like forward secrecy and double ratchet.