You can only know if you choose to read the code and compile from source. You can trust, in that your read the code and just install the app, or let others read the code for you. If reputable sources tell you it’s good, most of the time it’s good. How can you trust Signal more? Well you… shouldn’t. You could try to use a decompilation tool, don’t know if that works on Android’s apps though.

Are others different, like Signal

Signal’s encryption is sound, but there’s an uncomfortable fact that it uses google play services dependencies (like for maps and other things, I think). There are articles (1, 2) that discuss that it has functionality that may allow an other process (the google play services process) to read the signal app’s state or even directly it’s memory because of that, which can mean the contents of the screen or the in-memory cache of decrypted messages.

Security audits often only audit the app’s own source code, without the dependencies that it uses.
The google play services dependency could have a “flaw” today, or it could grow a new “feature” one day, allowing what I described above.

May or may not be connected, that Moxie (signal founder) is vehemently against any kinds of forks, including those that just get rid of non-free dependencies (like the google play services dependencies). The other comments of his are also telling.

Because of these, I have ruled for myself that I’ll not promote them as a better system, and I’ll not install Signal on my phone, because I think it gives a false sense of security, and for other things like still requiring an identity connected identifier (a phone number) for registration.
However if there were people whom I can only reach through Signal, there’s Molly. They maintain 2 active forks, one of which is rid of problematic dependencies, and I would probably use that. Molly-FOSS is not published on the official F-droid repository, but they have their own, so the F-droid app can still be used to install it and keep it updated.

and how do I know?

It’s hard, unfortunately, and in the end you need to trust a service and the app you use for it.

F-droid apps are auditable, they are forbidden from having non-free (non-auditable) dependencies, and popular apps available in the official repository are usually fine.

With google play, again the truth is uncomfortable.

On Android, the app’s signing key (a cryptographic key) makes it possible to verify that the app that you are going to install has not been modified by third parties.
Several years ago Google has mandated that all app developers are required to hand in their signing keys, so that google can sign the apps instead of them, basically impersonating them. Unfortunately this also means that unless the app’s total source code is available (along with all the source code of it’s dependencies), it’s impossible to know if google has done modifications to the app that they make accessible on the google play store. This in itself is already a huge trust issue to me, but what is even worse is that they can just install custom modified versions for certain users on a case by case basis, with the same signing key that once meant that it was not modified by third parties like google, and no one will know it ever.

Just an example to show that the above is possible: the amazon web store similarly also requires the developer to hand over the app’s signing key, and they admit in the documentation that they add their own tracking code to every published app.