I am the person who originally asked for help. In our case the solution was to change the authenticationMode in our WCF Custom adapter from “MutualCertificate” to “MutualCertificateDuplex”. The difference being the use of symmetric encryption vs asymmetric; the duplex setting calls in asymmetric and apparently that’s what the Java service is using.
To be clear; we have a send port using WCF-Custom. On the binding tab we selected customBinding. Under the binding element there is a Security group; the authenticationMode is there.