Actually I'm having the same problem here but I'm also working on it.
Ciphering mode is not completely implemented in the YateBTS source code. So we need to complete this part.
To enable ciphering, BTS (only BTS can instantiate ciphering mode) sends Ciphering Mode Command (RR) message to the MS, then MS sends back the Ciphering Mode Complete (RR) message to the BTS and ciphering mode acknowledgement will be completed.
Currently, I'm trying to modify the source and send all changes as a patch on Github. I'll let you know if that happened.