Fix RSA-AES state machines

Fix RSA-AES state machines

If there isn't enough data, then the client reading the hash will fall
down and try to read the subtype instead, which isn't correct.

Invert the logic so we get a more consistent way through where we only
break out when there is insufficient data.

Do the same for the server code, for consistency.