I'm not sure of the times of the dolmuses out of Kaya at present but you could get a Taxi from there up to Hisaronu. From Hisaronu the dolmuses into Fethiye run until about midnight.
It's also worth asking at Cin Bal if they could give you a lift either to Hissy or even all the way home.