I've had a few occasions where I have been asked a question which has a yes or no answer and when that happens

Well, i say brain no worky - brain ends up thinking of the answer in English and then thumbs through it's filing system until it finds the correct phrase. By which time the person who has asked the question, now a little bored of watching the tumbleweed blow by, translates the question for me and I'm feeling a bit of a tool so I giggle and answer in English

I really freeze on yes and no unless it's a possesive question where the clue is in the question!
Any tips for dealing with this?