It is, of course, natural for participants to want to look at you because you understand and speak their language. Try to discourage this by minimizing eye contact.
Since turning toward people while speaking invites eye contact, the interpreter always speaks in a more neutral direction. When you focus on a notepad, gaze at the floor, or look out above an audience, the participants are more likely to look at those with whom they are interacting because they are not getting eye contact from you.