He's there to keep time and coordinate a big group. Do you really think everybody can really hear each other that well?
I'm thinking a symphony orch concert where it's pretty darn quiet, they can hear a pin drop. I agree that the conductor is the visual focal point. As long as he knows where he's at the musicians can read him, know pretty much where they're at by reading his body english/emotional sensitivity.
I think you'll agree that his main gig is rehearsing?
I'm under the impression that someone else does that and he just whips them into shape. I know a few people that play in orchestras and when I get an answer, I'll PM you.
Edit: I guess I'm thinking of the concertmaster, the string section leader.
Here's a bit about the conductor.