I think that your take on Our Man in Jazz is similar to some of the feelings about the record when it was released, especially by the "music is logic" critics. I've always enjoyed the record - it was one of the first jazz records I bought and it took me a few listens to begin to hear some of what was going on - and I enjoy it now more than I did then. To my ears, Sonny sounds comfortable with what he's playing, and the group sounds like they're listening to each other, even though they're coming from different places. Sonny was exploring when the record was made and, even though he left the musicians (except for Bob Cranshaw) and general freedom of the Cherry group behind, I hear some of the exploration carried over into the RCA recordings that followed this one.
As an aside, when I was going to school in Buffalo in 1964, Sonny Rollins appeared at a local club with a group that included Grant Green. Unfortunately, I missed that gig and I've wondered since what the group sounded like.