Except that you don't know what else they might have said in their lives which might have been gaffes. If all we had of Brigham Young was D&C 136, you wouldn't be any wiser about him neither. You judge him by a different standard than you use to judge the OT prophets.
Jesus vouched for the OT prophets.
That's good enough for me.
He NEVER vouched for Brigham Young.