That assumes that "draws" means "will believe", and it doesnt. Drawing in John 12:32 refers to being brought before judgement.Sure -- go to a public place and ask 100 people if they are drawn to Christ's Crucifixion. If even 1 says, "no," then Christ cannot mean He draws all men to Himself without exception.
John 12:31 Now is the judgment of this world: now shall the prince of this world be cast out.
32 And I, if I be lifted up from the earth, will draw all men unto me.
All men are drawn to be judged. See how that works? No need to redefine "all men" to fit a doctrine, just plain biblical teaching.