If you take Trump at his word that he wasn't referring to the Nazis and white supremacists when he was saying "very fine people," you have to imagine there were people in this group who weren't white supremacists.
Trump language is passive-aggressive to the core; he’s so used to lieutenants reading into his mealy-mouthed pronouncements as “actions” that Snopes not pointing out this behavior is a fundamental flaw in the function of the site, esp. in this hyper-disinformation era.
What's the German expression, if you have a Nazi sitting at a table with 11 other people, you have 12 Nazis? I think it's a bit more poetic but that's the gist