I don't think this is really a rational issue. I think a lot of people, white Americans, think in term of America as a country founded by White Americans, and made of mostly White Americans, and that that's what America mostly is, with democracy, and 'great' things whether it was conquering the west, becoming the world's top economy and military power and so on.
I think most feel 'slavery was wrong', but that 'times change' and seemed right to a lot of people then, but we've improved. I think they don't think of themselves as 'White Supremacists' or hostile to other races, that they're for equality, but that it's in a sense of 'black people are 10% of the country', and we have some immigrants from all over the world, and that's fine, and that they can be equal and they can vote and so on, but they think of it in a way that those non-white people don't change the results in elections. So it's sort of 'you black Americans are totally equally, but your 10% of the won't decide the election, and that's fine'.
But if you start to get to an idea where White Americans can become a minority, can be ruled by government non-white people select that they oppose, that's when you get into the "our country" stuff, when they've 'lost the country', when they talk about wanting the country back.
As I said I don't think this is a rational thing many think about. I think it's more just how they feel, and then if they see a black person screaming about 'whites are racist', they feel attacked and offended and that they don't like that group who is saying that.
Funny thing - the actually powers that be don't give a crap about any of this. They'll support whites, or non-whites, or mixes or anything else that results in money and power for them. And if nothing else, whites and non-whites should unite against the plutocrats, not have racial conflicts.