According to ordinary language philosophy, 'racism' applies in any sense it can be used intelligibly.
So, if we are referring to actions that may not result from racist intent but which stem from disregard for (and negatively impact) the material or cultural situation of a race, it is intelligible for us to describe that action as racist even though the person who performed it has no commitment to the belief, and therefore may not be intelligibly referred to as racist even though their actions are racist. This would cover 'what we say' as well, because speaking is an action.
However, if a person has racist intentions but lacks the power to impose them in any damaging manner, we would describe that person as racist but would be unable to determine any of his actions to be racist, because, again, he lacks the power to perform significantly racist actions.
With its politically, morally, and historically loaded connatations, clarity and emphasis on context on a case-by-case basis is the closest you can come to discovering the nature of an attribute like racism.