Exactly. This is why I always laugh at those that think that just because one person is bigger than another that the biggest will always win unless there is some "defect" (ie glass jaw) in the bigger person. That line of thinking seems to be chronic in people that are "big".
Well more power is always useful in HTH. However...
Energy (power) = mass x velocity, in physics... so faster also means more energy/power.
There's also the question of body mechanics. If you know how to generate power using the whole body, you hit harder than someone who is just using arm strength. I've known some wiry small guys that could hit like a frigging mule.
After a lot of thought on the matter, I think one of the biggest advantages a big guy has in most fights is the psychological aspect, particularly the intimidation factor. Confidence and aggression are both strong advantages in a physical fight, and typically the big man has those on his side because he figures his size gives him the edge. Contrariwise, the smaller guy, unless he's a well-trained or veteran fighter, is likely to be nervous about the other guy's size and presumed strength and often let it affect his mindset and sap his own confidence and aggression.
Now, when we're talking about trained and/or veteran fighters, especially OUTSIDE the ring where there are no rules, that formula doesn't always hold. A man who knows what he's capable of and has confidence in it will be a dangerous opponent regardless of size differences either way.
Common wisdom has it that "A good BIG man will usually beat a good SMALL man, all else being equal." There's a certain amount of truth in that, but there's also the caveat that "all else being equal" rarely applies in real life. Especially not in the street, where weapons or accomplices can appear without warning.
It's a lot more true in boxing, for instance, where reach and raw hitting power and mass to absorb hits are more important, due to the highly limiting rule-set. Not as much the case in UFC-type bouts, where we've seen middleweight MMA guys take down massive Sumotori and even Brock Lesnar got his clock cleaned. Even less so in the street.
One of my favorite instructors was a guy I'll call Mike. Mike was about 5'8, stocky, almost bald, red complexion, happy disposition, smiled a lot. Mike didn't
look tough at all. Mike was VERY tough. He was a metalworker, hard-muscled and strong, with black belts in karate and judo, as well as some boxing and wrestling experience. Had hands like hammers and a grip like a hydraulic press. He could sling half a dozen men around the dojo like rag-dolls while still smiling.
It's sort of like Louis Lamour said in his Westerns: "Every man out here with a gun on his hip, still breathing, is an undefeated champion." The only way to find out for sure what somebody's got is the hard way; judging by looks can be VERY deceiving.