Sounds like you didn't actually see what occurred that day. First and foremost, virtually everyone who went into the Capitol building was a Trump supporter. Regardless of whether or not they were present at the Ellipse. There's no difference between those who listened to Trump's speech at the Ellipse, with those who heard his speech from outside the Ellipse. Secondly, Trump told them to march to the Capitol at about 12:15. He ended his speech at about 1:10. The Capitol building was breached at about 2:15. The walk from the Ellipse to the Capitol was between a 30 to 45 minute walk. Plenty of time to march from the Ellipse to the Capitol and break in. Thirdly, members of the Proud Boys and Oath Keepers were at the Ellipse, and they, especially the former, were instrumental in violently breaking into the Capitol building.They were miles away from the Capitol when the protesters stormed the building and were threatening the Congress members. The rally-goers arrived on the scene long after the mayhem had begun.
Also, the day following the riots, there were videos of rally-goers pleading with the mob to stop the violence. Those videos were quickly pulled by the anti-Trump people, and never seen again.
Trump had no idea that the Capitol was under seige during his speech at the Ellipse rally. His Security Team didn't advise him of the mayhem until AFTER his speech was over. Trump had intended to march alongside the rally-goers, but obviously his Security Team couldn't allow that.