Yes, our right to speech is negative in nature - within boundaries (defamation, classified information, etc), the government is forbidden from interfering with our speech, either by restricting our expression, or by compelling it.
If I had a positive right to consensual sex, for example, that would mean the government was required to ensure I recieved consensual sex. The problem there quickly becomes - what if no one wants to have sex with me? Should the government enable me to recieved what, after all, is my right, by taking away the rights of someone else?
Instead, liberty regarding sexual expression is negative in nature, within boundaries. Matt Gaetz can't defend his actions with a minor by pointing to a positive right to recieved consensual sex; because there is no such thing, we've criminalized sexual interaction by age, as well as by relationship (prostitution), as well as by location (crossing state lines, no strip clubs next to schools, etc.)