If I rent a vacation house near the Frio, can I still get on the river if the park is at capacity?

Just access the river from a location that's outside of the park and you'll be good to go.

Indeed day pass or you could buy the 2018 pass and you can go to any statepark in the state as many times as you would like for the next 6 months

You can get on the river at many other locations, rather than the park itself.

No but if you get a day pass first and then try to get in you can.