I know that there are specific paid tour times, which allows entrance into the house for that time only. Gift shop & grounds are open to the public without fee as I recall...
I don't know why you couldn't do that.
Thanks! Your answer is awaiting moderation.