We went on a holiday weekend and I'm very glad we had advanced reservations
I think that you'll be fine. I went on the first Saturday of the month, when they have costumed actors and it wasn't too bad. It an interesting tour, I'm glad that I did it.
We walked in and were on a tour in 15 minutes, but it was a miserable rainy day. That said, the staff was very accommodating, so I think you'll be OK. If you need to wait for the next tour I think you'll find the museum and arboretum enough to keep you busy.
Thanks! Your answer is awaiting moderation.