I recommend the weekdays in the afternoon, 2-3 hrs before closing, if you plan on just visiting the garden. A bit earlier if you plan on stopping at the cafe. Temperature is usually great if its not raining, and it is less crowded.
Any time
Before it gets too hot. Stay away from busy weekends if possible.
Anytime you would not regret it
This would be a perfect time to visit as the temperature is cooler. But if you don't mind a little heat anytime is good.
Spring
Right now through July. CK the website and you will see the events taking place there.
Thanks! Your answer is awaiting moderation.