Is it possible to only visit the grounds at Hurst castle?

Not without a ticket, because you have to get in a bus to go to the castle and they won't let you get on without the wrist band, they scan it at the visitor center when you board the bus and at Hearst Castle when you get on the bus to come back to the visitor center. Buy a tour ticket and skip the movie and the guided tour and you should be able to just check out the outside gardens at the castle. Good luck!!

You have to purchase tickets to a tour, they take you on a bus.