No you can't drive past fences that lead to residences or vacation rentals like the castle, but you are welcome to park and walk around and take photos of anything you wish.
Troll Haven resides along 2 public roads, and oftentimes you will see the general public in the area taking photos and sightseeing.
Thanks! Your answer is awaiting moderation.