When I was there I saw a pregnant woman with a small child but they never left the area where the ferry docks. There's not much there except a few benches. Unless you are willing and able to hike 30 minutes or more it would be pretty boring just sitting by the dock. If you are able to hike it's a great place. I would recommend that if you do take small children you should start making your way back to the ferry with plenty of time spare. I don't think I would bring a 4 year old unless you plan on carrying them 30 minutes or so to the beach. Also once you reach the island you are stuck there untill the ferry leaves so make sure you come prepared to stay awhile. Everyone I went with had a great time but we were all adults.
Unless you come prepared for hiking or being on the beach there isn't much in the way of activities. If your little one likes walking trails and exploring the outdoors then you are all set. It is a gem of an island and has many different types of environments to see.
It depends on their stamina and the weather... when i went it was end of July and VERY hot and humid. The island is wild and you have to walk for at least 30-45 minutes to get to the beach, it's full of bugs, etc and there really isn't any great shelter if the little one got tired or too hot. If you're going in the fall or winter it might be more manageable.
Thanks! Your answer is awaiting moderation.