Parking would be your biggest issue. This is a small park, with no covered shelters. It's a great park for small group walks, hikes and picnics.
There are no covered picnic areas, but there are places to sit on the grass. Probably too small for such a group
The Gardens only have facilities for smaller groups. You might be more comfortable using one of Asheville's many wonderful parks. See https://beta.ashevillenc.gov/service/find-a-park-playground-or-recreation-center/ for locations and amenities.
It did not seem like a place for a brunch. I agree with Dorcas Shurberg.
Thanks! Your answer is awaiting moderation.