If you know who the owners are, then return the dog to the owner or call them as you said you have their phone number. Imagine someone sending your dog to the pound, knowing it belongs to you, because they were too inconvenienced to call you. Dogs aren't just animals you send off because you're inconvenienced, they're in more cases than not a member of the family. If you're actually concerned for the animal, make sure it gets returned home and not sent off to the pound.
You would want to contact your local animal control in your municipality.
Im not sure of all of the details you face but my default in this situation is to attempt to return any pets found to their owner immediately or remove them from danger if it's possible to do so safely. However, if it's a reoccurring issue, it may imply neglect and should have a proper investigation made to be certain. Sadly, I've one experience where I had missed some red flags and regret not calling the proper authorities. At the other end of this main road were owners with several dogs known for escaping and wandering they're way to the doggy daycare where I worked at. We had always felt responsible to get them home safe. One day, their dog was hit by a car in front of the facility and didn't make it. I learned it was not the first accident for the owners and that there was an investigation following other complaints of neglect. Perhaps our calls could've helped build a case for the investigators to help them. Maybe saved a life.
Thanks! Your answer is awaiting moderation.