I would like to get a new, faster router that also has more range. Do I need to bring the old one with me to get the new one?

Yes

Yes, bring the old one and they will upgrade your modem to a newer, faster, better one!

I definitely would, so they can see what you're working with at the least. If it's a router that you're renting from them, you'll definitely want to bring it woth you, cuz they won't give you a new one if the store without their old one back first. You can do it online or by phone as well and they will send it to you through the mail!