If I have a paid postal label, can I just drop the package off in a USPS dropoff box, or do I still have to go to an office?

If it fits in a mailbox then it's usually fine. There are some stipulations for mail being sent internationally or over a certain paid only with stamps.

You can just drop it off in a drop box. If it's already a paid label that's fine. If you put stamps on it, you can only put it in a drop box if it is under 13oz, otherwise you have to take it in to the office.