The Standard: Most hikers want their GPS units to collect geographical breadcrumbs as they hike, save key points of interest (POIs) at the push of a button, guide them to a designated spot, and let them follow a route they've previously recorded or loaded. These features are now fairly ubiquitous, but a few other options useful to backpackers aren't. The ideal unit also interacts with your computer (for uploading/downloading information); has a big screen (at least 2.5 inches) for reading detail on maps; and lets you turn off battery-draining features like digital compasses.
Variables: In addition to traditional GPS units, we're now seeing an explosion of GPS-enabled phones and Blackberries that combine multiple digital tools (GPS, camera, phone, text, calendar, browser, etc.) into one powerful handheld that can lock into satellites even outside of cell coverage. One cool use: imagine clicking a photo with a GPS waypoint embedded, then sending it to a Google Earth map wirelessly. The main downside to these phones is limited battery life.
Batteries: Think it doesn't matter? Think again. Many GPS units now come with rechargeable batteries, which limits their usefulness on extended (even weekend-long) backpacking trips. Unless your primary use is dayhiking or geocaching–or if you're only turning the unit on to record points of interest–make sure your GPS takes AA batteries. In cold weather, splurge for lithium AAs, which last substantially longer in sub-freezing temps.
Maps: For backcountry navigation, it's mighty helpful to load topo maps onto your GPS. Manufacturers offer sets by region, park, and in other configurations; you can also download digital topos from the Web or CDs.