If you're dealing with more than a few documents at a time, keeping the data on a USB drive synchronized across multiple machines is a hassle. Windows has had a native feature--the Briefcase--to do this since Windows 95. When you copy files from your computer into a Briefcase folder on a USB drive, they can be automatically synchronized with their originals if changes are made to them on another computer. It's one of the simplest ways to do this, and it's surprisingly underutilized.
If you want more flexible ways to sync, you'll need a third-party program. One of the first apps I used to do this was Microsoft's SyncToy. I used it to synchronize data not only to and from a USB drive, but to and from external removable drives as an impromptu backup application. Unfortunately, SyncToy has to be installed on a specific PC and can't run as a standalone application from the USB drive.
However, there's a plethora of programs for syncing data that can be run in a standalone fashion from a USB drive. I've grown fond of Allway Sync, which sports a staggering array of synchronization options and tracks all changes made to the files in a local database.
Another good one, with probably the most sophisticated sync options of the bunch, is 2BrightSparks' SyncBack, which includes goodies like being able to sync files still in use, versioning, innate support for FTP directories, and 256-bit encryption. Allway Sync is free; while there is a limited free version of SyncBack, the full version is priced at $30.
DataTraveler Secure has encryption built in
Some thumb drives come with encryption hardware built in. Kingston's DataTraveler Secure drives use hardware-based 256-bit AES encryption and come in sizes up to 8 Gbytes. That said, hardware-based encryption on a thumb drive is expensive--a 4-Gbyte DataTraveler costs as much as $120 retail--while software-based encryption can be implemented for free.