Yep. If you select the option to check for new content on every visit
to a web page, the browser has to find out from the server if its copy
of a file is stale compared with the server's copy. Why waste that time
rather than just get the file in the first place? After all, you have
the speed.
In fact, users of IE6/7 would often configure the advanced options to
always empty the TIF on exit from IE. They didn't want that stuff
hanging around. However, IE would sometimes forget to empty its TIF so
cleanup utilities showed up to make sure the TIF got cleaned out. Well,
with an empty TIF, the entire web page has to get downloaded on the next
visit.
High-speed users are not condemned to having to cache anything between
browser sessions (and don't need to cache much at all during a browser
session). As you noted, you are not in that crowd.