You need to have a lot of traffic for naive protocol based optimizations to be effective, and even then it is not obvious that you can gain much even when it does kick in.
Most people will be first time users of your site, and browser have only a limited space to store information about the last update time of all the pages a user visited.
But even if you actually get such a request, how exactly do you know when was the last update? The best answer to that question that wordpress has is the publish time of the latest published post, which might not represent at all when a theme change, or any other setting change happened. But even for such a naive assumption, you still need to load wordpress and do at least two DB queries. You will probably get some improvement, but not as much as using an object cache or page cache in the first place.
And in the real world, most of the slowness is because of the JS/CSS/media on the page, and if it is not cached (and unless you are facebook, it is unlikely t be cached), your page will still load slow even if the “last published” thing did work.