How to unpublish a page and still keep a draft?

When I logged out the URL generated a 404.

Correct.

Why can I see the page when logged in? Is this documented somewhere in the codex?

I think this is something that codex has but some parts are still not written, like this part for draft.
In the case of draft post your permalink will look like this:

?p=2898&preview=true

And it is not available when you are not logged out as you noted — 404 error page is what you will get.

The same error you will get also for the pending review status when you are not logged in, but the permalink will be the same as for the published post if you are logged in.

tech