There are pro’s and con’s for either pw or key-based authentication.
In some cases, for example, key-based authentication is less secure than password authentication. In other cases, its pw-based that’s less secure. In some cases, one is more convenient, in others, less.
It all boils down to this: When you do key-based authentication, you must secure your key with a passphrase. Unless you have ssh-agent running (ssh-agent frees you from entering your passphrase every time), you’ve gained nothing in terms of convenience. Security is disputable: the attack vector now shifted from the server to YOU, or your account, or your personal machine, (…) – those may or may not be easier to break.
Think outside of the box when deciding this. Whether you gain or loose in terms of security depends on the rest of your environment and other measures.
edit: Oh, just saw that you’re talking about a home server. I was in the same situation, “password” or “USB stick with key on it” always with me? I went for the former but changed the SSH listening port to something different than 22. That stops all those lame script kiddies brute forcing whole network ranges.