Load code for custom fields only on admin pages?

Loading this code won’t have big impact on loading times.

Of course you should make sure, that this code is only executed when it should be…

But yes, it should be possible. Just make sure you don’t use it to early (when is_admin() return incorrect value).

EDIT

The code you’ve posted here should work just fine. Of course you have to make sure that code placed in functions_admin.php is written correctly (for example you have to take care of using global variables in correct way).

To test if this is included, just add echo 'alamakota'; in first line of this included file and check if ‘alamakota’ is showing on every admin page (it should).

Also make sure that you don’t add filters/actions to hooks that are run before init hook. The won’t fire up (WordPress already called these hooks before you added your filters). See: http://codex.wordpress.org/Plugin_API/Action_Reference