You can add the action on the init itself, just increase the priority of the add_action
call. The higher the priority the later the function is called.
add_action('init', 'retrieve_my_terms', 9999);
But my suggestion is that you should do these kind of things as late as possible, preferably just before the first time they are used. There’s an action 'wp_loaded'
which gets called after full wordpress has been loaded & ready to start working on the output. That action might work for you.