Simply
if(!is_home() && !is_front_page()) add_action( 'thesis_hook_after_post', 'my_related_posts' );
will do that for you. It will not do the add_action() when we are at the home page or not any front page.
If it doesn’t work for you, may be you need to contact with the Thesis team.