Comments screen in backend, how to disable visual editor when replying to a comment

If understood correctly, you are referring to the Quick Edit Reply action.

There is no hook (a direct one at least) to hide that. It has to be CSS or jQuery. Like:

add_action( 'admin_head-edit-comments.php', 'hide_quick_reply_wpse_92435' );

function hide_quick_reply_wpse_92435()
{
    echo "<style>#qt_replycontent_toolbar {display:none}</style>";
}