Should I always prefer esc_attr_e & esc_html_e instead of _e?

Always escape output unless you have a great reason not to.

The main thing that you lose when escaping translated strings is the ability to use html tags and html entities in the original string and translations, but you should probably not get into a situation in which translators are required to know html just in order to translated several words in the first place.

If for whatever reason you have to include an html entity as part of the string you can use printf/sprintf to give the translator some flexibility on the location of such an entity in string with something like

printf('%s&%s',esc_html__('first part','td'),esc_html__('second part','td'))

where & is just an example for an html entity.

Obviously there might be more complex scenarios that might be impossible to escape in a sane way, but your guiding light should be to always escape by default.