how can i stop custom fields that have apostrophes from breaking my code

you could apply the content filters:

    echo do_shortcode(  "[av_button label="".apply_filters("the_content',(get_sub_field('speaker_file_label'))."' link='manually,".get_sub_field ('speaker_file_url')."' link_target="_blank" size="medium" position='left' color="theme-color"]" ).'<br>';

the above added pp marks <p></p>

try this:

    echo do_shortcode(  "[av_button label="".wptexturize( get_sub_field("speaker_file_label'))."' link='manually,".get_sub_field ('speaker_file_url')."' link_target="_blank" size="medium" position='left' color="theme-color"]" ).'<br>';