Are there any downsides to not using get_header(), get_sidebar and get_footer()?

If you’re creating a theme you intend to distribute, using these functions gives users and other developers an opportunity to override the files with a child theme. It’s just a predictable way to separate elements of your theme.

Each of them also trigger an action with the same name, which may be useful to some.

If you have some other template scheme, you can use locate_template to make them work with child themes.