Can you Use the Rest API to query a custom database table

WordPress has no system of assigning privileges or access to control to code that runs in functions.php or in an enabled plugin.

Therefore if you can get code to run into functions.php or a plugin in your WordPress environment, you can do anything that you can do in PHP in your web hosting environment, and this includes anything at all in WordPress.

So in your case, once you make your REST API endpoint work, the PHP there can do absolutely anything. You could access the table through the $wpdb object to use WordPress’s database connection and database helper functions. Or you could if you wanted to write your own PHP from scratch using direct mysqli_ calls if you wanted — this might be advisable if e.g. you would like some of your application logic to be independent from WP.