If you’re building something for public consumption (a plugin, a theme, etc) use admin-ajax.php like you should because that is the appropriate and accepted way to do things and gives your end users the power they need to change and modify things if they so choose.
Beyond that, the best you MAY be able to do is use the SHORTINIT constant. Define it in a custom php file, then require wp-load.php and do what you need to do. SHORTINIT stops most of the WordPress core from being loaded.
<?php
define('SHORTINIT', true);
require '/path/to/wp-load.php';
// you'll have the basic API here, including `get_option`. Do stuff.
If you do this outside the WP core, you’ll have to guess where wp-load.php may be — you won’t have any ABSPATH contant to guide you. That’s a risky bet unless you’re in full control of the system. In other words, if this is a custom, not publicly-released thing, go for it. Otherwise, use admin-ajax.php.