This doesn’t have much to do with AJAX itself, it is related to core principal of plugin/theme development – always use the latest possible hook to run your code. It sounds like the plugin in question does something on admin_init which is a very early hook that can be triggered from AJAX and XML-RPC in addition to the usual admin. A more proper hook would have probably been admin_menu or admin_notices assuming you have some general admin code that is not restricted to one page.
The idea of trying to detect what code path led to an execution of a code is violation of basic software development principals – functional code should have explicit input, OOP code should in addition refer only to the object state and nothing beyond. Your code should be written to be platform agnostic as possible, as if you might want to use it next year in drupal. In real life drupal is probably unrealistic, but the more self contained your code is the easier it is to test it with automatic unit test tools without going through the bother of setting the whole WP enviroment.
But is it possible at all? two ways come to my mind, IIRC one of the $_SERVER attributes contain the path to the php script being run, which in case of ajax will have a file name of ....\admin-ajax.php, or you can use the backtrace API of PHP to inspect the calling stack to learn where you came from.