You need to modify the TinyMCE settings object on instantiation:
Reference: TinyMCE documentation – content formatting
WordPress provides a filter for this very purpose called tiny_mce_before_init
which you can use as follows:
function modify_tinymce_settings($settings) {
$settings['indentation'] = '10px';
return $settings;
}
add_filter('tiny_mce_before_init', 'modify_tinymce_settings');
If that does not work precisely, then dump the output of $settings
in the hook above to see where values for indentation
are held, they may be nested as part of a multi-dimensional array.