Actually, that hook just lets you tie in to the beginning of executing that function, it doesn’t let you override anything.

If you want to replace this function entirely, I recommend two different options:

1. Don’t use it

Just define your own XMLRPC method (myNamespace.newMediaObject) and call that instead.

2. Replace it

You can tie in to the xmlrpc_methods filter the same way you would to add a new method and can replace the callback for metaWeblog.newMediaObject:

add_filter( 'xmlrpc_methods', 'myMediaHandler' );
function myMediaHandler( $methods ) {
    $methods[ 'metaWeblog.newMediaObject' ] = 'myMediaObject';

    return $methods;
}

function myMediaObject( $args ) {
    // ... custom functionality
}

Just be sure to maintain the same format with the $args array and to call/apply the same actions/filters in your custom method so that you don’t run into any surprises.