get_avatar filter is not working as per requirement

That’s not quite how to use add_filter. Generally you use it to attach your custom function to some WP (or plugin or theme) filter hook. Then you do the processing inside your custom function and have it return the type of result the original hook requires.

Here’s an example how to use pre_get_avatar to check for custom avatar early. If not found, get_avatar, where the filter is nested, continues with its normal routine to get Gravatar image. The get_avatar filter hooks runs after pre_get_avatar.

add_filter('pre_get_avatar', 'my_pre_get_avatar_filter', 10, 3);
function my_pre_get_avatar_filter( $avatar, $id_or_email, $args ) {
  $user_id = 0;
  // Can we get an user ID?
  if ( is_int( $id_or_email ) ) {
    $user_id = $id_or_email;
  } else if ( is_email( $id_or_email ) ) {
    $user = get_user_by( 'email', $id_or_email );
    if ( $user ) {
      $user_id = $user->ID;
    }
  } else if ( is_a( $id_or_email, 'WP_User' ) ) {
    $user_id = $id_or_email->ID;
  } else {
    return $avatar;
  }
  // If not, bail
  if ( ! $user_id ) {
    return $avatar; // by default this is null
  }
  // Check for custom avatar
  $custom_avatar = get_user_meta( $user_id, 'wp_user_avatar' );
  if ( is_numeric( $custom_avatar ) && $custom_avatar ) {
    // Here you could just get the attachment url and format the img html yourself to include any available $args
    $custom_avatar = wp_get_attachment_image($custom_avatar, 'thumbnail');
    // Return custom avatar if image exists
    if ( $custom_avatar ) {
      // this short circuits get_avatar and prevents it from continuing with the default process of checking for Gravatar image, which would be unnecessary as we already have an avatar image
      return $custom_avatar;
    }
  }
  // Return default
  return $avatar;
}