When using the wp_handle_sideload function then, as @xvilo pointed out in a comment, it isn’t creating the meta data wordpress needs to display the image in the media library. As I can not see where the wp_handle_sideload function is writing the data to the db, I don’t understand why the files are being displayed in your media library. So with your approach you are missing a call to wp_insert_attachment (as you said yourself) and wp_generate_attachment_metadata.
$temp_file = download_url( $url, 5 );
$file = array(
'name' => basename($url),
'type' => 'image/jpg',
'tmp_name' => $temp_file,
'error' => 0,
'size' => filesize($temp_file),
);
$overrides = array(
'test_form' => false,
'test_size' => true,
);
// Move the temporary file into the uploads directory
$results = wp_handle_sideload( $file, $overrides );
// Write attachment to db
$attachment = [
'post_title' => basename($results['file']),
'post_content' => '',
'post_status' => 'inherit',
'post_mime_type' => $results['type'],
];
$attachment_id = wp_insert_attachment( $attachment, $results['file'] )
// Generate the attachment's meta data
wp_generate_attachment_metadata( $attachment_id, $results['file'] );
Or you can just use the media_handle_sideload function, as this function is doing that already for you.
$attachment_id = media_handle_sideload( $file_array, 0 );
After that you can use
set_post_thumbnail( $post, $attachment_id );