How to register and enqueue JavaScript files without breaking plugin dependencies?

I would use jquery-masonry included in WordPress core:

function my_scripts() {
    wp_enqueue_script( 'jquery-masonry', true );
}
add_action( 'wp_enqueue_scripts', 'my_scripts' );

If you really need the standalone masonry library, you should use your “first way” but I would not use “my_masonry” as handle for the script, I would use “masonry” if you have not modified the code. Also, use the correct version number of the library (I think your are not really using 1.0.0).