jQuery instantly executes (a button click and css change) on load

The .click() function triggers a click. This is why this is happening. After the click .css() runs and turns the button orange.

If you want to change the colour on click then, as clear from the documentation, the event handler needs to be passed as a callback function to .click(), not chained to it:

$(document).ready(function() {
    $("button").click(function() {
        e.preventDefault();
        $(this).css("color", "orange");
    });
});

Note that e.preventDefault(); is supposed to go in the event handler, not where you had it.

Also, I think it’s worth pointing out that jQuery is unecessary for this, you can do it with ‘vanilla’ JS like so:

var buttons = document.querySelectorAll( 'button' );

for ( var i = 0; i < buttons .length; i++ ) {
    buttons[i].addEventListener( 'click', function( e ) {
        e.preventDefault();
        buttons[i].style.color="orange";
    }
}

Or, if you don’t need to support IE:

document.querySelectorAll( 'button' ).forEach( function( el ) {
    el.addEventListener( 'click', function( e ) {
        e.preventDefault();
        el.style.color="orange";
    }
} );