If I understand correctly, you’re only wanting to change the behavior of wptexturize within the href attribute of links. Here’s how I would go about that problem:
Don’t mess with the default behavior of wptexturize, that function is used in a lot of different filters and its behavior is pretty useful. And definitely don’t hack the core files for a simple case like this… its not necessary, and it’ll cause you no end of headaches down the road when you want to upgrade or anything.
If you need a filter that will urlencode the href attributes of links in your content, I would use the urlencode function built in to WordPress. This also takes care of converting spaces to plus signs and any other special cases in urls. You just have to make sure its applied before the wptexturize filter comes along. I’ve added it at priority 9, but any priority lower than 10 would work.
add_filter( 'the_content', 'replace_apostrophes_in_links', 9 );
function replace_apostrophes_in_links( $text ) {
return preg_replace_callback(
'/(<a[^>]*href="https://wordpress.stackexchange.com/questions/33012/)([^"]*)(.*?>)/',
create_function(
'$matches',
'return $matches[1] . urlencode( $matches[2] ) . $matches[3];' ),
$text );
}
A bit of explanation… the regular expression will find all <a> tags in your content. The callback function will then urlencode the href attribute, and return the result. And since this filter is applied before the default formatting filters, there won’t be any single quotes left in your links for WordPress to mangle.