Getting rid of the #038; when string replacing content

& is essentially synonym of &. In the_title filter wptexturize() runs with priority 1 (important!) and makes this replacement.

So by the time it gets to your format_title() at priority 11 – instead of replacing lone & symbol you replace (and break) chunk of & character entity.

So you can:

1. move your function to priority 0 and it will run before texturize
2. leave priority at 11 but replace $#38; instead of just &