altering %title

Unfortunately, your logic doesn’t quite work out. The reason is because the order of operations, just like math, works inside to outside.

So, your short_title() function is running with just passing the string %title, which then returns, I assume, the same string because that string meets your requirements. Then it passes %title onto previous_post_link and returns the full title as it did before.

As for a solution, outside of writing your own custom function, there is a filter previous_post_link that you could try to use, but if you’re only worried about the display side, you may be able to fix it with CSS using text-overflow.