Before trying to figure out if it would hurt performance (yes, it probably would), one should ask what the benefit of switching to Symfony would be. I have no idea, since I haven’t worked with Symfony.
In general, less code and fewer levels of abstraction are better. For example, WordPress uses a very thin DB layer, instead of an ORM.