How to filter manage categories listing

Yes, $terms contains all terms that are retrieved from database for display so whatever changes you make will cascade upwards to interface.

One issue I am not sure about if that will affect pagination of results in this case.