posts going to 2 pages/categories only show up in 1

First thing : you should not use category ids directly. Use slugs instead : it’s easier to maintain.

<?php query_posts('category_name=portfolio'); ?>

Second thing : why did you put a query parameter in the_post ? This function does not accept any parameter (see the Codex).

Third thing : the call to query_posts('cat=-4'); will show all posts EXCEPT the one that have the category N°4 (portfolio, if I guess well). This is not want you want, right ?

Want you want is query_posts('category_name=portfolio'); in your portfolio page and query_posts('category_name=blog'); in your blog page.