do_action appearing outside of menu?

I think the reason is echo vs return:

The function posts_logout_link from

http://plugins.svn.wordpress.org/logout-password-protected-posts/trunk/logout.php

is using echo.

Your code is therefore both returning and echoing values.

ps: You could consider using output buffering to fix this

ob_start();
do_action('posts_logout_link');
$logout_link=ob_get_clean();

return $menu . '<li>' . $logout_link . '</li>';

or modify the plugin code functions to your needs and place them into your functions.php file.