Make sure that you only attempt to set up the items after BP has set up its core navigation. You can ensure this by hooking to bp_setup_nav with a priority higher than 10.
Thus:
function bbg_setup_nav() {
bp_core_new_subnav_item( array(
'name' => 'Document List',
'slug' => 'group-document-list',
'parent_url' => $bp->loggedin_user->domain . $bp->groups->slug . "https://wordpress.stackexchange.com/",
'parent_slug' => $bp->groups->slug,
'screen_function' => 'group_document_list_function_to_show_screen',
'position' => 55
) );
}
add_action( 'bp_setup_nav', 'bbg_setup_nav', 100 );
Keep in mind that, as written, this code is intended to add a subnav tab to the Groups subnav of a user profile. If you want to attach it to a single group instead, you need to relativize it to groups, using a different parent_slug and parent_url:
// ...
'parent_slug' => bp_get_current_group_slug(),
'parent_url' => bp_get_group_permalink( groups_get_current_group() )
// ...
To be honest, though, if you want to be adding subnav items to BP Groups, your best bet is to use the BP Group Extension API http://codex.buddypress.org/developer-docs/group-extension-api/. You just fill in a few methods, and all the BP-specific navigation logic is done for you. If there are methods (like create_screen() you’re not going to use, just leave them blank.