Page navigation doesn’t seem to work when I add offset?

If you sets a manual offset value, pagination will not function because that value will override WordPress’s automatic adjustment of the offset for a given page.
In order to use an offset in WordPress queries without losing WordPress’s pagination features, you will need to manually handle some basic pagination calculations.

For example you can use this bellow code

$paged = ( get_query_var('paged') ) ? get_query_var('paged') : 1;
$offset = 1;
$ppp = 6;
$page_offset = $offset + ( ($paged-1) * $ppp );

$args = array(
    'offset' => $page_offset,
    'posts_per_page' => $ppp,
    'cat' => 5,
    'orderby' => 'date',
    'order' => 'DESC',
    'paged' => $paged
);

Try that, then let me know the result.
Note: Follow this https://codex.wordpress.org/Making_Custom_Queries_using_Offset_and_Pagination