Because of query mechanics either $paged or $offset is used, but not both at the same time. Basically if you want custom offset then it is assumed that you do not want pagination.
You would probably need to hook into post_limits and change generated LIMIT SQL to adjust offset, unless there is an easier way that I am missing.