There’s nothing wrong, the second page of your home page is still your home page. If you want to check if it’s only the first page, check if ! is_paged().
That said, you can’t just change posts per page and have it work properly, because when you’re on the 2nd page and beyond, the post it starts on is calculated based on the current number of posts per page- WordPress doesn’t know that your first page only had one post and the 2nd should start at post #2, it will just multiply posts per page by the page number, so posts will be skipped.