Update PHP version 5.3 to 7.2 or first WP 4.7.11 to 4.9.8?

You should update both. ASAP.

PHP 5.3 us not supported for over 3 years… So it may have security vulnerabilities…

AFAIR there are no known vulnerabilities in WP 4.7.11, but… It’s an old version too…

WP 4.7.11 should work fine on PHP 7.2 and WP 4.9.8 should also work on PHP 5.3 (but at least 5.6 is recommended).

So I would go with updating PHP first and then WP.