This will be brief, because there’s not much to elaborate on.
There are two parts to a WordPress site (similar for most websites):
- the content, and
- the database.
Yes, you can just copy all the files but you’ll also need to do a database dump. Your host will usually have a control panel you can log in to to do this (often providing a tool called phpMyAdmin which allows you to do a DB dump and a lot more).
Alternatively, there are many WordPress backup plugins out there that will take care of all of this for you.
EDIT: Expansion to answer below to help others who come across this.
A note about using the inbuilt exporter
Although you can export your content using the inbuilt Export function in WordPress, I generally don’t recommend this for making a reliable backup of your site. Yes it will backup your post content, but you won’t be able to quickly restore your entire site: in particular, your plugins, themes, settings, and specific WordPress version you were running won’t be there.
In most cases, it will take a lot more manual work to restore a site that you only have the exported content for, and it may not be possible to restore everything. This is why I always recommend taking a full backup of your WordPress directory and the database.
What files do you need to copy in the WordPress directory?
For a full backup, along with your database, you should copy your entire WordPress directory – that includes all wp-* files in the root plus a few others (eg. xmlrpc.php), as well as all three wp-* folders: wp-admin, wp-includes and wp-content.
Alternatively you could just backup wp-config.php and wp-content and download the rest from the WordPress website, but you’ll need to be careful that you download the exact WordPress version your database corresponds to (which will also be the version that you know your theme and plugins work on). Again, to make things simple at restore time, I usually recommend to just back everything up yourself.