Should I use Filesystem API for reading files or listing directories?

No you don’t, and not even sure if you can use it for listing files in directories. The only limitation on reading files is that they will be readable to the web server, and obviously plugin directories has to be readable for the web server to be able to read the PHP files.

Now, if you are trying to read something outside of the wordpress file structure you might need to use the filesystem API.