First, I have to tell you that this all depends on the server size and the amount of resources your WordPress site uses. That being said, I set my php memory limit to between 256 and 512 MB.
WordPress sites can be slow for a large variety of reasons. Performance often improves by tweaking database settings (and checking for slow queries), configuring page and object caching, using a separate object cache like Redis or Memcache, adding Varnish in front of your site or using external edge cache like Cloudflare or Cloudfront, etc.