Why is TCP accept() performance so bad under Xen?

Right now: Small packet performance sucks under Xen

(moved from the question itself to a separate answer instead)

According to a user on HN (a KVM developer?) this is due to small packet performance in Xen and also KVM. It’s a known problem with virtualization and according to him, VMWare’s ESX handles this much better. He also noted that KVM are bringing some new features designed alleviate this (original post).

This info is a bit discouraging if it’s correct. Either way, I’ll try the steps below until some Xen guru comes along with a definitive answer 🙂

Iain Kay from the xen-users mailing list compiled this graph:
netperf graph
Notice the TCP_CRR bars, compare “2.6.18-239.9.1.el5” vs “2.6.39 (with Xen 4.1.0)”.

Current action plan based on responses/answers here and from HN:

  1. Submit this issue to a Xen-specific mailing list and the xensource’s bugzilla as suggested by syneticon-dj
    A message was posted to the xen-user list, awaiting reply.

  2. Create a simple pathological, application-level test case and publish it.
    A test server with instructions have been created and published to GitHub. With this you should be able to see a more real-world use-case compared to netperf.

  3. Try a 32-bit PV Xen guest instance, as 64-bit might be causing more overhead in Xen. Someone mentioned this on HN. Did not make a difference.

  4. Try enabling net.ipv4.tcp_syncookies in sysctl.conf as suggested by abofh on HN. This apparently might improve performance since the handshake would occur in the kernel. I had no luck with this.

  5. Increase the backlog from 1024 to something much higher, also suggested by abofh on HN. This could also help since guest could potentially accept() more connections during it’s execution slice given by dom0 (the host).

  6. Double-check that conntrack is disabled on all machines as it can halve the accept rate (suggested by deubeulyou). Yes, it was disabled in all tests.

  7. Check for “listen queue overflow and syncache buckets overflow in netstat -s” (suggested by mike_esspe on HN).

  8. Split the interrupt handling among multiple cores (RPS/RFS I tried enabling earlier are supposed to do this, but could be worth trying again). Suggested by adamt at HN.

  9. Turning off TCP segmentation offload and scatter/gather acceleration as suggested by Matt Bailey. (Not possible on EC2 or similar VPS hosts)

Related Posts:

  1. EC2 ssh Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
  2. Permission denied (publickey). SSH from local Ubuntu to Amazon EC2 server
  3. What does Virtual memory size in top mean?
  4. Why do we use a OS Base Image with Docker if containers have no Guest OS?
  5. How to copy a large number of files quickly between two servers
  6. How to add a security group to a running EC2 Instance?
  7. What Linux distribution is the Amazon Linux AMI based on?
  8. What limits the maximum number of connections on a Linux server?
  9. Why don’t EC2 ubuntu images have swap?
  10. How do I grep recursively?
  11. Why do you need to put #!/bin/bash at the beginning of a script file?
  12. error: ‘Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)’ — Missing /var/run/mysqld/mysqld.sock
  13. No space left on device
  14. “E: Unable to locate package python-pip” on Ubuntu 18.04 [duplicate]
  15. -bash: syntax error near unexpected token `newline’ for display command
  16. How do I use grep to search the current directory for all files having the a string “hello” yet display only .h and .cc files?
  17. SSH using python script
  18. Changing the resolution of a VNC session in linux
  19. Building HelloWorld C++ Program in Linux with ncurses
  20. How does “cat << EOF" work in bash?
  21. Pseudo-terminal will not be allocated because stdin is not a terminal
  22. tar removing leading ‘/’ from member names
  23. X11 forwarding request failed on channel 0
  24. What is difference between arm64 and armhf?
  25. Opening a .tar.gz file with a single command
  26. How to make and apply SVN patch?
  27. Diff command along with Grep gives “Binary file (standard input) matches”
  28. can’t remove python pip
  29. How to download a file from server using SSH?
  30. connect to host localhost port 22: Connection refused
  31. Implementing shell in C and need help handling input/output redirection
  32. How to count lines in a document?
  33. What does `set -x` do?
  34. Can you Run Xcode in Linux?
  35. How to get the process ID to kill a nohup process?
  36. Writing a simple shell in C using fork/execvp
  37. Implementation of multiple pipes in C
  38. chmod: changing permissions of ‘my_script.sh’: Operation not permitted
  39. Using ls to list directories and their total sizes
  40. tar: add all files and directories in current directory INCLUDING .svn and so on
  41. What does set -e mean in a bash script?
  42. Is there an alternative sleep function in C to milliseconds?
  43. Gedit command not working in kali linux terminal
  44. Syntax error near unexpected token ‘then’
  45. Curl : connection refused
  46. make -j 8 g++: internal compiler error: Killed (program cc1plus)
  47. Bash export command
  48. How to edit gitignore file
  49. Pipe to/from the clipboard in a Bash script
  50. what does -zxvf mean in tar -zxvf filename? 
  51. Retrieve last 100 lines logs
  52. Linux Bash: Move multiple different files into same directory
  53. Adding a new entry to the PATH variable in ZSH
  54. Configuring Apache for localhost
  55. how to search for a directory from the terminal in ubuntu
  56. Android – Command not found
  57. How to use dos2unix?
  58. How to add a default include path for GCC in Linux?
  59. What does ‘bash -c’ do?
  60. How do I know the script file name in a Bash script?
  61. WordPress can’t find temporary folder, but folder it’s looking at has correct permissions
  62. How to configure both webserver and appserver for WordPress CMS?
  63. How to remove all plugins, posts, pages, and inactive themes in one line with wp-cli? WordPress bloatware removal
  64. Communcation with wordrpess and linux server [closed]
  65. Anyone else experiencing high rates of Linux server crashes during a leap second day?
  66. Can I nohup/screen an already-started process?
  67. df in linux not showing correct free space after file removal
  68. In my /etc/hosts/ file on Linux/OSX, how do I do a wildcard subdomain?
  69. Setting the hostname: FQDN or short name?
  70. Shell command to monitor changes in a file
  71. When does /tmp get cleared?
  72. Difference in sites-available vs sites-enabled vs conf.d directories (Nginx)?
  73. How do I sleep for a millisecond in bash or ksh
  74. Disk full, du tells different. How to further investigate?
  75. What does ‘set -e’ do, and why might it be considered dangerous?
  76. Filename length limits on linux?
  77. What useful things can one add to one’s .bashrc? [closed]
  78. How to add a timestamp to bash script log?
  79. best way to clear all iptables rules
  80. How to remove empty/blank lines from a file in Unix (including spaces)?
  81. How can I kill all stopped jobs?
  82. How to copy file preserving directory path in Linux?
  83. Postfix – how to retry delivery of mail in queue?
  84. How can I rename a Unix user?
  85. How to re-order windows, change the scroll shortcut, and modify the status bar contents in GNU Screen?
  86. Is it possible to alias a hostname in Linux?
  87. How do I list loaded Linux module parameter values?
  88. Does getting disconnected from an SSH session kill your programs?
  89. Tips for Securing a LAMP Server
  90. How to prevent a user from login in, but allow “su – user” in Linux?
  91. What is this IP address: 169.254.169.254?
  92. Should I install Linux applications in /var or /opt?
  93. I have a keypair. How do I determine the key length?
  94. Why drop caches in Linux?
  95. swap partition vs file for performance?
  96. Best way to disable swap in Linux
  97. Amazon Linux vs. Ubuntu for Amazon EC2 [closed]
  98. How to remove invalid characters from filenames?
  99. How can I zip/compress a symlink?
  100. How to find the physical volume(s) that hold a logical volume in LVM

Leave a Comment