Log all commands run by admins on production servers

Update:
2 more things that have popped up in the comments and in follow-up questions:

  • Using auditd this way will dramatically increase your log volume, especially if the system is heavily in use via commandline. Adjust your log retention policy.
  • Auditd logs on the host where they are created are just as secure as other files on the same box. Forward your logs to a remote log collection server like ELK or Graylog to preserve your logs’ integrity. Plus, adding to the point above, it allows to more aggressively delete old logs.

As was suggested by Michael Hampton, auditd is the correct tool for the job here.

I tested this on an Ubuntu 12.10 installation, so your mileage may vary on other systems.

  • Install auditd:

    apt-get install auditd

  • Add these 2 lines to /etc/audit/audit.rules:

    -a exit,always -F arch=b64 -F euid=0 -S execve
-a exit,always -F arch=b32 -F euid=0 -S execve

These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.

  • To get rid of auid=4294967295 messages in logs, add audit=1 to the kernel’s cmdline (by editing /etc/default/grub)

  • Place the line

    session required pam_loginuid.so

in all PAM config files that are relevant to login (/etc/pam.d/{login,kdm,sshd}), but not in the files that are relevant to su or sudo.
This will allow auditd to get the calling user’s uid correctly when calling sudo or su.

  • Restart your system now.

  • Let’s login and run some commands:

    $ id -u
    1000
    $ sudo ls /
    bin  boot  data  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  scratch  selinux  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old
    $ sudo su -
    # ls /etc
    [...]

This will yield something like this in /var/log/audit/auditd.log:

----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.239:576): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.239:576): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.239:576):  cwd="/home/user"
type=EXECVE msg=audit(1359968226.239:576): argc=2 a0="ls" a1="https://serverfault.com/"
type=SYSCALL msg=audit(1359968226.239:576): arch=c000003e syscall=59 success=yes exit=0 a0=10cfc48 a1=10d07c8 a2=10d5750 a3=7fff2eb2d1f0 items=2 ppid=26569 pid=26570 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)
----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.231:575): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.231:575): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.231:575):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968226.231:575): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968226.231:575): argc=3 a0="sudo" a1="ls" a2="https://serverfault.com/"
type=SYSCALL msg=audit(1359968226.231:575): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26569 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.523:578): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.523:578): item=0 name="/bin/su" inode=44 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.523:578):  cwd="/home/user"
type=EXECVE msg=audit(1359968229.523:578): argc=2 a0="su" a1="-"
type=SYSCALL msg=audit(1359968229.523:578): arch=c000003e syscall=59 success=yes exit=0 a0=1ceec48 a1=1cef7c8 a2=1cf4750 a3=7fff083bd920 items=2 ppid=26611 pid=26612 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="su" exe="/bin/su" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.519:577): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.519:577): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.519:577):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968229.519:577): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968229.519:577): argc=3 a0="sudo" a1="su" a2="-"
type=SYSCALL msg=audit(1359968229.519:577): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26611 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.543:585): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.543:585): item=0 name="/bin/bash" inode=6941 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.543:585):  cwd="/root"
type=EXECVE msg=audit(1359968229.543:585): argc=1 a0="-su"
type=SYSCALL msg=audit(1359968229.543:585): arch=c000003e syscall=59 success=yes exit=0 a0=13695a0 a1=7fffce08a3e0 a2=135a030 a3=7fffce08c200 items=2 ppid=26612 pid=26622 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="bash" exe="/bin/bash" key=(null)
----
time->Mon Feb  4 09:57:11 2013
type=PATH msg=audit(1359968231.663:594): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968231.663:594): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968231.663:594):  cwd="/root"
type=EXECVE msg=audit(1359968231.663:594): argc=3 a0="ls" a1="--color=auto" a2="/etc"
type=SYSCALL msg=audit(1359968231.663:594): arch=c000003e syscall=59 success=yes exit=0 a0=7fff8c709950 a1=7f91a12149d8 a2=1194c50 a3=7fff8c709510 items=2 ppid=26622 pid=26661 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)

The auid column contains the calling user’s uid, which allows you filter for commands run by this user with

 ausearch -ua 1000

This will even list commands the user ran as root.

Sources:

  • http://www.woitasen.com.ar/2011/11/auditing-user-actions-after-sudo/
  • http://linux.die.net/man/8/pam_loginuid
  • http://linux.die.net/man/8/auditd

Related Posts:

  1. No acceptable C compiler found in $PATH while installing the C compiler
  2. Failed to start LSB :Bring Up down Networking
  3. HAProxy vs. Nginx
  4. Nginx 403 forbidden for all files
  5. CentOS error – sudo: effective uid is not 0, is sudo installed setuid root?
  6. bash: mkvirtualenv: command not found
  7. rndc: connect failed: 127.0.0.1#953: connection refused
  8. HAProxy vs. Nginx
  9. How to add a timestamp to bash script log?
  10. How do you install Node.JS on CentOS?
  11. How do I update a CentOS server’s time from an authoritative time server?
  12. Best way to gracefully restart CentOS?
  13. Adding a directory to $PATH in CentOS?
  14. How to extend an ext4 partition and filesystem?
  15. My /var/log/btmp file is huge! What should I do?
  16. How can I run arbitrarily complex command using sudo over ssh?
  17. Centos 7 save iptables settings
  18. How to open port for a specific IP address with firewall-cmd on CentOS? [duplicate]
  19. How do I deal with a filename that starts with the hyphen (-) character?
  20. Bash: No such file or directory?
  21. How to split a string into an array in Bash?
  22. How to create a file in Linux from terminal window? [closed]
  23. What does export PS1=”\[\033[36m\]\u\[\033[m\]@\[\033[32m\]\h:\[\033[33;1m\]\w\[\033[m\]\$ ” mean in MacOS’ bash Terminal?
  24. How do I use a regex in a shell script?
  25. An example of how to use getopts in bash
  26. How do I execute a bash script in Terminal?
  27. Installing Homebrew on OS X
  28. Multi-line string with extra space (preserved indentation)
  29. No space left on device
  30. Loop through an array of strings in Bash?
  31. ‘\r’: command not found – .bashrc / .bash_profile [duplicate]
  32. Pass a password to ssh in pure bash
  33. How do I iterate over a range of numbers defined by variables in Bash?
  34. Echo newline in Bash prints literal \n
  35. How to install “make” in ubuntu?
  36. How can I declare and use Boolean variables in a shell script?
  37. Rails: Why “sudo” command is not recognized?
  38. javac : command not found
  39. Replace one substring for another string in shell script
  40. Parsing JSON with Unix tools
  41. How to check if a variable is set in Bash?
  42. mvn command not found in OSX Mavrerick
  43. Using Bash regex match (=~) where regex includes quotes (” characters)
  44. “[ ]” vs. “[[ ]]” in Bash shell
  45. Pseudo-terminal will not be allocated because stdin is not a terminal
  46. Cannot use mkdir in home directory: permission denied (Linux Lubuntu)
  47. Using find command in bash script
  48. Emulating a do-while loop in Bash
  49. Git Bash won’t run my python files?
  50. How to change the output color of echo in Linux
  51. Recursively find all files that match a certain pattern
  52. find: missing argument to -exec
  53. How do I set a variable to the output of a command in Bash?
  54. Grep ‘binary file matches’. How to get normal grep output?
  55. Trying to use bash on Windows and got no installed distributions message
  56. Bash if statement with multiple conditions throws an error
  57. What is the difference between “#!/usr/bin/env bash” and “#!/usr/bin/bash”?
  58. Laravel PHP Command Not Found
  59. bash : Bad Substitution
  60. Getting an “ambiguous redirect” error
  61. Is there a “goto” statement in bash?
  62. Difference between return and exit in Bash functions
  63. How to modify a global variable within a function in bash?
  64. How do I automatically restart a Minecraft Spigot server in the event of a crash or /stop when using screen?
  65. Using SED with wildcard
  66. npm install errors with Error: ENOENT, chmod
  67. How can I check if a program exists from a Bash script?
  68. Bash command to sum a column of numbers
  69. Is there anything in Zsh like .bash_profile?
  70. Running programs in parallel using xargs
  71. Java command not found on Linux
  72. How to output a multiline string in Bash?
  73. Shell script not running, command not found
  74. How to ssh from within a bash script?
  75. Bash script prints “Command Not Found” on empty lines
  76. E/Surface﹕ getSlotFromBufferLocked: unknown buffer: 0xab7519c0
  77. What is a simple explanation for how pipes work in Bash?
  78. Integer expression expected error in shell script
  79. Virtualenv not compatible with this system or executable
  80. Swift: print() vs println() vs NSLog()
  81. Error when using ‘sed’ with ‘find’ command on OS X: “invalid command code .”
  82. Core framework/helpers for logging stuff?
  83. Does the debug.log do log rotation?
  84. Same log message keeps on printing to debug.log file thousand of times
  85. Can I hook error_log(…)?
  86. Debug log file with rolling datestamp in filename?
  87. Debugging – logging database queries
  88. My account staying logged when the browser closed
  89. Debug info from request handler
  90. Check if array is empty in Bash
  91. How do I prevent accidental rm -rf /*?
  92. Run an interactive bash subshell with initial commands without returning to the (“super”) shell immediately
  93. What’s the best way to check if a volume is mounted in a Bash script?
  94. How can I implement ansible with per-host passwords, securely?
  95. Linux command line best practices and tips?
  96. How to run command as user who has /usr/sbin/nologin as Shell?
  97. How to get pid of just started process
  98. Is there a proper way to clear logs?
  99. How can I find the path to an executable in OSX
  100. How should an IT department choose a standard Linux distribution?

Leave a Comment