How to Monitoring All Executed Commands in Linux

Posted on Tuesday August 21

Introduction

To safeguards the production environment requires additional control. This can be achieve through logging the user activity and this will be rescued in the scenario like investigating suspicious activity happening on that environment.

The Linux kernel can record a lot of information about processes as they exit. In this article we will go though one of command that can help for audit trail and detective work.

The accton command enables process accounting, and specifies the file used for the audit trail i.e /var/account/pacct .

This file must already exist, so manually create an empty file first if necessary, carefully restricting access to prevent public viewing of the sensitive accounting data.

If the filename is omitted, then the accton command disables process accounting.

Prepare the file

   umask 077
  touch /var/account/pacct

Install psacct or acct

   RHEL :  yum install psacct
  Ubuntu : sudo apt-get install acct

Check the status of psacct if it’s not started then start it

   systemctl status psacct
  psacct.service - Kernel process accounting
  Loaded: loaded (/usr/lib/systemd/system/psacct.service; disabled; vendor preset: disabled)
  Active: inactive (dead)

Start the psacct service
Centos7.x/RHEL7.x

systemctl start psacct
systemctl status psacct

psacct.service - Kernel process accounting
Loaded: loaded (/usr/lib/systemd/system/psacct.service; enabled; vendor preset: disabled)
Active: **active (exited)** since Tue 2018-08-21 05:17:41 EDT; 56min ago
Main PID: 9124 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/psacct.service
Aug 21 05:17:41 host-192-0-2-74 systemd[1]: Starting Kernel process accounting...
Aug 21 05:17:41 host-192-0-2-74 accton[9124]: Turning on process accounting, file set to '/var/account/pacct'.
Aug 21 05:17:41 host-192-0-2-74 systemd[1]: Started Kernel process accounting.

Centos6.x/RHEl6.x

   /etc/init.d/psacct start
  /etc/init.d/acct start    # In Ubuntu

Enable this changes in Linux runlevel
```
   chkconfig psacct on
```

Note: Logrotate file for psacct, other wise your system will fill-up immediately. In centos/rhel the daily rotation specified in the file /etc/logrotate.d/psacct

/var/account/pacct {
    compress
    delaycompress
    notifempty
    daily
    rotate 31
    create 0600 root root
    postrotate
       if /usr/bin/systemctl --quiet is-active psacct.service ; then
           /usr/sbin/accton /var/account/pacct | /usr/bin/grep -v "Turning on process accounting, file set to '/var/account/pacct'." | /usr/bin/cat
       fi
    endscript
}

Summarizes accounting (SA)

The Linux sa summarizes information about previously executed commands as recorded in the acct file. In addition, it condenses this data into a summary file named savacct which contains the number of times the command was called and the system resources used. The information can also be summarized on a per-user basis;

Files Used

fileName	Description
acct	The raw system wide process accounting file
savacct	A summary of system process accounting sorted by command
usracct	A summary of system process accounting sorted by user ID

sa output summary

label	description
cpu	sum of system and user time in cpu minutes
re	Elapsed time in minutes
k	cpu-time averaged core usage, in 1k units
avio	average number of I/O operations per execution
tio	total number of I/O operations
k*sec	cpu storage integral (kilo-core seconds)
u	user cpu time in cpu seconds
s	system time in cpu seconds

Command Options

If no arguments are specified, sa will print information about all of the commands in the acct file

   # sa
  294     428.58re       0.01cp         0avio     23947k
  29     277.34re       0.00cp         0avio     14085k   ***other*
  14       0.00re       0.00cp         0avio     29616k   troff
  14      14.45re       0.00cp         0avio     29712k   man
  98       0.00re       0.00cp         0avio     29770k   man*
  14      14.44re       0.00cp         0avio     27584k   less
  14       0.01re       0.00cp         0avio     28288k   nroff
  14       0.01re       0.00cp         0avio      3190k   groff
  14       0.01re       0.00cp         0avio     29200k   grotty
  14       0.00re       0.00cp         0avio      3198k   tbl

User Summary sa -m
this command will print the number of processes and number of CPU minutes on a per-user basis

# sa -m 
                               330     446.58re       0.01cp         0avio     24026k
     root                      330     446.58re       0.01cp         0avio     24026k

Show userid and command name executed by each user with sa -u

   sa -u 
  ansible    0.00 cpu    28832k mem      0 io bash            *
  ansible    0.00 cpu    26992k mem      0 io touch           
  ansible    0.00 cpu    28864k mem      0 io bash            
  root       0.01 cpu    31264k mem      0 io su              
  root       0.00 cpu     1624k mem      0 io sa              
  root       0.00 cpu    34960k mem      0 io useradd         *
  root       0.00 cpu    34960k mem      0 io useradd         *
  root       0.00 cpu    34960k mem      0 io useradd         *
  root       0.00 cpu    34960k mem      0 io useradd         *
  root       0.00 cpu    34960k mem      0 io useradd         
  test1      0.00 cpu     3842k mem      0 io id              
  test1      0.00 cpu     2920k mem      0 io bash            *
  test1      0.00 cpu     1626k mem      0 io hostname

last but not least check the help sa -h to explore all the available options

Print statistics about users (AC)

The linux ac commands prints out a report of connect time (in hours) based on the logins/logouts in the current wtmp file. A total is also printed out.

This accounting file wtmp is maintained by init(8) and login(1). Neither ac nor login creates the wtmp if it doesn't exist, no accounting is done. To begin accounting, create the file with a length of zero.

NOTE: The wtmp file can get really big, really fast. You might want to trim it every once and a while.

Some use-case about ac comand line

print statistics of user day wise ac -d
Print totals for each day rather than just one big total at the end. The output looks like this:
```
   ac -d
  Aug  3  total 1.17
  Today  total  5.64
```
Print total time for each users ac -p

Print time totals for each user in addition to the usual everything-lumped-into-one value. It looks like:
```
   ac -p 
  bob 8.06
  goff  0.60
  maley 7.37
  root  0.12
  total  16.15
```

LASTCOMM

lastcomm prints out information about previously executed commands, it uses the file acct

Find the commands executed by the user test1

    # lastcomm test1
  
  bash S test1  pts/1  0.00 secs Tue Aug 21 06:09
  touch  test1  pts/1  0.00 secs Tue Aug 21 06:09
  bash  F  test1  pts/1  0.00 secs Tue Aug 21 06:09
  consoletype  test1  pts/1  0.00 secs Tue Aug 21 06:09
  grep test1  pts/1  0.00 secs Tue Aug 21 06:09

If no arguments are specified, lastcomm will print info about all of the commands in acct(the record file)

finding with lastcomm which user has added the new user, here you can see after few fail attempts the user is created by the root

   # lastcomm adduser
  adduser  root pts/1  0.00 secs Tue Aug 21 06:24
  adduser S  root pts/1  0.00 secs Tue Aug 21 06:24
  adduser F  root pts/1  0.00 secs Tue Aug 21 06:24
  adduser F  root pts/1  0.00 secs Tue Aug 21 06:24
  adduser F  root pts/1  0.00 secs Tue Aug 21 06:24

LAST

Last searches back through the file /var/log/wtmp and displays a list of all users logged in (and out) since that file was created.

 # last
root pts/1  192.0.2.67 Tue Aug 21 04:23 still logged in
root pts/0  192.0.2.67 Tue Aug 21 03:09 - 05:22  (02:12)
root tty1  Tue Aug 21 03:09 still logged in
reboot system boot  3.10.0-862.el7.x Tue Aug 21 03:08 - 06:29  (03:21)
root tty1  Thu Jul  5 06:59 - 07:00  (00:01)
reboot system boot  3.10.0-862.el7.x Thu Jul  5 06:58 - 07:00  (00:02)
wtmp begins Thu Jul  5 06:58:40 2018

Wtmp

This is not command it's the file on the Linux, Solaris, and BSD operating systems that keeps a history of all logins and logouts. On Linux systems. it is located at /var/log/wtmp. Various commands access wtmp to report login statistics, like who , last, lastb commands.

Monitor changes in File

In production environment certain file integrity needs to be maintained and should be reported for any accidental or purpose fully changes

The linux watch commands allows you to watch the program output change over time. By default, the program is run every 2 seconds. By default, watch will run until interrupted.

 watch -n 10 -d ls -l /etc/shadow

inotifywait from inotify-tools is useful if you want to run a command every time a file (or any files in a directory) change. For example:

 inotifywait -r -m -e modify /var/log | 
   while read path _ file; do 
       echo $path$file modified
   done

Next Reading : Most frequent ps command example to monitor linux process and troubleshooting

Thanku for reading !!! Give a Share for Support

Your Support Matters!

Instead of directly asking for donations, I'm thrilled to offer you all nine of my books for just $9 on leanpub By grabbing this bundle you not only help cover my coffee, beer, and Amazon bills but also play a crucial role in advancing and refining this project. Your contribution is indispensable, and I'm genuinely grateful for your involvement in this journey!

Any private key value that you enter or we generate is not stored on this site, this tool is provided via an HTTPS URL to ensure that private keys cannot be stolen, for extra security run this software on your network, no cloud dependency