How to Monitoring All Executed Commands in Linux

by Anish


Posted on Tuesday August 21


Referefce 8gwifi.org

Introduction

To safeguards the production environment requires additional control. This can be achieve through logging the user activity and this will be rescued in the scenario like investigating suspicious activity happening on that environment.

The Linux kernel can record a lot of information about processes as they exit. In this article we will go though one of command that can help for audit trail and detective work.

The accton command enables process accounting, and specifies the file used for the audit trail i.e /var/account/pacct .

This file must already exist, so manually create an empty file first if necessary, carefully restricting access to prevent public viewing of the sensitive accounting data.

If the filename is omitted, then the accton command disables process accounting.

  • Prepare the file

       umask 077
      touch /var/account/pacct
    
  • Install psacct or acct

       RHEL :  yum install psacct
      Ubuntu : sudo apt-get install acct
    
  • Check the status of psacct if it’s not started then start it

       systemctl status psacct
      psacct.service - Kernel process accounting
      Loaded: loaded (/usr/lib/systemd/system/psacct.service; disabled; vendor preset: disabled)
      Active: inactive (dead)
    
  • Start the psacct service
    Centos7.x/RHEL7.x

    systemctl start psacct
    systemctl status psacct
    
    psacct.service - Kernel process accounting
    Loaded: loaded (/usr/lib/systemd/system/psacct.service; enabled; vendor preset: disabled)
    Active: **active (exited)** since Tue 2018-08-21 05:17:41 EDT; 56min ago
    Main PID: 9124 (code=exited, status=0/SUCCESS)
    CGroup: /system.slice/psacct.service
    Aug 21 05:17:41 host-192-0-2-74 systemd[1]: Starting Kernel process accounting...
    Aug 21 05:17:41 host-192-0-2-74 accton[9124]: Turning on process accounting, file set to '/var/account/pacct'.
    Aug 21 05:17:41 host-192-0-2-74 systemd[1]: Started Kernel process accounting.
    

    Centos6.x/RHEl6.x

       /etc/init.d/psacct start
      /etc/init.d/acct start    # In Ubuntu
    
  • Enable this changes in Linux runlevel

       chkconfig psacct on
    
  • Note: Logrotate file for psacct, other wise your system will fill-up immediately. In centos/rhel the daily rotation specified in the file /etc/logrotate.d/psacct

    /var/account/pacct {
        compress
        delaycompress
        notifempty
        daily
        rotate 31
        create 0600 root root
        postrotate
           if /usr/bin/systemctl --quiet is-active psacct.service ; then
               /usr/sbin/accton /var/account/pacct | /usr/bin/grep -v "Turning on process accounting, file set to '/var/account/pacct'." | /usr/bin/cat
           fi
        endscript
    }

Summarizes accounting (SA)

The Linux sa summarizes information about previously executed commands as recorded in the acct file. In addition, it condenses this data into a summary file named savacct which contains the number of times the command was called and the system resources used. The information can also be summarized on a per-user basis;

Files Used

fileName Description
acct The raw system wide process accounting file
savacct A summary of system process accounting sorted by command
usracct A summary of system process accounting sorted by user ID

sa output summary

label description
cpu sum of system and user time in cpu minutes
re Elapsed time in minutes
k cpu-time averaged core usage, in 1k units
avio average number of I/O operations per execution
tio total number of I/O operations
k*sec cpu storage integral (kilo-core seconds)
u user cpu time in cpu seconds
s system time in cpu seconds

Command Options

  • If no arguments are specified, sa will print information about all of the commands in the acct file

       # sa
      294     428.58re       0.01cp         0avio     23947k
      29     277.34re       0.00cp         0avio     14085k   ***other*
      14       0.00re       0.00cp         0avio     29616k   troff
      14      14.45re       0.00cp         0avio     29712k   man
      98       0.00re       0.00cp         0avio     29770k   man*
      14      14.44re       0.00cp         0avio     27584k   less
      14       0.01re       0.00cp         0avio     28288k   nroff
      14       0.01re       0.00cp         0avio      3190k   groff
      14       0.01re       0.00cp         0avio     29200k   grotty
      14       0.00re       0.00cp         0avio      3198k   tbl
    
  • User Summary sa -m
    this command will print the number of processes and number of CPU minutes on a per-user basis

    # sa -m 
                                   330     446.58re       0.01cp         0avio     24026k
         root                      330     446.58re       0.01cp         0avio     24026k
    
  • Show userid and command name executed by each user with sa -u

       sa -u 
      ansible    0.00 cpu    28832k mem      0 io bash            *
      ansible    0.00 cpu    26992k mem      0 io touch           
      ansible    0.00 cpu    28864k mem      0 io bash            
      root       0.01 cpu    31264k mem      0 io su              
      root       0.00 cpu     1624k mem      0 io sa              
      root       0.00 cpu    34960k mem      0 io useradd         *
      root       0.00 cpu    34960k mem      0 io useradd         *
      root       0.00 cpu    34960k mem      0 io useradd         *
      root       0.00 cpu    34960k mem      0 io useradd         *
      root       0.00 cpu    34960k mem      0 io useradd         
      test1      0.00 cpu     3842k mem      0 io id              
      test1      0.00 cpu     2920k mem      0 io bash            *
      test1      0.00 cpu     1626k mem      0 io hostname        
    
  • last but not least check the help sa -h to explore all the available options

Print statistics about users (AC)

The linux ac commands prints out a report of connect time (in hours) based on the logins/logouts in the current wtmp file. A total is also printed out.

This accounting file wtmp is maintained by init(8) and login(1). Neither ac nor login creates the wtmp if it doesn't exist, no accounting is done. To begin accounting, create the file with a length of zero.

NOTE: The wtmp file can get really big, really fast. You might want to trim it every once and a while.

Some use-case about ac comand line

  • print statistics of user day wise ac -d
    Print totals for each day rather than just one big total at the end. The output looks like this:

       ac -d
      Aug  3  total 1.17
      Today  total  5.64
    
  • Print total time for each users ac -p

    Print time totals for each user in addition to the usual everything-lumped-into-one value. It looks like:

       ac -p 
      bob 8.06
      goff  0.60
      maley 7.37
      root  0.12
      total  16.15
    

LASTCOMM

lastcomm prints out information about previously executed commands, it uses the file acct

  • Find the commands executed by the user test1

        # lastcomm test1
      
      bash S test1  pts/1  0.00 secs Tue Aug 21 06:09
      touch  test1  pts/1  0.00 secs Tue Aug 21 06:09
      bash  F  test1  pts/1  0.00 secs Tue Aug 21 06:09
      consoletype  test1  pts/1  0.00 secs Tue Aug 21 06:09
      grep test1  pts/1  0.00 secs Tue Aug 21 06:09
    
  • If no arguments are specified, lastcomm will print info about all of the commands in acct(the record file)

  • finding with lastcomm which user has added the new user, here you can see after few fail attempts the user is created by the root

       # lastcomm adduser
      adduser  root pts/1  0.00 secs Tue Aug 21 06:24
      adduser S  root pts/1  0.00 secs Tue Aug 21 06:24
      adduser F  root pts/1  0.00 secs Tue Aug 21 06:24
      adduser F  root pts/1  0.00 secs Tue Aug 21 06:24
      adduser F  root pts/1  0.00 secs Tue Aug 21 06:24
    

LAST

Last searches back through the file /var/log/wtmp and displays a list of all users logged in (and out) since that file was created.

 # last
root pts/1  192.0.2.67 Tue Aug 21 04:23 still logged in
root pts/0  192.0.2.67 Tue Aug 21 03:09 - 05:22  (02:12)
root tty1  Tue Aug 21 03:09 still logged in
reboot system boot  3.10.0-862.el7.x Tue Aug 21 03:08 - 06:29  (03:21)
root tty1  Thu Jul  5 06:59 - 07:00  (00:01)
reboot system boot  3.10.0-862.el7.x Thu Jul  5 06:58 - 07:00  (00:02)
wtmp begins Thu Jul  5 06:58:40 2018

Wtmp

This is not command it's the file on the Linux, Solaris, and BSD operating systems that keeps a history of all logins and logouts. On Linux systems. it is located at /var/log/wtmp. Various commands access wtmp to report login statistics, like who , last, lastb commands.

Monitor changes in File

In production environment certain file integrity needs to be maintained and should be reported for any accidental or purpose fully changes

The linux watch commands allows you to watch the program output change over time. By default, the program is run every 2 seconds. By default, watch will run until interrupted.

 watch -n 10 -d ls -l /etc/shadow

inotifywait from inotify-tools is useful if you want to run a command every time a file (or any files in a directory) change. For example:

 inotifywait -r -m -e modify /var/log | 
   while read path _ file; do 
       echo $path$file modified
   done

Next Reading : Most frequent ps command example to monitor linux process and troubleshooting



Thanku for reading !!! Give a Share for Support

Asking for donation sound bad to me, so i'm raising fund from The Modern Cryptography CookBook for Just $9. Leanpub books Discount coupon first 100 reader. No hurry read the sample chapters here then decide.

Alternatively to support you can buy My all four Cryptography book Just $10.99


Referefce
  • The Modern Cryptography Book.
  • Go lang Cryptography for developers
  • Python Cryptography
  • Cryptography for JavaScript Developer

python Cryptography Topics
Topics
For Coffee/ Beer/ Amazon Bill and further development of the project Support by Purchasing, The Modern Cryptography CookBook for Just $9 Coupon Price

Cryptography for Python Developers

Cryptography for JavaScript Developers

Go lang ryptography for Developers