20 Sed (Stream Editor) Command Examples for Linux Users

Sed command or Stream Editor is very powerful utility offered by Linux/Unix systems. It is mainly used for text substitution , find & replace but it can also perform other text manipulations like insertion, deletion, search etc. With SED, we can edit complete files without actually having to open it. Sed also supports the use of regular expressions, which makes sed an even more powerful test manipulation tool.

In this article, we will learn to use SED command with the help some examples. Basic syntax for using sed command is,

sed OPTIONS… [SCRIPT] [INPUTFILE…]

Now let’s see some examples.

Example :1) Displaying partial text of a file

With sed, we can view only some part of a file rather than seeing whole file. To see some lines of the file, use the following command,

[lpicentral@localhost ~]$ sed -n 22,29p testfile.txt

here, option ‘n’ suppresses printing of whole file & option ‘p’ will print only line lines from 22 to 29.

Example :2) Display all except some lines

To display all content of a file except for some portion, use the following command,

[lpicentral@localhost ~]$ sed 22,29d testfile.txt

Option ‘d’ will remove the mentioned lines from output.

Example :3) Display every 3rd line starting with Nth line

Do display content of every 3rd line starting with line number 2 or any other line, use the following command

[lpicentral@localhost ~]$ sed -n '2-3p' file.txt

Example :4 ) Deleting a line using sed command

To delete a line with sed from a file, use the following command,

[lpicentral@localhost ~]$ sed Nd testfile.txt

where ‘N’ is the line number & option ‘d’ will delete the mentioned line number. To delete the last line of the file, use

[lpicentral@localhost ~]$ sed $d testfile.txt

Example :5) Deleting a range of lines

To delete a range of lines from the file, run

[lpicentral@localhost ~]$ sed '29-34d' testfile.txt

This will delete lines 29 to 34 from testfile.txt file.

Example :6) Deleting lines other than the mentioned

To delete lines other than the mentioned lines from a file, we will use ‘!’

[lpicentral@localhost ~]$ sed '29-34!d' testfile.txt

here ‘!’ option is used as not, so it will reverse the condition i.e. will not delete the lines mentioned. All the lines other 29-34 will be deleted from the files testfile.txt.

Example :7) Adding Blank lines/spaces

To add a blank line after every non-blank line, we will use option ‘G’,

[lpicentral@localhost ~]$ sed G testfile.txt

Example :8) Search and Replacing a string using sed

To search & replace a string from the file, we will use the following example,

[lpicentral@localhost ~]$ sed 's/danger/safety/' testfile.txt

here option ‘s’ will search for word ‘danger’ & replace it with ‘safety’ on every line for the first occurrence only.

Example :9) Search and replace a string from whole file using sed

To replace the word completely from the file, we will use option ‘g’  with ‘s’,

[lpicentral@localhost ~]$ sed 's/danger/safety/g' testfile.txt

Example :10) Replace the nth occurrence of string pattern

We can also substitute a string on nth occurrence from a file. Like replace ‘danger’ with ‘safety’ only on second occurrence,

[lpicentral@localhost ~]$ sed ‘s/danger/safety/2’ testfile.txt

To replace ‘danger’ on 2nd occurrence of every line from whole file, use

[lpicentral@localhost ~]$ sed 's/danger/safety/2g' testfile.txt

Example :11) Replace a string on a particular line

To replace a string only from a particular line, use

[lpicentral@localhost ~]$ sed '4 s/danger/safety/' testfile.txt

This will only substitute the string from 4th line of the file. We can also mention a range of lines instead of a single line,

[lpicentral@localhost ~]$  sed '4-9 s/danger/safety/' testfile.txt

Example :12) Add a line after/before the matched search

To add a new line with some content after every pattern match, use option ‘a’ ,

[lpicentral@localhost ~]$ sed '/danger/a "This is new line with text after match"' testfile.txt

To add a new line with some content a before every pattern match, use option ‘i’,

[lpicentral@localhost ~]$ sed '/danger/i "This is new line with text before match" ' testfile.txt

Example :13) Change a whole line with matched pattern

To change a whole line to a new line when a search pattern matches we need to use option ‘c’ with sed,

[lpicentral@localhost ~]$ sed '/danger/c "This will be the new line" ' testfile.txt

So when the pattern matches ‘danger’, whole line will be changed to the mentioned line.

Advanced options with sed

Up until now we were only using simple expressions with sed, now we will discuss some advanced uses of sed with regex,

Example :14) Running multiple sed commands

If we need to perform multiple sed expressions, we can use option ‘e’ to chain the sed commands,

[lpicentral@localhost ~]$  sed -e 's/danger/safety/g' -e 's/hate/love/' testfile.txt

Example :15) Making a backup copy before editing a file

To create a backup copy of a file before we edit it, use option ‘-i.bak’,

[lpicentral@localhost ~]$ sed -i.bak -e 's/danger/safety/g'  testfile.txt

This will create a backup copy of the file with extension .bak. You can also use other extension if you like.

Example :16) Delete a file line starting with & ending with a pattern

To delete a file line starting with a particular string & ending with another string, use

[lpicentral@localhost ~]$ sed -e 's/danger.*stops//g' testfile.txt

This will delete the line with ‘danger’ on start & ‘stops’ in the end & it can have any number of words in between , ‘.*’ defines that part.

Example :17) Appending lines

To add some content before every line with sed & regex, use

[lpicentral@localhost ~]$ sed -e 's/.*/testing sed &/' testfile.txt

So now every line will have ‘testing sed’ before it.

Example :18) Removing all commented lines & empty lines

To remove all commented lines i.e. lines with # & all the empty lines,  use

[lpicentral@localhost ~]$ sed -e 's/#.*//;/^$/d' testfile.txt

To only remove commented lines, use

[lpicentral@localhost ~]$ sed -e 's/#.*//' testfile.txt

Example :19) Get list of all usernames from /etc/passwd

To get the list of all usernames from /etc/passwd file, use

[lpicentral@localhost ~]$  sed 's/\([^:]*\).*/\1/' /etc/passwd

a complete list all usernames will be generated on screen as output.

Example :20) Prevent overwriting of system links with sed command

‘sed -i’ command has been know to remove system links & create only regular files in place of the link file. So to avoid such a situation & prevent ‘sed -i‘ from destroying the links, use ‘–follow-symklinks‘ options with the command being executed.

Let’s assume i want to disable SELinux on CentOS or RHEL Severs

[lpicentral@localhost ~]# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

These were some examples to show sed, we can use these reference to employ them as & when needed.

Continue reading

10 useful ncat (nc) Command Examples for Linux Systems

ncat or nc is networking utility with functionality similar to cat command but for network. It  is a general purpose CLI tool for reading, writing, redirecting data across a network. It is  designed to be a reliable back-end tool that can be used with scripts or other programs.  It’s also a great tool for network debugging, as it can create any kind of connect one can need.

ncat/nc can be a port scanning tool, or a security tool, or monitoring tool and is also a simple TCP proxy.  Since it has so many features, it is known as a network swiss army knife. It’s one of those tools that every System Admin should know & master.

In most of Debian distributions ‘nc’ is available and its package is automatically installed during installation. But in minimal CentOS 7 / RHEL 7 installation you will not find nc as a default package. You need to install using the following command.

[root@lpicentral ~]# yum install nmap-ncat -y

System admins can use it audit their system security, they can use it find the ports that are opened & than secure them. Admins can also use it as a client for auditing web servers, telnet servers, mail servers etc, with ‘nc’ we can control every character sent & can also view the responses to sent queries.

We can also cause it to capture data being sent by client to understand what they are upto.

In this tutorial, we are going to learn about how to use ‘nc’ command with 10 examples,

Example: 1) Listen to inbound connections

Ncat can work in listen mode & we can listen for inbound connections on port number with option ‘l’. Complete command is,

$ ncat -l port_number

For example,

$ ncat -l 8080

Server will now start listening to port 8080 for inbound connections.

Example: 2) Connect to a remote system

To connect to a remote system with nc, we can use the following command,

$ ncat IP_address port_number

Let’s take an example,

$ ncat 192.168.1.100 80

Now a connection to server with IP address 192.168.1.100 will be made at port 80 & we can now send instructions to server. Like we can get the complete page content with

GET / HTTP/1.1

or get the page name,

GET / HTTP/1.1

or we can get banner for OS fingerprinting with the following,

HEAD / HTTP/1.1

This will tell what software is being used to run the web Server.

Example: 3) Connecting to UDP ports

By default , the nc utility makes connections only to TCP ports. But we can also make connections to UDP ports, for that we can use option ‘u’,

$ ncat -l -u 1234

Now our system will start listening a udp port ‘1234’, we can verify this using below netstat command,

$ netstat -tunlp | grep 1234
udp           0          0 0.0.0.0:1234                 0.0.0.0:*               17341/nc
udp6          0          0 :::1234                      :::*                    17341/nc

Let’s assume we want to send or test UDP port connectivity to a specific remote host, then use the following command,

$ ncat -v -u {host-ip} {udp-port}

example:

[root@localhost ~]# ncat -v -u 192.168.105.150 53
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 192.168.105.150:53.

Example: 4) NC as chat tool

NC can also be used as chat tool, we can configure server to listen to a port & than can make connection to server from a remote machine on same port & start sending message. On server side, run

$ ncat -l 8080

On remote client machine, run

$ ncat 192.168.1.100 8080

Than start sending messages & they will be displayed on server terminal.

Example: 5) NC as a proxy

NC can also be used as a proxy with a simple command. Let’s take an example,

$ ncat -l 8080 | ncat 192.168.1.200 80

Now all the connections coming to our server on port 8080 will be automatically redirected to 192.168.1.200 server on port 80. But since we are using a pipe, data can only be transferred & to be able to receive the data back, we need to create a two way pipe. Use the following commands to do so,

$ mkfifo 2way
$ ncat -l 8080 0<2way | ncat 192.168.1.200 80 1>2way

Now you will be able to send & receive data over nc proxy.

Example: 6) Copying Files using nc/ncat

NC can also be used to copy the files from one system to another, though it is not recommended & mostly all systems have ssh/scp installed by default. But none the less if you have come across a system with no ssh/scp, you can also use nc as last ditch effort.

Start with machine on which data is to be received & start nc is listener mode,

$ ncat -l  8080 > file.txt

Now on the machine from where data is to be copied, run the following command,

$ ncat 192.168.1.100 8080 --send-only < data.txt

Here, data.txt is the file that has to be sent. –send-only option will close the connection once the file has been copied. If not using this option, than we will have press ctrl+c to close the connection manually.

We can also copy entire disk partitions using this method, but it should be done with caution.

Example: 7) Create a backdoor via nc/nact

NC command can also be used to create backdoor to your systems & this technique is actually used by hackers a lot. We should know how it works in order to secure our system. To create a backdoor, the command is,

$ ncat -l 10000 -e /bin/bash

‘e‘ flag attaches a bash to port 10000. Now a client can connect to port 10000 on server & will have complete access to our system via bash,

$ ncat 192.168.1.100 1000

Example: 8) Port forwarding via nc/ncat

We can also use NC for port forwarding with the help of option ‘c’ , syntax for accomplishing port forwarding is,

$ ncat -u -l  80 -c  'ncat -u -l 8080'

Now all the connections for port 80 will be forwarded to port 8080.

Example: 9) Set Connection timeouts

Listener mode in ncat will continue to run & would have to be terminated manually. But we can configure timeouts with option ‘w’,

$ ncat -w 10 192.168.1.100 8080

This will cause connection to be terminated in 10 seconds, but it can only be used on client side & not on server side.

Example: 10) Force server to stay up using -k option in ncat

When client disconnects from server, after sometime server also stops listening. But we can force server to stay connected & continuing port listening with option ‘k’. Run the following command,

$ ncat -l -k 8080

Now server will stay up, even if a connection from client is broken.

Continue reading

11 Useful split command examples for Linux/UNIX systems

As the name suggests ‘split‘ command is used to split or break a file into the pieces in Linux and UNIX systems. Whenever we split a large file with split command then split output file’s default size is 1000 lines and its default prefix would be ‘x’.

In this article we will discuss 11 useful split command examples for Linux Users. Apart from this we will also discuss how split files can be merge or reassembled into a single file. Syntax for split command:

# split {options} {file_name} {prefix}

Some of the important options of split command is shown below:

Example: 1) Split File into Pieces

Let’s assume we have file name with tuxlap.txt, Use below split command to break into the pieces

[root@lpicentral ~]# split tuxlap.txt
[root@lpicentral ~]# ll
total 32
-rw-------. 1 root root  980 Aug 12 00:11 anaconda-ks.cfg
-rw-r--r--. 1 root root 9607 Nov 11 03:22 tuxlap.txt
-rw-r--r--. 1 root root 8744 Nov 11 03:23 xaa
-rw-r--r--. 1 root root  863 Nov 11 03:23 xab
[root@lpicentral ~]#

As we can see the above output ‘tuxlab.txt‘ is split into two pieces with the name ‘xaa’ and ‘xab’.

Example: 2) Split Command with verbose option

We can run split command in verbose mode with option ‘–verbose‘, example is shown below:

[root@lpicentral ~]# split tuxlap.txt --verbose
creating file ‘xaa’
creating file ‘xab’
[root@lpicentral ~]#

Example: 3) Split files with customize line numbers (-l)

Let’s suppose we want to split a file with customize line numbers, let say I want max 200 lines per file.

To achieve this, use  ‘-l’ option in split command.

[root@lpicentral ~]# split -l200 tuxlap.txt --verbose
creating file ‘xaa’
creating file ‘xab’
creating file ‘xac’
creating file ‘xad’
creating file ‘xae’
creating file ‘xaf’
[root@lpicentral ~]#

Verify the lines of each file using below command

[root@lpicentral ~]# wc -l xa*
 200 xaa
 200 xab
 200 xac
 200 xad
 200 xae
  91 xaf
1091 total
[root@lpicentral ~]#

Example: 4) Split files with file size using option  -b

Using Split command we can split a file with file size. Use the following syntax to split files with size in bytes, KB , MB and GB

# split  -b{bytes}  {file_name}

# split  -b  nK      {file_name}    // n is the numeric value

# split  -b   nM    {file_name}      // n is the numeric value

# split  -b   nG     {file_name}     // n is the numeric value

Split file based on bytes:

[root@lpicentral ~]# split -b2000000 tuxlap.txt

Split file based on KB:

[root@lpicentral ~]# split -b 50K tuxlap.txt

Split file based on MB:

[root@lpicentral ~]# split -b 50M tuxlap.txt

Split file based on GB:

[root@lpicentral ~]# split -b 1G tuxlap.txt

Example: 5) Create Split files with numeric suffix instead of alphabetic (-d)

In the above examples we have seen that split command output files are created with alphabetic suffix like xaa, xab….. xan , Use ‘-d’ option with split command to create split output files with numeric suffix like x00, x01, … x0n

[root@lpicentral ~]# split -d tuxlap.txt

[root@lpicentral ~]# ll
total 1024256
-rw-------. 1 root root        980 Aug 12 00:11 anaconda-ks.cfg
-rwx------. 1 root root 1048576000 Nov 11 03:54 linux-lite.iso
-rw-r--r--. 1 root root     120010 Nov 11 04:39 tuxlap.txt
-rw-r--r--. 1 root root      11998 Nov 11 04:41 x00
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x01
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x02
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x03
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x04
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x05
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x06
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x07
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x08
-rw-r--r--. 1 root root      12000 Nov 11 04:41 x09
-rw-r--r--. 1 root root         12 Nov 11 04:41 x10
[root@lpicentral ~]#

Example: 6) Split file with Customize Suffix

With split command we can create split output files with customize suffix. Let’s assume we want to create split output files with customize suffix

Syntax:

# split  {file_name}  {prefix_name}

[root@lpicentral ~]# split tuxlap.txt split_file_

[root@lpicentral ~]# ll
total 1024248
-rw-------. 1 root root        980 Aug 12 00:11 anaconda-ks.cfg
-rwx------. 1 root root 1048576000 Nov 11 03:54 linux-lite.iso
-rw-r--r--. 1 root root      11998 Nov 11 04:56 split_file_aa
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ab
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ac
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ad
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ae
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_af
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ag
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ah
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_ai
-rw-r--r--. 1 root root      12000 Nov 11 04:56 split_file_aj
-rw-r--r--. 1 root root         12 Nov 11 04:56 split_file_ak
-rw-r--r--. 1 root root     120010 Nov 11 04:39 tuxlap.txt
[root@lpicentral ~]#

Example: 7) Generate n chunks output files with split command (-n)

Let’s suppose we want to split an iso file into 4 chunk output files. Use ‘-n’ option with split command limit the number of split output files.

[root@lpicentral ~]# split -n5 linux-lite.iso

Verify the Split out files using ll command.

[root@lpicentral ~]# ll
total 2048124
-rw-------. 1 root root        980 Aug 12 00:11 anaconda-ks.cfg
-rwx------. 1 root root 1048576000 Nov 11 03:54 linux-lite.iso
-rw-r--r--. 1 root root     120010 Nov 11 04:39 tuxlap.txt
-rw-r--r--. 1 root root  209715200 Nov 11 05:22 xaa
-rw-r--r--. 1 root root  209715200 Nov 11 05:22 xab
-rw-r--r--. 1 root root  209715200 Nov 11 05:22 xac
-rw-r--r--. 1 root root  209715200 Nov 11 05:23 xad
-rw-r--r--. 1 root root  209715200 Nov 11 05:23 xae
[root@lpicentral ~]#

Example: 8) Prevent Zero Size Split output files with option (-e)

There can be some scenarios where we split a small file into a large number of chunk files and zero size split output files can be created in such cases, so to avoid zero size split output file, use the option ‘-e’

[root@lpicentral ~]# split -n60 -e tuxlap.txt
[root@lpicentral ~]# ls -l x*
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xaa
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xab
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xac
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xad
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xae
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xaf
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xag
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xah
.............
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xce
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xcf
-rw-r--r--. 1 root root 2000 Nov 11 05:34 xcg
-rw-r--r--. 1 root root 2010 Nov 11 05:34 xch
[root@lpicentral ~]#

Example:9) Create Split output files of customize suffix length (-a option)

Let’s suppose we want to split an iso file and where size of each split output file is 500MB and suffix length is to be 3.  Use the following split command:

[root@lpicentral ~]# split -b 500M linux-lite.iso -a 3

[root@lpicentral ~]# ll
total 2048124
-rw-------. 1 root root        980 Aug 12 00:11 anaconda-ks.cfg
-rwx------. 1 root root 1048576000 Nov 11 03:54 linux-lite.iso
-rw-r--r--. 1 root root     120010 Nov 11 04:39 tuxlap.txt
-rw-r--r--. 1 root root  524288000 Nov 11 05:43 xaaa
-rw-r--r--. 1 root root  524288000 Nov 11 05:43 xaab
[root@lpicentral ~]#

Example: 10) Split ISO file and merge it into a single file.

Let’s suppose we have a Windows Server ISO file of size 4.2 GB and we are unable to scp this file to remote server because of its size.

To resolve such type of issues we can split the ISO into n number of pieces and will copy these pieces to remote sever and on the remote server we can merge these pieces into a single file using cat command,

[root@lpicentral ~]# split -b 800M Windows2012r2.iso Split_IS0_

View the split output files using ll command,

[root@lpicentral ~]# ll
total 8871788
-rw-------. 1 root root        980 Aug 12 00:11 anaconda-ks.cfg
-rw-r--r--. 1 root root  838860800 Nov 11 06:29 Split_IS0_aa
-rw-r--r--. 1 root root  838860800 Nov 11 06:29 Split_IS0_ab
-rw-r--r--. 1 root root  838860800 Nov 11 06:29 Split_IS0_ac
-rw-r--r--. 1 root root  838860800 Nov 11 06:29 Split_IS0_ad
-rw-r--r--. 1 root root  838860800 Nov 11 06:29 Split_IS0_ae
-rw-r--r--. 1 root root  347987968 Nov 11 06:29 Split_IS0_af
-rw-r--r--. 1 root root     120010 Nov 11 04:39 tuxlap.txt
-rwx------. 1 root root 4542291968 Nov 11 06:03 Windows2012r2.iso
[root@lpicentral ~]#

Now scp these files to remote server and merge these files into a single using cat command

[root@lpicentral ~]# cat Split_IS0_a* > Windows_Server.iso
[root@lpicentral ~]#

Example: 11) Verify the Integrity of Merge file using md5sum utility

As per Example 10, once the split output files are merged into a single file, then we can check the integrity of actual & merge file with md5sum utility. Example is shown below:

[root@lpicentral ~]# md5sum Windows2012r2.iso
5b5e08c490ad16b59b1d9fab0def883a  Windows2012r2.iso
[root@lpicentral ~]#

[root@lpicentral ~]# md5sum Windows_Server.iso
5b5e08c490ad16b59b1d9fab0def883a  Windows_Server.iso
[root@lpicentral ~]#

As per the above output, it is confirm that integrity is maintained and we can also say split file are successfully restored to a single file.

Continue reading

10 Quick Tips About sudo command for Linux systems

Overview

sudo stands for superuser do. It allows authorized users to execute command as an another user. Another user can be regular user or superuser. However, most of the time we use it to execute command with elevated privileges.

sudo command works in conjunction with security policies, default security policy is sudoers and it is configurable via /etc/sudoers file. Its security policies are highly extendable. One can develop and distribute their own policies as plugins.

How it’s different than su

In GNU/Linux there are two ways to run command with elevated privileges:

◈ Using su command
◈ Using sudo command

su stands for switch user. Using su, we can switch to root user and execute command. But there are few drawbacks with this approach.

◈ We need to share root password with another user.
◈ We cannot give controlled access as root user is superuser
◈ We cannot audit what user is doing.

sudo addresses these problems in unique way.

1. First of all, we don’t need to compromise root user password. Regular user uses its own password to execute command with elevated privileges.
2. We can control access of sudo user meaning we can restrict user to execute only certain commands.
3. In addition to this all activities of sudo user are logged hence we can always audit what actions were done. On Debian based GNU/Linux all activities are logged in /var/log/auth.log file.

Later sections of this tutorial sheds light on these points.

Hands on with sudo

Now, we have fair understanding about sudo. Let us get our hands dirty with practical. For demonstration, I am using Ubuntu. However, behavior with another distribution should be identical.

Allow sudo access

Let us add regular user as a sudo user. In my case user’s name is lpicentral

1) Edit /etc/sudoers file as follows:

$ sudo visudo

2) Add below line to allow sudo access to user lpicentral:

lpicentral ALL=(ALL) ALL

In above command:

◈ lpicentral indicates user name
◈ First ALL instructs to permit sudo access from any terminal/machine
◈ Second (ALL) instructs sudo command to be allowed to execute as any user
◈ Third ALL indicates all command can be executed as root

Execute command with elevated privileges

To execute command with elevated privileges, just prepend sudo word to command as follows:

$ sudo cat /etc/passwd

When you execute this command, it will ask lpicentral’s password and not root user password.

Execute command as an another user

In addition to this we can use sudo to execute command as another user. For instance, in below command, user lpicentral executes command as a devesh user:

$ sudo -u devesh whoami
[sudo] password for lpicentral:
devesh

Built in command behavior

One of the limitation of sudo is – Shell’s built in command doesn’t work with it. For instance, history is built in command, if you try to execute this command with sudo then command not found error will be reported as follows:

$ sudo history
[sudo] password for lpicentral:
sudo: history: command not found

Access root shell

To overcome above problem, we can get access to root shell and execute any command from there including Shell’s built in.

To access root shell, execute below command:

$ sudo bash

After executing this command – you will observe that prompt sign changes to pound (#) character.

Recipes

In this section we’ll discuss some useful recipes which will help you to improve productivity. Most of the commands can be used to complete day-to-day task.

Execute previous command as a sudo user

Let us suppose you want to execute previous command with elevated privileges, then below trick will be useful:

$ sudo !4

Above command will execute 4th command from history with elevated privileges.

sudo command with Vim

Many times we edit system’s configuration files and while saving we realize that we need root access to do this. Because this we may lose our changes. There is no need to get panic, we can use below command in Vim to rescue from this situation:

:w !sudo tee %

In above command:

◈ Colon (:) indicates we are in Vim’s ex mode
◈ Exclamation (!) mark indicates that we are running shell command
◈ sudo and tee are the shell commands
◈ Percentage (%) sign indicates all lines from current line

Execute multiple commands using sudo

So far we have executed only single command with sudo but we can execute multiple commands with it. Just separate commands using semicolon (;) as follows:

$ sudo -- bash -c 'pwd; hostname; whoami'

In above command:

◈ Double hyphen (–) stops processing of command line switches
◈ bash indicates shell name to be used for execution
◈ Commands to be executed are followed by –c option

Run sudo command without password

When sudo command is executed first time then it will prompt for password and by default password will be cached for next 15 minutes. However, we can override this behavior and disable password authentication using NOPASSWD keyword as follows:

lpicentral ALL=(ALL) NOPASSWD: ALL

Restrict user to execute certain commands

To provide controlled access we can restrict sudo user to execute only certain commands. For instance, below line allows execution of echo and ls commands only

lpicentral ALL=(ALL) NOPASSWD: /bin/echo /bin/ls

Insights about sudo

Let us dig more about sudo command to get insights about it.

$ ls -l /usr/bin/sudo
-rwsr-xr-x 1 root root 145040 Jun 13  2017 /usr/bin/sudo

If you observe file permissions carefully, setuid bit is enabled on sudo. When any user runs this binary it will run with the privileges of the user that owns the file. In this case it is root user.

To demonstrate this, we can use id command with it as follows:

$ id
uid=1002(lpicentral) gid=1002(lpicentral) groups=1002(lpicentral)

When we execute id command without sudo then id of user lpicentral will be displayed.

$ sudo id
uid=0(root) gid=0(root) groups=0(root)

But if we execute id command with sudo then id of root user will be displayed.

Continue reading

Grep Command in Unix Shell script

Q) How to use the grep command in unix or linux bash scripts to search for a pattern match?

You might have used the grep command to search for a string in a file on the unix command line. Here we will see how to use the grep command in a bash script.

Consider the below file with data

> cat sample_file.txt
linux storage
unix distributed system
linux file server
debian server
fedora backup server

Let see the bash script that prints the lines which contain the word server in it. The bash script code is shown below:

>vi grep_script.sh

#!/bin/bash

WORD=server
INPUT_FILE=sample_file.txt
grep "$WORD" $INPUT_FILE

Now we will execute this script and see what the output is

> bash grep_script.sh

linux file server
debian server
fedora backup server

Here in the script we have hardcoded the filename and the search pattern. We will see how to pass these as arguments from the command line to the script. The bash script is:

>vi grep_script_command_line.sh

#!/bin/bash

INPUT_FILE=$1
WORD=$2
grep "$WORD" $INPUT_FILE

Now run the script as shown below:

> bash grep_script_command_line.sh sample_file.txt server

Continue reading

Cut Command in Unix ( Linux) Examples

Cut command in unix (or linux) is used to select sections of text from each line of files. You can use the cut command to select fields or columns from a line by specifying a delimiter or you can select a portion of text by specifying the range or characters. Basically the cut command slices a line and extracts the text.

Unix Cut Command Example

We will see the usage of cut command by considering the below text file as an example

> cat file.txt
unix or linux os
is unix good os
is linux good os

1. Write a unix/linux cut command to print characters by position?

The cut command can be used to print characters in a line by specifying the position of the characters. To print the characters in a line, use the -c option in cut command

cut -c4 file.txt
x
u
l

The above cut command prints the fourth character in each line of the file. You can print more than one character at a time by specifying the character positions in a comma separated list as shown in the below example

cut -c4,6 file.txt
xo
ui
ln

This command prints the fourth and sixth character in each line.

2 .Write a unix/linux cut command to print characters by range?

You can print a range of characters in a line by specifying the start and end position of the characters.

cut -c4-7 file.txt
x or
unix
linu

The above cut command prints the characters from fourth position to the seventh position in each line. To print the first six characters in a line, omit the start position and specify only the end position.

cut -c-6 file.txt
unix o
is uni
is lin

To print the characters from tenth position to the end, specify only the start position and omit the end position.

cut -c10- file.txt
inux os
ood os
good os

If you omit the start and end positions, then the cut command prints the entire line.

cut -c- file.txt

3. Write a unix/linux cut command to print the fields using the delimiter?

You can use the cut command just as awk command to extract the fields in a file using a delimiter. The -d option in cut command can be used to specify the delimiter and -f option is used to specify the field position.

cut -d' ' -f2 file.txt
or
unix
linux

This command prints the second field in each line by treating the space as delimiter. You can print more than one field by specifying the position of the fields in a comma delimited list.

cut -d' ' -f2,3 file.txt
or linux
unix good
linux good

The above command prints the second and third field in each line.

Note: If the delimiter you specified is not exists in the line, then the cut command prints the entire line. To suppress these lines use the -s option in cut command.

4. Write a unix/linux cut command to display range of fields?

You can print a range of fields by specifying the start and end position.

cut -d' ' -f1-3 file.txt

The above command prints the first, second and third fields. To print the first three fields, you can ignore the start position and specify only the end position.

cut -d' ' -f-3 file.txt

To print the fields from second fields to last field, you can omit the last field position.

cut -d' ' -f2- file.txt

5. Write a unix/linux cut command to display the first field from /etc/passwd file?

The /etc/passwd is a delimited file and the delimiter is a colon (:). The cut command to display the first field in /etc/passwd file is

cut -d':' -f1 /etc/passwd

6. The input file contains the below text

> cat filenames.txt
logfile.dat
sum.pl
add_int.sh

Using the cut command extract the portion after the dot.

First reverse the text in each line and then apply the command on it.

rev filenames.txt | cut -d'.' -f1

Continue reading

Learn Git Command with Practical Examples on Linux

Git is distributed version control system. It is primarily used by software developers for their source code management. Git is free and open source software and many large organizations use it to manage their huge code base.

Git user initial configuration

First we have to configure settings for Git user. We can make these settings for all repositories present on current system or for a particular repository. Let us understand this with example:

User identity

In Git we can specify user identity by providing its name and e-mail address. This information will be used during each commit operation. Execute below command in terminal to assign identity:

$ git config --global user.name "LPICentral User"
$ git config --global user.email "lpicentraluser@lpicentral.blogspot.com"

Editor

This setting configures editor, which will be used while providing commit message:

$ git config --global core.editor vim

Compression

This setting configures compression level to be used. Valid range for compression is -1 to 9. -1 value indicates zlib compression and is default compression level. 0 value means no compression, and 1 to 9 are various speed/size tradeoffs, 9 being slowest.

$ git config --global core.compression 2

Diff tool

This setting configures diff viewer tool. For example, below command configures vimdiff as a diff tool:

$ git config --global diff.tool vimdiff

In above commands we have used –global option everywhere, which will make this configuration global. It means that same configuration will be applied to all repositories present on current system. To make configuration repository specific just remove –global option.

List configuration

To list Git configuration execute below command in terminal:

$ git config -l

This command will generate following output:

core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.editor=vim
core.compression=2
user.name=LPICentral User
user.email=lpicentraluser@lpicentral.blogspot.com
diff.tool=vimdiff

Git repositories

Repository is a location where source code is stored. We can either create new repository or use existing repository. To create new empty repository execute below command:

$ mkdir my-repo.git
$ cd my-repo.git
$ git init --bare
Initialized empty Git repository in my-repo.git

This method will be useful when you are starting with new project. Another method is to use existing repository. Such a repository is hosted on remote server like GitHub. To download remote repository use clone command as follows:

$ git clone https://github.com/lpicentraluser/my-repo.git
In above command last argument is path of remote repository.

Git workflow

In this section we’ll discuss git workflow.

Introduce new changes

First step is to introduce new changes. It can be addition of new file or updating existing files. Let us create a new file and modify existing file

$ touch AUTHORS                                  # Create new file
$ echo "New Contents" >> README                  # Update existing file
.

Check repository status

Git is content tracking system, it will identify above two changes. Let us check repository status:

$ git status -s
 M README
?? AUTHORS

In above output letter M appears before README which indicates that existing file is modified. Whereas ?? appears before AUTHORS which indicates that this is new file and Git is not aware about it hence such a file is called untracked file.

Add file to changeset

Let us instruct Git to track this new file. We can achieve this using add command. This operation will start tracking changes made to this file.

$ git add AUTHORS

Let us check repository status now:

$ git status -s
A  AUTHORS
M README

Above output shows A before AUTHORS which indicates that this file is newly added under Git. We can add any number of files using this command.

Remove file from changeset

Let us instruct Git to untrack this newly added file. We can achieve this using reset command. This operation will remove file from changeset

$ git reset AUTHORS
$ git status -s
 M README
?? AUTHORS

Above output shows that AUTHORS file is not tracked by Git.

Commit changes

In Git, files which are part of changeset will form a commit. Each commit will get unique ID. Let us create changeset first

$ git add AUTHORS
$ git add README

Now let us commit changes to local repository with commit message. In below command -m argument indicates commit message.

$ git commit -m "Updated README and added AUTHORS"

When you execute above command it will generate following output:

[master 0b124eb] Updated README and added AUTHORS
 2 files changed, 1 insertion(+)
 create mode 100644 AUTHORS

Review changes

In this section we’ll discuss commands which will allow us to review repository changes.

View commit log

Repository can contain multiple commits by multiple authors. We can use log command to view all available commits:

$ git log

When you execute above command it will generate following output:

commit 0b124eb6d0109d837f6f9396c9937406abd3f456 (HEAD -> master)
Author: LPICentral User <lpicentraluser@lpicentral.blogspot.com>
Date:   Fri Jul 27 21:06:55 2018 +0530

    Updated README and added AUTHORS

This is commit we had created earlier. In above command:

◈ Hexadecimal ID represents a commit ID
◈ Author section of commit shows details about who made these changes
◈ Date section shows date and timestamp of commit

View short commit log

Above command will show detailed information about each commit. To view short description about each commit use –oneline option as follows:

$ git log --oneline

When you execute above command, it will generate following output:

0b124eb (HEAD -> master) Updated README and added AUTHORS

View commit

Commit ID is associated with each changeset. We can use this ID with show command to view commit contents.

$ git show 0b124eb6d0109d837f6f9396c9937406abd3f456

When you execute above command it will generate following output:

commit 0b124eb6d0109d837f6f9396c9937406abd3f456 (HEAD -> master)
Author: LPICentral User <lpicentraluser@lpicentral.blogspot.com>
Date:   Fri Jul 27 21:06:55 2018 +0530
    Updated README and added AUTHORS
diff --git a/AUTHORS b/AUTHORS
new file mode 100644
index 0000000..e69de29
diff --git a/README b/README
index 980a0d5..5680123 100644
--- a/README
+++ b/README
@@ -1 +1,2 @@
 Hello World!
+New Contents

View diff

Diff command allows us to review changes before creating changeset. Diff command shows the differences between repository and local workspace. Let us modify README file and view differences

$ echo "Generating diff" >> README
$ git diff
diff --git a/README b/README
index 5680123..3158230 100644
--- a/README
+++ b/README
@@ -1,2 +1,3 @@
 Hello World!
 New Contents
+Generating diff

In above command:

◈ + sign indicates changes which are added to file
◈ – sign indicates changes which are removed from file

Working with remote repositories

So far we were working with local repository only. Other developers cannot use changes which are made by us as they are local. Hence code collaboration is not possible. This section will describe commands which will allow us to interact with remote repositories

Publish changes to remote repository

We can publish local changes to remote repository so that other developers can use it. Before publishing changes we have to create changeset and local commit. We can publish changes using push command as follows:

$ git push

This command copies changes from local workspace to remote repository. In Git, this operation is referred as push operation.

Sync workspace with remote repository

Many developers can publish their workspace changes to remote repository. Git allows us to download those changes and sync workspace with repository. We can achieve this using pull command:

$ git pull

In Git, this operation is referred as pull operation.

Miscellaneous Git commands

This section discusses miscellaneous Git commands which will be useful to perform day-to-day tasks:

Modify latest commit

Git allows us to modify latest commit. We can use this method to avoid creation of new commit. It is mostly used to modify previous commit message. To modify latest commit use –amend option as follows:

$ echo "Yet another new change" >> README
$ git add README
$ git commit --amend -m "This is amended commit

Now let us check the commit log:

$ git log
commit 8bf67aec1d1de87f03ab6aae93940b17826fde1c (HEAD -> master)
Author: LPICentral User <lpicentraluser@lpicentral.blogspot.com>
Date:   Fri Jul 27 21:54:55 2018 +0530

    This is amended commit

If you observe above output carefully then we can see new commit message, its ID and new timestamp.

Remove untracked files

Untracked files are those which are unknown to Git. We can remove all untracked files using clean command.

Let us create few untracked files:

$ touch delete-me-1 delete-me-2 delete-me-3

To remove all above untracked file use clean command with -f option as follows:

$ git clean -f
Removing delete-me-1
Removing delete-me-2
Removing delete-me-3

Please note that this command will remove files permanently hence use it with caution.

View commits of particular author

If we use log command then it shows commits of all authors. To view commits of particular author use –author flag as follows:

$ git log --author=LPICentral

When you execute above command it will list all the commits of LPICentral authors as follows:

commit 8bf67aec1d1de87f03ab6aae93940b17826fde1c (HEAD -> master)
Author: LPICentral User <lpicentraluser@lpicentral.blogspot.com>
Date:   Fri Jul 27 21:54:55 2018 +0530

    This is amended commit

View history of each file line by line

To view line by line history we can use blame command.

$ git blame README

When you execute above command it will generate following output:

76294131 (LPICentral User         2018-07-27 21:12:11 -0700 1) Hello World!
8bf67aec (LPICentral User         2018-07-27 21:54:55 +0530 2) New changes
8bf67aec (LPICentral User         2018-07-27 21:54:55 +0530 3) Yet another changes

In above command:

◈ First column indicates commit ID
◈ Second column indicates author
◈ Third column indicates timestamps
◈ Last column indicates line number and file content

View diff from staging area

When you create changeset using add command then file is logically moved to staging area. Let us see with this example:

$ echo "Let us demonstrate staging area" >> README
$ git add README
$ git diff

Above command will not show any difference as file is move to staging area. Let us use –staged operation to view differences:

$ git diff --staged
diff --git a/README b/README
index 74729a2..8bc5ffd 100644
--- a/README
+++ b/README
@@ -1,3 +1,4 @@
 Hello World!
 New changes
 Yet another changes
+Let us demonstrate staging area

Continue reading

WC Command Examples - Count of Lines, Words, Characters - Unix / Linux

WC command in unix or linux is used to find the number of lines, words and characters in a file. The syntax of wc command is shown below:

wc [options] filenames

You can use the following options with the wc command.

-l : Prints the number of lines in a file.
-w : prints the number of words in a file.
-c : Displays the count of bytes in a file.
-m : prints the count of characters from a file.
-L : prints only the length of the longest line in a file.

Let see how to use the wc command with few examples. Create the following file in your unix or linux operating system.

> cat unix_wc.bat
Oracle Storage
unix distributed system
linux file server
debian server
Oracle backup server

WC Command Examples:

1. Printing count of lines

This is the most commonly used operation to find the number of lines from a file. Run the below command to display the number of lines:

wc -l unix_wc.bat
5 unix_wc.bat

Here in the output, the first field indicates count and second field is the filename

2. Displaying the number of words.

Just use the -w option to find the count of words in a file. This is shown below:

wc -w unix_wc.bat
13 unix_wc.bat

3. Print count of bytes, count of characters from a file

We can use the -c and -m options to find the number of bytes and characters respectively in a file.

> wc -c unix_wc.bat
92 unix_wc.bat
> wc -m unix_wc.bat
92 unix_wc.bat

4. Print the length of longest line

The -L option is used to print the number of characters in the longest line from a file.

wc -L unix_wc.bat
23 unix_wc.bat

In this example, the second line is the longest line with 23 characters.

5. Print count of lines, words and characters.

If you dont specify any option to the wc command, by default it prints the count of lines, words and characters. This is shown below:

wc unix_wc.bat
 5 13 92 unix_wc.bat

6. Wc help

For any help on the wc command, just run the wc --help on the unix terminal.

Continue reading

Chmod Command Examples in Unix / Linux

Chmod (change mode) is one of the most frequently used commands in unix or linux operating system. The chmod command is used to change the file or directory access permissions. To know about the access permissions of a file or directory, use the ls -l command as shown below:

$ ls -l sample.sh
-rwx-rw-r-- 1 matt deploy 94 Oct  4 03:12 sample.sh

Here in the above example: Use matt has the read, write and execute permissions on the file. Group deploy has read and write permissions. Others have only the read permission.
File and Directory permissions:

There are three different permissions. They are: 

◈ Read (4): Permitted to read the contents of the file. In case of directory, you can view all the files and sub-directories in that directory.
◈ Write (2): Permitted to write to the file. In case of directory, you can create files and sub-directories.
◈ Execute (1): Execute the file as a program/shell script. In case of directory, You can enter into that directory.

Here in the above, the numbers in the brackets represents the numeric values for the corresponding permissions. If you want to have a combination of permissions add the required numbers. For example, for read and execute, it is 4+1=5.

The syntax of chmod command is 

chmod [options] mode filename

THe important options are:

-R : recursively change the permissions of a directory.
-v : Verbose

Chmod Examples in Linux / Unix: 

1. Give read, write and execute permissions to everyone. 

Read, write and execute: 4+2+1=7

$ chmod 777 sample.sh

In the above example, you can see that the permissions are specified with a three digit number. The first digit is for user permissions, second is for group and third is for others permission. This type of representation is called octal representation. Alternatively, you can use the symbolic representation to give the permissions.

chmod ugo+rwx sample.sh

We will see more details about the symbolic representation later.

2. Give read permission to user, write permission to group and execute permission to others. 

$ chmod 421 sample.sh

3. Recursive permissions to directory 

To give read and write permissions to all the users to a directory (including files and subdirectories) use the recursive option -R.

chmod -R 666 /dir

Symbolic Representation of Permissions: 

The following symbols are used to represent the users, groups and others:

u : User
g : Group
o : Others a : All (user, group and others)

The following symbols represent the permissions:

r : read
w : write
x : execute

The following symbols represent the permissions grant or revoke:

+ : Additional permissions. Selected permissions are added.
- : Revoke the permissions. Selected permissions are revoked.
= : Specific permissions. Only selected permissions are assigned.

Examples: 

1. Remove write permission from group 

$ chmod g-w sample.sh

This will only removes the write permission for the group.

2. Add new permission execute to others 

$ chmod o+x sample.sh

In addition to the existing permissions, this will add execute permission to others.

3. Give only read permissions to the user 

$ chmod u=w sample.sh

This will remove the existing permissions to the user and gives only write permission to the user.

Continue reading