Echo Command in Linux with Examples

Echo is a Unix/Linux command tool used for displaying lines of text or string which are passed as arguments on the command line. This is one of the basic command in linux and most commonly used in shell scripts. In this tutorial, we will look at the different  options of echo command.

Basic Syntax

echo [option] [string]

1) Displaying a string on the terminal

To print text or a string on the terminal, use the syntax

echo [string]

$ echo "Welcome to Linux"

Sample Output

Welcome to Linux

2) Declare a variable and echo it value

Assume you have a variable x which is assigned the value 45  i.e.

$ x=10

You can print the value of variable x by executing the command

$ echo The value of x is $x

Sample Output

The value of x is 10

3) Remove  Spaces in between text using -e and \b options

You can choose to remove spaces using the \b option in conjunction with the -e parameter.

Note:

The -e parameter is used for the interpretation of backslash interpreters

Let's assume you have the text string

Linux is an opensource operating system

To remove spaces between the words, run

$ echo -e "Linux \bis \ban \bopensource \boperating \bsystem"

Sample Output

Linuxisanopensourceoperatingsystem

4) Create new lines in between text  using \n option
To create a new line after each word in a string use the -e operator  with the \n option  as shown

$ echo -e "Linux \nis \nan \nopensource \noperating \nsystem"

Sample Output

Linux
is
an
opensource
operating
system

5) Create horizontal tab space in between text  using \t option

If you want to create tab spaces in between words in  a string use the  -e operator  with the \t option  as shown

$ echo -e "Linux \tis \tan \topensource \toperating \tsystem"

Sample Output

Linux is an opensource operating system

5) Create vertical tab spaces using the \v option

You can decide to get a bit fancy and create vertical tab spaces using the -e operator  with the \v option  as shown

$ echo -e "Linux \vis \van \vopensource \voperating \vsystem"

Sample Output

Linux
       is
           an
               opensource
                          operating
                                     system

6) Double vertical tab using \n and \v option simultaneously

You can double the vertical tab spacing as shown below using the \n and \v options as shown

$ echo -e "Linux \n\vis \n\van \n\vopensource \n\voperating \n\vsystem"

Sample Output

Linux

is

an

opensource

operating

system

7) Print all files and folders using the * option

You can print all files and folders in your current working directory using the command

$ echo *

This has the same output as the ls command

Sample Output

To print files of  a specific type run

$ echo *.file_extension

For example

$ echo *.pdf

Sample Output

8) Using the carriage return '\r' option

The 'r' option gives you the carriage return i.e. any word(s) before the \r are omitted in the output

$ echo -e "Linux \r is an opensource operating system"

Sample Output

is an opensource operating system

9) Omit echoing trailing newline

The -n option is used for omitting trailing newline. This is shown in the example below

$ echo -n "Linux is an opensource operating system"

Sample Output

Linux is an opensource operating systemjames@buster:/$

Continue reading

How to Time a Command in Linux

If you want to be a good Linux administrator, therefore you should know time command. It is used to determine how long a given command takes to run.

It is useful for testing the performance of your scripts and commands (ie helps to find execution time for shell scripts or time taken for a command to finish).

How to Use Linux time command

To use time command, just execute time with the command/program you want to run as input. Please check below example

$ time ping lpicentral.blogspot.com

output

PING lpicentral.blogspot.com (104.27.115.15) 56(84) bytes of data.

64 bytes from 104.27.115.15 (104.27.115.15): icmp_seq=1 ttl=58 time=1.77 ms

64 bytes from 104.27.115.15 (104.27.115.15): icmp_seq=2 ttl=58 time=2.12 ms

64 bytes from 104.27.115.15 (104.27.115.15): icmp_seq=3 ttl=58 time=1.65 ms

--- lpicentral.blogspot.com ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 10016ms

rtt min/avg/max/mdev = 1.118/1.476/2.124/0.362 ms

real    0m10.536s

user    0m0.002s

sys     0m0.007s

The real signifies the wall clock time the 'ping' command took from execution till termination, user and sys are the time taken by ping the user space and kernel space.

How to Make time command writes its output to a file

To write the time command output to a file instead of the print out screen, use the -o command line option, which expects a file name/path as input.

$ /usr/bin/time -o /home/smart/time-output.txt ping lpicentral.blogspot.com

Now we will display ping's output on stdout, while the time command output will be written to the text file.

Note: We used /usr/bin/time instead of time because the shell built-in time command doesn't offer the -o option.

How to appends its output to an existent file

To append the time command output to an existent file instead of overwrite it, use the -a command line option.

$ /usr/bin/time -a /home/smart/time-output.txt ping lpicentral.blogspot.com

How to get Detailed output of  linux time command

We can use the -v command line option to produce detailed output.

$ time -v ping lpicentral.blogspot.com

   

    output

    Command being timed: "ping lpicentral.blogspot.com"

    User time (seconds): 0.00

    System time (seconds): 0.00

    Percent of CPU this job got: 0%

    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:11.77

    Average shared text size (kbytes): 0

    Average unshared data size (kbytes): 0

    Average stack size (kbytes): 0

    Average total size (kbytes): 0

    Maximum resident set size (kbytes): 3064

    Average resident set size (kbytes): 0

    Major (requiring I/O) page faults: 0

    Minor (reclaiming a frame) page faults: 158

    Voluntary context switches: 14

    Involuntary context switches: 0

    Swaps: 0

    File system inputs: 0

    File system outputs: 0

    Socket messages sent: 0

    Socket messages received: 0

    Signals delivered: 0

    Page size (bytes): 4096

    Exit status: 0

How to customizing time command output

There are a large number of formatting options as shown in the following list

C - Name and command line arguments used

D - Average size of the process's unshared data area in kilobytes

E - Elapsed time in a clock format

F - Number of page faults

I - Number of file system inputs by the process

K - Average total memory use of the process in kilobytes

M - Maximum resident set the size of the process during the lifetime in Kilobytes

O - Number of file system outputs by the process

P - Percentage of CPU that the job received

R - Number of minor or recoverable page faults

S - Total number of CPU seconds used by the system in kernel mode

U - Total number of CPU seconds used by user mode

W - Number of times the process was swapped out of main memory

X - Average amount of shared text in the process

Z - System's page size in kilobytes

c - Number of times the process was context-switched

e - Elapsed real time used by the process in seconds

k - Number of signals delivered to the process

p - Average unshared stack size of the process in kilobytes

r - Number of socket messages received by the process

s - Number of socket messages sent by the process

t - Average resident set size of the process in kilobytes

w - Number of time the process was context-switched voluntarily

x - Exit status of the command

We can use the formatting switches as follows:

$ time -f "Elapsed Time = %E, Inputs %I, Outputs %O"

The output for the above command would be something like this:

Elapsed Time = 0:01:00, Inputs 2, Outputs 1

If we want to add a new line as part of the format string use the newline character as follows:

$ time -f "Elapsed Time = %E \n Inputs %I \n Outputs %O"

Linux Time Command Versions

There are three-time command Versions, Bash, Zsh and Gnu time command. We can use the type command to determine whether time is a binary or a built-in keyword.

$ type time

  output

  # Bash

  time is a shell keyword

  # Zsh

  time is a reserved word

  # GNU time (sh)

  time is /usr/bin/time

Continue reading

id command in Linux with examples

The Linux Professional Institute currently offers three different certification programs. The core certification program LPIC contains three different levels addressing distinct aspects of Linux system administration.

id command

id command in Linux is used to find out user and group names and numeric ID’s (UID or group ID) of the current user or any other user in the server. This command is useful to find out the following information as listed below:

◈ User name and real user id.
◈ Find out the specific Users UID.
◈ Show the UID and all groups associated with a user.
◈ List out all the groups a user belongs to.
◈ Display security context of the current user.

Synopsis:

id [OPTION]… [USER]

Options:

◈ -g : Print only the effective group id.
◈ -G : Print all Group ID’s.
◈ -n : Prints name instead of number.
◈ -r : Prints real ID instead of numbers.
◈ -u : Prints only the effective user ID.
◈ –help : Display help messages and exit.
◈ –version : Display the version information and exit.

Note: Without any OPTION it prints every set of identified information i.e. numeric ID’s.

Examples:

◈ To print your own id without any Options:

id

The output shows the ID of current user UID and GID.

◈ To find a specific users id: Now assume that we have a user named master, to find his UID we will use the command:

id -u master

◈ To find a specific users GID: Again assuming to find GID of master, we will use the command:

id -g master

◈ To find out UID and all groups associated with a username: In this case we will use the user “master” to find UID and all groups associated with it, use command:

id master

◈ To find out all the groups a user belongs to: Displaying the UID and all groups a user “master” belongs to:

id -G master

◈ To display a name instead of numbers: By default the id command shows us the UDI and GID in numbers which a user may not understand, with use of -n option with -u, -g and -G, use command(s)

id -ng master 

or

id -nu master

or

id -nG master

◈ To display real id instead of effective id: To show the real id with the use of -r option with -g, -u and -G, use command(s):

id -r -g master

id -r -u master

id -r -G master

Continue reading

Tailor-Made Suits for Computers: TUXEDO Computers

Turning a hobby into a profession may sound trite, but it best describes the way I’ve worked to create TUXEDO Computers.

In 2001 Windows XP came out, so I had to switch from my Windows 95. Yes, I was one of the many Windows 98 deniers. I tried to delay the upgrade as long as possible. So if I have to get used to a different user interface anyway, why not change and get used to Linux right away?

No sooner said than done. In 2002 I started to switch to Linux. Since, at that time, fast internet lines were hardly available on the one hand and much too expensive on the other hand, I ordered a software box. That was a SuSE Linux 7.3 with 7 CDs and thousands of pages of manuals. A DVD, which would have prevented the CD change orgy, was included, but due to the lack of a suitable drive I could not use it. I continued my journey with Mandrake Linux (later Mandriva), Debian, and a trip to Gentoo. A few Knoppix installations later, I stayed with Debian for a few years until I switched to Ubuntu.

Immediately after the first obstacles with SuSE, the idea grew in me to create a website - today one would call it a blog - which accompanied my transition to Linux, on which I could share my experiences and report on my difficulties, in order to save other users some problems, and above all, time.

I wanted to make people less afraid of Linux by giving easy-to-understand instructions and help.

After a short time this website/blog spawned a forum to answer questions and to stimulate general discussions among the readers. Frequently, people were asking where to get Linux. All this begged the question, “Why send customers away and not offer what they want?” Besides, hosting and domains for the Website and forum had to be paid. So in 2004 the “Linux-Onlineshop.de” was born.

The beginning was mostly thick boxes of software, a few books, and many fan articles. From there, we added stuffed animals, cups, mouse pads, baseball caps, and small stickers.

Eventually, people started asking about hardware. Which notebook did we recommend? Which one was particularly well supported? Which desktop PC was most suitable? What do I have to pay attention to when buying hardware? And of course, WHERE do you get the right devices?

It was not even possible to dream of pre-installed, pre-configured, and fully equipped devices at that time!

For a long time, I referred them to an external source, but more and more other customer problems arose, especially when it came to the detailed configuration of the devices and their peripherals.

Why not offer it by yourself?

In the end, I had the idea to provide those interested people with the devices, so that they could be sure that everything would work. I started with PCs, because they are comparatively simple: All components are plugged in and can be installed as well as removed. This has remained the case to this day. So I sat down and assembled and installed my own PCs.

In 2006, we added notebooks and here the composition was already more difficult. Due to their limited dimensions, notebooks are not as modular and flexible as desktop PCs. My company was still too small to manufacture them directly in the Far East. But with the right partners, and with time, the whole concept matured and I could offer the first notebooks. Over the years and with increased possibilities, these have become better and better and we continue to work constantly to improve their quality.

In the meantime, the little plant that (again sounding like a cliché) started in the basement, has grown into a medium-sized company, and is now called TUXEDO Computers. Today, we have around 30 employees in Germany, in Königsbrunn, near Augsburg, providing support, development, marketing, etc. The production in Leipzig has another 80 employees.

Devices that just work

It has always been important to me that our computers, and everything connected to them, just work. This means that my staff and I invest plenty of time and manpower to develop our own drivers, so that they work perfectly between the hardware and the installed software. We don’t avoid the weeks of testing or selection, nor the validation effort that has to be done with new hardware and software. It is imperative that our devices function flawlessly and that our customers are happy. Only linux-compatible options are selected for the devices, and we include our own tools for system control such as the Control Center, the Fan Control, and the Keyboard Control.

We also spend a lot of time on additional services for our customers. In addition to a web service for easy recovery or reinstallation of the devices, we also offer a cloud service. Of course, everything on our German servers is fully encrypted, according to the strictest security and data protection rules, as demanded by German law.

Customers value support and assistance

Customers often want to know that they can depend on support and consulting services as well. There are users who are, sometimes sensibly, worried about independent installations, or just don’t know what to do. This is understandable. We do the preparatory work with the installation and configuration of the operating system, so that the device only needs to be unpacked, and switched on. Everything works immediately. Simply. Out of the box.

We also respond to, and respect, people’s need for independence and freedom. With our devices, and their equipment we can make a big contribution. Data protection is a permanent issue for us because we have many customers from different professional groups who work with sensitive or valuable data. So, we offer possibilities such as BIOS-side shutdown of webcams, microphones, management engines, and radio technologies, including full hard disk encryption.

Our company, TUXEDO Computers, will continue to develop in the next years and offer customers new possibilities, but not at any cost, and not at the expense of our customers. The quality of our products and services must always be at the highest level. In the end, it is important to me that we remain true to ourselves, and to never forget where our beginnings lie. In Linux and Open Source.

Continue reading

What You Should Know About the Sudo Command

New users to Linux (especially Ubuntu) eventually become aware of the Sudo command. Many users never use it for anything other than getting past "permission denied" messages — but Sudo does so much more.

About Sudo

A common misconception about Sudo is that it is used solely to provide root permissions to an ordinary user. In fact, the Sudo command allows you to run a command as any user, with the default generally being the root.

How to Grant User Sudo Permissions

Ubuntu users typically take the ability to run the Sudo command for granted. That's because, during installation, a default user is created, and the default user in Ubuntu is always set up with Sudo permissions. If you are using other distributions or have other users within Ubuntu, however, the user likely needs to be granted permissions to run the Sudo command.

Only a few people should have access to the Sudo command, and they should be system administrators. Users should be given only the permissions they need to perform their jobs.

To grant users Sudo permissions, you just need to add them to the Sudo group. When creating a user, use the following command:

$sudo useradd -m -G sudo

The above command will create a user with a home folder and add the user to the Sudo group. If the user already exists, then you can add the user to the Sudo group using the following command:

$sudo usermod -a -G sudo

A Neat Sudo Trick for When You Forget to Run It

Here's one of those terminal command tricks you can learn from seasoned experts — in this case, for getting past the "permission denied" message. If it's a long command, you can go up through the history and put Sudo in front of it, you can type it out again, or you can use the following simple command, which runs the previous command using Sudo:

$sudo !!

How to Switch to Root User Using Sudo

The Su command is used to switch from one user account to another. Running the Su command on its own switches to the superuser account. Therefore, to switch to the superuser account using Sudo, simply run the following command:

$sudo su

How to Run a Sudo Command in the Background

If you want to run a command that requires superuser privileges in the background, run the Sudo command with the -b switch, as shown here:

$sudo -b

Note that, if the command being run requires user interaction, this won't work.

An alternative way to run a command in the background is to add an ampersand to the end, as follows:

$sudo &

How to Edit Files Using Sudo Privileges

The obvious way to edit a file using superuser privileges is to run an editor such as GNU nano, using Sudo as follows:

$sudo nano

Alternatively, you can use the following syntax:

$sudo -e

How to Run a Command as Another User Using Sudo

The Sudo command can be used to run a command as any other user. For example, if you are logged in as user "john" and you want to run the command as "terry," then you'd run the Sudo command in the following way:

$sudo-u terry

If you want to try it out, create a new user called "test" and run the following Whoami command:

$sudo -u test whoami

How to Validate Sudo Credentials

When you run a command using Sudo, you'll be prompted for your password. For a period afterward, you can run other commands using Sudo without entering your password. If you wish to extend that period, run the following command:

$sudo -v

Continue reading

How to Kill Processes Using Linux

Most of the time you will want a program to end by its own means, or, if it is a graphical application, by using the appropriate menu option or by using the cross in the corner.

Every so often a program will hang, in which case you will need a method for killing it. You might also want to kill a program that is running in the background that you no longer need to run.

This guide provides a method for killing all versions of the same application that are running on your system.

How to Use the 'killall' Command

The killall command kills all the processes by name. That means if you have three versions of the same program running the killall command will kill all three.

For example, open a small program such an image viewer. Now open another copy of the same image viewer. For this example, we've chosen Xviewer which is a clone of Eye Of Gnome.

Now open a terminal and type in the following command:

killall

For example to kill all instances of Xviewer type the following:

killall xviewer

Both instances of the program you have chosen to kill will now close.

Kill the Exact Process

killall may produce strange results. Well here is one reason why. If you have a command name which is more than 15 characters long then the killall command will only work on the first 15 characters. If therefore you have two programs which share the same first 15 characters both programs will be canceled even though you only wanted to kill one.

To get around this you can specify the following switch which will only kill files matching the exact name.

killall -e

Ignore Case When Killing Programs

To make sure the killall command ignores the case of the program name that you provide use the following command:

killall -I
killall --ignore-case

Kill All Programs In the Same Group

When you run a command such as the following one it will create two processes:

ps -ef | less

One command is for the ps -ef part which lists all of the running processes on your system and the output is piped to the less command.

Both programs belong to the same group, which is bash.

To kill both programs at once you can run the following command:

killall -g

For example to kill all commands running in a bash shell run the following:

killall -g bash

Incidentally to list all the running groups run the following command:

ps -g

Get Confirmation Before Killing Programs

Obviously, the killall command is quite a powerful command and you don't want to accidentally kill the wrong processes.

Using the following switch you will be asked whether you are sure before each process is killed.

killall -i

Kill Processes That Have Been Running for a Certain Amount of Time

Imagine you have been running a program and it is taking a lot longer than you hoped it would.

You can kill the command in the following way:

killall -o h4

The h in the above command stands for hours.

You can also specify any one of the following:

◈ s - seconds
◈ m - minutes
◈ h - hours
◈ d - days
◈ w - weeks
◈ M - months
◈ y - years

Alternatively, if you wish to kill commands that have only just started running you can use the following switch:

killall -y h4

This time the killall command will kill all programs running for less than 4 hours.

Don't Tell Me When a Process Isn't Killed

By default if you try and kill a program that isn't running you will receive the following error:

programname: no process found

If you don't want to be told if the process wasn't found use the following command:

killall -q

Using Regular Expressions

Instead of specifying the name of a program or command you can specify a regular expression so that all processes which match the regular expression are closed by the killall command.

To use a regular expression use the following command:

killall -r

Kill Programs for a Specify User

If you want to kill a program that is being run by a specific user you can specify the following command:

killall -u

If you want to kill all of the processes for a particular user you can omit the program name.

Wait for killall to Finish

By default, killall will return straight back to the terminal when you run it but you can force killall to wait until all the processes specified have been closed before returning you to the terminal window.

To do this run the following command:

killall -w

If the program never dies then killall will also continue to live on.

Signals Signals Signals

By default, the killall command sends the SIGTERM signal to programs to get them to close and this is the cleanest method for killing programs.

There are however other signals you can send using the killall command and you can list them using the following command:

killall -l

The list returned will be something like this:

◈ HUP
◈ INT
◈ QUIT
◈ ILL
◈ TRAP
◈ ABRT
◈ IOT
◈ BUS
◈ FPE
◈ KILL
◈ USR1
◈ SEGV
◈ USR2
◈ PIPE
◈ ALRM
◈ TERM
◈ STKFLT
◈ CHLD
◈ CONT
◈ STOP
◈ TSTP
◈ TTIN
◈ TTOU
◈ URG
◈ XCPU
◈ XFSZ
◈ VTALRM
◈ PROG
◈ WYNCH
◈ IO
◈ PWR
◈ SYS
◈ UNUSED

That list is extremely long. To read about what these signals mean run the following command:

man 7 signal

Generally, you should use the default SIGTERM option but if the program refuses to die you can use SIGKILL which forces the program to close albeit in an undignified way.

Other Ways to Kill a Program

However, to save you the effort of clicking the link here is a section showing what those commands are and why you might use those commands over killall.

The first one is the kill command. The killall command as you have seen is great at killing all the versions of the same program. The kill command is designed to kill one process at a time and is, therefore, more targetted.

To run the kill command you need to know the process ID of the process you wish to kill. For this, you can use the ps command.

For example to find a running version of Firefox you can run the following command:

ps -ef | grep firefox

You will see a line of data with the command /usr/lib/firefox/firefox at the end. At the beginning of the line, you will see your user ID and the number after the user ID is the process ID.

Using the process ID you can kill Firefox by running the following command:

kill -9 <processid>

Another way to kill a program is by using the xkill command. This is generally used to kill misbehaving graphical applications.

To kill a program such as Firefox open a terminal and run the following command:

xkill

The cursor will now change to a large white cross. Hover the cursor over the window you wish to kill and click with the left mouse button. The program will exit immediately.

Another way to kill a process is by using the Linux top command. The top command lists all the running processes on your system.

All you have to do to kill a process is press the k key and enter the process ID of the application you wish to kill.

Earlier in this section, we highlighted the kill command and it required you finding the process using the ps command and then kill the process using the kill command. This is not the simplest option by any means.

For one thing, the ps command returns loads of information you don't need. All you wanted was the process ID. You can get the process ID more simply by running the following command:

pgrep firefox

The result of the above command is simply the process ID of Firefox. You can now run the kill command as follows:

kill <processid>

(Replace <processid> with the actual process ID returned by pgrep).

It is actually easier, however, to simply supply the program name to pkill as follows:

pkill firefox

Finally, you can use a graphical tool such as the one supplied with Ubuntu called System Monitor. To run "System Monitor" press the super key (Windows key on most computers) and type "sysmon" into the search bar. Click on the system monitor icon when it appears.

Continue reading

17 useful rsync (remote sync) Command Examples in Linux

As the name suggests, rsync command is used to sync (or copy) files and directories locally and remotely. One of the important feature of rsync is that it works on “delta transfer algorithm”, means it will only sync or copy the changes from source to destination instead of copying the whole file which ultimately reduce amount of data sent over network. Linux geeks generally use rsync command to manage day to day backup, mirroring, and restoration activities. It uses remote shell like SSH while synchronizing the files from local machine to remote machine and any user in the system can use rsync command as it does not require root or sudo  privileges.

In this article we will discuss 17 useful rsync command examples in Linux, these examples will help specially Linux beginners to manage their sync, mirroring, and backup task more efficiently.

Rsync command is available for all UNIX and Linux like operating systems. When we do minimal installation of CentOS & RHEL OS then rsync package is not part of default installation, so to install rsync on CentOS & RHEL, run the beneath yum command

~]# yum install rsync -y

In Debian Like operating Systems (Ubuntu & Linux Mint) use below apt command to install rysnc tool,

~]# apt install rsync -y

Syntax of rsync command:

Local Sync: # rsync {options} {Source} {Destination}

Remote Sync pull: # rsync {options}  <User_Name>@<Remote-Host>:<Source-File-Dir>  <Destination>

Remote Sync Push: # rsync  <Options>  <Source-Files-Dir>   <User_Name>@<Remote-Host>:<Destination>

Some of the commonly used options in rsync command are listed below:

◈ -v, –verbose                             Verbose output

◈ -q, –quiet                                 suppress message output

◈ -a, –archive                              archive files and directory while synchronizing ( -a equal to following options -rlptgoD)

◈ -r, –recursive                           sync files and directories recursively

◈ -b, –backup                              take the backup during synchronization

◈ -u, –update                              don’t copy the files from source to destination if destination files are newer

◈ -l, –links                                   copy symlinks as symlinks during the sync

◈ -n, –dry-run                             perform a trial run without synchronization

◈ -e, –rsh=COMMAND            mention the remote shell to use in rsync

◈ -z, –compress                         compress file data during the transfer

◈ -h, –human-readable           display the output numbers in a human-readable format

◈ –progress                                 show the sync progress during transfer

Let’s jump into the useful examples of rsync command

Example:1) Copy or Sync files locally (rsync -zvh)

Let’s assume we want to copy a file from pkumar user’s home directory to /opt/back folder, execute the below rsync command

[root@gateway ~]# rsync -zvh /home/pkumar/OpenStack-Networking.pdf /opt/backup

OpenStack-Networking.pdf

sent 4.09M bytes  received 35 bytes  2.73M bytes/sec

total size is 6.15M  speedup is 1.50

[root@gateway ~]#

In above we have used the options like -z for compression, -v for verbose output and -h for human readable output.

Example:2) Copy or Sync directory locally (rsync -zavh)

Let’s assume we want to copy or sync pkumar user’s home directory to /opt/backup folder, execute  the below rsync command with options -zavh,

[root@gateway ~]# rsync -zavh /home/pkumar /opt/backup

sending incremental file list

pkumar/

pkumar/.bash_logout

pkumar/.bash_profile

pkumar/.bashrc

pkumar/OpenStack-Networking.pdf

sent 4.09M bytes  received 96 bytes  8.19M bytes/sec

total size is 6.15M  speedup is 1.50

[root@gateway ~]#

Let’s verify whether directory is copied or not, execute below command,

[root@gateway ~]# ls -ld /opt/backup/*

-rwx------. 1 root   root   6153239 Apr  7 00:25 /opt/backup/OpenStack-Networking.pdf

drwx------. 2 pkumar pkumar      90 Apr  7 00:25 /opt/backup/pkumar

[root@gateway ~]#

As we can see “pkumar” directory is created on destination folder and its contents are also copied. In the above example if use trail / after source folder (/home/pkumar/) then rsync command will not create pkumar directory on destination but it will copy the contents only.

[root@gateway ~]# rsync -zavh /home/pkumar/ /opt/backup

Example:3) Copy files & directories recursively locally (rsync -zrvh or rsync -zavh)

Let’s assume we have multiple files and directories inside pkumar user home directory, use below rsync command to copy files and directories recursively, either use -a or -r option to copy files and directories recursively.

Note : In rsync command -a option is used for archiving during the copy or sync and apart from archiving -a option is also used for followings:

◈ recursively copy files and directory

◈ copy symlinks as symlinks

◈ preserve permissions

◈ preserve group

◈ preserve modification time

◈ preserve ownership

[root@gateway ~]# rsync -zrvh /home/pkumar /opt/backup

or

[root@gateway ~]# rsync -zavh /home/pkumar /opt/backup

Example:4) Copy or sync files and directories from local to remote system

Let’s suppose we want to copy the folder “/home/pkumar/lpicentral” from local machine to remote machine (192.168.1.29) under the /opt

[root@gateway ~]# rsync -zarvh /home/pkumar/lpicentral root@192.168.1.29:/opt

root@192.168.1.29's password:

…………………………………………………………

lpicentral/OTRS/

lpicentral/OTRS/Database-Selection-OTRS-Installation-CentOS7.jpg

lpicentral/OTRS/Install-OTRS-Accept-License.png

lpicentral/OTRS/Install-OTRS-CentOS7-RHEL7.jpg

lpicentral/OTRS/mysql-secure-installation-centos7-part2.jpg

sent 34.85M bytes  received 2.65K bytes  6.34M bytes/sec

total size is 37.77M  speedup is 1.08

……………………………………………………………

[root@gateway ~]#

Example:5) Copy or Sync files and directories from remote machine to local system

Let’s suppose we want to copy files and directories from remote machine(192.168.1.29) to our local system, in the below example I am copying remote folder “/opt/rpms_db” in my local machine under /tmp folder

[root@gateway ~]# rsync -zarvh root@192.168.1.29:/opt/rpms_db /tmp

root@192.168.1.29's password:

receiving incremental file list

……………………………………………………………

rpms_db/

rpms_db/httpd-2.4.6-88.el7.centos.x86_64.rpm

rpms_db/httpd-tools-2.4.6-88.el7.centos.x86_64.rpm

rpms_db/postfix-2.10.1-7.el7.x86_64.rpm

rpms_db/pytalloc-2.1.13-1.el7.x86_64.rpm

rpms_db/samba-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-client-libs-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-common-4.8.3-4.el7.noarch.rpm

rpms_db/samba-common-libs-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-common-tools-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-libs-4.8.3-4.el7.x86_64.rpm

sent 484 bytes  received 15.45M bytes  1.07M bytes/sec

total size is 16.37M  speedup is 1.06

[root@gateway ~]#

Above command will automatically create a folder “rpms_db” under the /tmp folder in our local machine.

[root@gateway ~]# ls -l /tmp/

total 8

-rwx------. 1 root root  827 Apr  6 12:54 ks-script-avmuxw

drwxr-xr-x. 2 root root 4096 Apr  7 01:42 rpms_db

-rw-------. 1 root root    0 Apr  6 12:35 yum.log

[root@gateway ~]# ls -l /tmp/rpms_db/

total 16028

……………………………………

-rw-r--r--. 1 root root 2844028 Nov 12 09:28 httpd-2.4.6-88.el7.centos.x86_64.rpm

-rw-r--r--. 1 root root   92504 Nov 12 09:28 httpd-tools-2.4.6-88.el7.centos.x86_64.rpm

-rw-r--r--. 1 root root 2558136 Nov 12 09:44 postfix-2.10.1-7.el7.x86_64.rpm

-rw-r--r--. 1 root root   17404 Nov 12 09:44 pytalloc-2.1.13-1.el7.x86_64.rpm

-rw-r--r--. 1 root root  696472 Nov 12 09:46 samba-4.8.3-4.el7.x86_64.rpm

-rw-r--r--. 1 root root 5070800 Nov 12 09:46 samba-client-libs-4.8.3-4.el7.x86_64.rpm

-rw-r--r--. 1 root root  210984 Nov 12 10:19 samba-common-4.8.3-4.el7.noarch.rpm

-rw-r--r--. 1 root root  167512 Nov 12 09:46 samba-common-libs-4.8.3-4.el7.x86_64.rpm

-rw-r--r--. 1 root root  458380 Nov 12 09:46 samba-common-tools-4.8.3-4.el7.x86_64.rpm

-rw-r--r--. 1 root root  282332 Nov 12 09:46 samba-libs-4.8.3-4.el7.x86_64.rpm

……………………………………

[root@gateway ~]#

Example:6) Specify remote shell during synchronization (rsync -e ssh)

In rsync command we can specify the remote shell like ssh during synchronization, let’s assume we want to use secure communication between source to destination machine via rsync then we should use ssh as remote shell, example is shown below

[root@gateway ~]# rsync -zarvh -e ssh  root@192.168.1.29:/opt/rpms_db  /tmp

root@192.168.1.29's password:

receiving incremental file list

………………………………………………………………………………

rpms_db/

rpms_db/httpd-2.4.6-88.el7.centos.x86_64.rpm

rpms_db/httpd-tools-2.4.6-88.el7.centos.x86_64.rpm

rpms_db/postfix-2.10.1-7.el7.x86_64.rpm

rpms_db/pytalloc-2.1.13-1.el7.x86_64.rpm

rpms_db/samba-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-client-libs-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-common-4.8.3-4.el7.noarch.rpm

rpms_db/samba-common-libs-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-common-tools-4.8.3-4.el7.x86_64.rpm

rpms_db/samba-libs-4.8.3-4.el7.x86_64.rpm

sent 484 bytes  received 15.45M bytes  1.82M bytes/sec

total size is 16.37M  speedup is 1.06

[root@gateway ~]#

Example:7) Display Synchronization progress in rsync command output

If you want to see the sync or copy progress in rsync command then use “–progress“, example is shown below

[root@gateway ~]# rsync -avh --progress root@192.168.1.29:/opt/rpms_db /tmp

root@192.168.1.29's password:

receiving incremental file list

……………………………………………………………………………………………………..

rpms_db/

rpms_db/httpd-2.4.6-88.el7.centos.x86_64.rpm

          2.84M 100%   35.22MB/s    0:00:00 (xfr#6, to-chk=18/25)

rpms_db/httpd-tools-2.4.6-88.el7.centos.x86_64.rpm

         92.50K 100%    1.13MB/s    0:00:00 (xfr#7, to-chk=17/25)

rpms_db/postfix-2.10.1-7.el7.x86_64.rpm

          2.56M 100%   14.44MB/s    0:00:00 (xfr#17, to-chk=7/25)

rpms_db/samba-4.8.3-4.el7.x86_64.rpm

        696.47K 100%    3.71MB/s    0:00:00 (xfr#19, to-chk=5/25)

rpms_db/samba-client-libs-4.8.3-4.el7.x86_64.rpm

          5.07M 100%   19.90MB/s    0:00:00 (xfr#20, to-chk=4/25)

rpms_db/samba-common-4.8.3-4.el7.noarch.rpm

        210.98K 100%  844.42kB/s    0:00:00 (xfr#21, to-chk=3/25)

rpms_db/samba-common-libs-4.8.3-4.el7.x86_64.rpm

        167.51K 100%  667.70kB/s    0:00:00 (xfr#22, to-chk=2/25)

rpms_db/samba-common-tools-4.8.3-4.el7.x86_64.rpm

        458.38K 100%    1.77MB/s    0:00:00 (xfr#23, to-chk=1/25)

rpms_db/samba-libs-4.8.3-4.el7.x86_64.rpm

        282.33K 100%    1.09MB/s    0:00:00 (xfr#24, to-chk=0/25)

sent 484 bytes  received 16.38M bytes  3.64M bytes/sec

total size is 16.37M  speedup is 1.00

[root@gateway ~]#

Example:8) Copy the directory structure without copying files

There are some scenarios where we want to copy the directory structure skiping files only from local machine to remote or vice versa.

Let’s suppose we want to copy the directory structure of “/home/pkumar” from local machine to remote machine (192.168.1.29) under /opt folder.

root@gateway ~]# rsync -av -f"+ */" -f"- *" /home/pkumar root@192.168.1.29:/opt/

root@192.168.1.29's password:

building file list ... done

………………………………………………

pkumar/

sent 43 bytes  received 19 bytes  17.71 bytes/sec

total size is 0  speedup is 0.00

[root@gateway ~]#

Example:9) Resume large file transfer after getting failed in scp

There are some scenarios in linux admin profile where we have started copying a larger file using scp command, but it got terminated in the middle and we can’t afford to start copying it again using scp because of its large size and time consumption.

So in this situation rsync command can used as it  can start copying the file from where it left off or terminated, or in other words rsync can transfer the files which are partially copied using scp command. Example is shown below,

[root@gateway ~]# scp root@192.168.1.29:/root/ubuntu-18.04-desktop-amd64.iso /opt

root@192.168.1.29's password:

ubuntu-18.04-desktop-amd64.iso                   28%  526MB  61.5MB/s   00:21 ETA

^CKilled by signal 2.

[root@gateway ~]#

[root@gateway ~]# rsync -P --rsh=ssh root@192.168.1.29:/root/ubuntu-18.04-desktop-amd64.iso /opt

root@192.168.1.29's password:

ubuntu-18.04-desktop-amd64.iso

  1,921,843,200 100%   18.47MB/s    0:01:39 (xfr#1, to-chk=0/1)

[root@gateway ~]#

Example:10) Delete files at destination if it is not present in source (–delete)

If you have already synced files from source to destination and from source you have deleted the files then you can force rsync command to delete the files on destination using the –delete option, example is shown below

[root@app ~]# rsync -avz --delete /opt/rpms_db root@192.168.1.28:/tmp/rpms_db

root@192.168.1.28's password:

sending incremental file list

deleting rpms_db/apr-util-1.5.2-6.el7.x86_64.rpm

deleting rpms_db/apr-1.4.8-3.el7_4.1.x86_64.rpm

rpms_db/

sent 862 bytes  received 105 bytes  276.29 bytes/sec

total size is 15,947,152  speedup is 16,491.37

[root@app ~]#

Example:11) Put limit on file transfer size (–max-size)

If you don’t want to transfer or copy the large files using rsync then use the option ‘–max-size={specify-size-here}’, let’s assume we don’t we don’t want to transfer the files whose size is more than 500K,

Note: To specify the size in MB use M and for GB use G.

[root@app ~]# rsync -avz --max-size='500K' /opt/rpms_db root@192.168.1.28:/tmp

Example:12) Do not sync/copy the modified file at destination

There are some scenarios where we modified the files at destination and we don’t want to overwrite the modified files from source while syncing, so this can be easily achieved using -u option in rsync command

[root@gateway ~]# rsync -avzu root@192.168.1.29:/opt/lpicentral /home/pkumar/lpicentral

root@192.168.1.29's password:

receiving incremental file list

sent 37 bytes  received 4,960 bytes  1,427.71 bytes/sec

total size is 37,771,649  speedup is 7,558.87

[root@gateway ~]#

Example:13) Remove files from source after synchronization (–remove-source-files)

Let’s suppose you want to delete files from source once the synchronization is completed using rsync command. In the below example, folder from local system “/home/pkumar/lpicentral.tgz” is synced to remote system (192.168.1.29), once the synchronization is completed, it will delete the file from source.

[root@gateway ~]# rsync --remove-source-files -zvh /home/pkumar/lpicentral.tgz root@192.168.1.29:/opt

root@192.168.1.29's password:

lpicentral.tgz

sent 597 bytes  received 43 bytes  182.86 bytes/sec

total size is 518  speedup is 0.81

[root@gateway ~]#

[root@gateway ~]# ls -l /home/pkumar/lpicentral.tgz

ls: cannot access /home/pkumar/lpicentral.tgz: No such file or directory

[root@gateway ~]#

Example:14) Dry run rsync command before actual synchronization

There can be some situations where we are not about behavior of rsync command so in such cases it is better to do dry run of rsync

[root@gateway ~]# rsync --dry-run --remove-source-files -zvh /home/pkumar/projects.tar  root@192.168.1.29:/opt

root@192.168.1.29's password:

projects.tar

sent 51 bytes  received 19 bytes  7.37 bytes/sec

total size is 981.24M  speedup is 14,017,682.29 (DRY RUN)

[root@gateway ~]#

Example:15) Include and Exclude files during synchronization with rsync

There are some situations where we want to copy or sync files & directories of specific type and want to exclude files or directories of specific type. Rsync command supports both include and exclude options.

In the below example want to copy the files of type pdf and rpm and exclude png file types.

[root@gateway ~]# rsync -avz -e ssh --include '*.pdf *.rpm' --exclude '*.png' /home/pkumar/lpicentral root@192.168.1.29:/opt

root@192.168.1.29's password:

sending incremental file list

lpicentral/

lpicentral/OpenStack-Networking.pdf

lpicentral/httpd-2.4.6-88.el7.centos.x86_64.rpm

lpicentral/httpd-tools-2.4.6-88.el7.centos.x86_64.rpm

lpicentral/postfix-2.10.1-7.el7.x86_64.rpm

sent 9,469,912 bytes  received 96 bytes  2,705,716.57 bytes/sec

total size is 11,647,907  speedup is 1.23

[root@gateway ~]#

Example:16) Put restriction on data transfer speed in rsync (–bwlimit=<KB/s>)

In rsync command we can set the bandwidth limit for data transfer from one machine to another, use ‘–bwlimit=<KB/S>‘ option in rsync command to put limit on data transfer speed.

Let’s assume we want to set maximum data transfer rate (speed) is 600 KB/s with rsync, example is shown below

[root@gateway ~]# rsync -avz --progress --bwlimit=600 /home/pkumar/lpicentral root@192.168.1.29:/opt

root@192.168.1.29's password:

sending incremental file list

lpicentral/

lpicentral/OpenStack-Networking.pdf

6,153,239 100%  910.02kB/s    0:00:06 (xfr#1, to-chk=6/8)

lpicentral/httpd-2.4.6-88.el7.centos.x86_64.rpm

2,844,028 100%  615.28kB/s    0:00:04 (xfr#2, to-chk=5/8)

lpicentral/httpd-tools-2.4.6-88.el7.centos.x86_64.rpm

92,504 100%  507.51kB/s    0:00:00 (xfr#3, to-chk=4/8)

lpicentral/pkumar-2.png

0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=3/8)

lpicentral/pkumar-3.png

0 100%    0.00kB/s    0:00:00 (xfr#5, to-chk=2/8)

lpicentral/pkumar.png

0 100%    0.00kB/s    0:00:00 (xfr#6, to-chk=1/8)

lpicentral/postfix-2.10.1-7.el7.x86_64.rpm

2,558,136 100%  594.80kB/s    0:00:04 (xfr#7, to-chk=0/8)

sent 9,470,087 bytes  received 153 bytes  485,653.33 bytes/sec

total size is 11,647,907  speedup is 1.23

[root@gateway ~]#

Example:17) View the difference in files & directories between source and destination

Use “-i” option in rsync command to list the difference in files and directories between source and destination. Example is shown below

[root@gateway ~]# rsync -avzi /home/pkumar/lpicentral root@192.168.1.29:/opt

root@192.168.1.29's password:

sending incremental file list

.d..t...... lpicentral/

<f.st...... lpicentral/projects.txt

sent 438 bytes  received 45 bytes  138.00 bytes/sec

total size is 11,648,064  speedup is 24,116.07

[root@gateway ~]#

As per above command output, there is difference in file called “projects.txt” on destination. Following are the meaning of the keywords in above output,

◈ d: indicates change in destination file

◈ f: indicates a file

◈ t: indicates change in timestamps

◈ s: indicates change in size

Continue reading

10 Basic Linux Commands That Every Linux Newbies Should Remember

Linux has a big impact on our Lives. At least, your android phone has Linux kernel on it. However, getting started with Linux just make you discomfort for the first time. Because on Linux, you usually should use terminal commands instead of just clicking the launcher icon (as you did on Windows). But don’t worry, We will give you 10 basic Linux commands & important commands that will help you get started.

10 Basic Linux Commands That Help Newbies Get Started

When We talk about Linux commands, what we are really talking about is the Linux system itself. These small number of 10 Basic Linux commands will not make you a genius or a Linux expert instead, it’ll help you get started with Linux. It’ll help Linux newbies to perform daily basic tasks in Linux using these Linux basic commands or I should say Linux top commands(because of their usage).

So let’s get started with the list of 10 Linux Basic commands –

1. sudo

This SuperUserDo is the most important command Linux newbies will use. Every single command that needs root’s permission, need this sudo command. You can use sudo before each command that requires root permissions –

$ sudo su

2. ls (list)

Just like the other, you often want to see anything in your directory. With list command, the terminal will show you all the files and folders of the directory that you’re working in. Let’s say I’m in the /home folder and I want to see the directories & files in /home.

/home$ ls
ls in /home returns the following -
imad lost+found

3. cd

​Changing directory (cd) is the main command that always is in use in the terminal. It’s one of the most Linux basic commands. Using this is easy. Just type the name of the folder you want to go in from your current directory. If you want to go up just do it by giving double dots (..) as the parameter.
​
Let’s say I’m in /home directory and I want to move in usr directory which is always in the /home. Here is how I can use cd commands –

/home $ cd usr/home/usr $

4. mkdir

Just changing directory is still incomplete. Sometimes you want to create a new folder or subfolder. You can use mkdir command to do that. Just give your folder name after mkdir command in your terminal.

~$ mkdir folderName

5. cp

copy-and-paste is the important task we need to do to organize our files. Using cp will help you to copy-and-paste the file from the terminal. First, you determine the file you want to copy and type the destination location to paste the file.

$ cp src des

Note: If you’re copying files into the directory that requires root permission for any new file, then you’ll need to use sudo command. 

6. rm

rm is a command to remove your file or even your directory. You can use -f if the file need root permission to be removed. And also you can use -r to do recursive removal to remove your folder.

$ rm myfile.txt

7. apt-get

This command differs distro-by-distro. In Debian based Linux distributions, to install, remove and upgrade any package we’ve Advanced Packaging Tool (APT) package manager. The apt-get command will help you install the software you need to run in your Linux. It is a powerful command-line tool which can perform installation, upgrade, and even removing your software.​In other distributions, such as Fedora, Centos there are different package managers. Fedora used to have yum but now it has dnf.

$ sudo apt-get update
$ sudo dnf update

8. grep

You need to find a file but you don’t remember its exact location or the path. grep will help you to solve this problem. You can use the grep command to help to find the file based on given keywords.

$ grep user /etc/passwd

9. cat

As a user, you often need to view some of text or code from your script. Again, one of the Linux basic commands is cat command. It will show you the text inside your file.

$ cat CMakeLists.txt

10. poweroff

And the last one is poweroff. Sometimes you need to poweroff directly from your terminal. This command will do the task. Don’t forget to add sudo at the beginning of the command since it needs root permission to execute poweroff.

$ sudo poweroff

Continue reading