Expr Command Examples in Unix / Linux

This continuation to my previous post on bc command - calculator. In this article we will see how to use the expr command in unix or linux system for doing arithmetic operations.

The syntax of expr command is 

expr [expression]

Let see how to use the expr command in unix or linux with examples. Most of the examples are straightforward. I will provide description as and when required.

Note: You have to provide the space between the values and the operands. Otherwise the expr command may throw error or print them as a string.

Arithmetic Operator Examples:

1. Sum of numbers 

$ expr 5 + 3

8

$ expr 1 + 2 + 3

6

$ expr 5+3

5+3

Here in the third expr command, space is not provided between the literals. The expr command treated it as a string and printed on the terminal.

2. Difference between two numbers 

$ expr 10 - 6

4

3. Multiplying numbers 

$ expr 7 \* 9

63

Here the * is shell builtin operator, that is why it needs to escaped with backslash.

4. Dividing numbers 

$ expr 6 / 4

1

The division operator returns only the arithmetic quotient.

5. Remainder or modulus 

$ expr 6 % 4

2

Comparision or Relational Operator Examples: 

You can use the following comparision operators with the expr command:

◈ Val1 < Val2 : Returns 1 if val1 is less than val2. otherwise zero.
◈ Val1 <= Val2 : Returns 1 if val1 is less than or equal to val2. otherwise zero.
◈ Val1 > Val2 : Returns 1 if val1 is greater than val2. otherwise zero.
◈ Val1 >= Val2 : Returns 1 if val1 is greater than or equal to val2. otherwise zero.
◈ Val1 = Val2 : Returns 1 if val1 is equal to val2. otherwise zero.
◈ Val1 != Val2 : Returns 1 if val1 is equal to val2. otherwise zero.
◈ val1 | val2 : Returns val1 if val1 is neither null nor zero. Otherwise val2.
◈ val1 & val2 : Returns val1 if both val1 and val2 is neither null nor zero. Otherwise 0.

Note: You have to escape most of the operators with backslash as they are shell built in.

$ expr 1 \< 2

1

$ expr 1 \<= 1

1

$ expr 2 \> 5

0

$ expr 2 \>= 5

0

$ expr 7 = 7

1

$ expr 9 != 18

1

$ expr 2 \| 5

2

$ expr 0 \| 5

5

$ expr 2 \& 5

2

$ expr 6 \& 3

6

$ expr 6 \& 0

0

$ expr 0 \& 3

0

String Function Examples: 

1. Length of string

The length function is used to find the number of characters in a string.

$ expr length linux

5

$expr length linux\ system

12

$expr length "linux system"

If you have spaces in your string escape them with backslash or quote them with double quotes.

2. Find Substring 

You can extract a portion of the string by using the substr function. The syntax of substr function is

substr string position length

Here position is the character position in the string. length is the number of chracters to extract from the main string. An example is shown below:

$ expr substr unixserver 5 6
server

3. Index of the substring 

You can find the position of a string in the main string using the index function. The syntax of index function is shown below:

index string chars

If the chars string is found in the main string, then the index function returns the position of the chars. Otherwise it returns 0. See the following examples:

$ expr index linux nux
3

$expr index linux win
0

4. Matching a regexp 

The match function is used to find anchored pattern match of regexp in the string. The syntax of match function is shown below:

match string pattern

The match function returns the number of characters in the pattern is a match is found. Otherwise, it returns 0. Alternative synatx is

string : pattern

The following examples shows how to use the match function:

$ expr match linuxserver lin
3

$ expr match linuxserver server
0

Here in the second expr, the pattern (server) exists in the main string. However the pattern does not start from the beggining of the main string. Thats why the match function returns 0.

Continue reading

SSH/Zip Command Examples in Unix / Linux

SSH Command 

SSH client utility in unix or linux server is used to logging into a remote host and execute commands on the remote machine. The rlogin and rsh commands can also be used to login into the remote machine. However these are not secure. The ssh command provides a secure connection between two hosts over a insecure network.

The syntax ssh command is

ssh [-l username] hostname | user@remote-hostname [command]

Let see the examples of ssh command.

SSH Command Examples:

1. Logging to a remote server

You can login to a remote server from the local host as shown below:

localhost:[~]> ssh -l username remote-server
username@remote-server password:
remote-server:[~]>

Alternatively you can use the below ssh command for connecting to remote host:

localhost:[~]> ssh username@remote-server
username@remote-server password:
remote-server:[~]>

Note: If you are logging for the first time, then it will prints a message that host key not found and you can give yes to continue. The host key of the remote server will be cached and added to the .ssh2/hostkeys directory in your home directory. From second time onwards you just need to enter the password.

2. Logging out from remote server

Simply enter the exit command on the terminal to close the connection. This is shown below:

remote-server:[~]>exit
logout
Connection to remote-server closed.
localhost:[~]>

3. Running remote commands from local host

Sometimes it is necessary to run the unix commands on the remote server from the local host. An example is shown below:

localhost:[~]> ssh user@remote-host "ls test"
online-backup.dat
oracle-storage.bat
unix-dedicated-server.txt

The ssh command connects to the remote host, runs the ls command, prints the output on the local host terminal and exits the connection from remote host.

Let see whether the ls command actually displayed the correct result or not by connecting to the remote host.

localhost:[~]> ssh user@remote-host
user@remotehost password:
remotehost:[~]> cd test
remotehost:[~/test]> ls
online-backup.dat
oracle-storage.bat
unix-dedicated-server.txt

4. Version of the SSH command

We can find the version of SSH installed on the unix system using the -V option to the ssh. This is shown below:

> ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

5. Debugging the SSH Client

When we are not able to connect to the remote host, it is good to debug and find the exact error messages that causing the issue. Use the -v option for debugging the ssh client.

ssh -v user@remote-host
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to remote-host [172.22.200.140] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/identity type -1
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_dsa type 2
debug1: loaded 3 keys
..........
..........

6. Copying files between remote host and local host.

We can use the scp command to copy the files securely between the local host and remote host using the ssh authentication.

To copy the file from local host to remote hosts /var/tmp/ directory, run the below scp command.

scp filename user@remote-host:/var/tmp/

To copy the file from remote hosts /usr/local/bin/ directory to local hosts current directory, run the below scp command.

scp user@remote-host:/usr/local/bin/add.sh.

Zip Command

Zip is used to compress the files to reduce file size and also used as file package utility. zip is available in many operating systems like unix, linux, windows etc.

If you have a limited bandwidth between two servers and want to transfer the files faster, then zip the files and transfer.

The syntax of zip command is

zip [options] zipfile files_list

The options of zip command are:

-d : Removes the file from the zip archive
-u : Updates the file in the zip archive
-m : Deletes the original files after zipping.
-r : Recursively zips the files in a directory
-x : Exclude the files in creating the zip
-v : verbose mode
-1 : Compresses the files faster
-9 : Compresses the files better
-f : freshen only changed files.
zipfile : creates the zip file with name as zipfile.zip
files_list : list of files to be zipped.

Zip Command Examples:

The files in my current directory are listed below:

docs/linux.pdf
docs/oracle.pdf
docs/unix.pdf
linux-virtual-server.bat
unix-server.dat

Here docs is a directory which contains the files linux.pdf, unix.pdf and oracle.pdf. We will see how to use zip command with examples.

1. Creating a zip file

The zip command in unix or linux system creates an archive with the specified files. This is shown below:

> zip archive linux-virtual-server.bat unix-server.dat
  adding: linux-virtual-server.bat (deflated 80%)
  adding: unix-server.dat (deflated 80%)
> ls
archive.zip  docs  linux-virtual-server.bat  unix-server.dat

The above command creates the zip file with name archive.zip

2. Extracting files from zip

To extract files from the zip, use the unzip command in unix system. This is shown below:

> unzip archive.zip
Archive:  archive.zip
  inflating: linux-virtual-server.bat
  inflating: unix-server.dat
> ls
archive.zip  linux-virtual-server.bat  unix-server.dat

3. Removing file from a zip file

After creating a zip file, you can remove a file from the archive using the -d option. To remove the file unix-server.dat from the archive, run the below zip command:

> zip -d archive.zip unix-server.dat
deleting: unix-server.dat

> unzip archive.zip
Archive:  archive.zip
  inflating: linux-virtual-server.bat

4. Update existing zip file

You can update the files in already created zip file. If any of the files are modified after zipping, you can fresh the zip file with only those modified files using the -f option.

> zip -f archive.zip
freshening: linux-virtual-server.bat (stored 0%)

Another way is using the -u option. This option can be used to update the specified list of files or add new files to the existing zip file.

> zip -u archive.zip  linux-virtual-server.bat temp
updating: linux-virtual-server.bat (deflated 79%)
  adding: temp (stored 0%)

5. Recursively zip files in directory.

To zip a directory recursively, use the -r option with the zip command. This example is shown below:

> zip -r dir_archive docs
  adding: docs/ (stored 0%)
  adding: docs/unix.pdf (stored 0%)
  adding: docs/oracle.pdf (stored 0%)
  adding: docs/linux.pdf (stored 0%)

6. Excluding files in zipping

Let say you are zipping all the files in the current directory and want to exclude some unwanted files. You can exclude these unwanted files using the -x option.

zip exclude_archive * -x linux-virtual-server.bat

The above command zips all the files in the current directory except the file linux-virtual-server.bat

7. Faster compressing

You can compress the files very fast using the -1 option with zip command. An example is shown below with and without using fast compression.

> zip -1 fast_archive linux-virtual-server.bat
  adding: linux-virtual-server.bat (deflated 79%)

>zip normal_archive linux-virtual-server.bat
  adding: linux-virtual-server.bat (deflated 80%)

If you use fast compression, the archive file created will occupy more space (size) when compared to normal compression.

8. Better compression.

To reduce more amount of size the files occupied, you can use the -9 option with the zip command. This gives a better compression.

> zip -9 better_archive linux-virtual-server.bat
  adding: linux-virtual-server.bat (deflated 81%)

Compare the deflated percentages in the example 7 and 8.

Continue reading

Join Command in Unix/Linux Examples

Join command is one of the text processing utility in Unix/Linux. Join command is used to combine two files based on a matching fields in the files. If you know SQL, the join command is similar to joining two tables in a database.

The syntax of join command is

join [options] file1 file2

The join command options are

-1 field number : Join on the specified field number in the first file
-2 field number : Join on the specified field number in the second file
-j field number : Equivalent to -1 fieldnumber and -2 fieldnumber
-o list : displays only the specified fields from both the files
-t char : input and output field delimiter
-a filenumber : Prints non matched lines in a file
-i : ignore case while joining

Unix Join Command Examples

1. Write a join command to join two files on the first field?

The basic usage of join command is to join two files on the first field. By default the join command matches the files on the first fields when we do not specify the field numbers explicitly. Let's say we have two files emp.txt and dept.txt

> cat emp.txt
10 mark
10 steve
20 scott
30 chris
> cat dept.txt
10 hr
20 finance
30 db

Here we will join on the first field and see the output. By default, the join command treats the field delimiter as space or tab.

> join emp.txt dept.txt
10 mark hr
10 steve hr
20 scott finance
30 chris db

Important Note: Before joining the files, make sure to sort the fields on the joining fields. Otherwise you will get incorrect result.

2. Write a join command to join the two files? Here use the second field from the first file and the first field from the second file to join.

In this example, we will see how to join two files on different fields rather than the first field. For this consider the below two files as an example

> cat emp.txt
mark 10 1
steve 10 1
scott 20 2
chris 30 3
> cat dept.txt
10 hr 1
20 finance 2
30 db 3

From the above, you can see the join fields are the second field from the emp.txt and the first field from the dept.txt. The join command to match these two files is

> join -1 2 -2 1 emp.txt dept.txt
10 mark 1 hr 1
10 steve 1 hr 1
20 scott 2 finance 2
30 chris 3 db 3

You can also see that the two files can also be joined on the third filed. As the both the files have the matching join field, you can use the j option in the join command.

Here -1 2 specifies the second field from the first file (emp.txt) and -2 1 specifies the first field from the second file (dept.txt)

> join -j 3 emp.txt dept.txt
1 mark 10 10 hr
1 steve 10 10 hr
2 scott 20 20 finance
3 chris 30 30 db

3. Write a join command to select the required fields from the input files in the output? Select first filed from first file and second field from second file in the output.

By default, the join command prints all the fields from both the files (except the join field is printed once). We can choose what fields to be printed on the terminal with the -o option. We will use the same files from the above example.

> join -o 1.1 2.2 -1 2 -2 1 emp.txt dept.txt
mark hr
steve hr
scott finance
chris db

Here 1.1 means in the first file select the first field. Similarly, 2.2 means in the second file select the second field

4. Write a command to join two delimited files? Here the delimiter is colon (:)

So far we have joined files with space delimiter. Here we will see how to join files with a colon as delimiter. Consider the below two files.

> cat emp.txt
mark:10
steve:10
scott:20
chris:30
> cat dept.txt
10:hr
20:finance
30:db

The -t option is used to specify the delimiter. The join command for joining the files is

> join -t: -1 2 -2 1 emp.txt dept.txt
10:mark:hr
10:steve:hr
20:scott:finance
30:chris:db

5. Write a command to ignore case when joining the files?

If the join fields are in different cases, then the join will not be performed properly. To ignore the case in join use the -i option.

> cat emp.txt
mark,A
steve,a
scott,b
chris,C
> cat dept.txt
a,hr
B,finance
c,db

> join -t, -i -1 2 -2 1 emp.txt dept.txt
A,mark,hr
a,steve,hr
b,scott,finance
C,chris,db

6. Write a join command to print the lines which do not match the values in joining fields?

By default the join command prints only the matched lines from both the files which means prints the matched lines that passed the join condition. We can use the -a option to print the non-matched lines.

> cat P.txt
A 1
B 2
C 3
> cat Q.txt
B 2
C 3
D 4

Print non pairable lines from first file.

> join -a 1 P.txt Q.txt
A 1
B 2 2
C 3 3

Print non pairable lines from second file.

> join -a 2 P.txt Q.txt
B 2 2
C 3 3
D 4

Print non pairable lines from both file.

> join -a 1 -a 2 P.txt Q.txt
A 1
B 2 2
C 3 3
D 4

Continue reading

Objectives: LPIC-2 Exam 201 (Linux Engineer)

Topic 200: Capacity Planning

200.1 Measure and Troubleshoot Resource Usage

Weight: 6

Description: Candidates should be able to measure hardware resource and network bandwidth, identify and troubleshoot resource problems.

Key Knowledge Areas:

◈ Measure CPU usage
◈ Measure memory usage
◈ Measure disk I/O
◈ Measure network I/O
◈ Measure firewalling and routing throughput
◈ Map client bandwidth usage
◈ Match / correlate system symptoms with likely problems
◈ Estimate throughput and identify bottlenecks in a system including networking

The following is a partial list of the used files, terms and utilities:

◈ iostat
◈ netstat
◈ w
◈ top
◈ sar
◈ processes blocked on I/O
◈ blocks out
◈ vmstat
◈ pstree, ps
◈ Isof
◈ uptime
◈ swap
◈ blocks in

200.2 Predict Future Resource Needs

Weight: 2

Description: Candidates should be able to monitor resource usage to predict future resource needs.

Key Knowledge Areas:

◈ Use monitoring and measurement tools to monitor IT infrastructure usage.
◈ Predict capacity break point of a configuration
◈ Observe growth rate of capacity usage
◈ Graph the trend of capacity usage
◈ Awareness of monitoring solutions such as Icinga2, Nagios, collectd, MRTG and Cacti

The following is a partial list of the used files, terms and utilities:

◈ diagnose
◈ predict growth
◈ resource exhaustion

Topic 201: Linux Kernel

201.1 Kernel Components

Weight: 2

Description: Candidates should be able to utilize kernel components that are necessary to specific hardware, hardware drivers, system resources and requirements. This objective includes implementing different types of kernel images, identifying stable and development kernels and patches, as well as using kernel modules.

Key Knowledge Areas:

◈ Kernel 2.6.x, 3.x and 4.x documentation

Terms and Utilities:

◈ /usr/src/linux/
◈ /usr/src/linux/Documentation/
◈ zImage
◈ bzImage
◈ xz compression

201.2 Compiling a kernel

Weight: 3

Description: Candidates should be able to properly configure a kernel to include or disable specific features of the Linux kernel as necessary. This objective includes compiling and recompiling the Linux kernel as needed, updating and noting changes in a new kernel, creating an initrd image and installing new kernels.

Key Knowledge Areas:

◈ /usr/src/linux/
◈ Kernel Makefiles
◈ Kernel 2.6.x/3.x make targets
◈ Customize the current kernel configuration.
◈ Build a new kernel and appropriate kernel modules.
◈ Install a new kernel and any modules.
◈ Ensure that the boot manager can locate the new kernel and associated files.
◈ Module configuration files
◈ Use DKMS to compile kernel modules.
◈ Awareness of dracut

Terms and Utilities:

◈ mkinitrd
◈ mkinitramfs
◈ make
◈ make targets (all, config, xconfig, menuconfig, gconfig, oldconfig, mrproper, zImage, bzImage, modules, modules_install, rpm-pkg, binrpm-pkg, deb-pkg)
◈ gzip
◈ bzip2
◈ module tools
◈ /usr/src/linux/.config
◈ /lib/modules/kernel-version/
◈ depmod
◈ dkms

201.3 Kernel runtime management and troubleshooting

Weight: 4

Description: Candidates should be able to manage and/or query a 2.6.x, 3.x or 4.x kernel and its loadable modules. Candidates should be able to identify and correct common boot and run time issues. Candidates should understand device detection and management using udev. This objective includes troubleshooting udev rules.

Key Knowledge Areas:

◈ Use command-line utilities to get information about the currently running kernel and kernel modules
◈ Manually load and unload kernel modules
◈ Determine when modules can be unloaded
◈ Determine what parameters a module accepts
◈ Configure the system to load modules by names other than their file name.
◈ /proc filesystem
◈ Content of /, /boot/ , and /lib/modules/
◈ Tools and utilities to analyze information about the available hardware
◈ udev rules

Terms and Utilities:

◈ /lib/modules/kernel-version/modules.dep
◈ module configuration files in /etc/
◈ /proc/sys/kernel/
◈ /sbin/depmod
◈ /sbin/rmmod
◈ /sbin/modinfo
◈ /bin/dmesg
◈ /sbin/lspci
◈ /usr/bin/lsdev
◈ /sbin/lsmod
◈ /sbin/modprobe
◈ /sbin/insmod
◈ /bin/uname
◈ /usr/bin/lsusb
◈ /etc/sysctl.conf, /etc/sysctl.d/
◈ /sbin/sysctl
◈ udevmonitor
◈ udevadm monitor
◈ /etc/udev/

Topic 202: System Startup

202.1 Customizing SysV-init system startup

Weight: 3

Description: Candidates should be able to query and modify the behaviour of system services at various targets / run levels. A thorough understanding of the systemd, SysV Init and the Linux boot process is required. This objective includes interacting with systemd targets and SysV init run levels.

Key Knowledge Areas:

◈ Systemd
◈ SysV init
◈ Linux Standard Base Specification (LSB)

Terms and Utilities:

◈ /usr/lib/systemd/
◈ /etc/systemd/
◈ /run/systemd/
◈ systemctl
◈ systemd-delta
◈ /etc/inittab
◈ /etc/init.d/
◈ /etc/rc.d/
◈ chkconfig
◈ update-rc.d
◈ init and telinit

202.2 System Recovery

Weight: 4

Description: Candidates should be able to properly manipulate a Linux system during both the boot process and during recovery mode. This objective includes using both the init utility and init-related kernel options. Candidates should be able to determine the cause of errors in loading and usage of bootloaders. GRUB version 2 and GRUB Legacy are the bootloaders of interest. Both BIOS and UEFI systems are covered.

Key Knowledge Areas:

◈ BIOS and UEFI
◈ NVMe booting
◈ GRUB version 2 and Legacy
◈ grub shell
◈ boot loader start and hand off to kernel
◈ kernel loading
◈ hardware initialisation and setup
◈ daemon/service initialisation and setup
◈ Know the different boot loader install locations on a hard disk or removable device.
◈ Overwrite standard boot loader options and using boot loader shells.
◈ Use systemd rescue and emergency modes.

Terms and Utilities:

◈ mount
◈ fsck
◈ inittab, telinit and init with SysV init
◈ The contents of /boot/, /boot/grub/ and /boot/efi/
◈ EFI System Partition (ESP)
◈ GRUB
◈ grub-install
◈ efibootmgr
◈ UEFI shell
◈ initrd, initramfs
◈ Master boot record
◈ systemctl

202.3 Alternate Bootloaders

Weight: 2

Description: Candidates should be aware of other bootloaders and their major features.

Key Knowledge Areas:

◈ SYSLINUX, ISOLINUX, PXELINUX
◈ Understanding of PXE for both BIOS and UEFI
◈ Awareness of systemd-boot and U-Boot

Terms and Utilities:

◈ syslinux
◈ extlinux
◈ isolinux.bin
◈ isolinux.cfg
◈ isohdpfx.bin
◈ efiboot.img
◈ pxelinux.0
◈ pxelinux.cfg/
◈ uefi/shim.efi
◈ uefi/grubx64.efi

Topic 203: Filesystem and Devices

203.1 Operating the Linux filesystem

Weight: 4

Description: Candidates should be able to properly configure and navigate the standard Linux filesystem. This objective includes configuring and mounting various filesystem types.

Key Knowledge Areas:

◈ The concept of the fstab configuration
◈ Tools and utilities for handling swap partitions and files
◈ Use of UUIDs for identifying and mounting file systems
◈ Understanding of systemd mount units

Terms and Utilities:

◈ /etc/fstab
◈ /etc/mtab
◈ /proc/mounts
◈ mount and umount
◈ blkid
◈ sync
◈ swapon
◈ swapoff

203.2 Maintaining a Linux filesystem​

Weight: 3

Description: Candidates should be able to properly maintain a Linux filesystem using system utilities. This objective includes manipulating standard filesystems and monitoring SMART devices.

Key Knowledge Areas:

◈ Tools and utilities to manipulate and ext2, ext3 and ext4
◈ Tools and utilities to perform basic Btrfs operations, including subvolumes and snapshots
◈ Tools and utilities to manipulate XFS
◈ Awareness of ZFS

Terms and Utilities:

◈ mkfs (mkfs.*)
◈ mkswap
◈ fsck (fsck.*)
◈ tune2fs, dumpe2fs and debugfs
◈ btrfs, btrfs-convert
◈ xfs_info, xfs_check, xfs_repair, xfsdump and xfsrestore
◈ smartd, smartctl

203.3 Creating and configuring filesystem options

Weight: 2

Description: Candidates should be able to configure automount filesystems using AutoFS. This objective includes configuring automount for network and device filesystems. Also included is creating filesystems for devices such as CD-ROMs and a basic feature knowledge of encrypted filesystems.

Key Knowledge Areas:

◈ autofs configuration files
◈ Understanding of automount units
◈ UDF and ISO9660 tools and utilities
◈ Awareness of other CD-ROM filesystems (HFS)
◈ Awareness of CD-ROM filesystem extensions (Joliet, Rock Ridge, El Torito)
◈ Basic feature knowledge of data encryption (dm-crypt / LUKS)

Terms and Utilities:

◈ /etc/auto.master
◈ /etc/auto.[dir]
◈ mkisofs
◈ cryptsetup

Topic 204: Advanced Storage Device Administration

204.1 Configuring RAID

Weight: 3

Description: Candidates should be able to configure and implement software RAID. This objective includes using and configuring RAID 0, 1 and 5.

Key Knowledge Areas:

◈ Software raid configuration files and utilities

Terms and Utilities:

◈ mdadm.conf
◈ mdadm
◈ /proc/mdstat
◈ partition type 0xFD

204.2 Adjusting Storage Device Access

Weight: 2

Description: Candidates should be able to configure kernel options to support various drives. This objective includes software tools to view & modify hard disk settings including iSCSI devices.

Key Knowledge Areas:

◈ Tools and utilities to configure DMA for IDE devices including ATAPI and SATA
◈ Tools and utilities to configure Solid State Drives including AHCI and NVMe
◈ Tools and utilities to manipulate or analyse system resources (e.g. interrupts)
◈ Awareness of sdparm command and its uses
◈ Tools and utilities for iSCSI
◈ Awareness of SAN, including relevant protocols (AoE, FCoE)

Terms and Utilities:

◈ hdparm, sdparm
◈ nvme
◈ tune2fs
◈ fstrim
◈ sysctl
◈ /dev/hd*, /dev/sd*, /dev/nvme*
◈ iscsiadm, scsi_id, iscsid and iscsid.conf
◈ WWID, WWN, LUN numbers

204.3 Logical Volume Manager

Weight: 3

Description: Candidates should be able to create and remove logical volumes, volume groups, and physical volumes. This objective includes snapshots and resizing logical volumes.

Key Knowledge Areas:

◈ Tools in the LVM suite
◈ Resizing, renaming, creating, and removing logical volumes, volume groups, and physical volumes
◈ Creating and maintaining snapshots
◈ Activating volume groups

Terms and Utilities:

◈ /sbin/pv*
◈ /sbin/lv*
◈ /sbin/vg*
◈ mount
◈ /dev/mapper/
◈ lvm.conf

Topic 205: Network Configuration

205.1 Basic networking configuration

Weight: 3

Description: Candidates should be able to configure a network device to be able to connect to a local, wired or wireless, and a wide-area network. This objective includes being able to communicate between various subnets within a single network including both IPv4 and IPv6 networks.

Key Knowledge Areas:

Utilities to configure and manipulate ethernet network interfaces
Configuring basic access to wireless networks

Terms and Utilities:

◈ ip
◈ ifconfig
◈ route
◈ arp
◈ iw
◈ iwconfig
◈ iwlist

205.2 Advanced Network Configuration and Troubleshooting

Weight: 4

Description: Candidates should be able to configure a network device to implement various network authentication schemes. This objective includes configuring a multi-homed network device and resolving communication problems.

Key Knowledge Areas:

◈ Utilities to manipulate routing tables
◈ Utilities to configure and manipulate ethernet network interfaces
◈ Utilities to analyze the status of the network devices
◈ Utilities to monitor and analyze the TCP/IP traffic

Terms and Utilities:

◈ ip
◈ ifconfig
◈ route
◈ arp
◈ ss
◈ netstat
◈ lsof
◈ ping, ping6
◈ nc
◈ tcpdump
◈ nmap

205.3 Troubleshooting Network Issues

Weight: 4

Description: Candidates should be able to identify and correct common network setup issues, to include knowledge of locations for basic configuration files and commands.

Key Knowledge Areas:

◈ Location and content of access restriction files
◈ Utilities to configure and manipulate ethernet network interfaces
◈ Utilities to manage routing tables
◈ Utilities to list network states.
◈ Utilities to gain information about the network configuration
◈ Methods of information about the recognized and used hardware devices
◈ System initialization files and their contents (SysV init process)
◈ Awareness of NetworkManager and its impact on network configuration

Terms and Utilities:

◈ ip
◈ ifconfig
◈ route
◈ ss
◈ netstat
◈ /etc/network/, /etc/sysconfig/network-scripts/
◈ ping, ping6
◈ traceroute, traceroute6
◈ mtr
◈ hostname
◈ System log files such as /var/log/syslog, /var/log/messages and the systemd journal
◈ dmesg
◈ /etc/resolv.conf
◈ /etc/hosts
◈ /etc/hostname, /etc/HOSTNAME
◈ /etc/hosts.allow, /etc/hosts.deny

Topic 206: System Maintenance

206.1 Make and install programs from source

Weight: 2

Description: Candidates should be able to build and install an executable program from source. This objective includes being able to unpack a file of sources.

Key Knowledge Areas:

◈ Unpack source code using common compression and archive utilities
◈ Understand basics of invoking make to compile programs
◈ Apply parameters to a configure script
◈ Know where sources are stored by default

Terms and Utilities:

◈ /usr/src/
◈ gunzip
◈ gzip
◈ bzip2
◈ xz
◈ tar
◈ configure
◈ make
◈ uname
◈ install
◈ patch

206.2 Backup operations

Weight: 3

Description: Candidates should be able to use system tools to back up important system data.

Key Knowledge Areas:

◈ Knowledge about directories that have to be include in backups
◈ Awareness of network backup solutions such as Amanda, Bacula, Bareos and BackupPC
◈ Knowledge of the benefits and drawbacks of tapes, CDR, disk or other backup media
◈ Perform partial and manual backups.
◈ Verify the integrity of backup files.
◈ Partially or fully restore backups.

Terms and Utilities:

◈ /bin/sh
◈ dd
◈ tar
◈ /dev/st* and /dev/nst*
◈ mt
◈ rsync

206.3 Notify users on system-related issues

Weight: 1

Description: Candidates should be able to notify the users about current issues related to the system.

Key Knowledge Areas:

◈ ​Automate communication with users through logon messages
◈ Inform active users of system maintenance

Terms and Utilities:

◈ /etc/issue
◈ /etc/issue.net
◈ /etc/motd
◈ wall
◈ /sbin/shutdown
◈ systemctl

Continue reading

Delete Directory, Files - rm, rmdir command in Unix / Linux

How to delete directories and files in unix/linux

Unix provides rmdir and rm commands to remove the directories and files. Let see each command in detail.

Unix rmdir command syntax

The syntax of rmdir command is

rmdir [options] directories

The rmdir command options are

-p : Removes directory and its parent directories
-v : Provides the diagnostic information of the directory processed

Unix rmdir command examples

1. Write a unix/linux command to remove a directory?

The rmdir command deletes only the empty directories. If a directory contains files or sub directories, then the rmdir command fails.

rmdir docs/
rmdir: docs/: Directory not empty

Here the docs directory is not empty, that is why the rmdir command failed to remove the directory. To remove the docs directory first we have to make the directory empty and then delete the directory.

rm doc/*
rmdir docs/

We will see later how to remove non-empty directories with a single command.

2. Write a unix/linux command to remove the directory and its parent directories?

As mentioned earlier the -p option allows the rmdir command to delete the directory and also its parent directories.

rmdir -p docs/entertainment/movies/

This rmdir command removes the docs directory completely. If you don’t use the -p option, then it only deletes the movies directory.

3. Write a unix/linux command to remove directories using pattern matching?

You can specify the directory names using the regular expressions and can delete them.

rm doc*

This rm command deletes the directories like doc, documents, doc_1 etc.

Now we will see the rm command in unix.

Unix rm command syntax

The syntax of rm command is

rm [options] [directory|file]

The rm command options are

f : Removes all files in a directory without prompting the user.
i : Interactive: prompts the user for confirmation before deleting a file.
R or r : Recursively remove directories and sub directories.

The rm command can be used to delete both the files and directories. The rm command also deletes the non-empty directories.

Unix rm command examples

1. Write a unix/linux command to remove a file?

This is the basic feature of rm command. To remove a file, logfile.dat, in the current directory use the below rm command

rm logfile.dat

2. Write a unix/linux command to remove all the files in a directory?

use the * regular pattern as the file list in rm command for deleting all the files in the current directory.

rm *

3. Write a unix/linux command to delete empty directory?

The rm command can also be used to delete the empty directory. The command for this is

rm docs/

If the directory is non-empty, then the above command fails to remove the directories.

4. Write a unix/linux command to delete directories recursively (delete non empty directories)?

As mentioned earlier, the -r option can be used to remove the directories and sub directories.

rm -r docs

This removes the docs directory even if it is non-empty.

Continue reading

Grep Command in Unix and Linux Examples

Grep is the frequently used command in Unix (or Linux). Most of us use grep just for finding the words in a file. The power of grep comes with using its options and regular expressions. You can analyze large sets of log files with the help of grep command.

Grep stands for Global search for Regular Expressions and Print.

The basic syntax of grep command is

grep [options] pattern [list of files]

Let see some practical examples on grep command.

1. Running the last executed grep command

This saves a lot of time if you are executing the same command again and again.

!grep

This displays the last executed grep command and also prints the result set of the command on the terminal.

2. Search for a string in a file

This is the basic usage of grep command. It searches for the given string in the specified file.

grep "Error" logfile.txt

This searches for the string "Error" in the log file and prints all the lines that has the word "Error".

3. Searching for a string in multiple files.

grep "string" file1 file2
grep "string" file_pattern

This is also the basic usage of the grep command. You can manually specify the list of files you want to search or you can specify a file pattern (use regular expressions) to search for.

4. Case insensitive search

The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".

grep -i "UNix" file.txt

5. Specifying the search string as a regular expression pattern.

grep "^[0-9].*" file.txt

This will search for the lines which starts with a number. Regular expressions is huge topic and I am not covering it here. This example is just for providing the usage of regular expressions.

6. Checking for the whole words in a file.

By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.

grep -w "world" file.txt

7. Displaying the lines before the match.

Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.

grep -B 2 "Error" file.txt

This will prints the matched lines along with the two lines before the matched lines.

8. Displaying the lines after the match.

grep -A 3 "Error" file.txt

This will display the matched lines along with the three lines after the matched lines.

9. Displaying the lines around the match

grep -C 5 "Error" file.txt

This will display the matched lines and also five lines before and after the matched lines.

10. Searching for a sting in all files recursively

You can search for a string in all the files under the current directory and sub-directories with the help -r option.

grep -r "string" *

11. Inverting the pattern match

You can display the lines that are not matched with the specified search sting pattern using the -v option.

grep -v "string" file.txt

12. Displaying the non-empty lines

You can remove the blank lines using the grep command.

grep -v "^$" file.txt

13. Displaying the count of number of matches.

We can find the number of lines that matches the given string/pattern

grep -c "sting" file.txt

14. Display the file names that matches the pattern.

We can just display the files that contains the given string/pattern.

grep -l "string" *

15. Display the file names that do not contain the pattern.

We can display the files which do not contain the matched string/pattern.

grep -L "string" *

16. Displaying only the matched pattern.

By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.

grep -o "string" file.txt

17. Displaying the line numbers.

We can make the grep command to display the position of the line which contains the matched string in a file using the -n option

grep -n "string" file.txt

18. Displaying the position of the matched string in the line

The -b option allows the grep command to display the character position of the matched string in a file.

grep -o -b "string" file.txt

19. Matching the lines that start with a string

The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.

grep "^start" file.txt

20. Matching the lines that end with a string

The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.

grep "end$" file.txt

Continue reading

Objectives: LPIC-1 Exam 101

Topic 101: System Architecture

101.1 Determine and configure hardware settings

Weight: 2

Description: Candidates should be able to determine and configure fundamental system hardware.

Key Knowledge Areas:

◈ Tools and utilities to list various hardware information (e.g. lsusb, lspci, etc.)
◈ Tools and utilities to manipulate USB devices
◈ Conceptual understanding of sysfs, udev, dbus



The following is a partial list of the used files, terms and utilities:

◈ /sys/
◈ /proc/
◈ /dev/
◈ modprobe
◈ lsmod
◈ lspci
◈ lsusb

101.2 Boot the system

Weight: 3

Description: Candidates should be able to guide the system through the booting process.

Key Knowledge Areas:

◈ Provide common commands to the boot loader and options to the kernel at boot time
◈ Demonstrate knowledge of the boot sequence from BIOS to boot completion
◈ Understanding of SysVinit and systemd
◈ Awareness of Upstart
◈ Check boot events in the log files

Terms and Utilities:

◈ dmesg
◈ BIOS
◈ bootloader
◈ kernel
◈ initramfs
◈ init
◈ SysVinit
◈ systemd

101.3 Change runlevels / boot targets and shutdown or reboot system

Weight: 3

Description: Candidates should be able to manage the SysVinit runlevel or systemd boot target of the system. This objective includes changing to single user mode, shutdown or rebooting the system. Candidates should be able to alert users before switching runlevels / boot targets and properly terminate processes. This objective also includes setting the default SysVinit runlevel or systemd boot target. It also includes awareness of Upstart as an alternative to SysVinit or systemd.

Key Knowledge Areas:

◈ Set the default runlevel or boot target
◈ Change between runlevels / boot targets including single user mode
◈ Shutdown and reboot from the command line
◈ Alert users before switching runlevels / boot targets or other major system events
◈ Properly terminate processes

Terms and Utilities:

◈ /etc/inittab
◈ shutdown
◈ init
◈ /etc/init.d/
◈ telinit
◈ systemd
◈ systemctl
◈ /etc/systemd/
◈ /usr/lib/systemd/
◈ wall

Topic 102: Linux Installation and Package Management

102.1 Design hard disk layout

Weight: 2

Description: Candidates should be able to design a disk partitioning scheme for a Linux system.

Key Knowledge Areas:

◈ Allocate filesystems and swap space to separate partitions or disks
◈ Tailor the design to the intended use of the system
◈ Ensure the /boot partition conforms to the hardware architecture requirements for booting
◈ Knowledge of basic features of LVM

Terms and Utilities:

◈ / (root) filesystem
◈ /var filesystem
◈ /home filesystem
◈ /boot filesystem
◈ swap space
◈ mount points
◈ partitions

102.2 Install a boot manager

Weight: 2

Description: Candidates should be able to select, install and configure a boot manager.

Key Knowledge Areas:

◈ Providing alternative boot locations and backup boot options
◈ Install and configure a boot loader such as GRUB Legacy
◈ Perform basic configuration changes for GRUB 2
◈ Interact with the boot loader

The following is a partial list of the used files, terms and utilities:

◈ menu.lst, grub.cfg and grub.conf
◈ grub-install
◈ grub-mkconfig
◈ MBR

102.3 Manage shared libraries

Weight: 1

Description: Candidates should be able to determine the shared libraries that executable programs depend on and install them when necessary.

Key Knowledge Areas:

◈ Identify shared libraries
◈ Identify the typical locations of system libraries
◈ Load shared libraries

Terms and Utilities:

◈ ldd
◈ ldconfig
◈ /etc/ld.so.conf
◈ LD_LIBRARY_PATH

102.4 Use Debian package management

Weight: 3

Description: Candidates should be able to perform package management using the Debian package tools.

Key Knowledge Areas:

◈ Install, upgrade and uninstall Debian binary packages
◈ Find packages containing specific files or libraries which may or may not be installed
◈ Obtain package information like version, content, dependencies, package integrity and installation status (whether or not the package is installed)

Terms and Utilities:

◈ /etc/apt/sources.list
◈ dpkg
◈ dpkg-reconfigure
◈ apt-get
◈ apt-cache
◈ aptitude

102.5 Use RPM and YUM package management

Weight: 3

Description: Candidates should be able to perform package management using RPM and YUM tools.

Key Knowledge Areas:

◈ Install, re-install, upgrade and remove packages using RPM and YUM
◈ Obtain information on RPM packages such as version, status, dependencies, integrity and signatures
◈ Determine what files a package provides, as well as find which package a specific file comes from

Terms and Utilities:

◈ rpm
◈ rpm2cpio
◈ /etc/yum.conf
◈ /etc/yum.repos.d/
◈ yum
◈ yumdownloader

Topic 103: GNU and Unix Commands

103.1 Work on the command line

Weight: 4

Description: Candidates should be able to interact with shells and commands using the command line. The objective assumes the Bash shell.

Key Knowledge Areas:

◈ Use single shell commands and one line command sequences to perform basic tasks on the command line
◈ Use and modify the shell environment including defining, referencing and exporting environment variables
◈ Use and edit command history
◈ Invoke commands inside and outside the defined path

Terms and Utilities:

◈ bash
◈ echo
◈ env
◈ export
◈ pwd
◈ set
◈ unset
◈ man
◈ uname
◈ history
◈ .bash_history

103.2 Process text streams using filters

Weight: 3

Description: Candidates should should be able to apply filters to text streams.

Key Knowledge Areas:

◈ Send text files and output streams through text utility filters to modify the output using standard UNIX commands found in the GNU textutils package

Terms and Utilities:

◈ cat
◈ cut
◈ expand
◈ fmt
◈ head
◈ join
◈ less
◈ nl
◈ od
◈ paste
◈ pr
◈ sed
◈ sort
◈ split
◈ tail
◈ tr
◈ unexpand
◈ uniq
◈ wc

103.3 Perform basic file management

Weight: 4

Description: Candidates should be able to use the basic Linux commands to manage files and directories.

Key Knowledge Areas:

◈ Copy, move and remove files and directories individually
◈ Copy multiple files and directories recursively
◈ Remove files and directories recursively
◈ Use simple and advanced wildcard specifications in commands
◈ Using find to locate and act on files based on type, size, or time
◈ Usage of tar, cpio and dd

Terms and Utilities:

◈ cp
◈ find
◈ mkdir
◈ mv
◈ ls
◈ rm
◈ rmdir
◈ touch
◈ tar
◈ cpio
◈ dd
◈ file
◈ gzip
◈ gunzip
◈ bzip2
◈ xz
◈ file globbing

103.4 Use streams, pipes and redirects

Weight: 4

Description: Candidates should be able to redirect streams and connect them in order to efficiently process textual data. Tasks include redirecting standard input, standard output and standard error, piping the output of one command to the input of another command, using the output of one command as arguments to another command and sending output to both stdout and a file.

Key Knowledge Areas:

◈ Redirecting standard input, standard output and standard error
◈ Pipe the output of one command to the input of another command
◈ Use the output of one command as arguments to another command
◈ Send output to both stdout and a file

Terms and Utilities:

◈ tee
◈ xargs

103.5 Create, monitor and kill processes

Weight: 4

Description: Candidates should be able to perform basic process management.

Key Knowledge Areas:

◈ Run jobs in the foreground and background
◈ Signal a program to continue running after logout
◈ Monitor active processes
◈ Select and sort processes for display
◈ Send signals to processes

Terms and Utilities:

◈ &
◈ bg
◈ fg
◈ jobs
◈ kill
◈ nohup
◈ ps
◈ top
◈ free
◈ uptime
◈ pgrep
◈ pkill
◈ killall
◈ screen

103.6 Modify process execution priorities

Weight: 2

Description: Candidates should should be able to manage process execution priorities.

Key Knowledge Areas:

◈ Know the default priority of a job that is created
◈ Run a program with higher or lower priority than the default
◈ Change the priority of a running process

Terms and Utilities:

◈ nice
◈ ps
◈ renice
◈ top

103.7 Search text files using regular expressions

Weight: 2

Description: Candidates should be able to manipulate files and text data using regular expressions. This objective includes creating simple regular expressions containing several notational elements. It also includes using regular expression tools to perform searches through a filesystem or file content.

Key Knowledge Areas:

Create simple regular expressions containing several notational elements
Use regular expression tools to perform searches through a filesystem or file content

Terms and Utilities:

◈ grep
◈ egrep
◈ fgrep
◈ sed
◈ regex(7)

103.8 Perform basic file editing operations using vi

Weight: 3

Description: Candidates should be able to edit text files using vi. This objective includes vi navigation, basic vi modes, inserting, editing, deleting, copying and finding text.

Key Knowledge Areas:

◈ Navigate a document using vi
◈ Use basic vi modes
◈ Insert, edit, delete, copy and find text

Terms and Utilities:

◈ vi
◈ /, ?
◈ h,j,k,l
◈ i, o, a
◈ c, d, p, y, dd, yy
◈ ZZ, :w!, :q!, :e!

Topic 104: Devices, Linux Filesystems, Filesystem Hierarchy Standard

104.1 Create partitions and filesystems

Weight: 2

Description: Candidates should be able to configure disk partitions and then create filesystems on media such as hard disks. This includes the handling of swap partitions.

Key Knowledge Areas:

◈ Manage MBR partition tables
◈ Use various mkfs commands to create various filesystems such as:
◈ ext2/ext3/ext4
◈ XFS
◈ VFAT
◈ Awareness of ReiserFS and Btrfs
◈ Basic knowledge of gdisk and parted with GPT

Terms and Utilities:

◈ fdisk
◈ gdisk
◈ parted
◈ mkfs
◈ mkswap

104.2 Maintain the integrity of filesystems

Weight: 2

Description: Candidates should be able to maintain a standard filesystem, as well as the extra data associated with a journaling filesystem.

Key Knowledge Areas:

◈ Verify the integrity of filesystems
◈ Monitor free space and inodes
◈ Repair simple filesystem problems

Terms and Utilities:

◈ du
◈ df
◈ fsck
◈ e2fsck
◈ mke2fs
◈ debugfs
◈ dumpe2fs
◈ tune2fs
◈ XFS tools (such as xfs_metadump and xfs_info)

104.3 Control mounting and unmounting of filesystems

Weight: 3

Description: Candidates should be able to configure the mounting of a filesystem.

Key Knowledge Areas:

◈ Manually mount and unmount filesystems
◈ Configure filesystem mounting on bootup
◈ Configure user mountable removable filesystems

Terms and Utilities:

◈ /etc/fstab
◈ /media/
◈ mount
◈ umount

104.4 Manage disk quotas

Weight: 1

Description: Candidates should be able to manage disk quotas for users.

Key Knowledge Areas:

◈ Set up a disk quota for a filesystem
◈ Edit, check and generate user quota reports

Terms and Utilities:

◈ quota
◈ edquota
◈ repquota
◈ quotaon

104.5 Manage file permissions and ownership

Weight: 3

Description: Candidates should be able to control file access through the proper use of permissions and ownerships.

Key Knowledge Areas:

◈ Manage access permissions on regular and special files as well as directories
◈ Use access modes such as suid, sgid and the sticky bit to maintain security
◈ Know how to change the file creation mask
◈ Use the group field to grant file access to group members

Terms and Utilities:

◈ chmod
◈ umask
◈ chown
◈ chgrp

104.6 Create and change hard and symbolic links

Weight: 2

Description: Candidates should be able to create and manage hard and symbolic links to a file.

Key Knowledge Areas:

◈ Create links
◈ Identify hard and/or soft links
◈ Copying versus linking files
◈ Use links to support system administration tasks

Terms and Utilities:

◈ ln
◈ ls

104.7 Find system files and place files in the correct location

Weight: 2

Description: Candidates should be thoroughly familiar with the Filesystem Hierarchy Standard (FHS), including typical file locations and directory classifications.

Key Knowledge Areas:

◈ Understand the correct locations of files under the FHS
◈ Find files and commands on a Linux system
◈ Know the location and purpose of important file and directories as defined in the FHS

Terms and Utilities:

◈ find
◈ locate
◈ updatedb
◈ whereis
◈ which
◈ type
◈ /etc/updatedb.conf

Continue reading

Top Examples of Awk Command in Unix

Awk is one of the most powerful tools in Unix used for processing the rows and columns in a file. Awk has built in string functions and associative arrays. Awk supports most of the operators, conditional blocks, and loops available in C language.

One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility.

The basic syntax of AWK:

awk 'BEGIN {start_action} {action} END {stop_action}' filename

Here the actions in the begin block are performed before processing the file and the actions in the end block are performed after processing the file. The rest of the actions are performed while processing the file.

Examples:

Create a file input_file with the following data. This file can be easily created using the output of ls -l.

-rw-r--r-- 1 center center  0 Dec  8 21:39 p1
-rw-r--r-- 1 center center 17 Dec  8 21:15 t1
-rw-r--r-- 1 center center 26 Dec  8 21:38 t2
-rw-r--r-- 1 center center 25 Dec  8 21:38 t3
-rw-r--r-- 1 center center 43 Dec  8 21:39 t4
-rw-r--r-- 1 center center 48 Dec  8 21:39 t5

From the data, you can observe that this file has rows and columns. The rows are separated by a new line character and the columns are separated by a space characters. We will use this file as the input for the examples discussed here.

1. awk '{print $1}' input_file

Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively. This awk command will print the first column in each row as shown below.

-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--

To print the 4th and 6th columns in a file use awk '{print $4,$5}' input_file

Here the Begin and End blocks are not used in awk. So, the print command will be executed for each row it reads from the file. In the next example we will see how to use the Begin and End blocks.

2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file

This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block.

3. In this example we will see how to execute the awk script written in a file. Create a file sum_column and paste the below script in that file

#!/usr/bin/awk -f
BEGIN {sum=0} 
{sum=sum+$5} 
END {print sum}

Now execute the the script using awk command as

awk -f sum_column input_file.

This will run the script in sum_column file and displays the sum of the 5th column in the input_file.

4. awk '{ if($9 == "t4") print $0;}' input_file

This awk command checks for the string "t4" in the 9th column and if it finds a match then it will print the entire line. The output of this awk command is

-rw-r--r-- 1 pcenter pcenter 43 Dec  8 21:39 t4

5. awk 'BEGIN { for(i=1;i<=5;i++) print "square of", i, "is",i*i; }'

This will print the squares of first numbers from 1 to 5. The output of the command is

square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25

Notice that the syntax of “if” and “for” are similar to the C language.

Awk Built in Variables: 

You have already seen $0, $1, $2... which prints the entire line, first column, second column... respectively. Now we will see other built in variables with examples.

FS - Input field separator variable:

So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter.

6. awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2}' input_file

This will print the result as

39 p1
15 t1
38 t2
38 t3
39 t4
39 t5

OFS - Output field separator variable:

By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter. For example

7. awk '{print $4,$5}' input_file

The output of this command will be

center 0
center 17
center 26
center 25
center 43
center 48

We can change this default behavior using the OFS variable as

awk 'BEGIN {OFS=":"} {print $4,$5}' input_file

center:0
center:17
center:26
center:25
center:43
center:48

Note: print $4,$5 and print $4$5 will not work the same way. The first one displays the output with space as delimiter. The second one displays the output without any delimiter.

NF - Number of fileds variable:

The NF can be used to know the number of fields in line

8. awk '{print NF}' input_file

This will display the number of columns in each row.

NR - number of records variable:

The NR can be used to know the line number or count of lines in a file.

9. awk '{print NR}' input_file

This will display the line numbers from 1.

10. awk 'END {print NR}' input_file

This will display the total number of lines in the file.

String functions in Awk:
Some of the string functions in awk are:

index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)

Advanced Examples:

1. Filtering lines using Awk split function

The awk split function splits a string into an array using the delimiter.

The syntax of split function is
split(string, array, delimiter)

Now we will see how to filter the lines using the split function with an example.

The input "file.txt" contains the data in the following format

1 U,N,UNIX,000
2 N,P,SHELL,111
3 I,M,UNIX,222
4 X,Y,BASH,333
5 P,R,SCRIPT,444

Required output: Now we have to print only the lines in which whose 2nd field has the string "UNIX" as the 3rd field( The 2nd filed in the line is separated by comma delimiter ).
The ouptut is:

1 U,N,UNIX,000
3 I,M,UNIX,222

The awk command for getting the output is:

awk '{ 
        split($2,arr,","); 
        if(arr[3] == "UNIX") 
        print $0 
} ' file.txt

Continue reading