Example Uses Of The Linux grep Command

Introduction

The Linux grep command is used as a method for filtering input.

GREP stands for Global Regular Expression Printer and therefore in order to use it effectively, you should have some knowledge about regular expressions.

In this article, I am going to show you a number of examples which will help you understand the grep command.

01. How To Search For A String In A File Using GREP

Imagine you have a text file called books with the following children's book titles:

◈ Robin Hood

◈ Little Red Riding Hood

◈ Peter Pan

◈ Goldilocks And The Three Bears

◈ Snow White And The Seven Dwarfs

◈ Pinnochio

◈ The Cat In The Hat

◈ The Three Little Pigs

◈ The Gruffalo

◈ Charlie And The Chocolate Factory

To find all the books with the word "The" in the title you would use the following syntax:

grep The books

The following results will be returned:

◈ Goldilocks And The Three Bears

◈ Snow White And The Seven Dwarfs

◈ The Cat In The Hat

◈ The Three Little Pigs

◈ The Gruffalo

◈ Charlie And The Chocolate Factory

In each case, the word "The" will be highlighted.

Note that the search is case sensitive so if one of the titles had "the" instead of "The" then it would not have been returned.

◈ To ignore the case you can add the following switch:

grep the books --ignore-case

You can also use the -i switch as follows:

◈ grep -i the books

02. Search For A String In A File Using Wildcards

The grep command is very powerful. You can use a multitude of pattern matching techniques to filter results.

In this example, I will show you how to search for a string in a file using wildcards.

Imagine you have a file called places with the following Scottish place names:

aberdeen

aberystwyth

aberlour

inverurie

inverness

newburgh

new deer

new galloway

glasgow

edinburgh

If you want to find all the places with inver in the name use the following syntax:

grep inver* places

The asterisk (*) wildcard stands for 0 or many. Therefore if you have a place called inver or a place called inverness then both would be returned.

Another wildcard you can use is the period (.). You can use this to match a single letter.

grep inver.r places

The above command would find places called inverurie and inverary but wouldn't find invereerie because there can only be one wildcard between the two r's as denoted by the single period.

The period wildcard is useful but it can cause problems if you have one as part of the text you are searching.

To find all the about.coms you could just search using the following syntax:

grep *about* domainnames

The above command would fall down if the list contained the following name in it:

◈ everydaylinuxuser.com/about.html

You could, therefore, try the following syntax:

grep *about.com domainnames

This would work ok unless there was a domain with the following name:

aboutycom.com

To really search for the term about.com you would need to escape the dot as follows:

grep *about\.com domainnames

The final wildcard to show you is the question mark which stands for zero or one character.

For example:

grep ?ber placenames

The above command would return aberdeen, aberystwyth or even berwick.

03. Search For Strings At The Beginning And End Of Line Using grep

The carat (^) and the dollar ($) symbol allow you to search for patterns at the beginning and end of lines.

Imagine you have a file called football with the following team names:

◈ Blackpool

◈ Liverpool

◈ Manchester City

◈ Leicester City

◈ Manchester United

◈ Newcastle United

◈ FC United Of Manchester

If you wanted to find all the teams that began with Manchester you would use the following syntax:

grep ^Manchester teams

The above command would return Manchester City and Manchester United but not FC United Of Manchester.

Alternatively you can find all the teams ending with United using the following syntax:

grep United$ teams

The above command would return Manchester United and Newcastle United but not FC United Of Manchester.

04. Counting The Number Of Matches Using grep

If you don't want to return the actual lines that match a pattern using grep but you just want to know how many there are you can use the following syntax:

grep -c pattern inputfile

If the pattern was matched twice then the number 2 would be returned.

05. Finding All The Terms That Don't Match using grep

Imagine you have a list of place names with the countries listed as follows:

◈ aberdeen scotland

◈ glasgow scotland

◈ liverpool england

◈ colwyn bay

◈ london england

You may have noticed that colwyn bay has no country associated with it.

To search for all the places with a country you could use the following syntax:

grep land$ places

The results returns would be all the places except for colwyn bay.

This obviously only works for places which end in land (hardly scientific). 

You can invert the select using the following syntax:

grep -v land$ places

This would find all the places that didn't end with land.

06. How To Find Empty Lines In Files Using grep

Imagine you have an input file which is used by a third party application which stops reading the file when it finds an empty line as follows:

◈ aberdeen scotland

◈ inverness scotland

◈ liverpool england

◈ colwyn bay wales

When the application gets to the line after liverpool it will stop reading meaning colwyn bay is missed entirely.

You can use grep to search for blank lines with the following syntax:

grep ^$ places

Unfortunately this isn't particularly useful because it just returns the blank lines.

You could of course get a count of the number of blank lines as a check to see if the file is valid as follows:

grep -c ^$ places

It would however be more useful to know the line numbers that have a blank line so that you can replace them. You can do that with the following command:

grep -n ^$ places

07. How To Search For Strings Of Uppercase Or Lowercase Characters Using grep

Using grep you can determine which lines in a file have uppercase characters using the following syntax:

grep '[A-Z]' filename

The square brackets [] let you determine the range of characters. In the above example it matches any character which is between A and Z.

Therefore to match lowercase characters you can use the following syntax:

grep '[a-z]' filename

If you want to match only letters and not numerics or other symbols you can use the following syntax:

grep '[a-zA-Z]' filename

You can do the same with numbers as follows:

grep '[0-9]' filename

08. Looking For Repeating Patterns Using grep

You can use curly brackets {} to search for a repeating pattern.

Imagine you have a file with phone numbers as follows:

◈ 055-1234

◈ 055-4567

◈ 555-1545

◈ 444-0167

◈ 444-0854

◈ 4549-2234

◈ x44-1234

You know the first part of the number needs to be three digits and you want to find the lines that do not match this pattern.

From the previous example you know that [0-9] returns all numbers in a file.

In this instance we want the lines that start with three numbers followed by a hyphen (-). You can do that with the following syntax:

grep "^[0-9][0-9][0-9]-" numbers

As we know from previous examples the carat (^) means that the line must begin with the following pattern.

The [0-9] will search for any number between 0 and 9. As this is included three times it matches 3 numbers. Finally there is a hyphen to denote that a hyphen must succeed the three numbers.

By using the curly brackets you can make the search smaller as follows:

grep "^[0-9]\{3\}-" numbers

The slash escapes the { bracket so that it works as part of the regular expression but in essence what this is saying is [0-9]{3} which means any number between 0 and 9 three times.

The curly brackets can also be used as follows:

{5,10}

{5,}

The {5,10} means that the character being searched for must be repeated at least 5 times but no more than 10 whereas the {5,} means that the character must be repeated at least 5 times but it can be more than that.

09. Using The Output From Other Commands Using grep

Thus far we have looked at pattern matching within individual files but grep can use the output from other commands as the input for pattern matching.

A great example of this is using the ps command which lists active processes.

For example run the following command:

ps -ef

All of the running processes on your system will be displayed.

You can use grep to search for a particular running process as follows:

ps -ef | grep firefox

Continue reading

Linux: GRUB

GRUB, or the GRand Unified Boot loader, was for a long time one of the most common Linux boot loaders. You can install GRUB into the MBR of your bootable hard drive or into the partition boot record of a partition. You can also install it on removable devices such as floppy disks, CDs, or USB keys. It is a good idea to practice on a floppy disk or USB key if you are not already familiar with GRUB. The examples in this tutorial show you how.

Note: Most GRUB examples in this tutorial use CentOS 6.

During Linux installation, you often specify your choice of boot manager. If you choose LILO, then you might not have GRUB installed. If you do not have GRUB installed and it is available for your distribution, then you need to install the package for it. This tutorial assumes that you already have the GRUB package installed.

GRUB (Legacy) has a configuration file that is usually stored in /boot/grub/grub.conf. If your file system supports symbolic links, as most Linux file systems do, you probably have /boot/grub/menu.lst as a symbolic link to /boot/grub/grub.conf.

The grub command (/sbin/grub, or, on some systems, /usr/sbin/grub) is a small but reasonably powerful shell that supports several commands for installing GRUB, booting systems, locating and displaying configuration files, and similar tasks. This shell shares much code with the second stage GRUB boot loader, so it is useful to learn about GRUB without having to boot to a second stage GRUB environment. The GRUB stage 2 runs either in menu mode, so that you can choose an operating system from a menu, or in command mode, where you specify individual commands to load a system. There are also several other commands, such as grub-install, that use the grub shell and help automate tasks such as installing GRUB.

Listing 1 shows a fairly complex GRUB configuration file. As you look through it, remember one important thing: GRUB, at least GRUB Legacy, counts drives, partitions, and things that need to be counted, starting at 0 rather than 1. The second entry for CentOS has a kernel line that is very long. Listing 1 shows it with a backslash (\) indicating where it was broken for publication.

Listing 1. /boot/grub/menu.lst GRUB configuration example

# grub.conf generated by anaconda
#
# You do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,5)
#          kernel /boot/vmlinuz-version ro root=/dev/hda6
#          initrd /boot/initrd-version.img
#boot=/dev/hda
default=0
timeout=60
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
#password --md5 $1$y.uQRs1W$Sqs30hDB3GtE957PoiDWO.

title Fedora 22 64-bit (sda5)
    root (hd0,4)
        kernel /boot/grub2/i386-pc/core.img

title Fedora 18 64-bit (sda7)
    root (hd0,6)
        kernel /boot/grub2/i386-pc/core.img

title CentOS 6 64-bit (sda11)
        root (hd0,10)
        configfile /boot/grub/menu.lst

title CentOS (2.6.32-504.23.4.el6.x86_64)
    root (hd0,10)
    kernel /boot/vmlinuz-2.6.32-504.23.4.el6.x86_64 ro \
           root=UUID=2f60a3b4-ef6c-4d4c-9ef4-50d7f75124a2 rd_NO_LUKS rd_NO_LVM \
           LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M \
           KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
    initrd /boot/initramfs-2.6.32-504.23.4.el6.x86_64.img

title Fedora 20 64-bit (sda10)
    root (hd0,9)
        configfile /boot/grub/menu.lst

title Ubuntu 12.04-LTS 64-bit (sda9)
    root (hd0,8)
        kernel /boot/grub/core.img

title Ubuntu 14.04 32-bit (sda12)
    root (hd0,11)
        kernel /boot/grub/core.img

title Slackware 13.37 64-bit (sda6)
    root (hd0,5)
        chainloader +1
        boot
   
title Open SUSE 11.4 64-bit (sda8)
    root (hd0,7)
        configfile /boot/grub/menu.lst

title Windows Example
    rootnoverify (hd0,0)
    chainloader +1
#####

The first set of options in Listing 1 control how GRUB operates. For GRUB, these are called menu commands, and they must appear before other commands. The remaining sections give per-image options for the operating systems that you want to allow GRUB to boot. "Title" is considered a menu command. Each instance of title is followed by one or more general or menu entry commands.

The menu commands that apply to all other sections in Listing 1 are:

#

Any line starting with a # is a comment and is ignored by GRUB. This particular configuration file was originally generated by anaconda, the Red Hat installer. You will probably find comments added to your GRUB configuration file if you install GRUB when you install Linux. The comments often serve as an aid to the system upgrade program so that you can keep your GRUB configuration current with upgraded kernels. Pay attention to any markers that are left for this purpose if you edit the configuration yourself.

default
Specifies which system to load if the user does not make a choice within a timeout. In Listing 1, default=0 means to load the first entry. Remember that GRUB counts from 0 rather than 1. If not specified, then the default is to boot the first entry, entry number 0.

timeout
Specifies a timeout in seconds before booting the default entry. Note that LILO uses tenths of a second for timeouts, while GRUB uses whole seconds.

splashimage
Specifies the background, or splash, image to be displayed with the boot menu. GRUB Legacy refers to the first hard drive as (hd0) and the first partition on that drive as (hd0,0), so the specification of splashimage=(hd0,0)/boot/grub/splash.xpm.gz means to use the file /boot/grub/splash.xpm.gz located on partition 1 of the first hard drive. Remember to count from 0. The image is an XPM file compressed with gzip. Support for splashimage is a patch that might or might not be included in your distribution.

password
Specifies a password that you must enter before you can unlock the menu and either edit a configuration line or enter GRUB commands. The password can be in clear text. GRUB also permits passwords to be stored as an MD5 digest, as in the commented out example in Listing 1. This is somewhat more secure, and most administrators set a password. Without a password, you have complete access to the GRUB command line.
Listing 1 shows a CentOS kernel, /boot/vmlinuz-2.6.32-504.23.4.el6.x86_64, on /dev/sda11 (hd0,10), plus several systems that are configured to chain load. Listing 1 also has examples of loading GRUB 2 via /boot/grub2/i386-pc/core.img and an example of a typical Windows XP chain loading entry, although this system does not actually have Windows installed. The commands used in these sections are:

title
Is a descriptive title that is shown as the menu item when Grub boots. You use the arrow keys to move up and down through the title list and then press Enter to select a particular entry.

root
Specifies the partition that will be booted. As with splashimage, remember that counting starts at 0, so the first Red Hat system that is specified as root (hd0,6) is actually on partition 7 of the first hard drive (/dev/hda7 in this case), while the first Ubuntu system, which is specified as root (hd1,10), is on the second hard drive (/dev/hdb11). GRUB attempts to mount this partition to check it and provide values to the booted operating system in some cases.

kernel
Specifies the kernel image to be loaded and any required kernel parameters. A kernel value like /boot/grub2/i386-pc/core.img usually means loading a GRUB 2 boot loader from the named root partition.

initrd
Is the name of the initial RAM disk, which contains modules needed by the kernel before your file systems are mounted.

savedefault
Is not used in this example. If the menu command default=saved is specified and the savedefault command is specified for an operating system, then booting that operating system causes it to become the default until another operating system with savedefault specified is booted. In Listing 1, the specification of default=0 overrides any saved default.

boot
Is an optional parameter that instructs GRUB to boot the selected operating system. This is the default action when all commands for a selection have been processed.

lock
Is not used in Listing 1. This does not boot the specified entry until a password is entered. If you use this, then you should also specify a password in the initial options; otherwise, a user can edit out your lock option and boot the system or add "single" to one of the other entries. It is possible to specify a different password for individual entries if you want.

rootnoverify
Is similar to root, except that GRUB does not attempt to mount the file system or verify its parameters. This is usually used for file systems such as NTFS that are not supported by GRUB. You might also use this if you want GRUB to load the master boot record on a hard drive (for example, to access a different configuration file or to reload your previous boot loader).

chainloader
Specifies that another file will be loaded as a stage 1 file. The value "+1" is equivalent to 0+1, which means to load one sector starting at sector 0; that is, load the first sector from the device specified by root or rootnoverify.

configfile
Specifies that the running copy of GRUB replaces its configuration file with one loaded from the target location. For this to work, it is advisable that the version of GRUB that is loading the new configfile is as current as the version that built it.

You now have some idea of what you might find in a typical /boot/grub/grub.conf (or /boot/grub/menu.lst) file. There are many other GRUB commands to provide extensive control over the boot process as well as help with installing GRUB and other tasks. You can learn more about these in the GRUB manual, which should be available on your system through the command info grub.

Before you learn how to deal with such a large GRUB configuration file, let's drop back to a smaller and simpler example. I use the file that CentOS 6 built for me when I installed it on /dev/sda11. This is shown in Listing 2. Again, we have used a backslash (\) to show where we broke long kernel lines for publication.

Listing 2. Basic GRUB configuration built by CentOS 6

# grub.conf generated by anaconda
#
# You do not have to rerun grub after making changes to this file
# NOTICE:  You do not have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /, eg.
#          root (hd0,10)
#          kernel /boot/vmlinuz-version ro root=/dev/sdd11
#          initrd /boot/initrd-[generic-]version.img
#boot=/dev/sdd11
default=0
timeout=5
splashimage=(hd0,10)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-504.23.4.el6.x86_64)
    root (hd0,10)
    kernel /boot/vmlinuz-2.6.32-504.23.4.el6.x86_64 ro \
           root=UUID=2f60a3b4-ef6c-4d4c-9ef4-50d7f75124a2 rd_NO_LUKS rd_NO_LVM \
           LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M \
           KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
    initrd /boot/initramfs-2.6.32-504.23.4.el6.x86_64.img
title CentOS 6 (2.6.32-504.el6.x86_64)
    root (hd0,10)
    kernel /boot/vmlinuz-2.6.32-504.el6.x86_64 ro \
           root=UUID=2f60a3b4-ef6c-4d4c-9ef4-50d7f75124a2 rd_NO_LUKS rd_NO_LVM \
           LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M \
           KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
    initrd /boot/initramfs-2.6.32-504.el6.x86_64.img
title Other
    rootnoverify (hd0,0)
    chainloader +1

Notice the command hiddenmenu that you did not see earlier. This causes GRUB to not display a menu, but rather boot the default entry as soon as the timeout expires. In our case this means the first entry (default=0) will be booted in 5 seconds (timeout=5). If you press Enter during this time, the menu will be displayed.

Once you have a GRUB configuration file, you need to install it, or preferably test it. I'll show you how to do the install first and then show you how to test it using a floppy drive (if you still have one) or a CD.

I'll install GRUB in the partition boot record of the partition containing my CentOS distribution. I use the grub-install command and specify the device where the 512-byte stage1 boot loader should go. In my example, that's /dev/sda11 or (hd0,10) using GRUB notation. See Listing 3. You need to have root authority to write the partition boot record. If you have added or deleted devices you might have to remove your /boot/grub/device.map file and allow grub-install to rebuild is as shown in our example. This won't happen often, but if grub-install throws some odd error that you don’t understand, you might find deleting the device.map file helpful.

Listing 3. Install GRUB Legacy in a partition boot record

[root@attic4-cent ~]# rm /boot/grub/device.map
rm: remove regular file `/boot/grub/device.map'? y
[root@attic4-cent ~]# grub-install /dev/sda11
Probing devices to guess BIOS drives. This might take a long time.
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

(fd0)   /dev/fd0
(hd0)   /dev/sda
(hd1)   /dev/sdb
(hd2)   /dev/sdc
(hd3)   /dev/sdd

As you already learned the standard DOS MBR can't boot a logical partition, so you'll need something else to get this system booted. One option would be to install GRUB in the MBR by doing grub-install /dev/sda which would also install GRUB in the MBR of our disk (/dev/sda). I'll also show you how to do it with GRUB 2 in a moment, but before you commit to either approach step, you might want to test out your setup using a GRUB boot CD.

Building a bootable GRUB rescue CD

Before you reboot your shiny new system, it might be a good idea to build a bootable GRUB CD. First, you prepare a CD image on your hard drive. You need a temporary directory, say grubcd, with subdirectories boot and boot/grub. You then need to copy the stage2_eltorito file from your GRUB distribution files to the grub subdirectory that you just created. Then, use genisoimage to create a bootable .iso image file that you can burn to CD with your favorite burning tool. Listing 4 shows how to create the CD image as grubcd.iso. You do not need root authority to do this. Our stage2_eltorito is in /usr/share/grub/x86_64-redhat. This location might be different on other systems, particularly a 32-bit system. Or, you might find it under /usr/lib/grub. You might be able to locate it using the locate command, also illustrated in Listing 4.

Listing 4. Creating a GRUB bootable CD image

[ian@attic4-cent ~]$ mkdir mkdir -p grubcd/boot/grub
[ian@attic4-cent ~]$ ls /usr/share/grub/
x86_64-redhat
[ian@attic4-cent ~]$ ls /usr/share/grub/x86_64-redhat/stage2_eltorito
/usr/share/grub/x86_64-redhat/stage2_eltorito
[ian@attic4-cent ~]$ locate stage2_eltorito
/usr/share/grub/x86_64-redhat/stage2_eltorito
[ian@attic4-cent ~]$ cp /usr/share/grub/x86_64-redhat/stage2_eltorito grubcd/boot/grub
[ian@attic4-cent ~]$ genisoimage -R -b boot/grub/stage2_eltorito -no-emul-boot \
> -boot-load-size 4 -boot-info-table -o grubcd.iso grubcd
I: -input-charset not specified, using utf-8 (detected in locale settings)
Size of boot image is 4 sectors -> No emulation
Total translation table size: 2048
Total rockridge attributes bytes: 760
Total directory bytes: 4576
Path table size(bytes): 34
Max brk space used 22000
241 extents written (0 MB)

You can boot this CD in an arbitrary PC; it does not have to be one with a Linux system on it. If you boot the CD, it will load the GRUB shell from the CD. When you boot, you get a GRUB boot prompt. Press the tab key or use the help command to see a list of commands available to you. Try help commandname to get help on the command called commandname.

One last thing before you reboot with the CD: You can practice some of the GRUB commands that are available in the GRUB shell from your Linux command line. Listing 5 illustrates the grub command and some of the commands available, including the ability to display the menu and see that it is what you want. Some commands, such as find, require root authority, so my example uses that. Also note that when you attempt to load the first config entry by pressing Enter, GRUB crashes with a segmentation fault. Remember that you can practice some of the GRUB shell commands from the Bash command line, but not all. Up and down arrow keys might not work either. Again, a long kernel line is split using \.

Listing 5. The GRUB command line

[root@attic4-cent ~]# grub
Probing devices to guess BIOS drives. This might take a long time.


    GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]
grub> help rootnoverify
help rootnoverify
rootnoverify: rootnoverify [DEVICE [HDBIAS]]
    Similar to `root', but don't attempt to mount the partition. This
    is useful for when an OS is outside of the area of the disk that
    GRUB can read, but setting the correct root device is still
    desired. The items mentioned in `root' which derived
    from attempting the mount will NOT work correctly.
grub> find /boot/grub/menu.lst
find /boot/grub/menu.lst
 (hd0,0)
 (hd0,7)
 (hd0,10)
grub> configfile (hd0,10)/boot/grub/menu.lst
configfile (hd0,10)/boot/grub/menu.lst

Press any key to enter the menu


    GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

-------------------------------------------------------------------
 0: CentOS (2.6.32-504.23.4.el6.x86_64)
 1: CentOS 6 (2.6.32-504.el6.x86_64)
 2: Other
-------------------------------------------------------------------

      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, 'e' to edit the
      commands before booting, 'a' to modify the kernel arguments
      before booting, or 'c' for a command-line.

The selected entry is 0     Highlighted entry is 0:

  Booting 'CentOS (2.6.32-504.23.4.el6.x86_64)'

root (hd0,10)
 Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz-2.6.32-504.23.4.el6.x86_64 ro \
root=UUID=2f60a3b4-ef6c-4d4c-9ef4-50d7f75124a2 rd_NO_LUKS rd_NO_LVM \
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=128M  \
KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
   [Linux-bzImage, setup=0x3400, size=0x3f2790]
Segmentation fault (core dumped)

In this example, there are GRUB configuration files on three different partitions on the first hard drive, including the one built for CentOS on (hd0,10) or /dev/sda11. Listing 5 loads the GRUB menu from (hd0,10) using the configfile command.

You can explore these grub commands in the GRUB manual. Try typing info grub in a Linux terminal window to open the manual.

If you still have a floppy disk, you can install GRUB on a floppy using a command such as
grub-install /dev/fd0
, where /dev/fd0 corresponds to your floppy drive. You should unmount the floppy before installing GRUB on it.

Booting with GRUB legacy

Now you are ready to reboot your system using the GRUB CD that you just built. If your BIOS is not set up to boot automatically from a CD or DVD if present, then you might need to press some system-specific key (F8 on my BIOS) to choose a boot device other than your hard drive. The CD boots to a GRUB prompt as shown in Figure 1.

Figure 1. Booting your GRUB CD

In this example, I used the find command to find GRUB config files called menu.lst and found 3, including my CentOS GRUB configuration file on device (hd0,10) or /dev/sda11. I then used the root command to set )hd0,10) as the root for further file operations. I installed GRUB in the partition boot record of (hd0,10), So I use the chainloader command to tell grub to boot whatever boot loader is in the first sector of (hd0,10). Finally I use the boot command to boot this new loader (GRUB again in our case). The result is shown in Figure 2

Figure 2. The CentOS Grub menu

In this case press Enter to see the menu. Otherwise the hiddenmenu option simply displays the line being booted and a countdown timer.

Editing in the GRUB shell

Now, I show you how to use the GRUB shell to edit the configuration. For this purpose, you will boot in single user mode, but you can change any of the lines or even add or delete whole configuration lines if necessary. For example, you can add a complete root line if you had forgotten it. You press e to edit the configuration, then use the down arrow to highlight the kernel line. The result is shown in Figure 3.

Figure 3. Editing the kernel line

Press e again and then type the word single at the end of the line, as shown in Figure 4.

Figure 4. Editing the kernel line

Finally, press Enter to return to the screen you saw in Figure 2, then b to boot CentOS into single user mode. When the system boots, you will have a root prompt. You can use this mode to make emergency repairs to a system that won't boot normally.

At this point, you can repeat the previous process to boot into normal graphical mode or whatever mode you had set up your system to boot to. If you wanted GRUB to control all booting on the system, you would now do 

grub-install /dev/sda

to install GRUB in the MBR of /dev/sda. You'll see other ways to manage your booting as you proceed through this tutorial.

Continue reading

LPI Certifications Guide: Overview And Career Paths

The Linux Professional Institute offers several vendor-neutral Linux certifications designed for entry- to senior-level professionals. See what the LPI program has to offer and how you can build your career in this in-demand niche.

The Linux Professional Institute (LPI) is a nonprofit organization based in Toronto, Canada, that promotes the use of Linux, open source and free software. One way in which the organization furthers its mission is to provide vendor-neutral Linux certifications to IT professionals around the globe. With "more than 500,000 exams delivered" to candidates and 400 training partners, LPI stakes a claim as the largest vendor-neutral Linux certification body in the world.

With input from private industry, academia and individuals, Linux experts provide input on exam questions to ensure that they're rigorous, accurate and apply to any standard Linux system.

LPI Certification Program Overview

The LPI certification program is simple and includes three certifications that build on one another:

◈ LPIC-1: Linux Administrator — Entry-level certification that recognizes individuals who can install and configure a workstation running Linux, maintain the system from the command line and configure a basic network

◈ LPIC-2: Linux Engineer — Mid-level certification designed for professionals who administer small- to medium-sized mixed networks

◈ LPIC-3: Linux Enterprise Professional — Senior-level certification that identifies Linux professionals who plan, conceptualize, design, implement and troubleshoot Linux installations in enterprise environments

Each LPIC certification requires you to pass one or two multiple-choice exams, each of which costs $188 and all of which are administered by Pearson VUE. All LPIC certs are valid for five years.

Those just beginning their Linux careers may find the Linux Essentials certification a good place to start before jumping into the more advanced LPIC certifications. Linux Essentials covers basic skills, such as command-line editing and the Linux operating system (processes, programs and components).

LPIC-1: Linux Administrator

In the LPI certification program, the LPIC-1: Linux Administrator is considered a junior-level Linux certification that requires you to pass two exams: 101-400 and 102-400. There are no prerequisites.

◈ The LPIC-1 101-400 exam covers system architecture, the nuts and bolts of Linux installation, basic package management, GNU and Unix commands, devices and file systems.

◈ The LPIC-1 102-400 exam tests you on customizing the shell environment, writing and running scripts, and managing databases and running SQL commands. You must also know how to configure settings for user interfaces and the desktop, perform administrative tasks and manage system services, create network connections and secure Linux systems.

LPI Partner Certifications

LPI and CompTIA have a 2-in-1 Linux certification program, which lets candidates acquire LPIC-1 certification after achieving the CompTIA Linux+ Powered by LPI. (This program included the SUSE Certified Linux Administrator (CLA) certification – a 3-in-1 offer – but this third leg of the tripod expired on August 30, 2016.)

You first need to get an LPI ID by registering at the LPI website. Next, take the CompTIA Linux+ Powered by LPI exams and indicate that you want your exam scores sent to LPI upon successful completion. You'll soon receive notification of your LPIC-1 certification from LPI.

LPIC-2: Linux Engineer

The LPIC-2: Linux Engineer is an advanced-level Linux credential that requires a current LPIC-1 certification as a prerequisite. To achieve the LPIC-2, you must pass exams 201-450 and 202-450:

◈ The LPIC-2 201-450 exam dives into capacity planning, manipulating the Linux kernel, configuring system startup services and boot loaders, and configuring and maintaining file systems and devices. You will also be tested on advanced storage device administration, networking configuration and system maintenance.

◈ The LPIC-2 202-450 exam focuses mainly on networking-related topics, such as Domain Name Server (DNS), web services, file sharing, network client management, e-mail services and router configuration. The exam also covers security topics like secure shell (SSH), port testing and configuring OpenVPN.

LPIC-3: Linux Enterprise Professional

The LPIC-3: Linux Enterprise Professional is the pinnacle of the LPI certification program and is considered expert level. Therefore, you should have several years of hands-on experience installing, managing, integrating, networking and troubleshooting Linux in an enterprise environment.

To earn your LPIC-3 credential, you must achieve LPIC-2 certification as a prerequisite and pass one of these 300-series exams:

◈ Mixed Environment (exam 300-100): This exam focuses on OpenLDAP configuration, OpenLDAP as an authentication backend, and highly advanced levels of Samba administration, among other topics.

◈ Security (exam 303): To pursue this exam, be sure you're well versed in access controls and cryptography, as well as application, operations and network security.

◈ Virtualization and High Availability (exam 304-200): This exam covers virtualization (of course), along with load balancing, cluster management and cluster storage.

You can take the LPIC-2 exams and an LPIC-3 exam in any order. That means you can knock out the LPIC-3 exam of your choice, then circle back and sit for the LPIC-2 exams.


LPI Linux Essentials

The Linux Essentials Professional Development Certificate (PDC) is LPI's entry-level certification. It doesn't serve as a prerequisite for the LPIC-1, but it's a great way for people who are relatively new to Linux to begin validating their skills. The certificate is beneficial for many different industry professionals, from developers, to administrators and engineers, and data analysts. By studying for and taking the exam, you also gain certification prep experience, which will be a benefit if you choose to pursue other LPIC certs.

Achieving the Linux Essentials PDC indicates you are familiar with open source applications versus closed source, know the basics of the Linux operating system, and can run commands on the command line, manage files, perform backup and restore operations and write basic scripts.

Passing a single exam (LPI 010-150) is required to earn the certificate, which doesn't expire. By today's standards, the fee is quite affordable, coming in at a mere $110, making it attractive to those interested in exploring Linux certifications.

Related Jobs and Training Resources

Considering the focus of LPI Linux certifications, the vast majority of related positions are along the lines of system administrators, network administrators, system engineers and technical support specialist. But you will occasionally stumble across job listings seeking LPI certification for cloud administrator, cybersecurity engineer and technical education specialist.

Some positions specifically look for Linux engineers with programming skills. For example, one employer was looking for Puppet/Linux engineers to streamline Puppet workflow and assist with implementation and post-implementation support. Another position called for an operating systems programmer who can design, develop and implement new system tools, and write scripts in BASH and Python or other administrative scripting languages.

Linux certification courses are available through many different channels, although LPI recommends that you take courses through one of its approved training partners. The LPI Certification Marketplace is an online store chock-full of Linux references, certification study guides, practice exams, courseware, video training and practice labs.

Continue reading

LPIC-1: The Bash Shell and Commands and sequences

1. The Bash Shell

The bash shell is one of several shells available for Linux. It is also called the Bourne-again shell, after Stephen Bourne, the creator of an earlier shell (/bin/sh). Bash is substantially compatible with sh, but it provides many improvements in both function and programming capability. It incorporates features from the Korn shell (ksh) and C shell (csh), and is intended to be a POSIX-compliant shell.

Unless otherwise noted, the examples in this tutorial use Fedora 22, with a 4.0.4 kernel. Your results on other systems may differ.

Prerequisites

To get the most from the tutorials in this series, you should have a basic knowledge of Linux and a working Linux system on which you can practice the commands covered in this tutorial. Sometimes different versions of a program will format output differently, so your results may not always look exactly like the listings and figures shown here.

Before we delve deeper into bash, recall that a shell is a program that accepts and executes commands. It also supports programming constructs, allowing complex commands to be built from smaller parts. These complex commands, or scripts, can be saved as files to become new commands in their own right. Indeed, many commands on a typical Linux system are scripts.

Shells have some builtin commands, such as cd, break, and exec. Other commands are external.

Shells also use three standard I/O streams:

◈ stdin is the standard input stream, which provides input to commands.
◈ stdout is the standard output stream, which displays output from commands.
◈ stderr is the standard error stream, which displays error output from commands.

Input streams provide input to programs, usually from terminal keystrokes. Output streams print text characters, usually to the terminal. The terminal was originally an ASCII typewriter or display terminal, but it is now more often a window on a graphical desktop.

If you are using a Linux system without a graphical desktop, or if you open a terminal window on a graphical desktop, you will be greeted by a prompt, perhaps like one of the three shown in Listing 1.

Listing 1. Some typical user prompts

[ian@atticf20 ~]$
jenni@atticf20:data
$

Notice that these three prompts are all from my test system atticf20, but for different users. The first two are bash prompts and both show the logged in user, system name and current working directory. The third is the default prompt on my system for a ksh shell. Different distributions and different shells default to different prompts, so don’t panic if your distribution looks different. We'll cover how to change your prompt string in another tutorial in this series.

If you log in as the root user (or superuser), your prompt may look like one of those shown in Listing 2.

Listing 2. Superuser, or root, prompt examples

[root@atticf20 ~]#
atticf20:~#

The root user has considerable power, so use it with caution. When you have root privileges, most prompts include a trailing pound sign (#). Ordinary user privileges are usually delineated by a different character, commonly a dollar sign ($). Your actual prompt may look different than the examples in this tutorial. Your prompt may include your user name, hostname, current directory, date, or time that the prompt was printed, and so on.

Note: Some systems, such as Debian and Debian-based distributions such as Ubuntu, do not allow root login and require all privileged (root) commands to be executed using the sudo command. In this case, your prompt will not change, but you will know that you have to use sudo to execute commands that an ordinary user does not have the power to execute.

These tutorials include code examples that are cut and pasted from real Linux systems using the default prompts for those systems. Our root prompts have a trailing #, so you can distinguish them from ordinary user prompts, which have a trailing $. This convention is consistent with many books on the subject. If something doesn't appear to work for you, check the prompt in the example.

2. Commands and sequences

So now that you have a prompt, let's look at what you can do with it. The shell's main function is to interpret your commands so you can interact with your Linux system. On Linux (and UNIX®) systems, commands have a command name, and then options and parameters. Some commands have neither options nor parameters, and some have one but not the other.

If a line contains a # character, then all remaining characters on the line are ignored. So a # character may indicate a comment as well as a root prompt. Which it is should be evident from the context.

Echo

The echo command prints (or echos) its arguments to the terminal as shown in Listing 3.

Listing 3. Echo examples

[ian@atticf20 ~]$ echo Word
Word
[ian@atticf20 ~]$ echo A phrase
A phrase
[ian@atticf20 ~]$ echo Where     are   my   spaces?
Where are my spaces?
[ian@atticf20 ~]$ echo "Here     are   my   spaces." # plus comment
Here     are   my   spaces.

In the third example of Listing 3, all the extra spaces were compressed down to single spaces in the output. To avoid this, you need to quote strings, using either double quotes (") or single quotes ('). Bash uses white space, such as blanks, tabs, and new line characters, to separate your input line into tokens, which are then passed to your command. Quoting strings preserves additional white space and makes the whole string a single token. In the example above, each token after the command name is a parameter, so we have respectively 1, 2, 4, and 1 parameters.

The echo command has a couple of options. Normally, echo will append a trailing new line character to the output. Use the -n option to suppress this. Use the -e option to enable certain backslash escaped characters to have special meaning. Some of these are shown in Table 1.

Table 1. Echo and escaped characters

Escape sequence	Function
\a	Alert (bell)
\b	Backspace
\c	Suppress trailing newline (same function as -n option)
\f	Form feed (clear the screen on a video display)
\n	New line
\r	Carriage return
\t	Horizontal tab

Escapes and line continuation

There is a small problem with using backslashes in bash. When the backslash character (\) is not quoted, it serves as an escape to signal bash itself to preserve the literal meaning of the following character. This is necessary for special shell metacharacters, which we'll cover in a moment. There is one exception to this rule: a backslash followed by a newline causes bash to swallow both characters and treat the sequence as a line continuation request. This can be handy to break long lines, particularly in shell scripts.

For the sequences described above to be properly handled by the echo command or one of the many other commands that use similarly escaped control characters, you must include the escape sequences in quotes, or as part of a quoted string, unless you use a second backslash to have the shell preserve one for the command. Listing 4 shows some examples of the various uses of \.

Listing 4. More echo examples

[ian@atticf20 ~]$ echo -e "No new line\c"

No new line[ian@atticf20 ~]$ echo "A line with a typed

> return"

A line with a typed

return

[ian@atticf20 ~]$ echo -e "A line with an escaped\nreturn"

A line with an escaped

return

[ian@atticf20 ~]$ echo "A line with an escaped\nreturn but no -e option"

A line with an escaped\nreturn but no -e option

[ian@atticf20 ~]$ echo -e Doubly escaped\\n\\tmetacharacters

Doubly escaped

    metacharacters

[ian@atticf20 ~]$ echo Backslash \

> followed by newline \

> serves as line continuation.

Backslash followed by newline serves as line continuation.

Note that bash displays a special prompt (>) when you type a line with unmatched quotes. Your input string continues onto a second line and includes the new line character.

Bash shell metacharacters and control operators

Bash has several metacharacters, which when not quoted, also serve to divide input into words. Besides a blank, these are:

◈ |

◈ &

◈ ;

◈ (

◈ )

◈ <

◈ >

We will discuss some of these in more detail in other parts of this tutorial. For now, note that if you want to include a metacharacter as part of your text, it must be either quoted or escaped using a backslash (\) as shown in Listing 4.

The new line and certain metacharacters or pairs of metacharacters also serve as control operators. These are:

◈ ||

◈ &&

◈ &

◈ ;

◈ ;;

◈ |

◈ (

◈ )

Some of these control operators allow you to create sequences or lists of commands.

The simplest command sequence is just two commands separated by a semicolon (;). Each command is executed in sequence. In any programmable environment, commands return an indication of success or failure; Linux commands usually return a zero value for success and a non-zero value in the event of failure. You can introduce some conditional processing into your list using the && and || control operators. If you separate two commands with the control operator &&, then the second is executed if, and only if, the first returns an exit value of zero. If you separate the commands with ||, then the second one is executed only if the first one returns a non-zero exit code. Listing 5 shows some command sequences using the echo command. These aren't very exciting since echo returns 0, but you will see more examples later when we have a few more commands to use.

Listing 5. Command sequences

[ian@atticf20 ~]$ echo line 1;echo line 2; echo line 3

line 1

line 2

line 3

[ian@atticf20 ~]$ echo line 1&&echo line 2&&echo line 3

line 1

line 2

line 3

[ian@atticf20 ~]$ echo line 1||echo line 2; echo line 3

line 1

line 3

Exit

You can terminate a shell using the exit command. You may optionally give an exit code as a parameter. If you are running your shell in a terminal window on a graphical desktop, your window will close. Similarly, if you have connected to a remote system using ssh or telnet (for example), your connection will end. In the bash shell, you can also hold the Ctrl key and press the d key to exit.

Let's look at another control operator. If you enclose a command or a command list in parentheses, then the command or sequence is executed in a sub shell, so the exit command exits the sub shell rather than exiting the shell you are working in. Listing 6 shows a simple example in conjunction with && and || and two different exit codes.

Listing 6. Subshells and sequences

[ian@atticf20 ~]$ (echo In subshell; exit 0) && echo OK || echo Bad exit

In subshell

OK

[ian@atticf20 ~]$  (echo In subshell; exit 4) && echo OK || echo Bad exit

In subshell

Bad exit

Continue reading

Archiving Files from the Linux Command Line

I am sure if you have not already come across tar and gz files within Linux it will not be long before you become acquainted with each other. Of course, these file make up just a part of what we mean when we talk about archiving files from the Linux command line. Spending just a little time with Linux you will soon come across TAR files of some type, be they TAR, TGZ , TAR.GZ or some other format. The TAR archive, or Tape Archive, file is an oldie but a goodie having passed across from UNIX into Linux. The archive itself is a single file that can represent many files.

Compress or Not Compressed

These tar files do not have to compressed but they often are. However, even if it is not zipped up it will often consume less disk space than the files stored individually. Consider the following screenshot where we first look at the size of the directory and then create a tar archive file, uncompressed, and view the size of the tar file: it is smaller:

The output shows the directory to be 52K and the tar archive to be just 20K and no compression has been used. This relates to the way the filesystem uses blocks of disk space. Each new file has to start with its own new block. Often the block size is 4KB; this means for each 1KB file, for instance, will consume 4KB of disk space. If we combine these files into one file, a TAR file, then less space can be used to store the same amount of data.

Using Tar

The three main options we have with tar are:

◈ -c : create a new archive

◈ -x : extract an archive

◈ -t : verify or test and archive

Creating a TAR file

From the previous graphic we can see the creation of the TAR file. We normally include .tar as the last four characters of the file name so we easily identify this type of file. The option -f specifies the file name and must be followed by the same.

tar -cf labs.tar labs

In this example we archive the labs directory within current directory. The target file for the archive is: labs.tar, also within the current directory. The source directory labs remains intact and unaffected by the operation other than updating the last accessed time attribute of each file included in the archive. In order to backup a file it has to be read, hence the last accessed time of each file we archive will be updated to the time of the backup.

Viewing a TAR file

Once we have created the file we can verify the file contents with the -t option:

tar -tf labs.tar

Incidentally, a large TAR file may be read directly with the command less , this allows for the file contents to be paged through without the explicitly use of tar and less together.

less labs.tar

Extracting a TAR file

To extract the complete archive the command is simple and makes use of the -x option:

tar -xf labs.tar

The files will expand within the current directory unless the option -P is used both when the archive is created and expanded, in which case the files are expanded to the full path of the original files. If there are concerns that you may overwrite files then a couple of options exist that may help

◈ -k : prevents existing files being overwritten

◈ –keep-newer-files : will not overwrite if the target file is newer than the archive file

Should we want to extract only a single file or certain files from the archive we could use code similar to the following:

tar -xf labs.tar labs/file.sh

This would extract just the single file from the archive, we can use the -t option if we need to confirm the path too the file in the archive.

Compressing the archives

These archives can be compressed and uncompressed within the same TAR process. Using the options:

◈ -z : the use gzip for compression and gunzip for decompressing the file

◈ -j | uses bzip2 for compression and bunzip2 for decompressing.

tar -czf labs labs.tgz

tar -xzf labs.tgz

The first command creates the zipped archive and the second extracts the archive. The option -z must be used in both cases and also with -t for viewing. If we had used the option -j in the first instance then -j must be used to access the archive in future. A good naming standard is useful here so we know how to open or view the file; commonly these endings are used for file names:

◈ .tar : indicates a uncompressed file

◈ .tar.gz or .tgz : indicates a file compressed with gzip

◈ .tar.bz2 or .tbz2 | indicates a file where bzip2 was used to compress the archive

Ultimately, though, we can use the command file (/usr/bin/file) to identify the type of archive we have:

file labs.tar

This will identify the compression used, if any and confirm that the file is a TAR archive.

Continue reading

Basic Security and Identifying User Types

Weight: 2
Description: Various types of users on a Linux system.
Key Knowledge Areas:
Root and Standard Users.
System users.

The following is a partial list of the used files, terms and utilities:

/etc/passwd, /etc/group, id, who, w, sudo.
Nice to know: su.

The predominate administrative  account on Linux systems is the root account with the user ID (UID) of 0. To manage the Linux system you will need access to this account either directly or via sudo. As we discuss this further you will see that sudo is the preferred method of delegated administration as, in this way, the administrative users do not need access to the root password. Whereas logging into the system directly as root or using the substitute user command, su, knowledge of the root password is required.

Each Linux host must, at the minimum, have a local root account defined in the /etc/passwd file. The local user account store is /etc/passwd, passwords, on the other hand passwords are usually held in the /etc/shadow file. Each user, including the root user will need both a UID and a GID group ID. Users must belong to a minimum of one group; some systems such a Red Hat run a private group system where users belong to their own private groups, other systems including SUSE have a public group system where users belong to a shared groups : users. Local groups are recorded in the file /etc/group With root privileges users can be created and managed with the command useradd and groups with groupadd. Even though you would be expected to use the tools provided to manage users and groups there is nothing stopping  changes being made directly to the appropriate file.

/etc/passwd file

These are text files writable by the root account. The seven fields of the passwd file are delimited with a colon and are described as:

1. user name

2. password or a single x denotes the password is stored in /etc/shadow

3. UID

4. GID

5. Comments

6. Home directory path

7. Default user shell, (command line environment)

To display information about a user account, whether your own or another account the command id , (/usr/bin/id). By default the command will display the user name uid and gid and secondarygroups, but more specific information can be honed in upon with additional switches. The command finger, (/usr/bin/finger), can be used to display further information about users account information. The following screenshots first show the out from id then finger:

Output from id

Output from /usr/bin/finger

To return information on currently logged in users, perhaps if you need to bring a server down for maintenance, the use of the commands who (/usr/bin/who) or w (/usr/bin/w) are useful. With typical Linux humor the shorter command w, provides the more verbose output.

Output from who and the more verbose w

Controlling access to sudo, (/usr/bin/sudo) ,and what you are allowed to do with it, is a task that the root user will achieve through editing the file /etc/sudoers. So that mistakes are less likely to occur, root is encouraged to edit the file using visudo (/usr/sbin/visudo ) ; as this program closes the file is syntax checked helping prevent errors. Users can be delegated rights to run certain commands though the /etc/sudoers file; this way knowledge of the root password is not required when administering the host but the commands must be prefaced with sudo. On some systems, such as Ubuntu, the root account password is not shown during installation and all tasks are managed thorough sudo. This, is many ways is a correct administration model avoiding directly accessing the root account.

When using sudo, the command is prefaced with sudo

For ease of access to many administration commands consider adding the /sbin and usr/sbin directories into the PATH variable of your delegated administrators. In this way the previous command in the screenshot could be reduced to : sudo useradd -m bob .

When required if you have access to the root password you may use the su (/bin/su) command to substitute your current user id with the root user ID. (note the command su can be used to access any account you know the password to , not just root). It is possible to disallow remote access via SSH to root, you can still logon on as a standard account and su to root as required.

Continue reading