Linux gzip: How to work with compressed files

If you work much with Unix and Linux systems you'll eventually run into the terrific file compression utilities, gzip and gunzip. As their names imply, the first command creates compressed files (by gzip'ing them), and the second command unzip's those files.

In this post I take a quick look at the gzip and gunzip file compression utilities, along with their companion tools you may not have known about: zcat, zgrep, and zmore.

The Unix/Linux gzip command

You can compress a file with the Unix/Linux gzip command. For instance, if I run an ls -l command on an uncompressed Apache access log file named access.log, I get this output:

-rw-r--r--   1 al  al  22733255 Aug 12  2008 access.log

Note that the size of this file is 22,733,255 bytes. Now, if we compress the file using gzip, like this:

gzip access.log

we end up creating a new, compressed file named access.log.gz. Here's what that file looks like:

-rw-r--r--   1 al  al  2009249 Aug 12  2008 access.log.gz

Notice that the file has been compressed from 22,733,255 bytes down to just 2,009,249 bytes. That's a huge savings in file size, roughly 10 to 1(!).

There's one important thing to note about gzip: The old file, access.log, has been replaced by this new compressed file, access.log.gz. This might freak you out a little the first time you use this command, but very quickly you get used to it. (If for some reason you don't trust gzip when you first try it, feel free to make a backup copy of your original file.)

The Linux gunzip command

The gunzip ("g unzip") command works just the opposite of gzip, converting a gzip'd file back to its original format. In the following example I'll convert the gzip'd file we just created back to its original format:

gunzip access.log.gz

Running that command restores our original file, as you can see in this output:

-rw-r--r--   1 al  al  22733255 Aug 12  2008 access.log

The Linux file compress utilities (zcat, zmore, zgrep)

I used to think I had to uncompress a gzip'd file to work on it with commands like cat, grep, and more, but at some point I learned there were equivalent gzip versions of these same commands, appropriately named zcat, zgrep, and zmore. So, anything you would normally do on a text file with the first three commands you can do on a gzip'd file with the last three commands.

For instance, instead of using cat to display the entire contents of the file, you use zcat to work on the gzip'd file instead, like this:

zcat access.log.gz

(Of course that output will go on for a long time with roughly 22MB of compressed text.)

You can also scroll through the file one page at a time with zmore:

zmore access.log.gz
And finally, you can grep through the compressed file with zgrep:

zgrep '/java/index.html' access.log.gz

There are also two other commands, zcmp and zdiff, that let you compare compressed files, but I personally haven't had the need for them. However, as you can imagine, they work like this:

zmp file1.gz file2.gz

or

zdiff file1.gz file2.gz

Linux gzip / compress summary

As a quick summary, just remember that you don't have to uncompress files to work on them, you can use the following z-utilities to work on the compressed files instead:

◉ zcat
◉ zmore
◉ zgrep
◉ zcmp
◉ zdiff

Continue reading

Good (Free) Training is Not Hard to Come By

So now that we are indoors most of the time in keeping with social distancing guidelines, many of us are searching for ways to make this time productive. For whatever reasons you might find yourself in-between jobs or are already working for someone and want a change of career. If your interests lie within the realm of Linux and open source software, read on for some tips on how to move forward in learning more about this field.

One can find a plethora of lessons and tutorials on how to use and administer Linux throughout the internet. However, this article will focus on complete and coherent courses designed for those that are very new to Linux as well as seasoned system administrators.

There are some industry leaders in the field of technology that fully understand the importance of open source software and of Linux in general. In keeping with that understanding they offer some Linux training for free. Here are a few examples:

Cisco

Cisco Systems, one of the companies whose hardware and software powers a large portion of the internet’s backbone has long offered entry-level Linux training through their Cisco Networking Academy. Here are two free courses offered through this platform for those interested in Linux:

◉ Linux Unhatched - This is the “baby steps” introduction Linux course aimed at those that are completely new to Linux and are just starting out. This is an excellent resource for beginners.

◉ NDG Linux Essentials - Another course that originates from Network Development Group. This one lines up with the objectives for the Linux Professional Institute’s Linux Essentials exam. This course (and associated exam) presumes that you have some small amount of Linux knowledge and then expands on that to provide a more solid foundation. Those who complete this course should be fully prepared to sit for the Linux Essentials exam. As with the previous Cisco course offering, you could also register for this course at Network Development Group’s website.

IBM

Another large technology company that understands the importance of open source technology is IBM. Since the early 2000’s, IBM has maintained free training materials for Linux on their website.

Learn Linux 101 - This site offers updated training for those seeking to earn the Linux Professional Institute’s LPIC-1 Linux system administrator’s certification. These pages are free to use and no registration for a course is required. If you already have experience working with Linux in a production environment, then this course will help you bolster those base skills and prepare you for the first of the Linux Professional Institute’s professional-level certification track.

Community Supplied Training

Not to be outdone by the titans of the industry, many individuals have taken it upon themselves to provide their own quality Linux training to the world for free. Here are some examples:

◉ Guy Hummel’s Linux Survival website offers an in-browser shell simulator for you to follow along with a list of lessons listed on the left side of the page. Each group of lessons is broken down into modules with each ending with a quiz to test your newly-gained knowledge. This course introduces students to the basics of using Linux, all without having to install anything on their own systems.

◉ Ravi Saive has constructed a rather thorough series of tutorials at his site TecMint.com. Each group of tutorials are listed by a topic (such as installing Linux, package management, etc.) and are well organized. Beginners to Linux can definitely obtain a lot from this site, but the tutorials are written with the moderately experienced system administrator in mind. Nonetheless, this site is an excellent resource for those looking to learn how to use and maintain a Linux installation. There are a number of ads on the site, but they do not detract from the content too much.

◉ Last, but certainly not least, the Linux Professional Institute has been working on learning materials for those that are looking to obtain an LPI certification. This project contains lesson plans that have been submitted by community members and peer reviewed for clarity and correctness. These lessons are available in a number of languages (and more are planned for the near future) and are designed to assist an exam candidate in their studies. You can find these lessons, as well as information on joining the project, at the Learning Materials website.

Software Development

Perhaps you are more interested in creating software to run on a system? Here is a quick (and by no means exhaustive) list of sites that offer structured lessons in a particular programming language. There are a plethora of other sites available for each of the programming languages listed, these are just meant to provide you with a starting point.

◉ C and C++ - The C programming language has been around since the 1970’s, but its importance is still present even in modern projects. The Linux kernel, which is the core of any Linux distribution, is written in the C language. It is a very low level language meaning that the source code that the programs are written in speak directly to particular instructions that a computer’s processor adheres to. Oftentimes these low level programming languages are cryptic to type and read, but the benefit is in their speed in which these applications execute. A good starting point for someone wishing to learn about C or its slightly-newer object-oriented (a programming paradigm that you could learn more about in these courses) counterpart C++ can be found at C Programming’s tutorial site.

◉ Python - Likely the most popular programming language regardless of computing platform that is used is Python. Python is found in numerous applications that run on a Linux system (as well as Windows and macOS). Python is used for every day simple administration applications up to full-scale data analysis and artificial intelligence research. Python is easier to learn than a low level programming language as its syntax is less cryptic and much easier to read. There is no need to compile a Python program as it uses an interpreter to execute its code on a given system. This makes Python highly portable and able to run on numerous systems without having to rewrite anything for a given processor or operating system. If you are looking to get started in the exciting world of Python development, have a look at Python Programming’s website. If you are just starting out, go straight to the basics page to get going.

◉ HTML, CSS, JavaScript, PHP, SQL - Are you aspiring to become a web developer? A great resource is the W3Schools tutorial site. This site is more structured in a manner that has tutorials than an actual classroom approach. Nonetheless, this site offers a great springboard into the field of web development. This site covers everything from the skeletal structure of a web page in HTML up to advanced sites that use e-commerce methods with databases storing information behind the website. Take note that there are other websites available that provide further details on a given topic, particularly with web security, than what you would find at W3Schools. Nonetheless, this site is a wonderful resource for those that are just getting started.

As you can see, there are numerous locations on the internet that can assist you with your journey to become a qualified systems administrator, a developer, or both. These sites do not require any payment and are completely free to use at your pace. While you may be at home with extra free time on your hands, take a look at these sites and see what piques your interest. Whatever path you choose, practice whatever it is you are learning to become more proficient at it and ask questions at the site’s forums or mailing lists for assistance. As the saying goes: “We are all in this together.” So let us make the most of it.

Source: lpi.org

Continue reading

The Unix/Linux lpstat command

The Linux lpstat command lets you look at the progress of your print request(s). The name "lpstat" stands for "line printer statistics".

lpstat command examples

The "lpstat" command, used with no options, may return no output if there are no print jobs queued.

lpstat

It's generally better off to use an option with "lpstat". The "lpstat" command with the "-t" option gives you total information about the printer status. This option is generally the most useful, but the printout can be lengthy if you have a large number for printers configured.

lpstat -t

To look at Linux printer queue information for a specific printer, use the -p option. Here's an example of looking at the long list of output information (-l option) for a printer named "Sales":

lpstat -p Sales -l

The -r option shows whether the print scheduler is currently on or off:

lpstat -r

Finally, the -u option shows print request status information for a given user. Here's what the command looks like when looking at the printer queue information for a user named "fred":

lpstat -u fred

Continue reading

The Linux lp printing command

The lp command is used to print files on Unix and Linux systems. The name "lp" stands for "line printer". As with most Unix commands there are a fairly large number of options available to enable flexible printing capabilities.

Let's look at some lp printing commands examples.

Linux lp printing command examples

lp /etc/passwd

This command prints the "/etc/passwd" file to the default printer. If you do not use the "-d" option, the file is printed to the default printer destination.

lp -dSales .profile

This command prints the ".profile" file to the printer named "Sales". The -d option specifies the destination.

lp -dSales file1 file2 file3

This command prints the three files "file1", "file2", and "file3" to the printer named "Sales".

lp -i Sales-101 -H hold

This command places the print request Sales-101 on hold.

lp -i Sales-101 -H resume

This command resumes the print request Sales-101. The print request starts printing from page one unless you instruct it otherwise.

lp -i Sales-101 -H resume -P 4-

This command resumes the print request Sales-101, starting with page 4 of the print job. Note that the final hyphen (following the "4") is required.

The "lp" command can also be used as part of a pipeline. For instance, the following command will print the output of the "ps -ef" command to the default printer:

ps -ef | lp

Print the output of the "ps -ef" command to the default printer.

Continue reading

The Unix and Linux ps command

Unix/Linux processes FAQ: Can you share some examples of the Linux ps command? (Or, how do I use theps command?)

The basic Linux ps command

If you run the ps command by itself, it only shows very basic information about the processes you are currently running. For example, if you issue the basic command like this without any arguments:

ps

you'll see output from this command looks something like this:

  PID   TTY         TIME CMD
 4343  ttys000    0:00.35 -bash
 2617   ttys001    0:00.65 -bash
18201  ttys003    0:00.27 -bash

The PID column shows the process-id, the second column shows the TTY (terminal) the process is attached to, the TIME column shows how much CPU time the process has used, and the CMD column shows the command that is running. In this case I can tell (from experience) that I have three bash shells (terminals) running on my current system.

In practice I never run the ps command without any arguments like this, but I wanted to show this to help us get started.

There are many ways to customize the output of the ps command. For instance, I can add the f argument to get "full" information about each process. Used by itself, the f argument shows "full" information about just my processes. As an example, this command:

ps -f

leads to this output:

  UID   PID  PPID   C     STIME TTY           TIME CMD
  501  4343  4342   0   0:00.18 ttys000    0:00.35 -bash
  501  2617  2616     0   0:00.41 ttys001    0:00.66 -bash
  501 18201 18200   0   0:00.15 ttys003    0:00.27 -bash

As you can see this adds a few more columns of output to my ps command, including UID (user-id), PPID (parent process-id), and a couple of other columns I don't really look at.

Showing information about every process

As mentioned, those two ps command examples just show information about your processes. If you're a Linux system administrator, you're typically interested in information about all the processes running on the system. To show every process running on the system, we add the e argument to our previous ps command, like this:

ps -ef

This leads to much more output:

UID      PID  PPID  C STIME TTY     TIME CMD
root         1     0  0 Oct21 ?        00:00:01 init [3]                                           
root         2     1  0 Oct21 ?        00:00:00 [migration/0]
root         3     1  0 Oct21 ?        00:00:00 [ksoftirqd/0]
root         4     1  0 Oct21 ?        00:00:00 [watchdog/0]
root         5     1  0 Oct21 ?        00:00:00 [migration/1]
root         6     1  0 Oct21 ?        00:00:00 [ksoftirqd/1]
root         7     1  0 Oct21 ?        00:00:00 [watchdog/1]
root         8     1  0 Oct21 ?        00:00:00 [events/0]
root         9     1  0 Oct21 ?        00:00:00 [events/1]
root        10     1  0 Oct21 ?        00:00:00 [khelper]
root        11     1  0 Oct21 ?        00:00:00 [kthread]
root        15    11  0 Oct21 ?        00:00:00 [kblockd/0]
root        16    11  0 Oct21 ?        00:00:00 [kblockd/1]
root        17    11  0 Oct21 ?        00:00:00 [kacpid]
root        91    11  0 Oct21 ?        00:00:00 [cqueue/0]
root        92    11  0 Oct21 ?        00:00:00 [cqueue/1]
root        95    11  0 Oct21 ?        00:00:00 [khubd]
root        97    11  0 Oct21 ?        00:00:00 [kseriod]
root       158    11  0 Oct21 ?        00:00:00 [pdflush]

That was actually just the first 20 lines of output from this ps command on my CentOS Linux test system. This command actually generated 99 lines of output, and I cropped it to just show the first 20 lines (using ps -ef | head -20).

As mentioned, that's the older way to list processes on a Unix system (and it may still be preferred on Unix systems like HP-UX, AIX, and Solaris; I don't really know, I just use Linux and Mac OS X these days). I wanted to show these options to you (a) to help you learn about the ps command, and (b) see that there are other ps command options than what most people use on a day to day basis. Given that background, let's take a look at how the ps command is typically used on Linux systems.

How the Linux ps command is typically used

Now that you've seen some ps command arguments and sample output, here's how I run the ps command about 80% of the time:

ps auxwww | more

The first 20 lines of output from this command look like this:

USER   PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2064   620 ?    Ss   Oct21   0:01 init [3]                                           
root         2  0.0  0.0      0     0 ?        S<   Oct21   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Oct21   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   Oct21   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S<   Oct21   0:00 [migration/1]
root         6  0.0  0.0      0     0 ?        SN   Oct21   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S<   Oct21   0:00 [watchdog/1]
root         8  0.0  0.0      0     0 ?        S<   Oct21   0:00 [events/0]
root         9  0.0  0.0      0     0 ?        S<   Oct21   0:00 [events/1]
root        10  0.0  0.0      0     0 ?        S<   Oct21   0:00 [khelper]
root        11  0.0  0.0      0     0 ?        S<   Oct21   0:00 [kthread]
root        15  0.0  0.0      0     0 ?        S<   Oct21   0:00 [kblockd/0]
root        16  0.0  0.0      0     0 ?        S<   Oct21   0:00 [kblockd/1]
root        17  0.0  0.0      0     0 ?        S<   Oct21   0:00 [kacpid]
root        91  0.0  0.0      0     0 ?        S<   Oct21   0:00 [cqueue/0]
root        92  0.0  0.0      0     0 ?        S<   Oct21   0:00 [cqueue/1]
root        95  0.0  0.0      0     0 ?        S<   Oct21   0:00 [khubd]
root        97  0.0  0.0      0     0 ?        S<   Oct21   0:00 [kseriod]
root       158  0.0  0.0      0     0 ?        S    Oct21   0:00 [pdflush]

As you can see, this output is similar to the earlier output, but the columns are different. Before I talk about these ps command arguments, let me show a few more examples. Here's how I look at all httpd processes running on my Linux system:

$ ps auxwww | grep http

root      2928  0.0  0.6  17648  7120 ?           Ss   Oct21   0:00 /usr/local/apache2/bin/httpd -k start
nobody    2949  0.0  0.6  17648  6492 ?        S    Oct21   0:00 /usr/local/apache2/bin/httpd -k start
nobody    2950  0.0  0.6  17648  6492 ?        S    Oct21   0:00 /usr/local/apache2/bin/httpd -k start
nobody    2951  0.0  0.6  17648  6492 ?        S    Oct21   0:00 /usr/local/apache2/bin/httpd -k start
nobody    2952  0.0  0.6  17648  6492 ?        S    Oct21   0:00 /usr/local/apache2/bin/httpd -k start
nobody    2953  0.0  0.6  17648  6492 ?        S    Oct21   0:00 /usr/local/apache2/bin/httpd -k start
root     18508  0.0  0.0   3916   688 pts/0      S+   11:12   0:00 grep http

Here's how I list all my mysql processes:

$ ps auxwww | grep mysql

root      2837  0.0  0.1   4528  1236 ?        S    Oct21   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid
mysql     2897  0.0  1.7 136700 17952 ?        Sl   Oct21   0:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
root     18510  0.0  0.0   3916   712 pts/0    S+   11:13   0:00 grep mysql

Now that you've seen a few ps command examples, this is what the arguments to this ps command mean:

◉ The a argument means "show all processes", not just my processes. (There's a bit more to it than that, but this is usually close enough.)

◉ The u argument adds "user information" columns to the output. (Try your ps command without the 'u', and you'll see a major difference in the columns that are displayed.)

◉ The x lifts the BSD-style "must have a tty" restriction, meaning it will show processes that are not associated with a terminal (tty)>

◉ The w means "wide output". Use this option twice for unlimited width.

Examples from the Linux ps command man page

My intention for this article was to help get a new Unix or Linux user get started with the ps command. Before going, there are two more things I want to share with you.

First, there are many different variations of the ps command that you can use if you want to. For instance, I just looked at the Linux ps man page, and found these examples:

EXAMPLES
To see every process on the system using standard syntax:
   ps -e
   ps -ef
   ps -eF
   ps -ely

To see every process on the system using BSD syntax:
   ps ax
   ps axu

To print a process tree:
   ps -ejH
   ps axjf

To get info about threads:
   ps -eLf
   ps axms

To get security info:
   ps -eo euser,ruser,suser,fuser,f,comm,label
   ps axZ
   ps -eM

To see every process running as root (real & effective ID) in user format:
   ps -U root -u root u

To see every process with a user-defined format:
   ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm
   ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
   ps -eopid,tt,user,fname,tmout,f,wchan

Print only the process IDs of syslogd:
   ps -C syslogd -o pid=

Print only the name of PID 42:
   ps -p 42 -o comm=

Continue reading

Unix/Linux ‘cut’ command examples

Linux cut command FAQ: Can you share some cut command examples?

The Linux cut command is a really great command filter to know. You can use it to do all sorts of cool things when processing text files and text in command pipelines.

Using the cut command with /etc/passwd

For a first cut command example, I'll use the /etc/passwd file on my Unix system. I use this file because fields in the file are separated by the ":" character, which make it very easy to work with.

Using that file, here's a Linux cut command that prints the first field (first column) of every line in that file:

cut -f1 -d: /etc/passwd

This cut command can be read as:

◉ Print the first field (-f1)
◉ Fields are delimited by the ":" character (-d:)
◉ Use the /etc/passwd file as input

The output of this command will vary by Unix and Linux systems, but it will be the first field of your /etc/passwd file, which contains the name of the users on your system. This will look something like:

nobody
lpi
george
fred

and so on.

Using the cut command in a command pipeline

You can also use cut in a Unix/Linux command pipeline. For example, although you can get this information in other ways, you can use the cut command with the ls command to get a list of filenames in the current directory, like this:

ls -al | cut -c44-

Notice that in this command I'm using the cut command "-c" option. This command can be read as:

◉ Run the "ls -al" command
◉ Pipe the output of that command into cut
◉ The cut command prints everything from each line starting at column 44 through the end of the line (-c44-)

As you can see, the -c option lets you deal with columns or character positions. In this example I wanted all the output from each line starting in column 44 and going to the end of the line, but if I wanted only columns 40 through 50, I could have specified that like this instead:

ls -al | cut -c44-50

That particular command doesn't make much sense in the real world, since filenames can all be different lengths, but it does show how to use a range of columns with the Linux cut command.

More Unix cut command examples

There are many other uses of the Unix cut command, but I wanted to share these with you to get started. The cut command is a great Unix shell script tool, and I highly recommend being familiar with it.

Continue reading