Table of contents
If you have some experience in building enterprise applications, chances are you have used a Linux system. The Linux command-line interface (CLI) is an essential feature that developers appreciate. The CLI is powerful and flexible, allowing developers to perform complex tasks quickly and efficiently. In this post, I would like to list down 10 most useful commands which every developer should know.
I am intentionally omitting some of the basic but very widely used commands e.g. ls, cd,mkdir, rm, cp, mv
and so on with an assumption that they are too basic and should be already known to most of the regular users.
1 . tail (display the last lines of a file)
tail
is a Linux command that displays the last part of a file or multiple files. It can be useful for monitoring log files or keeping track of changes to a file in real time.
Here are some commonly used options for tail
and some examples of how to use them:
-n
or--lines
: This option specifies the number of lines to display.Display the last 5 lines of a file:
tail -n 5 file1.txt ## will display first 5 lines for file1.txt
-f
or--follow
: This option will output the last part of a file in real-time as it is being written to. This is commonly used for monitoring log files.Display new log entries as they are added to the file.
tail -f /var/log/messages ## will start straming file from end.
-q
or--quiet
: This option suppresses the display of file headers when displaying multiple files.Display only the last 5 lines of each file without showing the filename headers.
tail -q -n 5 file1.txt file2.txt ## will show content from file1,file2 stacked with out name(file1.txt,file2.txt)
-c
or--bytes
: This option specifies the number of bytes to display.Display the last 100 bytes of
file1.txt
tail -c 100 file1.txt ## shows 100 bytes starting from end
-v
or--verbose
: This option displays the name of each file before the output.Display "==> file1.txt <==" before the last 5 lines of the file.
tail -v file1.txt ## ==> file1.txt <== followed by the content of file
These are just a few examples of how tail
can be used. By combining these options and using tail
with other commands, you can create powerful and flexible file monitoring and processing tools.
2 . head (display the first lines of a file)
head
is a command in Linux used to display the first few lines of a file. By default, it displays the first 10 lines of a file. Here are some examples of using head
with different options:
- Display the first 5 lines of a file:
head -n 5 file.txt
- Display the first 20 bytes of a file:
head -c 20 file.txt
- Display the first line of multiple files:
head -n 1 file1.txt file2.txt file3.txt
- Display the first few lines of a file, with a message indicating the file name:
head -v file.txt
- Display the first 5 lines of a file, and include the line numbers:
head -n 5 -v file.txt
Here, the options used are:
-n
: specify the number of lines to display.-c
: specify the number of bytes to display.-v
: display the name of the file before the output.
These are just a few examples of using head
with different options. There are many other options available that can be used to customize the output of this command which you can explore with tail --help
.
3 . cat (concatenate and display files)
The cat
command is used to concatenate and display files. It can also be used to create new files by combining existing files. Here are some examples of using the cat
command with various options:
- Display contents of a single file:
cat file.txt
- Combine multiple files and display their contents:
cat file1.txt file2.txt file3.txt
- Display line numbers along with file contents:
cat -b file.txt
1 In web applications, it's common to have long-running processes that need to be executed asynchronously.
2 When these processes take a long time to complete,
3 it's important to provide feedback to the user so
4 that they know what's happening and can cont
- Append one file to another:
cat file1.txt >> file2.txt
- Create a new file by combining existing files:
cat file1.txt file2.txt > newfile.txt
In the first example, cat
is used to display the contents of a single file.
In the second example, cat
is used to combine multiple files and display their contents.
In the third example, the -n
option is used to display line numbers along with file contents.
In the fourth example, the >>
operator is used to append the contents of one file to another.
In the fifth example, the >
operator is used to create a new file by combining the contents of two existing files.
4 . diff ( compare two files line by line)
The diff
command in Linux is used to compare two files line by line and show the differences between them. It can also be used to compare two directories and their contents.
Here are some common options used with the diff
command:
-u
or--unified
: shows the differences in a unified format, making it easier to read and understand.-c
or--context
: shows the differences in a context format, which includes a few lines before and after each difference to provide more context.-r
or--recursive
: compares directories and their contents recursively.-i
or--ignore-case
: ignores case differences in the files being compared.-w
or--ignore-all-space
: ignores all whitespace differences in the files being compared.
Here are some examples of using the diff
command with different options:
Example 1: Compare two files using the default output format
diff file1.txt file2.txt
Example 2: Compare two files using the unified output format
diff -u file1.txt file2.txt
Example 3: Compare two directories and their contents recursively
diff -r dir1/ dir2/
Example 4: Compare two files ignoring case differences
diff -i file1.txt file2.txt
Example 5: Compare two files ignoring all whitespace differences
diff -w file1.txt file2.txt
In these examples, file1.txt
and file2.txt
are files being compared, and dir1/
and dir2/
are directories being compared. The output of the diff
command shows the differences between the files or directories. By using different options, the output format and the behavior of the diff
command can be customized.
5 . patch (apply a diff file to a file or directory)
patch
is a command-line tool that allows you to apply a patch file to a file or directory. A patch file is a file that contains the differences between two versions of a file or directory. When you apply a patch file, the changes in the patch file are applied to the file or directory, resulting in a new version of the file or directory.
Here are some examples of how to use the patch
command with different options:
Example 1: Apply a patch to a file
Suppose you have a patch file called patch.diff
and a file called file.txt
. To apply the patch to the file, you can use the following command:
patch file.txt patch.diff
This command will apply the changes in the patch file to the file.
Example 2: Apply a patch to a directory
Suppose you have a patch file called patch.diff
and a directory called dir/
. To apply the patch to the directory and all its files, you can use the following command:
patch -p1 < patch.diff
The -p1
option tells patch
to strip one level of the directory path from the files in the patch file. This is necessary because the patch file contains the full path to the files.
Example 3: Create a patch file
Suppose you have two files called file1.txt
and file2.txt
and you want to create a patch file that contains the differences between the two files. To create the patch file, you can use the following command:
diff -u file1.txt file2.txt > patch.diff
This command will create a patch file called patch.diff
that contains the differences between file1.txt
and file2.txt
.
Example 4: Ignore whitespace changes
Suppose your patch file contains changes that only affect whitespace. To ignore these changes when applying the patch, you can use the following command:
patch -l file.txt patch.diff
The -l
option tells patch
to ignore changes in whitespace.
Example 5: Apply a patch with dry-run
Suppose you want to see what changes a patch file will make before applying it. To do this, you can use the following command:
patch --dry-run file.txt patch.diff
The --dry-run
option tells patch
to simulate the patching process without actually making any changes to the file. This can be useful for testing a patch file before applying it.
These are just a few examples of how to use the patch
command. patch
provides many options for customizing the patching process, so be sure to consult the manual (man patch
) for more information.
6 . tar (extract files from the archive/create an archive from files )
tar
is a command-line utility in Linux that is used to create and extract archives from one or more files or directories. The name "tar" stands for "tape archive," as it was originally designed to write data to tape drives.
Here are some of the commonly used options with tar
:
-c
(create) - create a new archive-x
(extract) - extract files from an archive-v
(verbose) - display progress and filenames while processing-f
(file) - specify the filename of the archive-z
(gzip) - compress or uncompress the archive using gzip
Here are five examples of using tar
with different options:
- Create a new archive of all files in the current directory:
tar -cvf archive.tar *
- Extract all files from an archive:
tar -xvf archive.tar
- Extract all files from a gzip-compressed archive:
tar -xzvf archive.tar.gz
- Create a gzip-compressed archive of a directory:
tar -czvf archive.tar.gz directory/
- Extract a specific file from an archive:
tar -xvf archive.tar file.txt
These are just a few examples of how tar
can be used to create and extract archives. The utility offers many more options and can be used in more complex ways for a variety of use cases.
7 . sed (stream editor for filtering and transforming text)
sed
(stream editor) is a extremly powerful tool used for filtering and transforming text. It can be used to modify text files in place, or to process text output from other commands. sed
works by reading input line by line, applying a set of rules (expressions) to each line, and then printing the modified result to the output.
Here are 10 examples of sed
commands with explanations of their options:
Replace all occurrences of a string in a file with another string:
sed -i 's/old_string/new_string/g' file.txt
The
-i
option is used to modify the file in place. Thes
command replaces the first occurrence ofold_string
withnew_string
. Theg
flag at the end specifies that all occurrences should be replaced.Remove all occurrences of a string in a file:
sed -i '/string_to_remove/d' file.txt
The
/string_to_remove/d
expression deletes all lines that containstring_to_remove
. The-i
option is used to modify the file in place.Add a line to the beginning of a file:
sed -i '1i\new_line' file.txt
The
1i\
command insertsnew_line
at line 1 of the file. The-i
option is used to modify the file in place.Add a line to the end of a file:
sed -i '$a\new_line' file.txt
The
$a\
command appendsnew_line
to the end of the file. The-i
option is used to modify the file in place.Remove empty lines from a file:
pythonCopy codesed -i '/^$/d' file.txt
The
/^$/d
expression deletes all lines that are empty (contain only whitespace). The-i
option is used to modify the file in place.Replace the first occurrence of a string in a file on a specific line number:
sed -i '3s/old_string/new_string/' file.txt
The
3s/old_string/new_string/
expression replaces the first occurrence ofold_string
withnew_string
on line number 3. The-i
option is used to modify the file in place.Remove all whitespace from the beginning and end of each line:
sed -i 's/^[[:space:]]*//;s/[[:space:]]*$//' file.txt
The
s/^[[:space:]]*//
expression removes all whitespace from the beginning of each line. Thes/[[:space:]]*$//
expression removes all whitespace from the end of each line. The-i
option is used to modify the file in place.Remove a specific line from a file:
sed -i '5d' file.txt
The
5d
expression deletes line number 5 from the file. The-i
option is used to modify the file in place.Replace a string in a file only on lines that match a specific pattern:
sed -i '/pattern/s/old_string/new_string/' file.txt
The
/pattern/
expression matches all lines that containpattern
. Thes/old_string/new_string/
expression replaces the first occurrence ofold_string
withnew_string
on all matching lines. The-i
option is used to modify the file.Remove all consecutive duplicate lines from a file:
sed -i '$!N; /^\(.*\)\n\1$/!P; D' file.txt
In conclusion,
sed
is a versatile command-line tool that can be used to perform a wide range of text-processing tasks. Its ability to read and modify files line by line makes it particularly useful for working with large text files, and its powerful regular expression capabilities provide a flexible and efficient way to search for and manipulate text. With practice, developers can become proficient at usingsed
to streamline their text processing workflows and automate common tasks.
8 . awk (pattern scanning and processing language)
AWK is a programming language designed for text processing and data extraction. It is especially useful for processing files that contain structured data, such as CSV files, log files, and configuration files. AWK provides a range of built-in functions for manipulating strings, performing arithmetic operations, and manipulating data structures such as arrays.
Here are 10 examples of how to use AWK, along with explanations of the options used:
- Print the first field of each line in a file:
awk '{print $1}' filename.txt
This command tells AWK to print the first field of each line in the file filename.txt
. The $1
variable refers to the first field of the current line.
- Print the second field of each line in a file:
awk '{print $2}' filename.txt
This command tells AWK to print the second field of each line in the file filename.txt
. The $2
variable refers to the second field of the current line.
- Print the last field of each line in a file:
awk '{print $NF}' filename.txt
This command tells AWK to print the last field of each line in the file filename.txt
. The $NF
variable refers to the last field of the current line.
- Print all fields of each line in a file:
awk '{print $0}' filename.txt
This command tells AWK to print the entire line for each line in the file filename.txt
. The $0
variable refers to the entire line.
- Print the total number of lines in a file:
awk 'END {print NR}' filename.txt
This command tells AWK to print the number of records (NR
) at the end of the file. The END
keyword specifies that this action should be taken after all lines have been processed.
- Print the total number of fields in a file:
awk '{print NF}' filename.txt | sort -rn | head -1
This command tells AWK to print the number of fields for each line in the file filename.txt
. The output is then piped to sort -rn
to sort the results in reverse numerical order, and head -1
is used to print the first line (i.e., the line with the highest number of fields).
In conclusion, AWK is a powerful tool for processing text files and extracting data. By learning AWK, developers can streamline their workflows and automate common text-processing tasks.
9 . grep (search for a pattern in a file)
grep
is a command-line tool used for searching text in files or directories. It can search for a specific pattern in a file and return the lines that contain that pattern. Here is the basic syntax of the grep
command:
grep [OPTIONS] PATTERN [FILE...]
In this syntax, OPTIONS
are the different options available for the grep
command, PATTERN
is the text pattern to search for, and FILE
is the file or files to search in.
Here are 7 examples of using grep
command with some of the commonly used options:
Search for a pattern in a file:
grep "pattern" file.txt
Search for a pattern in multiple files:
grep "pattern" file1.txt file2.txt
Search for a pattern in all files in a directory:
grep "pattern" *
Search for a pattern in a file, ignoring case:
grep -i "pattern" file.txt
Search for a pattern in a file, with line numbers:
grep -n "pattern" file.txt
Search for a pattern in a file, showing only the matching text:
grep -o "pattern" file.txt
Search for a pattern in all files in a directory recursively:
grep -r "pattern" directory/
Find all the files which do not have a specific pattern:
grep -v "pattern" file.txt
The
-v
option ingrep
is used to invert the search and display only the lines that do not contain the specified patternThese are just a few examples of the many options available with
grep
. Thegrep
command can be a powerful tool for searching text in files and directories, and can be used in combination with other commands to perform complex operations on text files.
10. cut (remove sections from each line of a file)
cut
is a command-line utility in Linux that is used to extract parts of a file or a stream of data by specifying a delimiter. Here is a brief overview of the options available for cut
:
-d <delimiter> - Specify a delimiter character
-f <field_list> - Specify a list of fields to extract
Here are 7 examples of using cut
with different options:
- Extract the first field of each line of a file:
cut -d',' -f1 file.txt
This command uses ,
as the delimiter and extracts the first field of each line of file.txt
.
- Extract the first 3 characters of each line of a file:
cut -c1-3 file.txt
This command extracts the first three characters of each line of file.txt
.
- Extract the last field of each line of a file:
cut -d',' -fNF file.txt
This command uses ,
as the delimiter and extracts the last field of each line of file.txt
.
- Extract the 2nd and 3rd fields of each line of a file:
cut -d',' -f2,3 file.txt
This command uses ,
as the delimiter and extracts the 2nd and 3rd fields of each line of file.txt
.
- Extract the characters between the 10th and 20th positions of each line of a file:
cut -c10-20 file.txt
This command extracts the characters between the 10th and 20th positions of each line of file.txt
.
- Extract the first and last fields of each line of a file:
cut -d',' -f1,NF file.txt
This command uses ,
as the delimiter and extracts the first and last fields of each line of file.txt
.
- Extract the fields in reverse order of each line of a file:
cut -d',' --output-delimiter=' ' -f$(awk -F, '{print NF; exit}')-1 file.txt
This command uses ,
as the delimiter and extracts the fields of each line of file.txt
in reverse order, separated by a space. The awk
command is used to get the number of fields in the file and pass it to cut
to extract the fields in reverse order.
- Extract fields 2 through 4 of a file using a tab delimiter:
bashCopy codecut -d$'\t' -f2-4 file.txt
This command uses a tab character as the delimiter and extracts fields 2 through 4 of each line of file.txt
.
In conclusion, cut
is a versatile and useful command-line utility for extracting fields or characters from a file or a stream of data. It is often used in conjunction with other commands such as awk
and sed
to manipulate and process text files. By mastering cut
, developers can become more efficient at working with data and automating tasks on a Linux system.
Honorable mention
While the commands covered in this post are certainly useful for developers, it's important to note that there are many more Linux commands worth exploring and becoming familiar with. Due to the limitations of length, I did not cover them here.
For example, curl
and wget
are powerful tools for downloading files from the web, top
and ps
provide information on system processes, kill
is useful for terminating processes, and ping
and traceroute
are essential for checking network connectivity. ssh
and scp
are valuable for remote login and file transfer, respectively.
By using the --help
option or referring to Linux command manuals, developers can continue to learn about new commands and improve their workflow. I welcome any feedback or comments from our readers and hope that this post has helped expand your knowledge of commonly used Linux commands.
Happy Learning.