If you have some experience in building enterprise applications, chances are you have used a Linux system. The Linux command-line interface (CLI) is an essential feature that developers appreciate. The CLI is powerful and flexible, allowing developers to perform complex tasks quickly and efficiently. In this post, I would like to list down 10 most useful commands which every developer should know.
I am intentionally omitting some of the basic but very widely used commands e.g. ls, cd,mkdir, rm, cp, mv and so on with an assumption that they are too basic and should be already known to most of the regular users.
1 . tail (display the last lines of a file)
tail is a Linux command that displays the last part of a file or multiple files. It can be useful for monitoring log files or keeping track of changes to a file in real time.
Here are some commonly used options for tail and some examples of how to use them:
-n or --lines: This option specifies the number of lines to display.
Display the last 5 lines of a file:
tail -n 5 file1.txt
-f or --follow: This option will output the last part of a file in real-time as it is being written to. This is commonly used for monitoring log files.
Display new log entries as they are added to the file.
tail -f /var/log/messages
-q or --quiet: This option suppresses the display of file headers when displaying multiple files.
Display only the last 5 lines of each file without showing the filename headers.
tail -q -n 5 file1.txt file2.txt
-c or --bytes: This option specifies the number of bytes to display.
Display the last 100 bytes of file1.txt
tail -c 100 file1.txt
-v or --verbose: This option displays the name of each file before the output.
Display "==> file1.txt <==" before the last 5 lines of the file.
tail -v file1.txt
These are just a few examples of how tail can be used. By combining these options and using tail with other commands, you can create powerful and flexible file monitoring and processing tools.
2 . head (display the first lines of a file)
head is a command in Linux used to display the first few lines of a file. By default, it displays the first 10 lines of a file. Here are some examples of using head with different options:
- Display the first 5 lines of a file:
head -n 5 file.txt
- Display the first 20 bytes of a file:
head -c 20 file.txt
- Display the first line of multiple files:
head -n 1 file1.txt file2.txt file3.txt
- Display the first few lines of a file, with a message indicating the file name:
head -v file.txt
- Display the first 5 lines of a file, and include the line numbers:
head -n 5 -v file.txt
Here, the options used are:
-n: specify the number of lines to display.
-c: specify the number of bytes to display.
-v: display the name of the file before the output.
These are just a few examples of using head with different options. There are many other options available that can be used to customize the output of this command which you can explore with tail --help.
3 . cat (concatenate and display files)
The cat command is used to concatenate and display files. It can also be used to create new files by combining existing files. Here are some examples of using the cat command with various options:
- Display contents of a single file:
cat file.txt
- Combine multiple files and display their contents:
cat file1.txt file2.txt file3.txt
- Display line numbers along with file contents:
cat -b file.txt
1 In web applications, it's common to have long-running processes that need to be executed asynchronously.
2 When these processes take a long time to complete,
3 it's important to provide feedback to the user so
4 that they know what's happening and can cont
- Append one file to another:
cat file1.txt >> file2.txt
- Create a new file by combining existing files:
cat file1.txt file2.txt > newfile.txt
In the first example, cat is used to display the contents of a single file.
In the second example, cat is used to combine multiple files and display their contents.
In the third example, the -n option is used to display line numbers along with file contents.
In the fourth example, the >> operator is used to append the contents of one file to another.
In the fifth example, the > operator is used to create a new file by combining the contents of two existing files.
4 . diff ( compare two files line by line)
The diff command in Linux is used to compare two files line by line and show the differences between them. It can also be used to compare two directories and their contents.
Here are some common options used with the diff command:
-u or --unified: shows the differences in a unified format, making it easier to read and understand.
-c or --context: shows the differences in a context format, which includes a few lines before and after each difference to provide more context.
-r or --recursive: compares directories and their contents recursively.
-i or --ignore-case: ignores case differences in the files being compared.
-w or --ignore-all-space: ignores all whitespace differences in the files being compared.
Here are some examples of using the diff command with different options:
Example 1: Compare two files using the default output format
diff file1.txt file2.txt
Example 2: Compare two files using the unified output format
diff -u file1.txt file2.txt
Example 3: Compare two directories and their contents recursively
diff -r dir1/ dir2/
Example 4: Compare two files ignoring case differences
diff -i file1.txt file2.txt
Example 5: Compare two files ignoring all whitespace differences
diff -w file1.txt file2.txt
In these examples, file1.txt and file2.txt are files being compared, and dir1/ and dir2/ are directories being compared. The output of the diff command shows the differences between the files or directories. By using different options, the output format and the behavior of the diff command can be customized.
5 . patch (apply a diff file to a file or directory)
patch is a command-line tool that allows you to apply a patch file to a file or directory. A patch file is a file that contains the differences between two versions of a file or directory. When you apply a patch file, the changes in the patch file are applied to the file or directory, resulting in a new version of the file or directory.
Here are some examples of how to use the patch command with different options:
Example 1: Apply a patch to a file
Suppose you have a patch file called patch.diff and a file called file.txt. To apply the patch to the file, you can use the following command:
patch file.txt patch.diff
This command will apply the changes in the patch file to the file.
Example 2: Apply a patch to a directory
Suppose you have a patch file called patch.diff and a directory called dir/. To apply the patch to the directory and all its files, you can use the following command:
patch -p1 < patch.diff
The -p1 option tells patch to strip one level of the directory path from the files in the patch file. This is necessary because the patch file contains the full path to the files.
Example 3: Create a patch file
Suppose you have two files called file1.txt and file2.txt and you want to create a patch file that contains the differences between the two files. To create the patch file, you can use the following command:
diff -u file1.txt file2.txt > patch.diff
This command will create a patch file called patch.diff that contains the differences between file1.txt and file2.txt.
Example 4: Ignore whitespace changes
Suppose your patch file contains changes that only affect whitespace. To ignore these changes when applying the patch, you can use the following command:
patch -l file.txt patch.diff
The -l option tells patch to ignore changes in whitespace.
Example 5: Apply a patch with dry-run
Suppose you want to see what changes a patch file will make before applying it. To do this, you can use the following command:
patch --dry-run file.txt patch.diff
The --dry-run option tells patch to simulate the patching process without actually making any changes to the file. This can be useful for testing a patch file before applying it.
These are just a few examples of how to use the patch command. patch provides many options for customizing the patching process, so be sure to consult the manual (man patch) for more information.
tar is a command-line utility in Linux that is used to create and extract archives from one or more files or directories. The name "tar" stands for "tape archive," as it was originally designed to write data to tape drives.
Here are some of the commonly used options with tar:
-c (create) - create a new archive
-x (extract) - extract files from an archive
-v (verbose) - display progress and filenames while processing
-f (file) - specify the filename of the archive
-z (gzip) - compress or uncompress the archive using gzip
Here are five examples of using tar with different options:
- Create a new archive of all files in the current directory:
tar -cvf archive.tar *
- Extract all files from an archive:
tar -xvf archive.tar
- Extract all files from a gzip-compressed archive:
tar -xzvf archive.tar.gz
- Create a gzip-compressed archive of a directory:
tar -czvf archive.tar.gz directory/
- Extract a specific file from an archive:
tar -xvf archive.tar file.txt
These are just a few examples of how tar can be used to create and extract archives. The utility offers many more options and can be used in more complex ways for a variety of use cases.
7 . sed (stream editor for filtering and transforming text)
sed (stream editor) is a extremly powerful tool used for filtering and transforming text. It can be used to modify text files in place, or to process text output from other commands. sed works by reading input line by line, applying a set of rules (expressions) to each line, and then printing the modified result to the output.
Here are 10 examples of sed commands with explanations of their options:
Replace all occurrences of a string in a file with another string:
sed -i 's/old_string/new_string/g' file.txt
The -i option is used to modify the file in place. The s command replaces the first occurrence of old_string with new_string. The g flag at the end specifies that all occurrences should be replaced.
Remove all occurrences of a string in a file:
sed -i '/string_to_remove/d' file.txt
The /string_to_remove/d expression deletes all lines that contain string_to_remove. The -i option is used to modify the file in place.
Add a line to the beginning of a file:
sed -i '1i\new_line' file.txt
The 1i\ command inserts new_line at line 1 of the file. The -i option is used to modify the file in place.
Add a line to the end of a file:
sed -i '$a\new_line' file.txt
The $a\ command appends new_line to the end of the file. The -i option is used to modify the file in place.
Remove empty lines from a file:
pythonCopy codesed -i '/^$/d' file.txt
The /^$/d expression deletes all lines that are empty (contain only whitespace). The -i option is used to modify the file in place.
Replace the first occurrence of a string in a file on a specific line number:
sed -i '3s/old_string/new_string/' file.txt
The 3s/old_string/new_string/ expression replaces the first occurrence of old_string with new_string on line number 3. The -i option is used to modify the file in place.
Remove all whitespace from the beginning and end of each line:
sed -i 's/^[[:space:]]*//;s/[[:space:]]*$//' file.txt
The s/^[[:space:]]*// expression removes all whitespace from the beginning of each line. The s/[[:space:]]*$// expression removes all whitespace from the end of each line. The -i option is used to modify the file in place.
Remove a specific line from a file:
sed -i '5d' file.txt
The 5d expression deletes line number 5 from the file. The -i option is used to modify the file in place.
Replace a string in a file only on lines that match a specific pattern:
sed -i '/pattern/s/old_string/new_string/' file.txt
The /pattern/ expression matches all lines that contain pattern. The s/old_string/new_string/ expression replaces the first occurrence of old_string with new_string on all matching lines. The -i option is used to modify the file.
Remove all consecutive duplicate lines from a file:
sed -i '$!N; /^\(.*\)\n\1$/!P; D' file.txt
In conclusion, sed is a versatile command-line tool that can be used to perform a wide range of text-processing tasks. Its ability to read and modify files line by line makes it particularly useful for working with large text files, and its powerful regular expression capabilities provide a flexible and efficient way to search for and manipulate text. With practice, developers can become proficient at using sed to streamline their text processing workflows and automate common tasks.
8 . awk (pattern scanning and processing language)
AWK is a programming language designed for text processing and data extraction. It is especially useful for processing files that contain structured data, such as CSV files, log files, and configuration files. AWK provides a range of built-in functions for manipulating strings, performing arithmetic operations, and manipulating data structures such as arrays.
Here are 10 examples of how to use AWK, along with explanations of the options used:
- Print the first field of each line in a file:
awk '{print $1}' filename.txt
This command tells AWK to print the first field of each line in the file filename.txt. The $1 variable refers to the first field of the current line.
- Print the second field of each line in a file:
awk '{print $2}' filename.txt
This command tells AWK to print the second field of each line in the file filename.txt. The $2 variable refers to the second field of the current line.
- Print the last field of each line in a file:
awk '{print $NF}' filename.txt
This command tells AWK to print the last field of each line in the file filename.txt. The $NF variable refers to the last field of the current line.
- Print all fields of each line in a file:
awk '{print $0}' filename.txt
This command tells AWK to print the entire line for each line in the file filename.txt. The $0 variable refers to the entire line.
- Print the total number of lines in a file:
awk 'END {print NR}' filename.txt
This command tells AWK to print the number of records (NR) at the end of the file. The END keyword specifies that this action should be taken after all lines have been processed.
- Print the total number of fields in a file:
awk '{print NF}' filename.txt | sort -rn | head -1
This command tells AWK to print the number of fields for each line in the file filename.txt. The output is then piped to sort -rn to sort the results in reverse numerical order, and head -1 is used to print the first line (i.e., the line with the highest number of fields).
In conclusion, AWK is a powerful tool for processing text files and extracting data. By learning AWK, developers can streamline their workflows and automate common text-processing tasks.
9 . grep (search for a pattern in a file)
grep is a command-line tool used for searching text in files or directories. It can search for a specific pattern in a file and return the lines that contain that pattern. Here is the basic syntax of the grep command:
grep [OPTIONS] PATTERN [FILE...]
In this syntax, OPTIONS are the different options available for the grep command, PATTERN is the text pattern to search for, and FILE is the file or files to search in.
Here are 7 examples of using grep command with some of the commonly used options:
Search for a pattern in a file:
grep "pattern" file.txt
Search for a pattern in multiple files:
grep "pattern" file1.txt file2.txt
Search for a pattern in all files in a directory:
grep "pattern" *
Search for a pattern in a file, ignoring case:
grep -i "pattern" file.txt
Search for a pattern in a file, with line numbers:
grep -n "pattern" file.txt
Search for a pattern in a file, showing only the matching text:
grep -o "pattern" file.txt
Search for a pattern in all files in a directory recursively:
grep -r "pattern" directory/
Find all the files which do not have a specific pattern:
grep -v "pattern" file.txt
The -v option in grep is used to invert the search and display only the lines that do not contain the specified pattern
These are just a few examples of the many options available with grep. The grep command can be a powerful tool for searching text in files and directories, and can be used in combination with other commands to perform complex operations on text files.
10. cut (remove sections from each line of a file)
cut is a command-line utility in Linux that is used to extract parts of a file or a stream of data by specifying a delimiter. Here is a brief overview of the options available for cut:
-d <delimiter> - Specify a delimiter character
-f <field_list> - Specify a list of fields to extract
Here are 7 examples of using cut with different options:
- Extract the first field of each line of a file:
cut -d',' -f1 file.txt
This command uses , as the delimiter and extracts the first field of each line of file.txt.
- Extract the first 3 characters of each line of a file:
cut -c1-3 file.txt
This command extracts the first three characters of each line of file.txt.
- Extract the last field of each line of a file:
cut -d',' -fNF file.txt
This command uses , as the delimiter and extracts the last field of each line of file.txt.
- Extract the 2nd and 3rd fields of each line of a file:
cut -d',' -f2,3 file.txt
This command uses , as the delimiter and extracts the 2nd and 3rd fields of each line of file.txt.
- Extract the characters between the 10th and 20th positions of each line of a file:
cut -c10-20 file.txt
This command extracts the characters between the 10th and 20th positions of each line of file.txt.
- Extract the first and last fields of each line of a file:
cut -d',' -f1,NF file.txt
This command uses , as the delimiter and extracts the first and last fields of each line of file.txt.
- Extract the fields in reverse order of each line of a file:
cut -d',' --output-delimiter=' ' -f$(awk -F, '{print NF; exit}')-1 file.txt
This command uses , as the delimiter and extracts the fields of each line of file.txt in reverse order, separated by a space. The awk command is used to get the number of fields in the file and pass it to cut to extract the fields in reverse order.
- Extract fields 2 through 4 of a file using a tab delimiter:
bashCopy codecut -d$'\t' -f2-4 file.txt
This command uses a tab character as the delimiter and extracts fields 2 through 4 of each line of file.txt.
In conclusion, cut is a versatile and useful command-line utility for extracting fields or characters from a file or a stream of data. It is often used in conjunction with other commands such as awk and sed to manipulate and process text files. By mastering cut, developers can become more efficient at working with data and automating tasks on a Linux system.
Honorable mention
While the commands covered in this post are certainly useful for developers, it's important to note that there are many more Linux commands worth exploring and becoming familiar with. Due to the limitations of length, I did not cover them here.
For example, curl and wget are powerful tools for downloading files from the web, top and ps provide information on system processes, kill is useful for terminating processes, and ping and traceroute are essential for checking network connectivity. ssh and scp are valuable for remote login and file transfer, respectively.
By using the --help option or referring to Linux command manuals, developers can continue to learn about new commands and improve their workflow. I welcome any feedback or comments from our readers and hope that this post has helped expand your knowledge of commonly used Linux commands.
Happy Learning.
