sed tools

sed is a powerful and simple text parsing and transformation tool, which can read text, edit text content (delete, replace, add, move, etc.) according to specified conditions, and finally output all lines or only some processed lines. sed can also implement quite complex text processing operations without interaction, and is widely used in Shell scripts to complete various automated processing tasks.

sed workflow

1. Read: sed reads a line of content from the input stream (file, pipeline, standard input) and stores it in a temporary buffer.
2. Execution: By default, all sed commands are executed sequentially in the pattern space, unless the address of the line is specified, the SED command will execute sequentially on all lines.
3. Display: Send the modified content to the output stream. After sending the data, the schema space will be emptied.

Use of sed command

sed [option]'operation'parameter
 sed [option] - f scriptfile parameter

Common sed command options

- e or -- expression=: Represents processing input text files with specified commands or scripts.
- f or -- file=: Represents that the input text file is processed with the specified script file.
- h or -- help: Display help.
- n,--quiet or silent: Represents only the results after processing.
- i: Edit text files directly.

"Operation" is used to specify the action behavior of the file operation, which is the command of sed. Usually, the format of operation parameters of "[n1[,n2]" is adopted. N1, N2 are optional, not necessarily exist, representing the number of rows selected for operation

Common Operational Options

A: Increase by adding a specified line below the current line.
c: Replace, replacing the selected row with the specified content.
d: Delete, delete selected rows.
i: Insert, insert a specified line above the selected line.
p: Print, if you specify rows at the same time, it means print the specified rows; if you do not specify rows, it means print all content; if there are non-print characters, it is output with ASCII code. It is usually used with the "-n" option.
s: Replace, replace the specified character.
y: Character conversion.

Usage demonstration

1. Output eligible text (p for normal output)

[[email protected] opt]# Sed-n'p'/etc/passwd//output all content, equivalent to the use of cat
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
......
sed -n '5p' /etc/passwd        //Output line 5
sed -n '3,5p'  /etc/passwd        //Output lines 3 to 5
sed -n 'p;n' /etc/passwd        //Output all odd lines, n for reading in the next line
sed -n 'n;p'  /etc/passwd        //Output the contents of all even lines
sed -n '1,5{p;n}' /etc/passwd  //Output odd rows between 1 and 5 rows
 sed -n '22,${n;p}' /etc/passwd  //Output even rows from line 22 to row bit

These are the basic uses of sed commands. When combined with regular expressions, the format of SED commands is slightly different. Regular expressions are surrounded by "/".

The sed command is used in conjunction with regular expressions

[[email protected] opt]# Sed-n'/sbin/p'/etc/passwd//Output row containing SBIN
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
......
[[email protected] opt]# Sed-n'2, / sbin/p'/etc/passwd// output starts at line 2 to the first line containing SBIN      
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[[email protected] opt]# Sed-n'/sbin/='/etc/passwd//output contains the line number where SBIN is located, and the equal sign (=) is used to output the line number.
2
3
4
5
6
7
8
......
[[email protected] opt]# Sed-n'/^ssh/p'/etc/passwd//Output lines beginning with SSH
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
[[email protected] opt]# Sed-n'/[0-9]$/p'test.txt//Output lines ending in numbers
PI=3.141592653589793238462643383249901429
[[email protected] opt]# Sed-n'/wood\>/p'test.txt//output lines containing the word wood, \\\\\\\\\
a wood cross!

2. Delete eligible text (d)
The nl command in the following command is used to calculate the number of lines in the file. Combining with this command, the result of command execution can be seen more intuitively.

nl test.txt | sed '3d'       //Delete line 3
nl test.txt | sed '3,5d'         //Delete lines 3-5
nl test.txt |sed '/cross/d'      //Delete rows containing cross es 
sed '/^[a-z]/d' test.txt           //Delete rows beginning with lowercase letters
sed '/\.$/d' test.txt             //Delete the line ending with ".".
sed '/^$/d' test.txt              //Delete all empty lines
[[email protected] opt]# nl test.txt | sed '3,5d'
1 he was short and fat.
2 He was wearing a blue polo shirt with black pants.
6   The year ahead will test our political establishment to the limit. 7    PI=3.141592653589793238462643383249901429
Note: If you delete duplicate empty lines, that is to say, only one continuous empty line is reserved, then you can execute the command "sed-e'/^$/{n;/^$/d}'test.txt". The effect is the same as "cat-s test.txt", where n means reading the next row of data.

3. Replace eligible texts
The command options of s (string substitution), c (whole line/block substitution) and y (character conversion) are needed for substitution with sed command.

sed 's/the/THE/' test.txt       //Replace THE first of THE lines with THE 
sed 's/l/L/3' test.txt              //Replace the third L in each row with L
sed 's/the/THE/g' test.txt          //Replace all THE thes in THE file with THE
sed 's/o//G'test.txt// Delete all o in the file (replace with empty string)
sed 's/^/#/' test.txt                //Insert # at the beginning of each line
sed '/the/s/^/#/' test.txt            //Insert # at the beginning of each line containing the
sed 's/$/EOF/' test.txt                 //Insert the string EOF at the end of each line 
sed '3,5s/the/THE/g' test.txt             //Replace all THE thes in lines 3 to 5 with THE
sed '/the/s/o/O/g' test.txt             //Replace o in all rows containing the with O

4. Migrating eligible texts

H copy to clipboard;
G G covers/appends the data in the clipboard to the specified row;
w is saved as a file;
r reads the specified file;
a. Additional specified content.
sed '/the/{H;d};$G' test.txt          //Migrate rows containing the to the end of the file, {;} for multiple operations sed'1,5 {H; d}; 17G'test.txt// / to move lines 1 to 5 after lines 17
sed '/the/w out.file' test.txt        //Save the row containing the as a file out.file
sed '/the/r /etc/hostname' test.txt       //After adding the contents of the file / etc/hostname to each line containing the
sed '3aNew' test.txt      //Insert a new line after line 3, New 
sed '/the/aNew' test.txt          //Insert a new line after each line that contains the new sed'3aNew1 nNew2'test.txt// / Insert a multi-line content after the third line, and the middle n represents a newline

5. Use scripts to edit files
In sed scripts, multiple editing instructions can be stored in a file (one editing instruction per line) and invoked through the "- f" option.

sed '1,6{H;d};18G' test.txt //Transfer lines 1-6 to lines 18
[[email protected] opt]# vim try.list
1,6H
1,6d
18G

[[email protected] opt]# sed -f  try.list test.txt

6.sed directly manipulates files
Write a script to adjust the vsftpd service configuration: prohibit anonymous users, but allow local users (also write).

[[email protected] opt]#  vim aaa.sh
#!/bin/bash
# Specify Sample File Path, Configuration File Path
SAMPLE="/usr/share/doc/vsftpd-3.0.2/EXAMPLE/INTERNET_SITE/vsftpd.conf " CONFIG="/etc/vsftpd/vsftpd.conf"

# Back up the original configuration file and check whether the backup file named / etc/vsftpd/vsftpd.conf.bak exists or not. If not, use the cp command to backup the file.
[ ! -e "$CONFIG.bak" ] && cp $CONFIG $CONFIG.bak # Adjust based on sample configuration to overwrite existing files
sed -e '/^anonymous_enable/s/YES/NO/g' $SAMPLE > $CONFIG
sed -i -e '/^local_enable/s/NO/YES/g' -e'/^write_enable/s/NO/YES/g' $CONFIG 
grep "listen" $CONFIG || sed -i '$alisten=YES' $CONFIG
# Start the vsftpd service and set it to run automatically after booting
systemctl restart vsftpd
systemctl enable vsftpd

[[email protected] opt]#  chmod +x aaa.sh 
[[email protected] opt]#  ./aaa.sh