> For the complete documentation index, see [llms.txt](https://riteshs4hu.gitbook.io/infosec-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://riteshs4hu.gitbook.io/infosec-notes/linux-server-administrator/linux-commands/string-processing.md).

# String Processing

## **`head`**: View the Beginning of a File

**Examples:**

* Display the first 10 lines of a file (default):

  ```bash
  head filename.txt
  ```
* Display the first N lines of a file:

  ```bash
  head -n 20 filename.txt
  ```
* Display the first N bytes of a file:

  ```bash
  head -c 100 filename.txt
  ```
* Display the first 10 lines of multiple files:

  ```bash
  head file1.txt file2.txt
  ```

***

## **`tail`**: View the End of a File

**Examples:**

* Display the last 10 lines of a file (default):

  ```bash
  tail filename.txt
  ```
* Display the last N lines of a file:

  ```bash
  tail -n 20 filename.txt
  ```
* Display the last N bytes of a file:

  ```bash
  tail -c 100 filename.txt
  ```
* Follow the output of a file in real-time (useful for log files):

  ```bash
  tail -f filename.txt
  ```
* Display the last 10 lines of multiple files:

  ```bash
  tail file1.txt file2.txt
  ```

***

## **`sort`**: Sort Lines of Text Files

**Examples:**

* Sort a file in ascending order:

  ```bash
  sort filename.txt
  ```
* Sort in descending order:

  ```bash
  sort -r filename.txt
  ```
* Sort by a specific column (e.g., column 2):

  ```bash
  sort -k 2 filename.txt
  ```
* Sort numerically (useful for numbers rather than strings):

  ```bash
  sort -n filename.txt
  ```
* Sort and remove duplicates:

  ```bash
  sort -u filename.txt
  ```
* Sort the contents of multiple files:

  ```bash
  sort file1.txt file2.txt
  ```

***

## **`wc`** : Count Words, Lines, and Characters in a File

**Examples:**

* Count lines, words, and characters in a file:

  ```bash
  wc file.txt
  ```
* Count only lines:

  ```bash
  wc -l file.txt
  ```
* Count only words:

  ```bash
  wc -w file.txt
  ```
* Count only characters:

  ```bash
  wc -c file.txt
  ```

***

## **`grep` : Search for Text in Files and Output**

`grep` (Global Regular Expression Print) used in Linux to search for patterns within files and text streams. It is commonly used for filtering logs, extracting specific data, and searching for keywords.

***

#### **Basic Syntax**

```bash
grep [OPTIONS] "pattern" filename
```

***

#### **Basic `grep` Examples**

* **Search for a word in a file:**

  ```bash
  grep "error" logfile.txt
  ```

  This will find and display all lines in `logfile.txt` containing the word "error".
* **Case-insensitive search:**

  ```bash
  grep -i "error" logfile.txt
  ```

  The `-i` option makes the search case-insensitive, so it matches "error", "Error", "ERROR", etc.
* **Search for a word in multiple files:**

  ```bash
  grep "warning" file1.txt file2.txt
  ```

  This searches for "warning" in both `file1.txt` and `file2.txt`.
* **Search recursively in directories:**

  ```bash
  grep -r "password" /etc/
  ```

  The `-r` (recursive) option searches for "password" in all files within `/etc/` and its subdirectories.
* **Search for a keyword in multiple files within a directory:**

  ```bash
  grep -r "TODO" /home/user/projects/
  ```

  This will search for "TODO" in all files within `/home/user/projects/`.

#### **Display Line Numbers**

* **Show matching lines with line numbers:**

  ```bash
  grep -n "error" logfile.txt
  ```

  This will display matching lines along with their line numbers.

#### **Invert Search (Show Non-Matching Lines)**

* **Display lines that do NOT match the pattern:**

  ```bash
  grep -v "success" logfile.txt
  ```

  The `-v` option inverts the match, showing lines that do not contain "success".

#### **Search for Whole Words Only**

* **Match whole words, avoiding partial matches:**

  ```bash
  grep -w "user" users.txt
  ```

  This ensures that only the exact word "user" is matched, not "username" or "superuser".

#### **Count the Number of Matches**

* **Count the occurrences of a pattern:**

  ```bash
  grep -c "fail" logfile.txt
  ```

  This will count and display the number of times "fail" appears in `logfile.txt`.

#### **Highlight Matches**

* **Enable colour highlighting for matches:**

  ```bash
  grep --color "error" logfile.txt
  ```

  This highlights the matched text for better visibility.

***

#### **Using Regular Expressions with `grep`**

#### **Match Lines Starting with a Pattern**

* **Find lines that start with "Error":**

  ```bash
  grep "^Error" logfile.txt
  ```

  The `^` symbol ensures that only lines starting with "Error" are matched.

#### **Match Lines Ending with a Pattern**

* **Find lines that end with "done":**

  ```bash
  grep "done$" logfile.txt
  ```

  The `$` symbol ensures that only lines ending with "done" are matched.

#### **Match Any Single Character**

* **Match lines containing "c.t", such as "cat" or "cut":**

  ```bash
  grep "c.t" words.txt
  ```

#### **Match Multiple Patterns**

* **Search for multiple words using `-e`:**

  ```bash
  grep -e "error" -e "failed" logfile.txt
  ```

  This will find lines containing either "error" or "failed".
* **Alternative method using `|` (OR operator):**

  ```bash
  grep "error\|failed" logfile.txt
  ```

#### **Match Digits**

* **Find lines containing numbers:**

  ```bash
  grep "[0-9]" logfile.txt
  ```

***

#### **Using `grep` with Other Commands**

#### **Filter Output with `grep`**

* **Find all running Apache processes:**

  ```bash
  ps aux | grep apache
  ```

  This filters the output of `ps aux`, displaying only lines that mention "apache".
* **Find all open network ports:**

  ```bash
  netstat -tulnp | grep LISTEN
  ```

#### **Search for a User in `/etc/passwd`**

* **Check if a user exists in the system:**

  ```bash
  grep "username" /etc/passwd
  ```

#### **Count Files in a Directory**

* **Find the number of `.txt` files in a directory:**

  ```bash
  ls -l | grep -c "\.txt"
  ```

***

## **`cut` : Command in Linux**

The `cut` command is used to extract specific fields(columns) of text from files or command output. It works by selecting columns, characters, or fields based on delimiters.

***

#### **1. Extract Specific Columns**

* **Extract the first column from a file (default delimiter is tab)**

  ```bash
  cut -f1 file.txt
  ```
* **Extract multiple columns (e.g., 1st and 3rd)**

  ```bash
  cut -f1,3 file.txt
  ```

***

#### **2. Extract Specific Characters**

* **Extract the first 5 characters from each line**

  ```bash
  cut -c1-5 file.txt
  ```
* **Extract characters from 5th to 10th position**

  ```bash
  cut -c5-10 file.txt
  ```

***

#### **3. Extract Fields Using a Delimiter**

* **Extract the first field from a comma-separated file**

  ```bash
  cut -d',' -f1 data.csv
  ```
* **Extract username from `/etc/passwd` (fields separated by `:`)**

  ```bash
  cut -d':' -f1 /etc/passwd
  ```
* **Extract user ID from `/etc/passwd`**

  ```bash
  cut -d':' -f3 /etc/passwd
  ```

***

## `awk`: Command in Linux

The `awk` command is a advance text-processing tool used for filtering and manipulating rows and columns in a file.

#### **Switches**

* `-F` → Specifies the delimiter.
* `-f` → Executes an AWK filter script file.

#### **Examples**

* **Print the first column using `:` as a delimiter:**

  ```bash
  awk -F: '{print $1}' filename
  ```
* **Print the first and fourth columns:**

  ```bash
  awk '{print $1, $4}' filename
  ```
* **Search for lines containing "root" and print the entire line:**

  ```bash
  awk '/root/ {print $0}' filename
  ```
* **Search for lines containing "root" and print the second field:**

  ```bash
  awk '/root/ {print $2}' filename
  ```
* **Print a header and footer while displaying all file contents:**

  ```bash
  awk 'BEGIN {print "Start File"} {print $0} END {print "End of File"}' filename
  ```
* **Run an AWK script from a file:**

  ```bash
  awk -f 'awk_script.awk' 'input_file'
  ```
* **Print line numbers along with each line:**

  ```bash
  awk '{print NR, $0}' filename
  ```
* **Print only lines where the third column is greater than 50:**

  ```bash
  awk '$3 > 50 {print $0}' filename
  ```
* **Calculate the sum of numbers in the second column:**

  ```bash
  awk '{sum += $2} END {print "Total:", sum}' filename
  ```
* **Replace "error" with "warning" in the output:**

  ```bash
  awk '{gsub(/error/, "warning"); print}' filename
  ```

***

## `sed`: Stream Editor

The `sed` (Stream Editor) command is used for modifying files by performing text replacements, deletions, insertions, and other transformations.

#### **Switches**

* `-i` → Saves the changes made.
* `-n` → Suppresses automatic printing of pattern space and prints only specified lines.
* `-e` → Allows running one or more `sed` commands.

#### **Common Commands**

* `a` → Append text after a line.
* `i` → Insert text before a line.
* `g` → Perform a global replacement.
* `p` → Print the modified output.
* `e` → Execute a shell command.
* `c` → Change the specified line(s).

#### **Examples**

* **Remove all double quotes from a line:**

  ```bash
  sed 's/"//g' filename
  ```
* **Replace "foo" with "bar" in the entire file:**

  ```bash
  sed 's/foo/bar/g' filename
  ```
* **Delete all lines containing "error":**

  ```bash
  sed '/error/d' filename
  ```
* **Insert "Header Line" at the beginning of the file:**

  ```bash
  sed -i '1i Header Line' filename
  ```
* **Append "End of File" at the last line:**

  ```bash
  sed -i '$a End of File' filename
  ```
* **Print only lines that contain "root":**

  ```bash
  sed -n '/root/p' filename
  ```
* **Replace "hello" with "hi" only on the second line:**

  ```bash
  sed '2s/hello/hi/' filename
  ```
* **Delete lines from 3 to 5 in a file:**

  ```bash
  sed '3,5d' filename
  ```
* **Replace only the first occurrence of "test" in each line:**

  ```bash
  sed 's/test/example/' filename
  ```
* **Execute a shell command inside `sed` to replace text dynamically:**

  ```bash
  sed -e 's/TIME/'"$(date)"'/g' filename
  ```