How to Merge Lines in Bash | ActiveDirectoryTools

Quick Answer: Merge Lines in Bash

To merge multiple lines into one in Bash, use tr: tr '\n' ' ' < file. Alternatively, use paste: paste -sd' ' file to join all lines with a space. For conditional merging, use sed or awk.

Quick Comparison: Line Merging Methods

Method	Syntax	Best For	Separator
tr	`tr '\n' ' '`	Simple merge	Flexible
paste -sd	`paste -sd' '`	All lines	Flexible
sed	`sed 'N;s/\n/ /'`	Pairs	Flexible
awk	`awk '{printf "%s", NR==1 ? $0 : " " $0}'`	Complex	Flexible
xargs	`xargs -J% %`	Compact	Single space

Bottom line: Use tr for simplicity, use awk for complex conditions.

Merging multiple lines into single lines is a common text processing task. Whether you’re combining CSV records, joining array elements, or consolidating log entries, understanding different merge techniques helps you process data efficiently.

Method 1: Using tr to Remove Newlines

The tr command is the simplest way to replace newlines with another character.

# Replace newlines with space
tr '\n' ' ' < file.txt

# Replace with comma
tr '\n' ',' < file.txt

# Replace with pipe
tr '\n' '|' < file.txt

# Remove newlines entirely
tr -d '\n' < file.txt

Example with sample file:

# Input file (input.txt)
apple
banana
cherry

# Command:
tr '\n' ',' < input.txt

# Output:
apple,banana,cherry,

Method 2: Using paste Command

The paste command merges lines from files with a specified delimiter.

# Merge every 2 lines with space
paste -d' ' - - < file.txt

# Merge every 3 lines with comma
paste -d',' - - - < file.txt

# Merge with pipe as delimiter
paste -d'|' - - < file.txt

# Merge entire file into one line
paste -sd',' < file.txt

Example:

# Input file
1
2
3
4
5
6

# Merge every 2 lines
paste -d' ' - - < input.txt

# Output:
1 2
3 4
5 6

# Merge all into one line
paste -sd',' < input.txt

# Output:
1,2,3,4,5,6

Method 3: Using sed

The sed command provides powerful line merging with pattern matching.

# Join pairs of lines
sed 'N;s/\n/ /' file.txt

# Join all lines
sed ':a;N;$!ba;s/\n/ /g' file.txt

# Join with custom separator (uses substitute)
sed ':a;N;$!ba;s/\n/,/g' file.txt

# Join lines that match a pattern
sed '/pattern/{N;s/\n/ /;}' file.txt

Example with sed:

# Input:
line1
line2
line3
line4

# Command: sed 'N;s/\n/ /' file.txt
# Output:
line1 line2
line3 line4

# Command: sed ':a;N;$!ba;s/\n/ /g' file.txt
# Output:
line1 line2 line3 line4

Method 4: Using awk

awk provides flexible line merging with custom processing.

# Merge all lines
awk '{printf "%s ", $0} END {print ""}' file.txt

# Merge with custom separator
awk '{printf "%s,", $0} END {print ""}' file.txt

# Merge specific number of lines
awk 'NR % 3 == 1 { row=$0; next } { row = row "," $0 } END { print row }' file.txt

# Merge and remove empty lines
awk 'NF { if (row) row = row " " $0; else row = $0 } END { print row }' file.txt

Example:

# Command:
awk '{printf "%s|", $0} END {print ""}' file.txt

# Output:
apple|banana|cherry|date|

Practical Examples

Example 1: Merge CSV Records

#!/bin/bash

# Merge multi-line CSV records into single lines
input_file="$1"
output_file="${input_file%.csv}_merged.csv"

# Example: Merge continuations (lines without leading quotes to previous line)
awk '
  /^"/ {
    if (row) print row;
    row=$0;
    next
  }
  {
    row = row " " $0
  }
  END {
    if (row) print row
  }
' "$input_file" > "$output_file"

echo "Merged file created: $output_file"

Example 2: Combine Array Elements

#!/bin/bash

# Merge array elements into single line
items=("apple" "banana" "cherry" "date")

# Method 1: Using printf
echo "${items[@]}" | tr ' ' ','
# Output: apple,banana,cherry,date

# Method 2: Using awk
printf '%s\n' "${items[@]}" | awk '{printf "%s,", $0} END {print ""}'
# Output: apple,banana,cherry,date

# Method 3: Using IFS
IFS=',' echo "${items[*]}"
# Output: apple,banana,cherry,date

Output:

apple,banana,cherry,date
apple,banana,cherry,date
apple,banana,cherry,date

Example 3: Join Split Log Lines

#!/bin/bash

# Merge continuation lines in log file (lines starting with space)
log_file="$1"

sed ':a;/^ /{N;s/\n/ /;ba;}' "$log_file"

Example usage:

# Input (continuation_log.txt)
[2024-12-25 10:15] Error in processing
 Continued error details
 More error information
[2024-12-25 10:16] Success

# Output:
[2024-12-25 10:15] Error in processing Continued error details More error information
[2024-12-25 10:16] Success

Example 4: Convert Multi-line Config to Single Line

#!/bin/bash

# Convert multi-line configuration format
config_file="$1"

# Input format:
# interface eth0
#   ip address 192.168.1.1
#   netmask 255.255.255.0
# interface eth1
#   ip address 192.168.2.1

sed ':a;/^ /{N;s/\n//;ba;}' "$config_file" | sed 's/^  / /g'

# Output: single line per interface

Example 5: Merge Query Results

#!/bin/bash

# Merge database query results from multiple lines to single line
database_output="$1"

# Convert multi-line results to CSV
sed ':a;N;$!ba;s/\n/,/g' "$database_output"

# Or with specific handling:
awk '
  NR == 1 { row = $0; next }
  { row = row "," $0 }
  END { print row }
' "$database_output"

Example 6: Function to Merge Lines

#!/bin/bash

# Reusable function for merging lines
merge_lines() {
  local input_file="$1"
  local separator="${2:- }"  # Default space separator
  local method="${3:-tr}"     # Method: tr, paste, sed, or awk

  case "$method" in
    tr)
      tr '\n' "$separator" < "$input_file"
      ;;
    paste)
      paste -sd"$separator" < "$input_file"
      ;;
    sed)
      sed ":a;N;\$!ba;s/\n/$separator/g" < "$input_file"
      ;;
    awk)
      awk -v sep="$separator" '{printf "%s%s", $0, sep} END {print ""}' < "$input_file"
      ;;
    *)
      echo "Unknown method: $method"
      return 1
      ;;
  esac
}

# Usage
merge_lines "file.txt" "," "awk"

Example 7: Conditional Line Merging

#!/bin/bash

# Merge lines based on conditions
# Example: Merge lines that are continuations of previous

file="$1"

awk '
  /^[A-Z]/ {
    if (row) print row
    row = $0
    next
  }
  {
    row = row " " $0
  }
  END {
    if (row) print row
  }
' "$file"

Performance Comparison

For merging lines in large files:

Method	Speed	Memory	Best For
tr	Fastest	Low	Simple newline replacement
paste -sd	Very Fast	Low	Entire file merging
sed	Fast	Low	Piping, pattern-based
awk	Fast	Medium	Complex merging logic

Best choice: Use paste -sd',' for simplicity, sed for complex patterns.

Important Considerations

Avoiding Extra Delimiters

When merging, the last line often has an extra delimiter:

# Remove trailing delimiter
tr '\n' ',' < file.txt | sed 's/,$//'

# Or using awk
awk '{printf "%s%s", sep, $0; sep=","} END {print ""}' file.txt

Preserving Whitespace

Some methods may trim whitespace:

# Preserve all whitespace
paste -d' ' - - < file.txt

# Or with awk
awk '{printf "%s ", $0} END {print ""}' file.txt

Large Files

For very large files, avoid loading entire content:

# Streaming approach with sed
sed 'N;s/\n/ /' largefile.txt

# Or paste for two at a time
paste -d' ' - - < largefile.txt

Key Points

Use tr '\n' ',' for simple newline replacement
Use paste -sd ',' for merging entire files
Use sed for pattern-based complex merging
Use awk for conditional merging logic
Remove trailing delimiters when needed
Consider performance with large files
Remember that paste works with pairs/triplets with - - -

Quick Reference

# Merge all lines into one with space
paste -sd' ' file.txt

# Merge all lines with comma
tr '\n' ',' < file.txt | sed 's/,$//'

# Merge with awk
awk '{printf "%s,", $0} END {print ""}' file.txt

# Merge every 2 lines
paste -d' ' - - < file.txt

# Merge with custom logic
sed ':a;N;$!ba;s/\n/ /g' file.txt

Recommended Pattern

#!/bin/bash

# For merging entire file into one line
paste -sd',' < input.txt > output.txt

# For merging pairs of lines
paste -d' ' - - < input.txt > output.txt

# For conditional merging with pattern
sed ':a;N;$!ba;s/\n/,/g' input.txt > output.txt