How to Loop Through Files in Directory in Bash

Quick Answer: How to Loop Through Files

Use a for loop with a glob pattern: for file in /path/*.txt; do [ -f "$file" ] || continue; process "$file"; done. For large directories or complex filtering, use find with a pipe: find /path -type f -name "*.txt" | while read -r file; do process "$file"; done.

Quick Comparison: File Looping Methods

Method	Speed	Memory	Best For
for loop + glob	Very fast	Low	Small directories, simple patterns
for loop + find	Medium	High	Complex patterns, complex filtering
find | while read	Very fast	Low	Large directories
find -exec	Fast	Low	Simple operations
find -print0 | xargs	Very fast	Low	Filenames with special characters

Bottom line: Use for file in *.txt for small directories. Use find | while read for large directories or complex needs.

---

Loop through files efficiently in Bash using various methods. Whether you’re processing specific file types, handling large directories, or applying complex filtering, understanding different looping techniques is essential for file automation scripts.

Method 1: For Loop with Glob Pattern

The simplest and fastest method for small directories. Bash glob patterns expand to match filenames, and the for loop processes each one. This is perfect for quick scripts where you know you’re working with a manageable number of files.

# Loop through all files in directory
for file in /path/to/directory/*; do
  [ -f "$file" ] || continue  # Skip non-files (directories, symlinks)
  echo "Processing: $file"
done

# Loop through files in current directory
for file in *; do
  [ -f "$file" ] || continue
  echo "File: $file"
done

# Loop through specific pattern
for file in *.txt; do
  [ -f "$file" ] || continue
  echo "Found: $file"
done

The [ -f "$file" ] || continue line checks if each item is actually a file. Without it, glob patterns might match directories too, and symlinks might behave unexpectedly. The continue statement skips non-files and moves to the next iteration. Always quote "$file" to handle filenames with spaces or special characters correctly.

Example:

$ for file in *.txt; do
>   [ -f "$file" ] || continue
>   echo "Processing: $file"
> done
Processing: file1.txt
Processing: file2.txt
Processing: notes.txt

When to use this method:

• Small to medium-sized directories (under 1000 files)
• You know the pattern you’re looking for (like *.txt)
• You want the fastest code for simple operations
• Memory usage doesn’t matter

Method 2: For Loop with Specific Patterns

Target specific file types or naming patterns.

# Loop only .txt files
for file in *.txt; do
  [ -f "$file" ] || continue
  echo "Text file: $file"
done

# Loop .log files
for file in *.log; do
  [ -f "$file" ] || continue
  echo "Log file: $file"
done

# Multiple patterns with brace expansion
for file in *.{txt,log,csv}; do
  [ -f "$file" ] || continue
  echo "Found: $file"
done

# Hidden files
for file in .*; do
  [ -f "$file" ] || continue
  echo "Hidden: $file"
done

Method 3: Using find with Pipe

This method processes files one at a time and is perfect for large directories. find searches disk as it goes, so it doesn’t load all filenames into memory at once. This is the most memory-efficient approach for large directories.

# Find and process
find /path/to/directory -type f -name "*.log" | while read -r file; do
  echo "Processing: $file"
done

# Find multiple patterns
find /path -type f \( -name "*.txt" -o -name "*.log" \) | while read -r file; do
  echo "Found: $file"
done

# Find modified in last 7 days
find /path -type f -mtime -7 | while read -r file; do
  echo "Recent: $file"
done

The -type f ensures we only get files (not directories). The pipe (|) feeds each filename to the while loop, which reads one at a time with -r (raw input, prevents backslash interpretation). This approach is incredibly flexible—find can filter by name, size, modification time, permissions, and more.

Example:

$ find . -type f -name "*.txt" | while read -r file; do
>   echo "Processing: $file"
> done
Processing: ./document.txt
Processing: ./notes.txt

When to use this method:

• Large directories (thousands of files)
• Complex filtering needs (size, age, permissions, etc.)
• Memory efficiency is critical
• You need recursive searching in subdirectories

One gotcha: pipes create subshells in Bash, so variables assigned inside the while loop won’t exist after it finishes. If you need to collect results, use process substitution instead: while read file; do ...; done < <(find...)

Method 4: Using find with -exec

Execute a command directly on each found file.

# Process with -exec
find /path/to/directory -type f -exec echo {} \;

# Execute script on each file
find /path -type f -exec process_file.sh {} \;

# Multiple operations
find /path -type f -exec chmod 644 {} \;

# Safer execution with +
find /path -type f -exec process {} +

Method 5: Using find with -print0

Handle filenames with spaces or special characters safely.

# Process files with spaces in names
find /path -type f -print0 | xargs -0 -I {} echo "Processing: {}"

# More efficient with xargs
find /path -type f -name "*.txt" -print0 | xargs -0 cat

# Count lines in multiple files
find /path -name "*.py" -print0 | xargs -0 wc -l

Practical Examples

Example 1: Process and Rename Files

#!/bin/bash

directory="${1:-.}"

for file in "$directory"/*.txt; do
  [ -f "$file" ] || continue

  # Extract filename
  filename=$(basename "$file")

  # Remove .txt and add .bak
  newname="${filename%.txt}.bak"
  newpath="$directory/$newname"

  mv "$file" "$newpath"
  echo "Renamed: $filename -> $newname"
done

Usage:

$ bash script.sh /path/to/files
Renamed: file1.txt -> file1.bak
Renamed: file2.txt -> file2.bak

Example 2: Count Lines in Multiple Files

#!/bin/bash

directory="${1:-.}"
total_lines=0
file_count=0

for file in "$directory"/*.txt; do
  [ -f "$file" ] || continue

  lines=$(wc -l < "$file")
  total_lines=$((total_lines + lines))
  ((file_count++))

  echo "$(basename "$file"): $lines lines"
done

echo "---"
echo "Total files: $file_count"
echo "Total lines: $total_lines"

Output:

file1.txt: 42 lines
file2.txt: 58 lines
file3.txt: 15 lines
---
Total files: 3
Total lines: 115

Example 3: Process Large Directory

#!/bin/bash

# Process large directory using find for memory efficiency
directory="/large/directory"
processed=0

find "$directory" -type f -name "*.log" | while read -r file; do
  ((processed++))

  # Show progress
  if [ $((processed % 100)) -eq 0 ]; then
    echo "Processed: $processed files"
  fi

  # Process file
  process_log "$file"
done

echo "Completed processing"

Example 4: Archive Old Files

#!/bin/bash

# Find and archive files older than 30 days
archive_dir="./archive"
mkdir -p "$archive_dir"

find .  -type f -mtime +30 | while read -r file; do
  # Get filename
  filename=$(basename "$file")

  # Move to archive
  mv "$file" "$archive_dir/$filename"
  echo "Archived: $filename"
done

Example 5: Validate File Contents

#!/bin/bash

# Process files and validate
for file in *.json; do
  [ -f "$file" ] || continue

  # Validate JSON
  if jq empty "$file" 2>/dev/null; then
    echo "✓ Valid: $file"
  else
    echo "✗ Invalid: $file"
  fi
done

Output:

✓ Valid: config.json
✗ Invalid: broken.json
✓ Valid: settings.json

Example 6: Process with Conditions

#!/bin/bash

# Process only readable files over 1MB
for file in *; do
  [ -f "$file" ] || continue           # Must be file
  [ -r "$file" ] || continue           # Must be readable
  [ -s "$file" ] || continue           # Must not be empty
  [ $(stat -c%s "$file") -gt 1048576 ] || continue  # Over 1MB

  echo "Processing large file: $file"
done

Example 7: Function for File Processing

#!/bin/bash

# Reusable function
process_files() {
  local pattern="$1"
  local operation="$2"

  for file in $pattern; do
    [ -f "$file" ] || continue

    case "$operation" in
      count)
        lines=$(wc -l < "$file")
        echo "$file: $lines lines"
        ;;
      size)
        size=$(stat -c%s "$file")
        echo "$file: $size bytes"
        ;;
      backup)
        cp "$file" "$file.bak"
        echo "Backed up: $file"
        ;;
      *)
        echo "Unknown operation"
        ;;
    esac
  done
}

# Usage
process_files "*.txt" "count"
process_files "*.txt" "backup"

Performance Comparison

For looping through files:

Method	Speed	Memory	Best For
for file in *	Very Fast	Low	Small directories
for file in $(find...)	Medium	High	Complex patterns
find | while read	Very Fast	Low	Large directories
find -exec	Fast	Low	Simple operations

Best choice: Use for file in * for small directories, find | while read for large ones.

Important Considerations

Always Verify File Type

# Check it's actually a file
[ -f "$file" ] || continue

# Check it's readable
[ -r "$file" ] || continue

# Check it's not empty
[ -s "$file" ] || continue

Handling Spaces in Filenames

Quote variables to handle spaces:

# Correct: quotes around variable
for file in *; do
  [ -f "$file" ] || continue
  echo "File: $file"
done

# Wrong: unquoted variable
for file in *; do
  echo "$file"  # Breaks if filename has spaces
done

Recursive File Processing

# Process all files recursively
find /path -type f | while read -r file; do
  echo "Processing: $file"
done

# Or with globstar (Bash 4+)
shopt -s globstar
for file in **/; do
  echo "$file"
done

Avoiding Subshell Issues

With pipes, remember variable assignments don’t persist:

# Variables in subshell are lost
find . -type f | while read file; do
  ((count++))  # This won't persist outside loop
done
echo $count    # Will be empty

# Solution: Use process substitution
while read file; do
  ((count++))
done < <(find . -type f)
echo $count    # Now this works

Key Points

• Use for file in pattern for small directories
• Use find | while read for large directories or complex filtering
• Always verify files with [ -f "$file" ]
• Quote variables: "$file" to handle spaces
• Use find -print0 with xargs -0 for special characters
• Remember pipes create subshells (variables won’t persist)
• Use process substitution < <(find...) to avoid subshell issues

Quick Reference

# Simple loop
for file in *.txt; do
  [ -f "$file" ] || continue
  echo "$file"
done

# With find
find . -type f -name "*.txt" | while read -r file; do
  echo "$file"
done

# Count files
count=$(find . -type f | wc -l)

# Process recursively
find . -type f -exec process {} \;

# Handle special characters
find . -type f -print0 | xargs -0 wc -l

Recommended Pattern

#!/bin/bash

directory="${1:-.}"

# For simple file processing
for file in "$directory"/*.txt; do
  [ -f "$file" ] || continue

  # Process file
  echo "Processing: $file"
done

# For large directories
find "$directory" -type f -name "*.log" | while read -r file; do
  # Process file
  echo "Processing: $file"
done