help-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Storing grep results in variable


From: Greg Wooledge
Subject: Re: Storing grep results in variable
Date: Fri, 6 Jun 2025 09:53:28 -0400
User-agent: Mutt/1.10.1 (2018-07-13)

On Fri, Jun 06, 2025 at 06:06:05 +0000, Heime via wrote:
> Can I improve this?
> 
> mysearch ()
>  {
>    local search_dir=$1
>    local search_str=$2
> 
>    if [[ -z "$search_dir" || -z "$search_str" ]]; then
>      echo "Usage: mysearch <directory> <search_string>"
>      return 1
>    fi
> 
>    find "$search_dir" -type f | while IFS= read -r file; do
>      matches=$(grep -n -- "$search_str" "$file" | sed 's/:/: /1' )
>      if [[ -n "$matches" ]]; then
>        echo "$file"
>        echo "$matches"
>        echo
>      fi
>    done
>  }

Storing a list in a string variable is never ideal.  You want to store
lists in array variables.  That's the first and most important improvement
I would recommend.

After that, the primary issue is how you separate the filenames from
each other.  Except... now that I've read your code a couple of times,
it appears you aren't storing just filenames.  You appear to be storing
filename/line number pairs, with the possibility of multiple matches
per file.  So, now we're back to the question of how to store these
things in a useful way.  The answer to that is going to depend primarily
on what you intend to *do* with these pairs, and you didn't tell us that.

The most obvious way (to me) to store these pairs would use two arrays.
Both arrays would be indexed by a "match number" which exists only to
tie the two results together.  The first array would store the filename,
and the second, the line number.

So then the problem becomes how to parse out the line number and filename
from each result.  The naive approach which you appear to be using right
now uses grep's line-oriented output, where the pairs are separated by
newlines.  This will break down as soon as you encounter a filename with
a newline in it.

hobbit:/tmp/x$ touch z 's pace' $'t\tab' $'new\nline'
hobbit:/tmp/x$ ls -l
total 0
-rw-r--r-- 1 greg greg 0 Jun  6 09:15 'new'$'\n''line'
-rw-r--r-- 1 greg greg 0 Jun  6 09:15 's pace'
-rw-r--r-- 1 greg greg 0 Jun  6 09:15 't'$'\t''ab'
-rw-r--r-- 1 greg greg 0 Jun  6 09:15  z
hobbit:/tmp/x$ for f in *; do echo payload >> "$f"; done
hobbit:/tmp/x$ grep -n p *
new
line:1:payload
s pace:1:payload
t       ab:1:payload
z:1:payload

Obviously, that's not viable.  However, if you're writing this function
to be used only on GNU/Linux systems, you can use GNU grep's --null
option to produce a stream with NULs in it.  Normally, this would be
used to produce a parseable list of filenames.  However, your desire
to include matching line numbers complicates this quite a bit.

hobbit:/tmp/x$ grep --null -n p * | hd
00000000  6e 65 77 0a 6c 69 6e 65  00 31 3a 70 61 79 6c 6f  |new.line.1:paylo|
00000010  61 64 0a 73 20 70 61 63  65 00 31 3a 70 61 79 6c  |ad.s pace.1:payl|
00000020  6f 61 64 0a 74 09 61 62  00 31 3a 70 61 79 6c 6f  |oad.t.ab.1:paylo|
00000030  61 64 0a 7a 00 31 3a 70  61 79 6c 6f 61 64 0a     |ad.z.1:payload.|
0000003f

As far as I can tell, what grep is writing here is:

    File name
    NUL
    Line number
    Colon
    Matching line text
    Newline

This is repeated for each match, and the filename may contain a newline
character.  Therefore, we can't just split the output on NULs or on
newlines.  We have to use an iterative approach -- read until the
next NUL, then read until the next newline, and repeat.

Also, just for the record, be warned that filenames may contain bytes
which are not valid characters in your locale.  Several versions of
bash have bugs which crop up when reading filenames containing invalid
partial characters.  The workaround for that is to override the locale
variable(s).[1]

Since we're using GNU grep, we can also use its -r option to recurse
for us, and drop the unnecessary find command.

So, in order to parse this into our two arrays, we would need something
like this:

files=()
lines=()
i=-1
while IFS= LC_ALL=C read -r -d '' file; do
    IFS=: read -r num text
    files[++i]=$file
    lines[i]=$num
done < <(grep --null -r -n -- "$search_str" "$seach_dir")

The "text" variable is unused, so we don't need to worry about the issues
with using multiple variables with read.[2]

This should handle any filename, in any locale, in any version of bash
that has indexed arrays and read -d ''.  It looks like the minimum bash
version for those is 2.04.  I don't know the minimum required version
of GNU grep to have the --null feature; you might need to research that.

If your needs aren't exactly met by this, then please feel free to
explain more precisely what you need.

[1] https://mywiki.wooledge.org/BashPitfalls#pf65
[2] https://mywiki.wooledge.org/BashPitfalls#pf47



reply via email to

[Prev in Thread] Current Thread [Next in Thread]