rg fails to find matches on Github Actions?

[roy-work]

Hi,

I have a CI job that I'm migrating from a rather unknown CI system named "Drone" to Github Actions.

This job runs shellcheck (a popular shell linter) against our shell. In order to find shell, among other things, we look for shebangs with rg. This works fine on Drone, but matches nothing on Github.

Here's what it looks like in our CI process on Actions; I'm going to intersperse this with some commentary:

+ printf ── cat sample file ───────────────\n
+ cat tools/bob/foo.sh
── cat sample file ───────────────
#!/bin/bash
az account set --subscription a-uuid-that-is-snipped

So, here at tools/bob/foo.sh, we have a shell script that, when I run rg, I will expect to be in the matches to rg. It isn't in a hidden dir, I do not believe it is covered by .gitignore (but I don't think that will matter, as we'll be passing -u to rg) and I don't think it is binary. So, I think rg should consider the file (I do not think rg does), and if considered, the file should match.

It isn't the only file that should match, I'm merely attempting to demonstrate that at least one file should match.

Let's run rg:

+ printf ── rg -uuu ───────────────\n
── rg -uuu ───────────────
+ rg -l -uuu -m 1 #!/bin/(bash|ash|sh) --debug
DEBUG|grep_regex::literal|/usr/share/cargo/registry/ripgrep-12.1.1/debian/cargo_registry/grep-regex-0.1.8/src/literal.rs:58: literal prefixes detected: Literals { lits: [Complete(#!/bin/bash), Complete(#!/bin/ash), Complete(#!/bin/sh)], limit_size: 250, limit_class: 10 }
DEBUG|globset|/usr/share/cargo/registry/ripgrep-12.1.1/debian/cargo_registry/globset-0.4.5/src/lib.rs:431: built glob set; 0 literals, 0 basenames, 12 extensions, 0 prefixes, 0 suffixes, 0 required extensions, 0 regexes
+ true

Here we've passed -uuu; this was a (vain) attempt to figure out if rg wasn't considering the file b/c of .gitignore or something. (But I don't know why that would be different between our two CI systems.) We've also passed --debug. If I run rg locally, it normally outputs much more (and finds many matches within our repository).

According to man rg,

rg -uuu is roughly equivalent to grep -r.

Okay then:

+ printf ── grep -r ───────────────\n
── grep -r ───────────────
+ grep -l -r <the *, expanded> -e #!/bin/\(bash\|ash\|sh\)
<many, many matched files>
tools/bob/foo.sh
<more matches>

I also tried strace'ing rg on Actions, and it just wasn't enlightening. The only thing that stands out to me is this call,

statx(AT_FDCWD, "-", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffcf6567c40) = -1 ENOENT (No such file or directory)

which occurs right between the two debug logs. This does seem like a odd call, and it made me double check that I didn't have like a random - in the command.

When I run rg locally, as near as I can tell, there it does for me,

statx(AT_FDCWD, "./", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=2272, ...}) = 0

But my local strace is pretty different from the one I get in Actions. (Much of it seems to be that, locally, FDs 0/1/2 are a terminal.)

For reference:

rg --version
ripgrep 12.1.1
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

We run things in containers, so the exact same rg binary is running on these two CI systems, which further perplexes me that it is behaving differently. It must be something about the state of the contain (e.g., the mounted source directory could be different between the two) but I've not successfully narrowed down what the operative differences is, yet. I have a feeling that if I knew, this'd be simple.

[BurntSushi]

What happens if you run rg -l -uuu -m 1 '#!/bin/(bash|ash|sh)' ./? That is, tell ripgrep to search the CWD. My guess is that ripgrep is somehow being tricked into searching stdin. It'd probably be good to make that clearer somehow in the --debug output.

If that doesn't work, can you try running rg with --trace instead of --debug and show the output here?

The "file not found" error for - is indeed a little perplexing, but also possibly hints at the stdin detection problem as well.

[roy-work]

Alright; I think this is pieced together. I had thought both Actions & Drone used pipes, and they do, for their outputs, it's for inputs that they differ. I'd never thought about it, b/c what is stdin in CI seems like an odd question.

For Drone, stdin is /dev/null; for Actions it is a pipe. I have no idea what's on the pipe. (I presume the write end was closed, but that's a weird choice on Github's part…)

"is stdin a pipe" is really getting into implementation specifics.

Yes of course; was still trying to figure out why it differs on Actions.

The money bit is […]

Yep, that answers it: /dev/null does not meet that criteria. (For anyone reading this and wondering why: /dev/null is a character device, which is neither a normal file (is_file), nor a pipe nor socket.)

Of course it does. Otherwise things like some-command | rg foo wouldn't work. Stdin is actually the most fundamental interface of a grep tool.

My point was not that rg can't do something like that (nor that some tools don't do stdin detection), just that in the case of rg, I don't know what behavior to expect, piping something into it. I would have expected to have needed to say something like rg --use-file-for-search-paths - or something. I.e., certainly rg can read stdin, but I don't know what it would use the data on it for.

I guess if you expect it to act like grep, that makes more sense. (And I … guess I don't. Although I do replace some use cases of grep with rg, I guess I don't think of it as replacing the <data> | grep use case of grep, rather only the $ grep -r * -e … cases.)

[BurntSushi]

but I don't know what it would use the data on it for

Ooooh, I see. Interesting. Sometimes I just kind of assume that everyone has used grep before. (Oh, I now see that even that is wrong here. It's that you didn't see ripgrep as a grep itself.)

Interestingly, you can actually put the patterns you want to search for on stdin and tell ripgrep to read the patterns from it:

$ echo foo > 2241.txt
$ echo foo | rg -f- 2241.txt
1:foo

(There is also an open feature request for giving ripgrep a file containing a list of paths to search. Which presumably would also support stdin in the same way that -f/--file does.)

I guess if you expect it to act like grep, that makes more sense. (And I … guess I don't. Although I do replace some use cases of grep with rg, I guess I don't think of it as replacing the | grep use case of grep, rather only the $ grep -r * -e … cases.)

Yes, when I built ripgrep, I very specifically went out of my way to make sure it worked in shell pipelines well. The only places where I tend to still use grep today are in shell scripts I share with others, because I don't want to assume they'll have ripgrep installed. Otherwise, I use rg everywhere I used to use grep.

for Actions it is a pipe. I have no idea what's on the pipe. (I presume the write end was closed, but that's a weird choice on Github's part…)

Strange indeed. Other environments/tools do similarly strange things. See #1892 for example.

[roy-work]

Oh, I now see that even that is wrong here. It's that you didn't see ripgrep as a grep itself.

Yeah, exactly.

Yes, when I built ripgrep, I very specifically went out of my way to make sure it worked in shell pipelines well.

Yeah, TIL for me … I had never thought of rg that way, though in hindsight, I don't really know why. For all I know is that it is because when I adopted it, "look how fast it searches the entire codebase" was for me, the first and killer application of it. Now, having had this discussion, I should remember to force my fingers to try to use rg in a <stuff> | grep pipeline next time I need to do that. At the very least, rg's regex syntax is much less annoying than grep's.

I should also say: thank you very much for you help in narrowing down where things were going wrong! We were perplexed that something so seemingly simple could somehow go wrong, and knew it had to be caused by something in the environments. rg is a wonderful tool, thank you for building it!

[BurntSushi]

No problem! And yeah, this is unfortunately one of ripgrep's worst failure modes. :-(

[c-git]

THANK YOU VERY MUCH for this tool. I really love it.

Please add a note to the Readme about this. I just spent a few hours trying to figure out what I was doing wrong. I completely understand now after reading this. But I had no idea why it wasn't working. I read the readme a few times looking for help but there wasn't anything there to help.