Hey all, I'm having trouble finding this one in Google and I'm confident that someone here can help me. I'm interested in the redirection syntax in a Bash script.

I'm using a variation on the first answer in this stackOverflow article, to parse multiple lines out of a piece of JSON that I got from an API: https://stackoverflow.com/questions/38364261/parse-json-to-array-in-a-shell-script

It uses a syntax like this:



My question is about this part of the script specifically:



I'm not sure why this redirection is two chevrons separated by a space instead of a single chevron or a double one without a space. I'm having trouble finding this particular syntax explained in my google searches.

The reason I need to know is, when I run my script at an SSH prompt on my Synology, this works and gets me a correctly-populated array that I can iterate through. But when I run the same script in the Synology Task Scheduler, I get the following error:



So my goal here is to understand the syntax, and therefore possibly understand why it works in one situation but doesn't work in the other. I've tried changing the syntax with various combinations of other stuff and I can't get it to work at the SSH prompt any other way (at least not so far anyway).

For reference, here is the full unedited code line from my code. All of the parsing statements (the sed/grep/cut stuff) is proven to work in all cases because I do it elsewhere in the program too. The difference with this line is the redirecting to the readarray command, that's the part that doesn't work in the Synology Task Scheduler.



So what's going wrong with that redirection which is working at the SSH prompt but failing in the Synology Task Scheduler?

Thanks for any tips and education you can offer!

I've fixed my problem by changing the syntax to the following:



This is more in line with how I expect things to work with regard to redirection, and this one seems to work both at the SSH prompt and from within Synology Task Scheduler.

What I don't understand is the syntax behind the other one, and why it didn't work in the Synology Task Scheduler (but worked at the SSH prompt).

For JSON, use jq, which is apparently installed on both of my Synology boxes already, so I'm assuming it's standard.

I'm going to guess that the Task Scheduler doesn't use /bin/bash, but uses /bin/sh, which is bash, but probably in a sh-compatible mode, with advanced bash features turned off.

Stop putting big hunks of script in the Task Scheduler dialog: put them in a script file (which can be properly versioned) and just call the script from Task Scheduler.

If you do that, you can use a bash shebang and get bash behaviour.

Thanks so much!

Agreed that I should be using jq for parsing Json. I had already read about that (and I think one of the threads that I linked also said the same thing). However I was initially composing and pre-testing the script on a system which didn't have jq installed, and I wasn't sure if the Synology would have it installed or not. Since I didn't have access to the Synolgy to check for sure at the time, I kind of went with the safer option just in case maybe jq isn't widely available and I shouldn't be counting on it. So I went with some plain-text parsing which, in my defense, worked perfectly for my particular situation. In any case, that wasn't related to the error.

Regarding the difference between bin/bash and bin/sh - It certainly sounds like a plausible explanation for the problem I saw, but I was already running it as a script file with "#!/bin/bash" at the top of the script. Not sure if the Synology Task Scheduler honors that or not.

So, assuming that the syntax was correct to begin with, what does the "< <( )" syntax mean in Bash? Just want to understand it. I've seen so many different weird kinds of ways to redirect output in Bash and it's hard to find explanations for any of them. For instance, in my final solution I used "<<<" because I saw that in a different example and it happened to work, but I'm not sure what that means either.

My understanding of redirects is from DOS of course, where a single chevron means redirect and overwrite, and two chevrons means redirect and append, and that's more or less it. I'm not understanding the spaces and triple chevrons.

Thanks!

Oh hey. Now that I think about it, all the scripts that I'm running in the Synology Task Manager are being launched like this (because I was following an example I saw from Google search results):

sh "/volume1/homes/admin/ScriptName.sh"

Is it the "sh" there that's the problem? Should that have been "bash" there?

Yes.

Or just simply: /volume1/homes/admin/ScriptName.sh
(assuming you have marked the script with executable permissions).

See? I'm learning.

Process substitution.

So "foo < <(someCommands)" means, first run "someCommands" in a subshell, capture the output in a temporary file, and then substitute the name of that temporary file back into the original command and run that. So in this case it would be equivalent to:

someCommands > tempFile
foo < tempFile
rm tempFile

Nine times out of ten you'd just write that as "someCommands | foo"... but using a pipe means executing foo in a subshell, and in the specific case of "foo" being readarray or read, you need the readarray or read to execute in the current shell, not a subshell, so that subsequent commands in the same shell can use the shell-variables it sets up.

Peter

Some detail: the "shebang" at the top of the script (#!/bin/bash) tells the kernel which program to use to load the script. Other tags ("MZ"+"PE", "\x7FELF", etc.) are treated in a similar way by the kernel loader.

But by using "sh script.sh" you told the kernel that you knew what you were doing and loaded the script with "sh". Which ignores the shebang, because it's just a comment.

Thank you so much, Peter and Roger, for those excellent and clear explanations.

In particular, the description of how process substitution works is super interesting. I wonder if it actually create a physical temp file, and if the temp file automatically gets cleaned up by the operating system. I'll google it and do more research.

The thing about running "sh script.sh" totally makes sense, and that's pretty much what I figured was happening as soon as you pointed out that there's a difference between "sh" and "bash" (and remembered that the script was being launched by "sh").

Thanks!

It might do. On Unixes (unlike Windows), you can delete (unlink) a file that's in use, so if you unlink the file immediately after opening it, it will be deleted when the process exits.

I'd never come across process substitution before, very neat.

Some reading around suggests that it typically doesn't actually create temporary files as such, just multiple pointers to the same pipe.

And that in cases where it is achieved with actual files, those are deleted when they're finished with.

https://unix.stackexchange.com/questions/17107/process-substitution-and-pipe
http://tldp.org/LDP/abs/html/process-sub.html

I once wrote a batch file that displayed "Hello, World" right on the screen! Of course I'd have to do some research and refresh my memory to figure out just how I did that.

Do you think I should go to Microsoft and offer to help them with the next version of Windows 10?



tanstaafl.

One week after I started this CS degree, my boss asked if I was ready to help on a model that I worked on ages ago when it was still in FORTRAN. They turned it into a a bird's nest of horrible code. Yes, the FORTRAN was easier to trace.

So I told him I knew how to echo a user's favorite animal back to them.

He didn't respond after that.

Updates on how this code was eventually used:
https://empegbbs.com/ubbthreads.php/topics/371856/Re:_How_to_make_YouTube_think_#Post371856