I have a bash script that looks like this:
#!/bin/bash
wget LINK1 >/dev/null 2>&1
wget LINK2 >/dev/null 2>&1
wget LINK3 >/dev/null 2>&1
wget LINK4 >/dev/null 2>&1
# ..
# ..
wget LINK4000 >/dev/null 2>&1
But processing each line until the command is finished then moving to the next one is very time consuming, I want to process for instance 20 lines at once then when they’re finished another 20 lines are processed.
I thought of wget LINK1 >/dev/null 2>&1 & to send the command to the background and carry on, but there are 4000 lines here this means I will have performance issues, not to mention being limited in how many processes I should start at the same time so this is not a good idea.
One solution that I’m thinking of right now is checking whether one of the commands is still running or not, for instance after 20 lines I can add this loop:
while [ $(ps -ef | grep KEYWORD | grep -v grep | wc -l) -gt 0 ]; do
sleep 1
done
Of course in this case I will need to append & to the end of the line! But I’m feeling this is not the right way to do it.
So how do I actually group each 20 lines together and wait for them to finish before going to the next 20 lines, this script is dynamically generated so I can do whatever math I want on it while it’s being generated, but it DOES NOT have to use wget, it was just an example so any solution that is wget specific is not gonna do me any good.
