Agnus Dei (jackal) wrote,
Agnus Dei
jackal

How to do Rate Limiting on While Looped Backgrounded/Forked Processes in Bash For Parallelism

MAX_BATCH_FILES_COUNT=5    # how many log files to work at the same time
open_files=0
for i in `find /x/FILES -name "*gz" |sort`; do
        let "open_files++"  # increment open files count
        # echo $i
        print_errors_count $i &  # this function does all my heavy work, and is forked into background
        pids[${open_files}]=$!;
        if [[ "$open_files" -ge "$MAX_BATCH_FILES_COUNT" ]]; then
                echo "WAITING NOW.  open files=$open_files"
                open_files=0   # Reset back to zero
                for pid in ${pids[*]}; do
                        wait $pid;   # this will wait for them all to exit before starting the next batch
                done;
        else
                echo "open_files=$open_files"
        fi
done
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

  • 0 comments