Shell/Bash Tips

Enable Javascript to display Table of Contents.

Read File Line by Line

Reading a file line by line:

#!/bin/bash

INPUT_FILE="/some/input/file"

while IFS= read -r LINE ; do

  echo "$LINE"

done < "$INPUT_FILE"

Source: cyberciti.biz

Read CVS Line by Line

Separator in this example is the semicolon:

#!/bin/bash

INPUT_FILE="/some/input/file.csv"

while IFS=; read -r FIRST SECOND THIRD ; do

  echo "$FIRST $SECOND $THIRD"

done < "$INPUT_FILE"

Source: cyberciti.biz

Parsing preprocessed data line by line

E.g. handling only the first 20 lines:

#!/bin/bash

INPUT_FILE="/some/input/file"

head -n 20 $INPUT_FILE | \
while IFS= read -r LINE ; do

  echo "$LINE"

done

Or handling only lines which contain some special token:

#!/bin/bash

INPUT_FILE="/some/input/file"

grep failed $INPUT_FILE | \
while IFS= read -r LINE ; do

  echo "$LINE"

done

Rename Files with Colon (e.g. Time) in filename for Windows

Windows does not allow colons in the filename and thus shows the file e.g. as "H1QVDA~Y.TXT". The following code will replace all colons in timestamps with a minus:

$ find . -name "*T[0-9]*:[0-9]*:[0-9]**" -exec /bin/sh -c "mv -v {} \$(echo '{}' | sed 's/\([0-9]\):\([0-9]\)/\1-\2/g')" \;
renamed './zmqping_2023-08-01T14:29:40Z.log' -> './zmqping_2023-08-01T14-29-40Z.log'
renamed './failed02_threads/zmqping_2023-08-02T14:04:37Z.log' -> './failed02_threads/zmqping_2023-08-02T14-04-37Z.log'
renamed './zmqping_2023-08-01T14:44:53Z.log' -> './zmqping_2023-08-01T14-44-53Z.log'
renamed './zmqping_2023-08-01T14:19:30Z.log' -> './zmqping_2023-08-01T14-19-30Z.log'
renamed './zmqping_2023-08-01T14:04:18Z.log' -> './zmqping_2023-08-01T14-04-18Z.log'
renamed './zmqping_2023-08-01T14:24:37Z.log' -> './zmqping_2023-08-01T14-24-37Z.log'
renamed './failed03_threads/zmqping_2023-08-02T14:37:19Z.log' -> './failed03_threads/zmqping_2023-08-02T14-37-19Z.log'
renamed './failed01/zmqping_2023-08-02T11:56:08Z.log' -> './failed01/zmqping_2023-08-02T11-56-08Z.log'
renamed './zmqping_2023-08-01T14:34:50Z.log' -> './zmqping_2023-08-01T14-34-50Z.log'
renamed './zmqping_2023-08-01T14:09:25Z.log' -> './zmqping_2023-08-01T14-09-25Z.log'
renamed './zmqping_2023-08-01T14:14:27Z.log' -> './zmqping_2023-08-01T14-14-27Z.log'
renamed './zmqping_2023-08-01T14:39:47Z.log' -> './zmqping_2023-08-01T14-39-47Z.log'
renamed './ok01/zmqping_2023-08-02T11:50:59Z.log' -> './ok01/zmqping_2023-08-02T11-50-59Z.log'
$

Shell Parameter Handling

Using "$@" (please note the double quotes) in a for loop will loop over all parameters, even if they contain a space:

#!/bin/sh
for FILE in "$@" ; do
	echo ">$FILE<"
done

Source: stackoverflow.com

Output Timestamping

unbuffer python3 -m unittest tests 2>&1 | ts -s "%.s" > master.txt

Output Buffering

For efficiency, command line programs buffer it's output when it's not written to a terminal. Some programs have options to disable this behavior (e.g. tcpdump -l or grep --line-buffered), others you have to start with stdbuf -oL YOUR_PROGRAM.

Source: BashFAQ/009

For Python I had to use unbuffer (from packet expect).

unbuffer python3 -m unittest tests 2>&1 | ts -s "%.s" > master.txt

Source: Stackoverflow