I have an infinite loop in bash that:

1. reads input from the user; and
2. echoes it without the newline, after also removing extra whitespace from the input.

#!/bin/bash -u

# Does not work.
while :; do
    read -p "Type, and press ENTER: "
    echo -n "$REPLY" 
done | sed -u -E 's/ +//g' 

When I run the program, I never see any output (so I have to use Ctrl-C to quit):

$ ./script.sh 
Type, and press ENTER: a   b   c    
Type, and press ENTER: d   e   f
Type, and press ENTER: g h i
Type, and press ENTER: ^C

If, however, I let the echo output the newline, like so,

# This works.
while :; do
    read -p "Type, and press ENTER: "
    echo "$REPLY"    # <--------------- -n removed
done | sed -u -E 's/ +//g' 

the input is able to get past sed. But, as I said, the program's requirement is to not include the newline. Hence, this isn't really a solution, just my observation on the behavior of echo -n inside of a loop.

As a workaround, moving sed inside the infinite loop works:

# Workaround, that works!
while :; do
    read -p "Type, and press ENTER: "
    echo "$REPLY" | sed -u -E 's/ +//g' 
done 

$ ./script.sh 
Type, and press ENTER: a  b  c
abc
Type, and press ENTER: d  e  f
def
Type, and press ENTER: ^C

Why doesn't my original loop work but the above workaroud does?

EDIT

1. Removed the irrelevant case of the finite loop.
2. The post now includes only valid cases that all consistently use sed -u for unbuffered output.

I have an infinite loop in bash that:

1. reads input from the user; and
2. echoes it without the newline, after also removing extra whitespace from the input.

#!/bin/bash -u

# Does not work.
while :; do
    read -p "Type, and press ENTER: "
    echo -n "$REPLY" 
done | sed -u -E 's/ +//g' 

When I run the program, I never see any output (so I have to use Ctrl-C to quit):

$ ./script.sh 
Type, and press ENTER: a   b   c    
Type, and press ENTER: d   e   f
Type, and press ENTER: g h i
Type, and press ENTER: ^C

If, however, I let the echo output the newline, like so,

# This works.
while :; do
    read -p "Type, and press ENTER: "
    echo "$REPLY"    # <--------------- -n removed
done | sed -u -E 's/ +//g' 

the input is able to get past sed. But, as I said, the program's requirement is to not include the newline. Hence, this isn't really a solution, just my observation on the behavior of echo -n inside of a loop.

As a workaround, moving sed inside the infinite loop works:

# Workaround, that works!
while :; do
    read -p "Type, and press ENTER: "
    echo "$REPLY" | sed -u -E 's/ +//g' 
done 

$ ./script.sh 
Type, and press ENTER: a  b  c
abc
Type, and press ENTER: d  e  f
def
Type, and press ENTER: ^C

Why doesn't my original loop work but the above workaroud does?

EDIT

1. Removed the irrelevant case of the finite loop.
2. The post now includes only valid cases that all consistently use sed -u for unbuffered output.

• bash

• 3 echo -n flushes output immediately, but sed always processes text one line at a time -- meaning it doesn't output anything until it's received a full line of input (including the newline terminator), and since echo -n doesn't output full lines that never happens. – Gordon Davisson Commented Mar 27 at 8:07
• 1 I think that even with -u sed is line oriented. You may try awk and change record separator but it has to be present in the input. – Arkadiusz Drabczyk Commented Mar 27 at 9:51
• 2 Aside: any time you find yourself using echo with arguments (e.g. -n) you should be using printf instead, see why-is-printf-better-than-echo. – Ed Morton Commented Mar 27 at 10:21
• 1 @EdMorton should be a real treat to read your link, thanks a TON. – Harry Commented Mar 27 at 10:32
• 1 Aside: tr -d ' ' doesn't operate on lines, so if you were using it instead of sed you wouldn't have this problem. – Charles Duffy Commented Mar 27 at 11:25

1 Answer 1

You got answers to the question you asked in the comments (buffering or not, sed processes a linefeed-separated block at a time) but consider using awk instead of a shell while-read loop for this as the latter is generally considered bad practice (see why-is-using-a-shell-loop-to-process-text-considered-bad-practice), e.g. using any awk:

$ cat tst.sh
#!/usr/bin/env bash

awk '
    BEGIN {
        prompt = "Type, and press ENTER: "
        printf "%s", prompt |"cat>&2"
    }
    {
        gsub(/ +/, "")
        printf "%s", $0
        printf "\n%s", prompt |"cat>&2"
        fflush()
    }
' "${@:--}"

$ ./tst.sh
Type, and press ENTER: a  b c
abc
Type, and press ENTER: foo   bar
foobar
Type, and press ENTER: $

$ ./tst.sh > out
Type, and press ENTER: a   b c

Type, and press ENTER: foo bar

Type, and press ENTER: $

$ cat out
abcfoobar$

If you do want to use a shell while-read loop it should start with:

while IFS= read -r -p "Type, and press EN"; do

instead of:

while :; do
    read -p "Type, and press EN"

so it doesn't consume backslash characters (e.g. converts a\tb to atb) or strip leading trailing white space (not actually an issue with REPLY but always do IFS= unless you need to not do it) and will stop if/when the read fails or encounters EOF (e.g. the user types Ctrl-D)

By the way, the reason sed processes one line at a time when called as:

while :; do
    printf foo | sed '...'
done

but not as:

while :; do
    printf foo
done | sed '...'

is that in the first case sed is called one line at a time (so each time printf foo executes sed gets called to process just the output from that one invocation of printf) while in the second sed is called once for all of the output that the loop will produce and sed will process its input buffer every time it sees a linefeed or it reaches the end of its input.