Echo: the command, the (several) builtins, and makefiles

As explained in this post I wanted to patch a symbol in a binary file, and for that needed to echo a specific byte value (xEA) into a sed command appearing in a Makefile recipe.

Now, you may be aware that the echo command lives in /bin/, but the shell also provide an echo shell builtin. Let's see how we can echo a byte with hex value EA in both cases:

$ man echo
       echo [SHORT-OPTION]... [STRING]...
       echo LONG-OPTION

DESCRIPTION
       Echo the STRING(s) to standard output.

       -n     do not output the trailing newline

       -e     enable interpretation of backslash escapes

       [...]

       If -e is in effect, the following sequences are recognized:

       [...]

       \xHH   byte with hexadecimal value HH (1 to 2 digits)

       NOTE: your shell may have its own version of echo, which usually super-
       sedes the version described here.  Please refer to your  shell's  docu-
       mentation for details about the options it supports.

So we need to use the -e option to enable escape sequences, and then use the \xEA sequence. Let's see the builtin.

$ man bash
  [...]
  SHELL BUILTIN COMMANDS
  [...]
       echo [-neE] [arg ...]
              Output the args, separated by spaces,  followed  by  a  newline.
              The  return  status  is 0 unless a write error occurs.  If -n is
              specified, the trailing newline is suppressed.  If the -e option
              is  given,  interpretation  of  the  following backslash-escaped
              characters is enabled.  The -E option disables  the  interpreta-
 [...]
              \xHH   the  eight-bit  character  whose value is the hexadecimal
                     value HH (one or two hex digits)

The bash echo builtin supports the same option and escape syntax, so it seems it's irrelevant wether we run the external command or the bash builtin.

Let's try it:

$ echo -e "\xEA"
�

Obviously the character can only be displayed correctly in a terminal supporting Latin-1 encoding. We start one with xterm -en iso8859-1 and will perform all following operations there.

echo -e "\xEA"
ê
echo -e \\xEA
ê

This is the shell builtin running here, but as expected the results are the same when using /bin/echo.¹

Good, let's try it in a Makefile:

echo1:
	echo -e "\xEA"

$ make echo1
echo -e "\xEA"
-e \xEA

Hey, quite unexpected! Seems echo here does not understand -e as an option! First, let's confirm that this is the shell builtin that's running:

type:
	type echo

$ make type
type echo
echo is a shell builtin

Ok. Let's try the standalone command instead.

echo2:
	/bin/echo -e "\xEA"

$ make echo2
/bin/echo -e "\xEA"
ê

So the builtin works ok in the terminal, but not from the Makefile? Does make do something special to the shell that runs its commands? Let a recipe be "run the echo command inside a bash shell".

echo3:
	bash -c "echo -e \"\xEA\""

$ make echo3
bash -c "echo -e \"\xEA\""
ê

Hum… Now that I think of it, there's no reason why make would necessarily use bash, and the doc actually says:

   How to update is specified by a RECIPE.  This is one or more lines to
be executed by the shell (normally 'sh'), but with some extra features
(*note Writing Recipes in Rules: Recipes.).

So the recipes are not run with bash, but with sh ("normally")! On my devuan system, sh is actually dash:

ls -l /bin/sh 
lrwxrwxrwx 1 root root 4 Jan  5  2023 /bin/sh -> dash

So the echo run by the recipe echo1 is actually dash's echo builtin. Let's look at dash's manual page:

     echo [-n] args...
            Print the arguments on the standard output, separated by spaces.
            Unless the -n option is present, a newline is output following the
            arguments.

            If any of the following sequences of characters is encountered
            during output, the sequence is not output.  Instead, the specified
            action is performed:

            \b      A backspace character is output.

            \c      Subsequent output is suppressed.  This is normally used at
                    the end of the last argument to suppress the trailing new-
                    line that echo would otherwise output.

            \e      Outputs an escape character (ESC).

            \f      Output a form feed.

            \n      Output a newline character.

            \r      Output a carriage return.

            \t      Output a (horizontal) tab character.

            \v      Output a vertical tab.

            \0digits
                    Output the character whose value is given by zero to three
                    octal digits.  If there are zero digits, a nul character
                    is output.

            \\      Output a backslash.

            All other backslash sequences elicit undefined behaviour.

Saw that? This builtin does not know about the -e option; escape sequences are always honored, except that the \xHH is unknown! However the \0OOO one is supported. Let's make sure:

$ sh -c "echo \"\0352\""
ê

Okay. By the way, if we don't like the escaped quotes \" we can escape the \ instead. So, while this obviously does not work

$ sh -c "echo \0352"
0352

this should (the first \ protects the second one so that echo actually receives \0352 as argument):

sh -c "echo \\0352"
0352

hoooo, except it doesn't!? Oh, well let's try this

$ sh -c "echo \\\0352"
$ ê

Do you get it? bash processes the echo \\\0352 string, so \\ is turned into \ (and \0 is unchanged²), so dash receives echo \\0352 as a command line, parses it as the command echo and the argument \0352. Let's try this directly in dash:

$ dash
$ echo \\0352
ê

So we should be able to put this in the Makefile:

echo4:
	echo \\\0352

$ make echo4
echo \\\0352
\0352

Oh my fxxx god, what's that? I must have missed something. Let's keep calm and proceed step by step:

echo5:
	sh -c "echo \"\0352\""

$ make echo5
sh -c "echo \"\0352\""
ê

Good. Let's see how make runs the shell:

5.3.2 Choosing the Shell

The program used as the shell is taken from the variable 'SHELL'.  If
this variable is not set in your makefile, the program '/bin/sh' is used
as the shell.  The argument(s) passed to the shell are taken from the
variable '.SHELLFLAGS'.  The default value of '.SHELLFLAGS' is '-c'
normally, or '-ec' in POSIX-conforming mode.

Our command should be run as sh -c "cmd..." . Let's check this .SHELLFLAGS variable:

shellflags:
	echo $(.SHELLFLAGS)

$ make shellflags 
echo -c
-c

Allright. So can we make sure what shell make is using?

shell:
	echo $$SHELL

   Because dollar signs are used to start 'make' variable references, if
you really want a dollar sign in a target or prerequisite you must write
two of them, '$$' (*note How to Use Variables: Using Variables.).  If

$ make shell
echo $SHELL
/bin/bash

Does it drive you crazy? It should not, because sh (dash) does not set the SHELL environment variable, and here we run bash that runs make that runs sh to run the echo command (should be sh's builtin echo). Let's look at the processes to confirm that.

pstree:
	pstree deleuzec

$ make pstree
[...]
st---bash-+-emacs-+-aspell
         |       |-qutebrowser-+-QtWebEngineProc
         |       |             |-QtWebEngineProc---QtWebEngineProc-+-7*[QtWebE+
         |       |             |                                   `-2*[QtWebE+
         |       |             |-QtWebEngineProc---3*[{QtWebEngineProc}]
         |       |             `-65*[{qutebrowser}]
         |       `-10*[{emacs}]
          `-xterm---luit---bash---make---pstree

Oh well, actually make did run pstree directly without any shell here… hum, well, just because it can!

5.3 Recipe Execution

When it is time to execute recipes to update a target, they are executed
by invoking a new sub-shell for each line of the recipe, unless the
'.ONESHELL' special target is in effect (*note Using One Shell: One
Shell.) (In practice, 'make' may take shortcuts that do not affect the
results.)

Let's force it to run a damn shell by providing a composite command:

pstree2:
	pstree ; echo $$SHELL

$ make pstree2
[...]
    |-st---bash-+-emacs-+-aspell
    |           |       |-qutebrowser-+-QtWebEngineProc
    |           |       |             |-QtWebEngineProc---QtWebEngineProc---9*+
    |           |       |             |-QtWebEngineProc---4*[{QtWebEngineProc}+
    |           |       |             `-64*[{qutebrowser}]
    |           |       `-10*[{emacs}]
    |           `-xterm---luit---bash---make---sh---pstree
[...]
/bin/bash

Ahhh! It's really running sh, good!

But then what happens with echo4 does not make sense. Could it be that it's executing the echo external command this time? Let's try:

echo7:
	echo -e \\\0352

echo8:
	echo -e \\\xEA

$ make echo7 echo8
echo -e \\\0352
ê
echo -e \\\xEA
ê

Yes it does! But at the very beginning we discovered it did not, it was executing dash's builtin instead. Recall the echo1 recipe, it did output -e \xEA!

echo1:
	echo -e "\xEA"

What do you think?

No idea?

The fxxxing quotes, I'll tell you! Here's what's happening:

• for echo7 and echo8, these are just a command with arguments, so make decides it can directly execute the command, without launching any shell. Thus the external echo command is run.
• for echo1, the quotes certainly convince make that this is a complex command line that needs to be executed through a shell. So it fires sh and gives it the command line. sh parses the command line and decides to use its builtin echo command.

Recall the sentence in make documentation above:

In practice, 'make' may take shortcuts that do not affect the results.

In this very specific case, it actually does affect the results…