# mamake and the MAM language #

MAM (Make Abstract Machine) is a simple rule-based make language
that is implemented in just eight four-letter commands and four attributes,
yet allows unlimited flexibility as it can execute arbitrary shell code.
The program implementing MAM, `mamake`,
is a portable C90 program written in a single file, `mamake.c`.
This allows ksh 93u+m,
or other programs using this build system,
to be built using only a standard C compiler and utilities installation
without any other dependencies or complications.

MAM was originally designed by Glenn Fowler at AT&T and intended as an
abstraction layer for `make` implementations such as AT&T `nmake`.
The [original documentation](https://web.archive.org/web/20041227143022/http://www2.research.att.com/~gsf/mam/mam.html)
for MAM specified a more extensive and markedly different language
than was actually implemented in `mamake.c`.
This file documents the MAM implementation that is currently in use.

Since fixing and maintaining AT&T `nmake` proved impractical, `mamake` is
used here as a full `make` replacement, with some features gradually added to
the language to facilitate human maintenance of the `Mamfile`s.

## Table of contents ##

* [General overview of the MAM language](#user-content-general-overview-of-the-mam-language)
    * [Strict and legacy modes](#user-content-strict-and-legacy-modes)
* [MAM variables](#user-content-mam-variables)
    * [Special expansion syntax](#user-content-special-expansion-syntax)
    * [Automatic variables](#user-content-automatic-variables)
* [Commands](#user-content-commands)
    * [Comments](#user-content-comments)
    * [Rules](#user-content-rules)
    * [Simple prerequisite rules](#user-content-simple-prerequisite-rules)
    * [Referencing previously defined rules](#user-content-referencing-previously-defined-rules)
    * [Setting MAM variables](#user-content-setting-mam-variables)
    * [Shell actions](#user-content-shell-actions)
        * [Viewpathing](#user-content-viewpathing)
        * [Execution](#user-content-execution)
        * [Strict level 2+ change](#user-content-strict-level-2-change)
    * [Declaring common code for shell actions](#user-content-declaring-common-code-for-shell-actions)
    * [Binding libraries](#user-content-binding-libraries)
        * […while scanning and sorting leaf directories](#user-content-while-scanning-and-sorting-leaf-directories)
        * […while building the current directory](#user-content-while-building-the-current-directory)
    * [Repeatedly iterating through a block](#user-content-repeatedly-iterating-through-a-block)
* [Debugging mamake](#user-content-debugging-mamake)
* [Appendix: Main changes from the AT&T version](#user-content-appendix-main-changes-from-the-att-version)

## General overview of the MAM language ##

MAM is a simple declarative language, easy to parse for machines and easy to
read for humans, in which targets are defined that correspond to files that
need to be generated or updated, or that are prerequisites. `mamake` reads
build scripts from a file, `Mamfile` by default, from start to finish. As it
encounters `make` *target*…`done` blocks, it ‘makes’ (updates) the *target*s
using the instructions within those blocks. Those instructions may declare
dependencies and/or specify shell command actions.

If a *target* is specified on the command line, `mamake` will update the
actions contained within that target and ignore the rest.

### Strict and legacy modes ###

By default, `mamake` remains fully backward compatible with Mamfiles as
originally generated by AT&T `nmake`. If the `MAMAKE_STRICT` variable is
set, some backward incompatible changes and deprecation warnings are
activated to ensure correct operation and to facilitate human maintenance of
the Mamfiles. A numeric value assigned to `MAMAKE_STRICT` indicates the
backward incompatibility level. The empty value is equivalent to level 1.
These are called the "strict levels" and their absence the "legacy mode",
also known as strict level 0.

Each time backward incompatible changes are introduced to `mamake` that would
break previous Mamfiles, those are made subject to a `MAMAKE_STRICT` value of one
higher than the previous highest one; details of those changes are documented
throughout this file and listed in the appendix below.
This makes it possible to test or backport old code using the current build
system. Current Mamfiles should use the highest strict level available.
The current highest available strict level is **4**.

## MAM variables ##

MAM variables are imported from the environment
or set via `setv` (see below).
They are referenced and expanded using the syntax `%{`*variable_name*`}`.
Except for the special expansion syntax described below,
it is an error to reference an undefined variable using a `%` expansion.

> *Obsolete:* At strict levels 3 and below, expansions may start with `${`
> instead of `%{`, making the syntax similar to `sh`(1), though the braces are
> *not* optional. This syntax potentially conflicts with the expansion of shell
> variables in shell actions. If an undefined MAM variable is expanded with the
> `$` syntax and the variable name is valid in `sh`(1) syntax, the expansion is
> left in place unexpanded, otherwise it is removed. At strict level 2 and 3,
> it is left unexpanded even if it is not a valid `sh`(1) variable name; this
> allows POSIX shell expansions like `${foo#*bar}` in shell actions.
> Old `$` and new `%` syntax may be mixed.

Variables always expand to their literal values, and variable references
(including in `setv` values) only work for previously defined variables.

> *Obsolete:* At strict levels 1 and 0, the expansion of MAM variable references
> is recursive, i.e., the value may itself contain other variable references.
> Beware: there is no reference loop detection; any variable referencing itself
> directly or indirectly will cause `mamake` to crash.

Note that, in shell actions (see `exec` below), MAM variables are expanded
before the script ever reaches the shell. Consequently, the use of single
shell quotes `'`…`'` does not stop their expansion as you might expect.

### Special expansion syntax ###

In `%{`*variable*`?`*str*`?`*x*`?`*y*`?}`,
if the string value of the *variable* is identical to *str*,
then the value *x* is substituted, otherwise *y*.
A *str* of `*` is treated specially: if the *variable* is set to a
non-empty string, then the value *x* is substituted, otherwise *y*.
The *x* and *y* values may result from nested variable references.
The last `?` is optional.

In `%{`*variable*`-`*x*`}`, the value of *variable* is substituted
if it is defined and non-empty, otherwise the value of *x* is substituted.
In `%{`*variable*`+`*x*`}`, *x* is substituted if the value of
*variable* is defined and non-empty, otherwise the reference is removed.
Note that, unlike in `sh`(1), no distinction is made between an undefined
variable and a defined variable with an empty value.

### Automatic variables ###

The following variables are set and updated automatically.
They are inspired by similar variables in `make` implementations,
but since `mamake` is different, so are these variables.

`%{@}` is the name of the rule currently being made.

`%{<}` is the name of the prerequisite rule (`make`…`done` or `prev`)
that was *last* processed within the current rule.

`%{^}` is a space-separated list of names of all the current rule's
previously processed prerequisites.

`%{?}` is a space-separate list of the current rule's previously processed
prerequisites that have been updated by a shell action (see `exec` below)
during the current `mamake` run. Prequisites that were already up to date,
or prerequisites that do not contain a shell action, are not included.

## Commands ##

MAM commands have the following basic form:

*command* [ *argument* [ *operand string* ] ]

The *command* name consists of four lower-case letters.
Unrecognized commands or attributes are an error.
The *argument* is a single word.
The *operand string* is any arbitrary text until the end of the line.

### Comments ###

`note` is the comment command and is ignored.
In the legacy mode, `info` and `meta` are also ignored.

### Rules ###

`make` *target* [ *attribute* … ]    
`done` [ *target* ]

A `make`…`done` block defines the rule named *target* using the other commands described here.
Unless the `virtual` attribute is used, *target* names the pathname of the file generated or referenced by the rule.

`mamake` processes the commands within the block if the *target* is out
of date or if the rule has the `virtual` attribute (see below).

The *target* may be repeated as the operand to the `done` command.
In that case, it is matched against the current `make` *target* and
any mismatch will produce a "mismatched done statement" error.
If it is omitted, the current `make` *target* is assumed.

Dependencies may be defined in two ways:
1. By nesting `make`…`done` blocks:
   the enclosing rule is the parent
   and the enclosed rules are the prerequisites.
2. By using the `prev` command (see **Referencing previously defined rules** below)
   to reference a previous `make`…`done` block.
   The dependency is defined as if that block were repeated at the `prev` command's location.

If the block contains one or more `exec` commands (see **Shell actions** below),
the `done` command executes the shell script defined by them.

A `make`…`done` rule may lack any `exec` action, in which case it declares a
dependency (a file that must be present and whose timestamp is propagated to
potentially outdate parent rules, such as a source code file that comes with
the distribution or a header file preinstalled by a previously run Mamfile).
The rule may include its own dependencies. If it does not have any dependencies
of its own, the `make`…`done` block is empty, with `make` immediately followed
by `done`. The one-line `makp` command (see below) may be used as a shorthand
for the latter case.

Making a prerequisite that is currently being made, or one that has already
been made, produces a warning; at strict level 3 and up, this is an error.

One or more *attribute*s may be specified by appending them to the `make` command.
(At strict levels \< 2, they may also be appended to the `done` command; the effect
is the same either way. At strict level 1, this produces a deprecation warning.)
Attributes apply to the current rule only and do not propagate down to nested rules.
The following *attribute*s are available:
* `dontcare`: Marks files that do not need to exist.
  If the file exists then its last-modified timestamp is checked and propagated,
  otherwise it is silently ignored. 
* `ignore`: The timestamp associated with the *target* is ignored in dependency resolution.
* `notrace`: Disables echoing (xtrace) of shell action commands.
  This does not disable the trace header for the containing rule (see *Shell actions* below).
* `virtual`: Marks a rule that is not associated with any file.
  The commands within are executed every time the rule is processed.
  By convention, a virtual rule with target `install` performs pre-installation.

> *Obsolete:*
> At strict level 4 and up, the following *attribute* is not available.
> * `implicit`: Equivalent to `dontcare`.
> 
> At strict level 1 and up, specifying the following *attribute*s is
> deprecated and will produce a warning; at strict level 2 and up,
> specifying these is an error.
> * `archive`: Ignored.
>   Historically used to mark the generation of an `ar`(1) archive.
> * `generated`: Marks rules that produce output files generated by a shell action.
>   The explicit assignment of this attribute is ignored at strict level 1.
>   The `exec` command implicitly assigns this attribute.
>   If a rule has this attribute, other rules dependent on this rule
>   will avoid applying viewpathing based on this rule.
> * `joint`: Ignored.
>   Historically used to mark one of a group of rules that are built by a single shell action.

### Simple prerequisite rules ###

`makp` *target* [ *attribute* … ]

`makp` creates a rule that declares a dependency on a prerequisite file named
by *target* in a manner equivalent to an empty `make` *target*/`done` block,
with the optional *attribute*s applied to the new rule. A nonexistent
prerequisite is an error unless a `virtual` or `dontcare` attribute is given.
Declaring a dependency on a prerequisite that is currently being made (i.e.:
directly or indirectly within that prerequisite's block) is an error.

### Referencing previously defined rules ###

`prev` *target*

If *target* matches a previously defined rule, `prev` adds a dependency on
that rule to the current rule. This is used to make a rule a prerequisite of
multiple `make`…`done` blocks without repeating the rule. It is an error
to specify attributes, because the attributes of the referenced rule are used.
In the legacy mode, attributes are silently ignored.

> *Obsolete:* If the strict level is \< 4, and if *target* does not match a
> previously defined rule, then the following applies. In the legacy mode,
> `prev` creates an empty dummy rule and ignores the *attribute*s; this is
> for backward compatibility. At strict levels 1 and up, `prev` in this
> context is equivalent to `makp` (see above). Declaring a dependency on a
> prerequisite that is currently being made produces a warning; at strict
> level 3, this is an error.

### Setting MAM variables ###

`setv` *variable* [ *defaultvalue* ]

Defines a new MAM *variable*, optionally assigning the initial *defaultvalue*.
If the variable already has a value, the `setv` command is ignored; assigning a new value is not possible.
When `mamake` starts, it imports all environment variables as MAM variables,
so any variable's default value can be overridden by exporting an environment variable by its name.
Any leading and trailing whitespace is stripped from the value. The value is
otherwise taken entirely literally, with no parsing of quotes, etc.

> *Obsolete:* If the strict level is less than 2 and the *defaultvalue*
> begins and ends with double quotes (`"`), those quotes are discarded,
> though double quotes elsewhere in the value are not treated specially.

When the *variable* is `CC`, `mamake` runs the `mamprobe` script
to probe the C compiler for flags and features,
or uses that script's stored results if not outdated.
The results are stored as a series of `setv` commands
in a file in the directory `%{INSTALLROOT}/lib/probe/C/mam`,
the file name being a hash of full path to the compiler indicated by `%{CC}`.
That results file is then read and included in the current Mamfile
as if it followed the `setv CC` command.

### Shell actions ###

`exec` `-` *code*

One or more `exec` commands within a `make`…`done` block
define a shell script that is run to generate the *target*.
The argument following `exec` is ignored; by convention it is `-`.
Each `exec` command appends a line of code to the shell script for the current rule.
It is customary for a rule's `exec` commands to be contiguous, but not necessary.

Before adding each line of code to the script,
MAM variable references (see **MAM variables** above)
are expanded; their literal values are inserted into the *code* line
(beware: no quoting is applied!).
Because variables are expanded when the line is encountered, the value
of the automatic variables for any `exec` line depends on the position
of the line in the rule.

#### Viewpathing ####

After MAM variable expansion, *viewpathing* is applied.
The first colon-separated element of `%{VPATH}` is considered
the object code directory and the second the source code directory;
viewpathing provides the first with a vew to the second.
Viewpathing applies two transformations.

The first is *prerequisite replacement*.
Each word (separated by whitespace, `;`, `(`, `)`, `` ` ``, `|`, `&` or `=`)
is searched for in the current rule's prerequisites,
and if it matches the name of a prerequisite that was not generated by a shell action,
it is replaced by the canonical path to it in the source directory,
ensuring that things like prerequisite headers are found.

The second is *include flag duplication*.
After every argument that looks like a compiler include directory path (i.e.,
starting with `-I`) with a relative path name (i.e., a directory path that does
*not* start with a `/`), another argument starting with `-I` is inserted with
that path name prefixed by the path to the source directory.  This mechanism
ensures that headers are found both in the object directory and in the source
directory.  It is processed regardless of the command; for example, it also
works for compiler flags passed to `iffe`(1).

Note that shell quotes are *not* treated specially. If an argument starting
with `-I` ends in a shell quote without preceding whitespace, that trailing
quote is repeated along with the prefixed path and causes a syntax error.

#### Execution ####

When `mamake` encounters the `done` command,
the script is executed by the shell whose path is in the `SHELL` environment variable
or, absent that, by `sh`(1).
Each shell action is run in a new instance of the shell.

Before executing the script, an empty line followed by a trace header
in the following format is written to standard error:

    # path/to/Mamfile: startline-endline: rule

During script execution, shell action comands are traced using the
shell's xtrace option, unless the rule has the `notrace` attribute.

#### Strict level 2+ change ####

At strict level 2 and up, `mamake` turns off global pathname expansion
(globbing) using `set -f` to make safer MAM variable expansion and shell
field splitting possible; this avoids unexpected pathname expansion if
a value contains `?`, `*` or `[`.
A shell action can override this using `set +f`;
this should only be done for individual commands in a `(`subshell`)`.

### Declaring common code for shell actions ###

`shim` `-` *code*

One or more `shim` commands declare a ‘shim’: a common section of `sh`(1)
code that will be automatically inserted in front of subsequent shell
actions upon execution. Like `exec`, `shim` combines multiple lines of
*code* into one section, with MAM variables expanded at declaration time and
viewpathing applied at execution time. The effect of `shim` is global.

One use case is defining a shell function that each shell action can call.
For example, such a function might invoke the compiler with a series of
compiler flags common to all compiler invocations, with other flags added via
arguments to the function, so the common flags do not need to be repeated in
every shell action.

Only one shim is active at a time, but it can be redefined. When the next
`exec` command is encountered, the shim is marked ready for use. The next
time a `shim` command is encountered after that, it starts a new shim from
scratch that affects subsequently executed shell actions.
A single `shim -` deactivates the shim.

### Binding libraries ###

`bind` `-l`*libraryname* [ `dontcare` ]

These commands are scanned for while sorting leaf directories for recursive
building, and executed as normal commands while building the current directory.

#### …while scanning and sorting leaf directories ####

Any leaf directories with names that start with `INIT` will always be built before
all others. For all other leaf directories, the presence of any `bind`
command of the form `bind -lfoo` anywhere in a leaf
directory's Mamfile causes the leaf directory named `libfoo` (if it exists)
to be a prerequisite of that leaf directory.
The prerequisite leaf directory does not have to be in the same parent
directory, as long as it is processed as part of the same scan.
At this stage, attributes are ignored.

#### …while building the current directory ####

The `bind` command takes an argument of the form `-l`*libraryname*, which
causes a MAM variable `mam_lib`*libraryname* to be defined (see **MAM variables** above).
The variable will contain either the compiler argument for linking to the library *libraryname*
(either the `-l`*libraryname* flag, or the full path in case of a static library)
or, if the `dontcare` attribute is specified, possibly the empty string.
Any library dependencies are also included (see below).
This can be used both for AST libraries shipped with the distribution and for system libraries.

If the current rule's name starts with `FEATURE/` or is `configure.h`, this
is all that the `bind` command does. Otherwise, it also does the following.

For each corresponding *.a library archive dependency built previously,
its time stamp is checked and the current target is marked as outdated if it is
newer, as if a `prev` had been executed for it.
The variable set by `bind` is global, but the marking of the target as
outdated applies to the current rule only, so it may be necessary to
repeat a `bind` command when statically linking executables that depend
on a library, otherwise they may not be relinked when the library changes.
The `mam_lib`*libraryname* variable will not be regenerated when repeating a `bind`.

There is also a mechanism to communicate library dependency information across Mamfiles and `mamake` invocations.
If a file named *libraryname*`.req` in the current directory
or an `%{INSTALLROOT}/lib/lib/`*libraryname* file
exists, `mamake` processes each of the words in the form `-l`*libraryname* in its contents
as if they were arguments to `bind` commands
and the resulting values are appended to the value of `mam_lib`*libraryname*
as dependencies separated by spaces.
`mamake` does not create these dependency files;
they are expected to be generated by Mamfile shell actions (see **Shell actions** above).
The `INIT` package preinstalls the `mkreq` and `mkreq-maplib` scripts for this purpose.
If no such dependency file exists, and the `dontcare` attribute is added,
then `mamake` compiles a small test program on the fly to check if the library exists;
if this fails, the `mam_lib`*libraryname* variable will be emptied.

Cross-directory dependencies on AST library headers (preinstalled in
`$INSTALLROOT/include/ast`) are similarly communicated via a file with
the path `%{INSTALLROOT}/lib/mam/`*libraryname*.
The `bind` command automatically includes this file as necessary.
Note that this may have side effects on the automatic variables.
The `INIT` package preinstalls the `mkdeps` script that Mamfile shell
actions should use to generate this file while building each library.
The generated file is expected to:
1. set the `INCLUDE_AST` variable to `%{INSTALLROOT}/include/ast`;
2. define a single virtual rule by the name of `_hdrdeps_lib`*libraryname*`_`
   which contains all the rules that define the library's public `include/ast`
   headers and how they depend on each other.
This way, Mamfiles can declare a dependency on a single header and all its
dependencies using a simple `prev %{INCLUDE_AST}/`*headername*`.h` command.
Any `bind -l`*libraryname* command will automatically apply the dependencies
defined in the corresponding file to the context of the current rule.
The non-existence of this file is not an error and is silently ignored.

### Repeatedly iterating through a block ###

`loop` *variable* *word* [ *word* … ]    
`done`

`loop` reads the lines contained between it and the corresponding `done`
repeatedly with a named *variable* set to each of the *word*s. The lines
are processed as part of the rule containing the loop.
The *variable* is restored to its previous state after the loop completes.

Note that `loop` causes repeated reading and processing of Mamfile lines,
*not* necessarily repeated execution. For instance, a loop can be used to
consolidate repetitive `make`…`done` rules. However, each rule is only made
once and subsequent rules by the same name are an error at strict level 3
and up, or skipped over at strict \< 3. So it only makes sense to do this
if the contained make target names are modified by the expansion of the
iteration *variable*.

`loop` requires a seekable input file (i.e.: not a pipe).

## Debugging mamake ##

If the environment variable `MAMAKE_DEBUG_PREFIX` is exported, its value is
split into whitespace-separated command arguments and inserted before every
`mamake` invocation. This allows debugging `mamake` with tools like
`valgrind`, `strace`, or `dtruss`, which are used by passing a command to
them as a set of operands.

## Appendix: Main changes from the AT&T version ##

Compared to the original AT&T version, ksh 93u+m made a number of
changes to `mamake` that facilitate correct operation and make it easier to
maintain Mamfiles by hand. The following lists the important changes.
* Introduced the notion of ‘strict mode’ levels that tidy things up by
  activating some backward incompatible changes and deprecation warnings.
* Indentation and word separators may use any whitespace (e.g. tabs), not only spaces.
* Fixed a bug that stopped a rule marked `virtual` (not associated with
  any file) from being executed if a file by that rule's name exists.
* Unrecognized commands and rule attributes throw an error instead of being silently ignored.
  This also applies to the `bind` command with an argument not starting with `-l`.
* It has been made optional to repeat the `make` target after `done`.
* The `notrace` attribute was added to disable xtrace for a rule's shell action.
* The automatic variables `%{@}`, `%{<}`, `%{^}` and `%{?}` have been added.
* An iteration block command, `loop`…`done`, has been added.
* A command to set common code for shell actions, `shim`, has been added.
* The `bind` command now reads library header dependency rules from a central
  rules file that is automatically generated for each library by the supplied
  `mkdeps` script.
* Attempting to make a rule that has already been made produces a warning.
* Attempting to declare a dependency on a rule currently being made produces a warning.
* MAM expansions may start with `%{` instead of `${`, avoiding conflicts
  and confusion with the shell's `${` syntax in shell actions.
* The `makp` command may be used instead of an empty `make`…`done`
  block to declare a simple prerequisite with optional attributes.
* **At strict level 1 and up:**
    * Appending attributes to `done` instead of `make` is deprecated
      and produces a warning.
    * The ignored `archive` and `joint` attributes are deprecated.
    * Explicitly specifying the `generated` attribute is deprecated.
    * The dummy `info` and `meta` commands are unavailable instead of ignored.
    * The `prev` may be used as an equivalent of `makp`.
    * When `prev` references a previously processed target,
      attributes are an error instead of being ignored.
    * The legacy `silent` and `ignore` command prefixes are unavailable.
* **At strict level 2 and up:**
    * Appending attributes to `done` instead of `make` is an error.
    * The `archive` and `joint` attributes are unavailable.
    * Explicitly specifying the `generated` attribute is an error.
    * All variable references are expanded to their literal values
      without scanning the values for recursive variable references.
    * `setv` does not remove leading and trailing `"` from the value.
    * Shell actions have pathname expansion (globbing) disabled by default.
* **At strict level 3 and up:**
    * Attempting to make a rule that has already been made is an error.
    * Attempting to declare a dependency on a rule currently being made is an error.
* **At strict level 4 and up:**
    * MAM expansions can no longer start with `${`. Only `%{` is recognized.
    * The `prev` command may no longer be used as an equivalent of `makp`.
    * The `implicit` attribute is not available.
