[Cryptech-Commits] [user/sra/alpha-releng/omnibus] 03/03: Document the build process.

[git at cryptech.is]

This is an automated email from the git hooks/post-receive script.

sra at hactrn.net pushed a commit to branch master
in repository user/sra/alpha-releng/omnibus.

commit 76f6dba6d19039d5634442a8731e4aba17a3330b
Author: Rob Austein <sra at hactrn.net>
AuthorDate: Wed Jun 29 21:33:09 2016 -0400

    Document the build process.
---
 README.md | 156 ++++++++++++++++++++++++++++++++++++++++++++------------------
 1 file changed, 110 insertions(+), 46 deletions(-)

diff --git a/README.md b/README.md
index e3f3eff..cca2619 100644
--- a/README.md
+++ b/README.md
@@ -1,66 +1,130 @@
 Alpha Releng
 ============
 
-Release engineering stuff for Cryptech Alpha.
+Release engineering stuff for Cryptech Alpha board and support software.
 
-Work in progress, currently a merge in progress of two separate
-release engineering repositories, one for HSM firmware, one for host
-software.
 
-This README is probably obsolete by the time you're reading it.
+## Overview ##
 
-## Old Firmware README ##
+The Makefiles and scripts in this repository attempt to automate the
+process of building packaged versions of the firmware and software for
+the Cryptech Alpha board.  At a high level, this consists of two main
+phases:
 
-Preliminary release engineering super-repository for building firmware
-for the Cryptech "Alpha" board.
+1.  Building a firmware package (Verilog and C code which runs on the
+    Alpha board itself, regardless the host system to which the Alpha
+    is connected); and
 
-Primary tasks here are to build a bitstream for the FPGA and the
-"bootstrap" and "hsm" images for the Alpha's CPU.
+2.  Building software packages (code which runs on the host to which
+    the Alpha is connected).
 
-Eventually there will be a lot of packaging and versioning glorp here,
-but let's start with basic build and clean targets.
+Since the firmware is the same for any given version of the Cryptech
+source code, and since the process of building the firmware binaries
+is both time-consuming and involves tools like the XiLinx Verilog
+synthesis toolkit and cross-compilers for multiple CPUs (the Alpha's
+Cortex M4 ARM CPU and the Atmel AVR ATtiny828 MCU used to implement
+the anti-tampering logic for the master key memory), we include a
+signed tarball containing pre-built binaries for all the firmware as
+part of the source tarball for the software packages.
 
-Current repository structure is, um, complicated.  On the RTL side, we
-have a tree of simple subrepositories, each representing one RTL core.
+You are of course free to build all of this for yourself, with or
+without modification, the pre-built versions are just a convenience.
 
-On the software side, we have a subrepository which has several
-subrepositories of its own: current thinking is that this should
-probably be replaced by separate repositories and Makefile VPATH
-magic, but this is what we have today so it's what we build with
-today.
 
-This README is probably obsolete by the time you're reading it.
+## Source Tree ##
 
-## Old Software README ##
+The source code itself, for both the firmware and the software, is in
+the source/ directory, with names corresponding to the canonical names
+of the relevant git repositories.  At the time this document was
+written, a few of the repositories in question had not yet been
+promoted to their expected permanent homes in the core/ or sw/ trees,
+but we expect this to be temporary.  Regardless, directory names
+within the source/ directory tree are intended to track the canonical
+names of the git repositories, whatever they may be at any given time.
 
-Preliminary release engineering super-repository for building software
-to work with the Cryptech "Alpha" board.
 
-Primary task here is to build the PKCS #11 library and any needed
-support tools for whichever platforms we support.  This will involve
-some packaging voodoo.
+## Firmware Build Process ##
 
-Our first targets for this are Debian and Ubuntu, probably the Jessie
-and Xenial releases, respectively.  If we really need to support
-multiple releases for each of these platforms, the packaging mechanics
-become more complicated, so we may just stop here for these platforms
-and assume we can fill any odd corners using the associated source
-package.
+The firmware build process consists of five main phases:
 
-Our next target for this is likely to be Mac OS X.  This should be
-relatively straightforward so long as we only have to support Homebrew
-and we don't have to produce Homebrew "bottles" (binary packages).  If
-we do need to bottle, we either need one or more Mac build machines or
-we need some kind of cross-compilation scheme (eg,
-https://github.com/tpoechtrager/osxcross).
+1.  Building a "shadow" directory tree populated with symlinks back to
+    the source tree.  This allows us to preserve binaries between
+    compilation runs where appropriate, without cluttering up the
+    source tree with various intermediate files which don't belong
+    there.  In particular, this allows to skip the Verilog synthesis
+    stage when nothing there has changed, which makes the overall
+    build process run significantly faster.
 
-Supporting Homebrew at all requires a bit of extra voodoo on top of
-supporting Debian packaging, but none of it looks particularly
-difficult, and the Debian packaging will produce the source tarball we
-need for the Homebrew formula, so integrating production of these two
-kinds of packaging makes some sense.
+2.  Synthesizing the Verilog source code into a bitstream.  This phase
+    generates a single bitstream file, suitable for loading into the
+    Alpha's FPGA chip.
 
-Windoze is not currently on the radar.  In theory, MinGW would suffice
-as a cross compiler if and when we have to do something about it.
+3.  Cross-compiling C code for the Alpha's ARM processor.  This
+    produces two images: the bootloader, and the HSM firmware itself.
 
-This README is probably obsolete by the time you're reading it.
+4.  Cross compiling C code for the Alpha's AVR MCU.  This produces a
+    single image, which runs the tamper-response controller.
+
+5.  Packaging all of the firmware created in the above steps into a
+    tarball, with some meta-data describing the package, including
+    SHA-2 digests of all of the firmware image files and a signature
+    over the entire meta-data block.
+
+The precise formats which show up in the firmware tarball are subject
+to change.  At the moment, they consist of:
+
+*  A raw bitstream (".bit" file) for the Xilinx Artix-7 FPGA.
+
+*  ELF and raw binary image files for the Cortex M4 ARM CPU; we supply
+   both .elf and .bin files because some tools want one format, some
+   want the other.
+
+*  A ".hex" image for the AVR MCU.
+
+The overall packaging for the firmware tarball is intentionally pretty
+boring: tar, gzip, gpg, and JSON.  The intent is that users be able to
+use our firmware tarball even if the scripts we provide are unsuitable
+for some reason.
+
+The final result of the firmware build process is the firmware
+tarball, which is written into the source/ tree for inclusion by the
+software packaging phase.
+
+
+## Software Build Process ##
+
+The software build process also consists of several phases, but is a
+bit more open-ended than the firmware build process.  Phases in the
+current build process:
+
+1.  Building a source tarball and Debian source package (".dsc").
+    The Debian and Ubuntu Linux distributions are our primary
+    development platforms, so we need to produce packages for them in
+    any case, and the process of producing a Debian-family source
+    package produces a source tarball as one of its outputs, so we get
+    that for free by doing this step first.
+
+2.  Running "pbuilder" to generate clean binary packages for the
+    "i386" and "amd64" architectures for Debian Jessie and Ubuntu
+    Xenial.
+
+3.  Loading all of the Debian and Ubuntu packages produced above into
+    the staging instances of a set of APT repositories.
+
+4.  Uploading changes from the staging repositories to our public APT
+    repositories.
+
+Planned, but missing from the above, is generation of a Homebrew
+formula for Mac OS X and loading that formula along with the
+corresponding source tarball to a public Homebrew "tap".  At least the
+initial version of this will likely be a source-only formula; in
+theory, it would be possible for us to cross-compile Mac OS X binaries
+without needing an expensive build farm (eg, using
+https://github.com/tpoechtrager/osxcross), but we haven't sorted
+through either the technical or licensing issues.
+
+In theory, it would also be possible to produce Windows binaries using
+the MinGW cross-compilation environment, but Windows is sufficiently
+different from all other platforms that even minimal Windows support
+would almost certainly require extensive source code changes, so we
+have not put any serious thought into build issues for Windows.