Safely Upgrading Boot Software
==============================

This document describes how to use the BlueField alternate boot
partition support feature to safely upgrade the boot software.  We
give the requirements that motivate the feature and explain the
software interfaces that are used to configure it.


The Problem
-----------

In most deployments, BlueField's ARM cores are expected to obtain
their software stack from an on-board Embedded Multi-Media Card (eMMC)
device.  Even in environments where the final OS kernel is not kept on
eMMC -- for instance, systems which boot over a network -- the initial
booter code will still come from eMMC.

Most software stacks need to be modified or upgraded at some point in
their lifetime.  Doing this on a fielded BlueField system involves
some risk; if there's a severe problem with the new software, system
operation may be impaired to the extent that it is impossible to do
further modifications to fix the problem.  Ideally, one would
like to be able to install a new software version on a BlueField
system, try it out, and then fall back to a previous version if the
new one does not work.  In some environments, it's important that this
fallback operation happen automatically, since there may be no
physical access to the system.  In others, there may be an external
agent, like a service processor, which could manage the process.


Solution Overview
-----------------

In order to satisfy the requests listed above, we do the following:

- Provide two software partitions on the eMMC, 0 and 1.  At any given
  time, one area is designated the primary partition, and the
  other the backup partition.  The primary partition is the one we'll
  boot from on the next reboot or reset.

- Allow software running on the ARM cores to declare that the primary
  partition is now the backup partition, and vice versa.  (For
  brevity, in the remainder of this document, we'll term this
  "swapping the partitions", although we're just modifying a pointer;
  the data on the partitions does not move.)

- Allow an external agent, like a service processor, to swap the
  primary and backup partitions.

- Allow software running on the ARM cores to reboot the system, while
  starting an upgrade watchdog timer.  If the upgrade watchdog
  expires, the system will reboot, but the primary and backup
  partitions will first be swapped.


Background
----------

Before explaining the implementation of the solution, we need to go
over some background on the BlueField boot process.

The BlueField boot flow comprises 4 main phases:

- Hardware loads ARM Trusted Firmware (ATF)
- ATF loads Unified Extensible Firmware Interface (UEFI); together ATF
  and UEFI make up the booter software
- UEFI loads the operating system, such as the Linux kernel
- The operating system loads applications and user data

When booting from eMMC, these stages make use of two different types
of storage within the eMMC part:

- ATF and UEFI come from a special area known as an eMMC boot
  partition.  Data from a boot partition is automatically streamed
  from the eMMC device to the eMMC controller under hardware control
  during the initial bootup.  Each eMMC device has two boot
  partitions, and the partition that is used to stream the boot data
  is chosen by a non-volatile configuration register in the eMMC.

- The operating system, applications, and user data come from the
  remainder of the chip, known as the user area.  This area is
  accessed via block-size reads and writes, done by a device driver or
  similar software routine.


The mlxbf-bootctl Program
-------------------

Access to all the boot partition management is via a program shipped
with the BlueField software called "mlxbf-bootctl".  The binary is
shipped as part of our Yocto image (in /sbin) and the sources are
shipped in the "src" directory in the BlueField Runtime Distribution.
A simple "make" command will build the utility.

The mlxbf-bootctl syntax is as follows, and will be explained in more detail
in the following sections::

  syntax: mlxbf-bootctl [--help|-h] [--swap|-s] [--device|-d MMCFILE]
                        [--output|-o OUTPUT] [--read|-r INPUT]
                        [--bootstream|-b BFBFILE] [--overwrite-current]
                        [--watchdog-swap interval | --nowatchdog-swap]

  --device: Use a device other than the default /dev/mmcblk0

  --bootstream: Write the specified bootstream to the alternate
    partition of the device.  This queries the base device
    (e.g. /dev/mmcblk0) for the alternate partition, and uses that
    information to open the appropriate boot partition device,
    e.g. /dev/mmcblk0boot0.

  --overwrite-current: Used with "--bootstream" to overwrite the current
    boot partition instead of the alternate one (not recommended)

  --output: Used with "--bootstream" to specify a file to write the boot
    partition data to (creating it if necessary), rather than using an
    existing master device and deriving the boot partition device.

  --read: Read a bootstream and convert it back to a BFBFILE specified
    by --bootstream.  For example use "mlxbf-bootctl --read /dev/mmcblk0boot0
    --bootstream current.bfb" to read the current bfb installed on boot
   partition zero.

  --watchdog-swap: Arrange to start the ARM watchdog with a countdown
    of the specified number of seconds until it fires; also, set the
    boot software so that it will swap the primary and alternate
    partitions at the next reset.

  --nowatchdog-swap: Ensure that after the next reset, no watchdog
    will be started, and no swapping of boot partitions will happen.


Updating the Boot Partition
---------------------------

To update the boot partition on the ARM cores, let us assume we have a
new bootstream file, e.g. "bootstream.new".  We would like to install
and validate this new bootstream.  To just update and hope for the
best, you can do::

  # mlxbf-bootctl --bootstream bootstream.new --swap
  # reboot

This will write the new bootstream to the alternate boot partition,
swap alternate and primary so that the new bootstream will be used on
the next reboot, then reboot to use it.

(You can also use --overwrite-current instead of --swap, which will
just overwrite your current boot partition, but this is not
recommended as there is no easy way to recover if the new booter code
does not bring the system up.)


Safely Updating with a BMC
--------------------------

With a BMC (board management processor) you can do better.  The ARM
cores simply notify the BMC prior to the reboot that an upgrade is
about to happen.  Software running on the BMC can then be implemented
that will watch the ARM cores after reboot.  If after some time the
BMC has not seen the ARM cores come up properly (for whatever
definition is appropriate for the application in question), it can use
its USB debug connection to the ARM cores to properly reset the ARM
cores, first setting a suitable mode bit that the ARM booter will
respond to by switching the primary and alternate boot partitions as
part of resetting into its original state.


Safely Updating from the ARM Cores
----------------------------------

Without a BMC, you can use the ARM watchdog to achieve similar
results.  If something goes wrong on the next reboot and the system
does not come up properly, it will reboot and return to the original
configuration.  In this case, you might run::

  # mlxbf-bootctl --bootstream bootstream.new --swap --watchdog-swap 60
  # reboot

With these commands, you will reboot the system, and if it hangs for
60 seconds or more, the watchdog will fire and reset the chip, the
booter will swap the partitions back again to the way they were
before, and the system will reboot back with the original boot
partition data.  Similarly, if the system comes up but panics and
resets, the booter will again swap the boot partition back to the way
it was before.

You must ensure that Linux after the reboot is configured to boot up
with the ``sbsa_gwdt`` driver enabled.  This is the SBSA (Server Base
System Architecture) Generic WatchDog Timer.  As soon as the driver is
loaded, it will start refreshing the watchdog and preventing it from
firing, which will allow your system to finish booting up safely.  In
the example above, we allow 60 seconds from system reset until the
Linux watchdog kernel driver is loaded.  At that point, your
application may open /dev/watchdog explicitly, and the application
would then become responsible for refreshing the watchdog frequently
enough to keep the system from rebooting.

For documentation on the Linux watchdog subsystem, see the Linux
watchdog documentation, e.g.

https://www.kernel.org/doc/Documentation/watchdog/watchdog-api.txt

For example, to disable the watchdog completely, you can run::

  # echo V > /dev/watchdog

You can incorporate other features of the ARM generic watchdog into
your application code using the programming API as well, if you wish.

Once the system has booted up, in addition to disabling or
reconfiguring the watchdog itself if you wish, you should also clear
the "swap on next reset" functionality from the booter by running::

  # mlxbf-bootctl --nowatchdog-swap

Otherwise, next time you reset the system (via reboot, external reset,
etc) it will assume a failure or watchdog reset occurred and swap the
eMMC boot partition automatically.

The above steps can be done manually, or can be done automatically by
software running in the newly-booted system.


Changing the Kernel or Userspace
--------------------------------

The above solutions simply update the boot partition to hold new
booter software (ATF and UEFI).  If you want to also provide a new
kernel image and/or new userspace, you should partition your eMMC into
multiple partitions appropriately.  For example, you might have a
single FAT partition that UEFI can read the kernel image file from,
but the new bootstream contains a UEFI bootpath pointing to an updated
kernel image.  Similarly, you might have two Linux partitions, and
your upgrade procedure would write a new filesystem into the "idle"
Linux partition, then reboot with the bootstream holding kernel boot
arguments that direct it to boot from the previously idle partition.
The details on how exactly to do this depend on the specifics of how
and what need to be upgraded for the specific application, but in
principle any component of the system can be safely upgraded using
this type of approach.
