Nvidia AGX Orin

Install Kairos on Nvidia AGX Orin

Warning

Despite the Flavor you may have selected to look into the docs. The Nvidia AGX Orin only works with Ubuntu 20.04

This page describes how to install Kairos on Nvidia AGX Orin in the eMMC.

Prerequisites

Nvidia AGX Orin
An USB type-C cable
A Linux host used to flash the Nvidia AGX Orin board
Jetson linux SDK download

You can find debugging information here: https://developer.ridgerun.com/wiki/index.php/NVIDIA_Jetson_Orin/In_Board/Getting_in_Board/Serial_Console

Flashing

We are going to write the partitions in the eMMC. In order to do this we will use the Nvidia SDK configured with a custom partitioning layout.

The partitions are:

OEM for storing cloud config files (/oem)
COS_STATE for storing the active/passive images to boot the system
EFI for storing the efi shell and grub to boot the system
RECOVERY - to store the recovery system
PERSISTENT - this is an optional partition to store the persistent data of the system. you can either write this in the eMMC or, for instance, to an external storage. It is enough to create a partition and label it as COS_PERSISTENT. There can be only one partition with such label, the first that matches wins.

Prepare the SDK

The Jetson Linux SDK is used to perform the flashing process.

Download the Jetson Linux SDK:

wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v3.1/release/jetson_linux_r35.3.1_aarch64.tbz2 -O tegra.bz2
tar xvf tegra.bz2

Now, we are going to prepare the rootfs and the bootloader. The Jetson Linux SDK requires the rootfs to generate the system.img file and continue the flashing process however, we will not use the image generated by the SDK as we will use a different set of images (see below). Here we also disable extlinux as Kairos uses GRUB:

cd Linux_for_Tegra
# Drop extlinux
echo "" > ./bootloader/extlinux.conf
# This is needed so the SDK doesn't complain of missing files (not really used in the flash process)
IMAGE=quay.io/kairos/ubuntu:20.04-core-arm64-nvidia-jetson-agx-orin-v3.2.3
docker run -ti --rm -v $PWD/rootfs:/rootfs quay.io/luet/base util unpack "$IMAGE" /rootfs
# workaround needed (SDK writes to the symlink)
rm rootfs/boot/initrd
# Extlinux is required by the SDK - so we fake it in our root (it will not be there eventually)
mkdir -p rootfs/boot/extlinux/
echo "" > rootfs/boot/extlinux/extlinux.conf

Prepare the images

You can either download pre-built Kairos images, or build your own from a container image. You can find Kairos core ubuntu images based on Ubuntu 20.04 here: https://quay.io/repository/kairos/ubuntu (search for nvidia in the tags)

We will download pre-built images from a container image. Pre-built images are using https://quay.io/repository/kairos/ubuntu and contains .img files that can be used for flashing. Img files are pushed automatically by the Kairos CI in https://quay.io/repository/kairos/ubuntu (search for tags with nvidia).

KAIROS_VERSION=v3.2.3
IMAGE=quay.io/kairos/ubuntu:20.04-core-arm64-nvidia-jetson-agx-orin-$KAIROS_VERSION-img
docker run -ti --rm -v $PWD/bootloader:/rootfs quay.io/luet/base util unpack "$IMAGE" /rootfs
mv bootloader/build/*.img bootloader

If you are customizing the image, or either modifying the default partition sizes you can build the images by running:

IMAGE=quay.io/kairos/ubuntu:20.04-core-arm64-nvidia-jetson-agx-orin-v3.2.3
docker run --privileged \
        -e container_image=$IMAGE \
        -e SIZE="15200" \
        -e STATE_SIZE="14000" \
        -e RECOVERY_SIZE="10000" \
        -e DEFAULT_ACTIVE_SIZE="4500" \
        -v $PWD/bootloader:/bootloader --entrypoint /prepare_arm_images.sh -ti --rm quay.io/kairos/osbuilder-tools

If you have instead the rootfs as a directory, you can create the required partitions with:

PATH=/rootfs/path
docker run --privileged \
        -e directory=/rootfs \
        -e SIZE="15200" \
        -e STATE_SIZE="14000" \
        -e RECOVERY_SIZE="10000" \
        -e DEFAULT_ACTIVE_SIZE="4500" \
	-v $PATH:/rootfs \
        -v $PWD/bootloader:/bootloader --entrypoint /prepare_arm_images.sh -ti --rm quay.io/kairos/osbuilder-tools

After running any of the commands above, the generated images files required for flashing will be inside the bootloader directory (bootloader/efi.img, bootloader/recovery_partition.img, bootloader/state_partition.img, bootloader/oem.img, bootloader/persistent.img ).

Note

The persistent image is optional, as you can store the system persistent data rather in an SD card or an NVME disk. The default persistent.img is of 2GB size. To create a persistent image manually of the size you prefer instead you can run:

# Create a 2GB filesystem for COS_PERSISTENT volume
truncate -s $((2048*1024*1024)) bootloader/persistent.img
mkfs.ext2 -L "COS_PERSISTENT" bootloader/persistent.img

Note that the size of the partitions you modify should be duly reported in the partition layout (see below).

Edit the parition layout

We are going now to modify the partition layout in bootloader/t186ref/cfg/flash_t234_qspi_sdmmc.xml which corresponds to the partitioning of the AGX Orin board. An example config file can be found in here.

wget 'https://kairos.io/examples/images/flash_t234_qspi_sdmmc.xml' -O bootloader/t186ref/cfg/flash_t234_qspi_sdmmc.xml

If you are editing the partition sizes and generating the images manually, use the example config file as a baseline and edit the size accordingly to the corresponding partitions (find the respective filename and compare the file size, see the notes below). Otherwise, if you want to use the original file, identify the “APP” partition ( <partition name="APP" id="1" type="data"> ), remove it , and add the following instead:

     <partition name="esp" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
	    <size> 20971520 </size>
	    <file_system_attribute> 0 </file_system_attribute>
	    <partition_type_guid> C12A7328-F81F-11D2-BA4B-00A0C93EC93B </partition_type_guid>
	    <allocation_attribute> 0x8 </allocation_attribute>
	    <percent_reserved> 0 </percent_reserved>
            <filename> efi.img </filename>
            <description> **Required.** Contains a redundant copy of CBoot. </description>
        </partition>
        
       <partition name="COS_RECOVERY" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 10485760000 </size>
            <allocation_attribute>  0x8 </allocation_attribute>
            <filename> recovery_partition.img </filename>
            <description>  </description>
        </partition>
        <partition name="COS_STATE" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 14680064000 </size>
            <allocation_attribute>  0x8 </allocation_attribute>
            <filename> state_partition.img </filename>
            <description>  </description>
        </partition>
        <partition name="COS_OEM" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 67108864 </size>
            <allocation_attribute>  0x8 </allocation_attribute>
            <filename> oem.img </filename>
            <description>  </description>
        </partition>
        <partition name="COS_PERSISTENT" type="data">
            <allocation_policy> sequential </allocation_policy>
            <filesystem_type> basic </filesystem_type>
            <size> 2147483648 </size>
            <allocation_attribute>  0x8 </allocation_attribute>
            <filename> persistent.img </filename>
            <description>  </description>
        </partition>

Note

The COS_PERSISTENT partition is optional. You can also use an SD card, or an nvme drive instead. The only requirement is to have the partition labeled as COS_PERSISTENT.

Note

If modifiying the parition sizes, you need to replace <size> of each partition in the XML:

stat -c %s bootloader/efi.img
stat -c %s bootloader/recovery_partition.img
stat -c %s bootloader/state_partition.img
stat -c %s bootloader/oem.img
stat -c %s bootloader/persistent.img

Flash

To flash the images to the Orin board

Put the board in recovery mode
Run:

sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1

Booting

The Orin board now should boot. If you are connected over the serial you can login with: kairos/kairos, similarly if you have plugged it to the network you should be able to SSH in as well.

Notes

USB Timeout error

It is possible that during flashing on certain kernel versions to see an error message:

[ 0.3623 ] tegrarcm_v2 --new_session --chip 0x23 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader.bin.encrypt --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --download bct_mb1 mb1_bct_MB1_sigheader.bct.encrypt
[ 0.3630 ] BR_CID: 0x80012344705DD25D1C00000019028240
[ 0.3932 ] Sending bct_br
[ 0.4409 ] ERROR: might be timeout in USB write.
[ 5.5325 ]

See also the relevant Nvidia discussions in the forum:

The solution here is trying with a different kernel version, as suggested in the Nvidia threads.

Default configuration

To customize the default cloud config of the board, generate the images mounting the cloud config you want in the images in /defaults.yaml:

IMAGE=quay.io/kairos/ubuntu:20.04-core-arm64-nvidia-jetson-agx-orin-v3.2.3
CLOUD_CONFIG=/cloud/config.yaml
docker run -v $CLOUD_CONFIG:/defaults.yaml --privileged \
        -e container_image=$IMAGE \
        -e STATE_SIZE="14000" \
        -e RECOVERY_SIZE="10000" \
        -e DEFAULT_ACTIVE_SIZE="4500" \
        -v $PWD/bootloader:/bootloader --entrypoint /prepare_arm_images.sh -ti --rm quay.io/kairos/osbuilder-tools

Debugging

Use the micro USB as debug serial port with minicom to debug any booting issues.

sudo minicom -D /dev/ttyACM0 -8 -b 115200

Flashing port

In order to flash the Nvidia AGX Orin you will need to use the USB Type-C ports. The Micro USB port is reserved only for debugging over the serial console.

Last modified September 5, 2024: Fix typo in nvidia agx installation instructions (b585e3f)