Qualcomm Qualcomm
Qualcomm QCM6490 · RB3 Gen2

Kodiak

Build · Flash · Boot

How the firmware stack is assembled from source,
programmed onto the board, and brought to life.

A build system for Qualcomm TZ open firmware development.

TF-A OP-TEE U-Boot Linux + UKI Buildroot QDL Flash

Agenda

Platform Overview

SoC · board · software stack

Build System Architecture

Top-level Makefile · component layout

Make Targets

Build targets and their outputs

Proprietary Platform Blobs

Skip Yocto — fetch blobs directly

Yocto Integration

Optional BSP for firmware blobs

Flashing

QDL / Firehose · flash-loader vs flash-efi

Flashing via LAVA

Remote lab boards · flash-lava · flash-lava-yocto

Boot Chain

PBL → XBL → TF-A → OP-TEE → U-Boot → Linux

Boot Loader Map

load addresses & partitions

Arm Exception Levels

EL3 / S-EL1 / EL2 at runtime

TF-A

BL31 runtime · FIP image

QTISECLIB

libqtisec.a · linked into BL31

OP-TEE OS

BL32 — Secure World Kernel

optee-client

libteec.so · tee-supplicant daemon

U-Boot

BL33 — UEFI bootloader

Linux Kernel

EL2 device tree · built-in drivers

Linux at EL2 — KVM

open hypervisor (KVM, not Gunyah)

xtest

OP-TEE test suite on the board

UKI Assembly

Unified Kernel Image → efi.bin

Platform Device Trees

U-Boot vs kernel DTB

Buildroot

Root filesystem → initramfs

Tests

on-board validation

Summary

Key facts and commands

Platform Overview

SoC

  • Qualcomm QCM6490
  • Kryo 670 (AArch64-only)
  • Adreno 643 GPU
  • Hexagon DSP (ADSP/CDSP)
  • WCSS (Wi-Fi/BT)

Board

  • Qualcomm RB3 Gen2
  • Development / EVK board
  • UFS storage
  • Serial: ttyMSM0
  • USB EDL / QDL flash

Software Stack

  • TF-A (BL2 + FIP)
  • OP-TEE OS (BL32)
  • U-Boot (BL33 / UEFI)
  • Linux/KVM (EL2) + Buildroot initramfs
  • Packaged as Unified Kernel Image

What's in the FIP

The Firmware Image Package is TF-A's single container for the boot-chain images loaded after BL2. TF-A packs three payloads, then wraps fip.bin as an ELF (uefi.elf) loaded at 0x9fc00000:

  • BL31 — TF-A EL3 runtime / secure monitor (opteed SPD)
  • BL32 — OP-TEE OS tee-raw.bin (Secure EL1)
  • BL33 — U-Boot u-boot.bin (Non-secure bootloader → UKI)
Not in the FIP
BL2 = tz.mbn — built by TF-A, signed with sectools, loaded separately by XBL

Everything above EL3 is open-source and built from source.
The low-level XBL firmware and signing keys remain Qualcomm proprietary.

Build System Architecture

Entry Point

  • kodiak.mk — top-level GNU Make
  • Calls into common.mk for shared targets
  • Toolchain: aarch64-none-linux-gnu-
  • All components are AArch64-only (64-bit NS user, kernel, S user, S kernel)

Key Directories

  • kodiak/input/ — pre-built blobs & flash support
  • kodiak/output/ — generated artifacts
  • kodiak/security/ — signing profile + seclib
  • kodiak/patches/ — kernel DTS patch
  • kodiak/ukify/ — EFI stub for UKI
make tfa —Bootloader pipeline→ make flash-loader
OP-TEE OS
tee-raw.bin
+
U-Boot
u-boot.bin
→
make tfa
build BL2 · sign tz.mbn
pack FIP → uefi.elf
→
make flash-loader
QDL / Firehose
make efi —Kernel + Root pipeline→ make flash-efi
Linux
Image · el2.dtb
+
Buildroot
rootfs.cpio.gz
→
make efi
uki.efi → efi.bin
→
make flash-efi
QDL / Firehose

Make Targets

Target What it does
Open Firmware — built from source
tfa BL31 — EL3 runtime firmware (TF-A)
optee-os BL32 — OP-TEE secure OS
u-boot BL33 — UEFI bootloader
linux Kernel + DTB — drivers built-in, no modules
buildroot Initramfs root filesystem
efi efi.bin — UKI bundling kernel + initramfs + DTB + cmdline
fetch-blobs Download Qualcomm firmware blobs — no Yocto build needed
flash-loader Flash tz.mbn + uefi.elf + blobs — first flash / boot-chain change
flash-efi Flash efi.bin (UKI) only — fast kernel iteration
Reference Firmware — Qualcomm Yocto BSP
yocto Build the complete reference Yocto BSP release
flash-yocto Install the reference release on the board — verify against the stock BSP

Proprietary Platform Blobs Skipping Yocto — direct download

Both upstream URLs are publicly accessible — no Qualcomm account required. make fetch-blobs automates all three steps below and populates kodiak/blobs/, which the flash targets use as a fallback.

Source: Qualcomm Software Center
sha256 25b19ccc…f85149

① Boot Binaries — 15 MB

SoC boot chain only — no DSP firmware.

  • XBL, XBL config, AOP
  • CPUCP, DEVCFG, HYP, SHRM
  • QUP FW, imagefv, prog_firehose.elf

② CDT File — 380 KB

Customer Device Tree.

  • Board-specific hardware config blob
  • Consumed by XBL early in boot
Source: CodeLinaro
sha256 0fe1c0b4…e0c557e

③ GPT Tables & rawprogram XMLs

Partition layout for QDL.

  • GPT bins + rawprogram / patch XMLs
  • Committed under platforms/qcs6490-rb3gen2/
  • make fetch-blobs does a shallow clone & copies them

④ DSP Firmware & Runtime

Loaded at runtime, baked into the rootfs.

  • Remoteproc images qcom/qcm6490/*.mbn·jsn
  • AudioReach tplg qcom/qcs6490/
  • Venus codec qcom/vpu-2.0/venus.mbn
  • FastRPC runtime fastrpc_shell + skels
  • Staged into kodiak/overlay/ → rootfs / UKI

Yocto Integration Optional BSP for firmware blobs

Why Yocto?

The flash procedure needs Qualcomm-built firmware binaries (XBL, AOP, CDSP firmware, WPSS, firehose programmer, GPT tables). These blobs are publicly available — no build required.

Faster: make fetch-blobs downloads them directly in minutes. make yocto is the multi-hour alternative when a full BSP image is needed.

make yocto — full BSP (hours)

  • Clones meta-qcom from GitHub
  • Builds via kas using rb3gen2-core-kit.yml
  • Produces core-image-base-rb3gen2-core-kit.rootfs.qcomflash/
  • Flash targets auto-copy from this deploy dir

Use make fetch-blobs instead to skip the build entirely.

What comes from Yocto vs this repo
From YoctoFrom this repo
XBL + XBL configTF-A BL2 (tz.mbn)
AOP firmwareFIP / uefi.elf
ADSP / CDSP FWLinux kernel
WPSS firmwareBuildroot initramfs
Prog firehose ELFUKI (uki.efi)
GPT tablesefi.bin (FAT32)
rawprogram XMLs

Flashing QDL — Qualcomm Download Protocol

The board is held in EDL (Emergency Download Mode) over USB. The host sends a firehose programmer ELF, then uses XML manifests to write partition images directly via the Sahara/Firehose protocol.

Prerequisite: the qdl flasher must be on PATH — build it from linux-msm/qdl.

make flash-loader

  • Programs the bootloader chain only
  • firmware blobs, XBL, GPT
  • TF-A (tz.mbn) + FIP (uefi.elf)
  • Use after a full rebuild / first provisioning
  • Follow with make flash-efi for the kernel

make flash-efi

  • Programs only the EFI partition (efi.bin)
  • Filters rawprogram0-only-kernel.xml from rawprogram0.xml
  • Fast kernel / initramfs iteration
  • Firmware untouched

Flash Artifacts — Lookup Order

Each required file is resolved in priority order — the first hit wins:

1
kodiak/input/
→
2
Yocto deploy dir
→
3
kodiak/blobs/
(make fetch-blobs)
→
error

Steps 2 and 3 are alternatives — Yocto deploy wins over kodiak/blobs/, so a make yocto build makes make fetch-blobs unused.

Flash Procedure — Step by Step

  • 1
    Enter EDL mode Hold Volume-Down while connecting USB, or issue the adb reboot edl command. The host sees a QDL USB device.
  • 2
    Load the firehose programmer QDL uploads prog_firehose_ddr.elf into SRAM via the Sahara protocol. The programmer initialises DDR and the UFS controller.
  • 3
    Write GPT partition tables gpt_main*.bin / gpt_backup*.bin are written first to lay out the partition map on UFS LUNs.
  • 4
    Flash partition images via rawprogram XML The XML manifests map each image file to a partition. flash-loader uses all rawprogram XMLs; flash-efi uses only the filtered EFI-partition XML.
  • 5
    Apply patches (patch*.xml) GPT corrections and CRC patches are applied. Board resets and boots from the freshly written images.

Flashing via LAVA remote lab boards — no local USB

flash-loader / flash-efi run qdl over USB against a board plugged into your machine. The LAVA lab boards are remote and not USB-reachable, so LAVA fetches a single flat flash tarball over HTTP onto its worker and runs qdl there (deploy: to: qdl + boot: method: qdl, storage: ufs). The make targets package the tarball and then submit it to a free qcs6490-rb3gen2 board via the LAVA API — one command, end to end (next slide).

make flash-lava

The LAVA counterpart of flash-loader + flash-efi combined. Packages one flat kodiak-flash.qcomflash.tar.gz uniting both sets: LUNs 1,2,4 (loader: firehose, XBL, GPTs, tz.mbn, uefi.elf, rawprogram4 with dtb.bin stripped) and LUN 0 (kernel: rawprogram0-only-kernel, efi.bin).

No rootfs.img — boots the flashed loader + kernel, but a login prompt needs a rootfs already on the board. Blobs are pulled on demand (fetch-blobs), each file resolved kodiak/input/ → Yocto deploy → kodiak/blobs/.

make flash-lava-yocto

The LAVA counterpart of flash-yocto. Packages the complete Yocto release — all LUNs + every patch, including rootfs.img — straight from the built .qcomflash directory, tarred as-is.

Self-sufficient: because rootfs.img is included, a job can boot to a login prompt on a bare board. Needs make yocto.

Tarball layout — path: matters

qdl runs relative to the path: inside the tarball. Both make targets emit a flat tarball (files at the root) → omit path:. Yocto's own emitted *.qcomflash.tar.gz is nested under core-image-base-rb3gen2-core-kit/ → set path: to it. Wrong path: ⇒ qdl fails instantly with failed to open "prog_firehose_ddr.elf".

LAVA Flash — One Command make does it all

make flash-lava / make flash-lava-yocto package the tarball and drive the whole LAVA round-trip via kodiak/lava/submit.sh — upload, submit, monitor. Build only, no lab: make flash-lava-package.

What the target runs, end to end:

  • 1
    Package the tarball The *-package target assembles kodiak/output/*.qcomflash.tar.gz (flat layout).
  • 2
    Mint a lab-reachable upload URL The worker fetches over HTTPS, so the tarball needs a URL. Minting one is an MCP-only op (create_artifact_upload) — LAVA's REST API has none and it won't answer plain curl. Since make can't speak MCP, submit.sh shells out to headless claude -p, which calls the tool and returns a token-guarded URL.
  • 3
    Upload + submit curl PUTs the tarball, then the job YAML from kodiak/lava/ (with <ARTIFACT_URL>/<ARTIFACT_TOKEN> filled) is POSTed to a free qcs6490-rb3gen2 via the REST API (Authorization: Token). The lab enters EDL automatically (bootToEDL) — no manual button.
  • 4
    Deploy → flash → boot LAVA downloads + extracts the tarball, runs qdl across all LUNs (storage: ufs), then reboots and can auto-login to confirm.
  • 5
    Monitor submit.sh polls to completion and exits non-zero unless health: Complete. A healthy run shows flash-qdl taking real time (0 s = failed). Full guide: kodiak/lava/README.md.

Token & MCP: mint a LAVA token at /api/tokens/; submit.sh reads it from ~/.claude.json (the X-Lava-Token the lava MCP server uses) or $LAVA_TOKEN. The MCP handles reads + the artifact upload; the actual submit/monitor go through the REST API.

Boot Chain

PBL — Primary Boot Loader Hardware ROM in SoC. Authenticates and loads XBL from UFS. Cannot be modified.
XBL — Secondary Boot Loader Qualcomm proprietary. Initialises DDR, clocks, power rails. Loads and verifies BL2 (tz.mbn), and loads the FIP (uefi.elf) into DRAM at 0x9FC0_0000.
BL2 — TF-A (tz.mbn, signed) Runs at EL3. Uses TF-A's memory-mapped IO backend (io_memmap) to read the FIP already resident in DRAM — no storage access — extracting BL31, BL32 (OP-TEE) and BL33 (U-Boot) by UUID, then hands off to BL31.
BL31 — TF-A EL3 Runtime (Secure Monitor) Resident EL3 firmware (links qtiseclib). Brings up PSCI / GIC, then dispatches to BL32 (OP-TEE) via the opteed SPD before entering BL33.
BL32 — OP-TEE OS Initialises in Secure EL1. Sets up Trusted Application infrastructure. Returns control to TF-A Secure Monitor.
BL33 — U-Boot (UEFI) Runs at Normal World EL2. Presents UEFI runtime. Reads EFI system partition. Discovers bootaa64.efi fallback, then uki.efi.
UEFI Boot — uki.efi U-Boot loads and executes the Unified Kernel Image. The UKI PE/COFF stub sets up the execution environment.
Linux Kernel (EL2, el2 DTB) Kernel boots with the EL2-specific device tree. Mounts the embedded initramfs. Loads OP-TEE kernel driver.
Buildroot Initramfs Busybox init, OpenSSH, networking. OP-TEE supplicant bridges TAs between user space and Secure World.

Boot Loader Map load addresses & partitions

Runtime load map (rb3gen2)
StageLoad addrSize
BL2 — TF-A (tz.mbn)0x1C00_E0001 MB
BL31 — EL3 monitor0x1C20_00001 MB
BL32 — OP-TEE0x1C30_00002 MB
BL33 — U-Boot0x9F80_00004 MB
FIP — uefi.elf0x9FC0_0000≤ 4 MB

BL2/BL31/BL32 live in on-chip IMEM (0x1C00_0000); BL33 & FIP in DDR (0x9F80_0000+). Console UART @ 0x994000 (ttyMSM0).

Flash partitions — built from source

The only three images built from source (everything else is a prebuilt Qualcomm blob):

  • tz_a / tz_b ← tz.mbn — signed ELF/MBN (BL2)
  • uefi_a / uefi_b ← uefi.elf — ELF-wrapped FIP (BL31+BL32+BL33)
  • efi ← efi.bin — FAT32 image (UKI: kernel + DTB + initramfs)
A/B slots & flashing
  • _a / _b — two slot copies of tz & uefi; same image in both
  • efi is single (no A/B)
  • flash-loader → tz + uefi (+ blobs)
  • flash-efi → efi only

Arm Exception Levels at Runtime

EL3 — Secure Monitor (TF-A)

Highest privilege. Handles SMCs between Normal and Secure worlds. After boot, TF-A remains resident as the Secure Monitor but only executes during world-switch.

Secure EL1 — OP-TEE OS

Runs in TrustZone Secure World. Hosts Trusted Applications. Communicates with Linux via the TEE subsystem SMC interface.

Non-secure EL2 — U-Boot & Linux

TF-A drops BL33 (U-Boot) into the Normal World at EL2; U-Boot then enters Linux at the same level — the kernel confirms it on the console: “All CPU(s) started at EL2.” The EL2 DTB overlay remaps IOMMU streams that behave differently without Gunyah.

Normal EL0 — User Space

Buildroot initramfs processes (Busybox, OpenSSH, tee-supplicant). OP-TEE client calls travel EL0 → EL2 (kernel) → EL3 (TF-A) → Secure EL1 (OP-TEE).

TF-A BL31 runtime · BL2 · FIP image

Build flags

  • PLAT: rb3gen2
  • SPD: opteed (OP-TEE dispatcher)
  • E=0: error extensions off
  • Needs: libqtisec.a (QTISECLIB)

Outputs

  • bl31.bin — TF-A EL3 runtime firmware (secure monitor, opteed SPD)
  • fip.bin = BL31 + BL32 (OP-TEE) + BL33 (U-Boot) → uefi.elf @ 0x9fc00000
  • bl2.elf — built by TF-A; signed with sectools into tz.mbn (OEM TEST key) → kodiak/output/

External blobs / tools

  • libqtisec.a — Qualcomm TZ security library (from coreboot/qc_blobs), MD5-verified
  • sectools — Qualcomm's open-source signing tool, fetched on demand to sign BL2 into tz.mbn

BL2 — tz.mbn (always signed)

tz.mbn is always generated by signing the freshly built BL2 — there is no checked-in pre-signed image. The tfa target builds BL2 from TF-A and signs it with the open-source Qualcomm Sec Tools (OEM TEST key):

make tfa

sectools secure-image bl2.elf \
  --outfile kodiak/output/tz.mbn \
  --image-id TZ-TEE \
  --security-profile kodiak/security/kodiak_tzt_security_profile.xml \
  --sign --signing-mode TEST

If signing fails (e.g. sectools unavailable) the tfa target aborts — there is no pre-signed image to fall back to.

QTISECLIB libqtisec.a — linked into BL31

What it is

QTISECLIB (Qualcomm Technologies Inc. Security Library) is a prebuilt, closed-source static library (libqtisec.a) that Qualcomm's upstream TF-A platform port (plat/qti) links into BL31 — the EL3 secure monitor.

The open plat/qti C code is a thin shim that calls into qtiseclib through a fixed interface. The SoC-specific power-management, CPU and security sequences Qualcomm does not open-source live inside the blob.

Provenance in this build
  • Not checked in — verify-qtiseclib fetches it on demand (a prereq of make tfa)
  • SoC variant sc7280 — covers QCM6490 / RB3 Gen2 (Kryo 670)
  • MD5-pinned & verified before it is used
  • Passed to TF-A as QTISECLIB_PATH=, statically linked into BL31 → FIP

QTISECLIB Role in BL31 · interface

Interface — qtiseclib_interface.h

TF-A calls a small, fixed set of entry points; qtiseclib implements them:

  • qtiseclib_bl31_platform_setup() — SoC EL3 bring-up at BL31 entry
  • PSCI backend — CPU on/off/suspend, system off/reset, power-state validation
  • CPU reset / warm-boot entry — secondary-core and resume paths
  • Secure-interrupt dispatch — EL3 ISR for QTI secure IRQs
  • Secure-memory assignment — access-control / carveout programming
How it fits the monitor
  • TF-A supplies the generic runtime — SMC dispatch, GIC framework, PSCI top-half, opteed SPD
  • qtiseclib supplies the SoC bottom-half — the actual hardware sequences
  • qtiseclib calls back into TF-A for console/log, mmap and locks (shared EL3 runtime)
  • The monitor then dispatches to OP-TEE (BL32, opteed) and U-Boot (BL33)

QTISECLIB Drivers & Features

Power Management · PSCI

  • CPU on/off, core & cluster power collapse
  • CPU & system suspend (deep sleep) / resume
  • System reset and power-off
  • Coordinated with AOSS / RPMh and PMIC rails

CPU & Cluster

  • Kryo 670 reset & warm-boot sequences
  • DSU cluster coherency enter / exit
  • Core errata workarounds applied at EL3

Interrupts & SMC

  • GICv3 secure-interrupt setup + EL3 ISR dispatch
  • QTI SiP SMC service backend

Memory Protection & Debug

  • TZ / XPU secure carveouts + mem-assign access control
  • SDI / download-mode & RAM crash-dump handling
  • Minimal EL3 bus / clock bring-up

QTISECLIB Being replaced by open-source drivers

The SoC logic bundled in libqtisec.a is being reimplemented as open TF-A drivers under plat/qti (renamed to the qcom namespace), so BL31 can be built without the blob. The series land in parallel upstream (TrustedFirmware Gerrit) and in qualcomm-linux/trusted-firmware-a.

PSCI · Power

  • Native PSCI backend
  • Backend-provided idle states
  • WFI-based CPU standby
  • Power & CPU topology

Power Infra

  • Resource Power Manager hardened (RPMh) — deinit after BL31 setup
  • Power Domain Controller (PDC) — chipset hooks
  • Inter-Processor Communication Controller (IPCC) — mailbox driver

Clocks

  • Clock driver framework
  • CPUCP clock-domain driver
  • Per-platform clock back-ends

Interrupts · GIC

  • QTI interrupt service
  • GICv3 group / IGRPMODR fixes
  • GICv3 redistributor routing API

Interconnect

  • Interconnect Bus (ICB) — micro-arbiter + core init
  • Network-on-Chip (NoC) — error logger
  • Interconnect Bus (ICB) — config-query driver

Platform Plumbing

  • SMEM driver
  • chipinfo part-gating query
  • Ring-buffer console
  • QTI → QCOM namespace rename

Two blobs, two replacements QTISECLIB → TF-A · QTEE → OP-TEE

QTISECLIB (the EL3/BL31 monitor blob) and QTEE (Qualcomm's secure OS) are different proprietary components with different open-source successors. Notably, the PAS/PIL firmware-authentication that QTEE used to provide is being taken over by OP-TEE — it lands in BL32, not in TF-A.

QTISECLIB → open TF-A drivers (EL3 / BL31)
  • Select the PSCI backend by QTISECLIB_PATH — build with the native backend when the blob is absent
  • Drop the platform banner requiring QTISECLIB — the dependency becomes optional
  • Once the driver set is complete, plat/qcom links no libqtisec.a
  • SoC-monitor functions only: PSCI/power, clocks, GIC, interconnect, sec_core, timers, watchdog, access control
QTEE → OP-TEE (Secure EL1 / BL32)
  • PAS / PIL — authenticate & bring up remoteproc firmware, now an OP-TEE PTA (qcom_pas, the path kodiak uses)
  • Qualcomm platform enablement (Wildcat / Nord / Bruin)
  • qcom crypto driver · software-key ICE PTA
  • Video / Camera PAS (IRIS codec, Camera ICP)
  • RPMB over UFS

Status tracked in the internal firmware-review dashboard — OP-TEE & TF-A, upstream + qualcomm-linux forks.

Open Firmware Initiative a company-wide effort for IoT targets

Open firmware on Qualcomm is the work of many teams across the company, aligned toward a common goal for IoT targets. The initiative was initially led by Tony Hamilton, Sumit Garg and Jorge Ramirez-Ortiz. The changes below are one contributor's slice.

TF-A / BL31 — QTISECLIB drivers · merged upstream (Gerrit)
  • sec_core driver — secondary-core boot / PSCI bring-up #45576 · #45577
  • qtimer driver — stub → real architected timer #45578 · #50251
  • watchdog driver — stub → real driver #45581 · #50252
  • interrupt dispatcher — extended for QTI secure IRQs #45580
  • access control driver — native XPU / secure-memory (kodiak + qcs9075) #45583 · #51964 · #51963
  • XPU bypass + mem_assign hardening; secure-region & PIMEM policy #46895 · #49087 · #50323 · #50565
  • QTI maintainers & qtiseclib-coupling docs #45610 · #51968
OP-TEE / BL32 — QTEE replacement · landing upstream (GitHub)
  • qcom_pas PTA — PAS/PIL firmware auth: ADSP, CDSP, Venus→IRIS; descriptor/ops refactor; early-TA
  • clk: qcom — PAS clock driver + Lucid-EVO PLL helper
  • PRNG driver · ramblur / pIMEM · GENI UART console
  • RPMB over UFS — UFS probe, key provisioning by CID, RPMB reinit #7881
  • Qualcomm MAINTAINERS & platform banner

QTISECLIB drivers merged in TrustedFirmware TF-A (Gerrit); the QTEE→OP-TEE / PAS work is landing in OP-TEE upstream and the qualcomm-linux forks.

OP-TEE OS BL32 — Secure World Kernel

What it is

OP-TEE is an open-source Trusted Execution Environment that runs in the Arm TrustZone Secure World (EL1-S). It hosts Trusted Applications (TAs) and handles Secure Monitor Calls (SMCs) from the Normal World.

Configuration

  • Platform: qcom-kodiak
  • Output: optee_os/out/arm/core/tee-raw.bin
  • Passed to TF-A as: BL32
Role in the Boot Chain

TF-A packs OP-TEE into the FIP image alongside U-Boot. At runtime, TF-A loads OP-TEE into Secure EL1 before handing off to U-Boot in Normal World EL2.

The Linux kernel communicates with OP-TEE via the tee-supplicant daemon and the CONFIG_OPTEE driver, both enabled in the kernel defconfig.

optee-client Normal World Runtime

Library — libteec.so

The TEE Client API library. User-space applications call it to open sessions with Trusted Applications and invoke commands across the Normal/Secure World boundary.

  • TEEC_InitializeContext / TEEC_FinalizeContext
  • TEEC_OpenSession — open a channel to a TA
  • TEEC_InvokeCommand — call a TA entry point
  • TEEC_CloseSession

Buildroot Package

  • Package: optee_client_ext
  • Init script: /etc/init.d/S30optee (auto-start)
  • User tee — runs supplicant, owns /dev/teepriv0
  • Group teeclnt — client apps; includes user test
Daemon — tee-supplicant

Runs in Normal World as a privileged daemon. Acts as the bridge between the OP-TEE Secure World and REE filesystem / hardware resources.

  • TA loading — reads /lib/optee_armtz/*.ta and passes TA binaries to the Secure World on demand
  • Secure storage — mediates RPMB and REE-filesystem secure storage RPCs from OP-TEE OS
  • Socket / time RPCs — handles other Secure World RPC calls that need REE access (entropy, sockets, time)

Without tee-supplicant running, TA loading and secure storage fail silently — always verify it is active before running xtest.

U-Boot BL33 — UEFI Bootloader

Build

  • Defconfig: qcm6490_defconfig
  • Output: u-boot/u-boot.bin
  • Cross-compiler: aarch64-none-linux-gnu-
  • Packed into FIP by TF-A as BL33

Why U-Boot?

U-Boot is configured in UEFI mode so the system presents a standard UEFI firmware interface, and the EFI partition is read using standard UEFI boot discovery.

Boot behaviour
  • Reads the EFI system partition (FAT32 efi.bin)
  • Falls back to EFI/BOOT/bootaa64.efi as the UEFI fallback path
  • Discovers and launches EFI/Linux/uki.efi (the Unified Kernel Image)
  • Hands kernel execution over at EL2 once the UKI is loaded

In short: BOOTSTD bootflow scan walks the UFS GPT partitions, finds the FAT ESP holding \EFI\BOOT\bootaa64.efi, and launches it → the UKI at EFI/Linux/uki.efi.

Linux Kernel + Kernel Patches

Build Config

  • Defconfig: defconfig (overridable)
  • Auto-enabled: CONFIG_TEE, CONFIG_OPTEE
  • Auto-enabled: CONFIG_EFI_BOOT, CONFIG_EFI_ZBOOT
  • Auto-enabled: CONFIG_DEVTMPFS, CONFIG_QCOM_WDT
  • PAS/DSP built-in: QCOM_PAS_TEE, QCOM_Q6V5_PAS, QCOM_FASTRPC, RPMSG/GLINK
  • Video: VIDEO_QCOM_VENUS=y, VIDEO_QCOM_IRIS=n
  • CONFIG_MODULES=n — every driver built-in (RAM initramfs, no module autoload)
  • Output: arch/arm64/boot/Image

Console

console=ttyMSM0,115200
qcom_scm.download_mode=1
earlycon
Kernel Patches — kodiak/patches/linux/

0001-...-el2-set-DSP-firmware-name.patch — EL2 device tree overlay: builds qcs6490-rb3gen2-el2.dtb, the variant needed when running at EL2 without the Gunyah hypervisor.

  • Enables the Qualcomm watchdog
  • Disables the GPU Zap shader (not needed at EL2)
  • Remaps IOMMU streams for ADSP, CDSP, WPSS remoteprocs
  • Overrides ADSP/CDSP firmware-name to qcom/qcm6490/ so the PAS driver finds the images
  • Enables the Venus video codec (status + SMMU streams + video-firmware node)

0002-venus-support-pas-framework-over-op-tee.patch — loads and authenticates the Venus firmware through the PAS path over OP-TEE instead of QTEE/SCM.

  • Selects OP-TEE via the /firmware/optee DT node (use_tz)
  • Firmware brought up with qcom_mdt_load + qcom_pas_auth_and_reset
  • Remote-state via qcom_pas_set_remote_state() (SCM fallback dropped)

Linux at EL2 — KVM, not Gunyah simplifies the Gunyah replacement · eases OP-TEE integration

  • EL2 is the hypervisor level; Kodiak gives it to Linux itself (Arm VHE)
  • The hypervisor is KVM — open source, replacing the proprietary Gunyah
  • Simplifies the Gunyah replacement — no proprietary EL2 component in the stack
  • Eases OP-TEE integration — OP-TEE passes resource tables to Linux (at EL2) for the PIL mappings the hypervisor used to do
Downstream — Gunyah at EL2
  • Gunyah (Type-1 hyp) owns EL2
  • Linux (HLOS) runs as a guest VM at EL1
  • Isolates VMs — stage-2 MMU, GIC virtualization
  • Resource Manager creates VMs, brokers memory/IRQ
  • PIL: hyp reassigns memory (HLOS→HYP→subsystem), delegates auth to QTEE
Kodiak — Linux + KVM at EL2
  • Linux owns EL2 directly (Arm VHE)
  • The hypervisor is KVM — open source, replacing Gunyah
  • XBL config gives EL2 to the non-secure world (kvm variant, not gunyah)
  • Guests still possible — hosted by KVM
  • Firmware auth via OP-TEE PAS (qcom_pas), not hyp→QTEE
  • During PIL, OP-TEE passes resource tables to Linux to set up the memory mappings Gunyah used to — possible now that Linux owns EL2
  • Needs the -el2 DTB (patch 0001): watchdog, GPU-zap off, DSP/WPSS SMMU remap
Security implications
  • Stage-2 is not gone — KVM still uses it to confine its guests; the EL2 host always runs stage-1 (true of any hypervisor)
  • Secure world unchanged: TF-A (EL3) + OP-TEE (S-EL1) sit above EL2 — TrustZone + XPU/SMMU + pIMEM enforce it in hardware; a compromised Linux still can't reach secure memory or keys
  • DSP carveout isolation = the secure-world-programmed XPU + the DSP's SMMU streams (EL3/TZ, EL-agnostic), not EL2 stage-2 — so Linux at EL2 is still denied DSP memory at the hardware bus level. Gunyah's stage-2 was only extra CPU-side defense-in-depth
  • Consequence: the TF-A/OP-TEE XPU / access-control correctness is now load-bearing (no hypervisor backstop)
  • Trade-off: the HLOS is no longer sandboxed by a smaller NS hypervisor — the whole Linux kernel becomes the top non-secure privilege (EL2)
  • Fine for a single-owner developer build; keep Gunyah where HLOS containment or multi-VM isolation is required

xtest OP-TEE Test Suite

What it covers

  • Core TEE OS functionality (panics, memory, threads)
  • GlobalPlatform TEE Internal Core API
  • GlobalPlatform TEE Client API
  • Cryptographic operations — AES, RSA, ECC, hash, ECDH
  • Secure Storage — RPMB and REE-filesystem backends
  • Regression suite and benchmarks

Buildroot Package

  • Package: optee_test_ext
  • Depends on: optee_client_ext, openssl
  • TAs installed to: /lib/optee_armtz/
  • Init: S30test-arm-ffa-user for ARM FF-A user tests
  • Source: OP-TEE/optee_test.git (via manifest)
Running xtest on the board

Connect via serial (ttyMSM0 115200) or SSH. Verify tee-supplicant is running before starting.

# Full test suite
xtest

# Regression tests only
xtest -t regression

# Single test case by number
xtest 1001

# List all test identifiers
xtest -l

# Help
xtest -h

Test results show OK / FAILED per case. A clean run ends with "RESULT: regression tests PASSED".

UKI Assembly Unified Kernel Image → efi.bin

A Unified Kernel Image bundles the kernel, device tree, initramfs and the kernel command line into a single PE/COFF EFI binary. make efi assembles it with systemd-ukify (over kodiak/ukify/linuxaa64.efi.stub) and stages it into a FAT32 EFI System Partition image, efi.bin, alongside the UEFI fallback entry.

Image
arm64 kernel
+
DTB
qcs6490-rb3gen2-el2
+
initramfs
rootfs.cpio.gz
+
cmdline
console=ttyMSM0…
→
ukify
uki.efi
→
efi.bin
FAT32 ESP

EFI System Partition — efi.bin (FAT32, 512 MB)

truncate -s 512M kodiak/output/efi.bin
mkfs.fat -F 32 -S 4096 kodiak/output/efi.bin
mmd   -i efi.bin ::/EFI/BOOT ::/EFI/Linux
mcopy -i efi.bin bootaa64.efi ::/EFI/BOOT/    # UEFI fallback
mcopy -i efi.bin uki.efi      ::/EFI/Linux/   # UKI
efi.bin
├── EFI/BOOT/bootaa64.efi   ← UEFI fallback entry
└── EFI/Linux/uki.efi       ← Unified Kernel Image

U-Boot's UEFI discovers and launches the UKI from EFI/Linux/uki.efi at boot; the EFI/BOOT/bootaa64.efi fallback covers default boot. make efi-clean removes uki.efi and efi.bin.

Platform Device Trees two DTBs, two delivery paths

U-Boot control DTB (BL33)
  • qcs6490-rb3gen2 — base board DTB
  • Source: upstream qcom DTS (OF_UPSTREAM)
  • Appended to u-boot.bin (OF_SEPARATE)
  • Multi-DTB FIT — one U-Boot image, many boards
  • Board picked at runtime via SMEM (QCOM_FIT_MULTIDTB)
Linux kernel DTB
  • qcs6490-rb3gen2-el2 — EL2 variant (kernel patch 0001)
  • Built from the kernel tree
  • Embedded in the UKI — .dtb section
  • Delivered in the ESP (efi.bin)
  • UKI stub installs it → kernel (EFI EFI_DTB)

What carries a DTB — and what does not

  • FIP (uefi.elf) — no DTB (BL31/BL32/BL33 only)
  • UKI (uki.efi) — the kernel DTB
  • Multi-DTB FIT — U-Boot control DTB only
  • No FIT in the TF-A / kernel path
  • dtb.bin partition — Yocto-only, unused (stripped)

Buildroot Root Filesystem → Initramfs

Target Format

Buildroot produces a compressed CPIO archive (rootfs.cpio.gz), embedded directly into the UKI as an initramfs and loaded entirely into RAM at boot.

Built-in drivers (no modules)

The kernel is built with CONFIG_MODULES=n: the RAM initramfs has no module-autoload mechanism, so every driver on the boot / OP-TEE / DSP path is compiled in (=y). There is no kernel module overlay — only firmware and the DSP runtime are staged into kodiak/overlay/.

Included Packages
  • Busybox (core utilities, watchdog support)
  • OpenSSH server + key utilities
  • OpenSSL CLI
  • DHCPCD (networking)
  • Ethtool, xinetd
  • GStreamer 1.x + v4l-utils — V4L2 video decode pipeline

Login banner: "OP-TEE embedded distrib for kodiak"

Tests on-board validation from the Normal World

qcom-tests — bundled runner

One interactive runner at /usr/bin/qcom-tests; PASS/FAIL per step:

  • ADSP FastRPC · CDSP FastRPC — OP-TEE PAS bring-up
  • VIDEO — venus encode → decode round-trip
  • CPU stress — soak (STRESS_SECS, stress-ng)
qcom-tests
Choice> 1 3    # space/comma list; blank = all
Individual tools
  • xtest — OP-TEE regression suite (xtest [N])
  • fastrpc_test -d 3 / -d 0 — CDSP / ADSP via OP-TEE PAS
  • v4l2-ctl --list-devices — venus codec nodes
  • gst-launch-1.0 — H.264 v4l2h264enc → v4l2h264dec

A passing FastRPC / video run means OP-TEE executed the PAS authenticate/reset correctly.

Summary

Build

  • 5 components built from source: OP-TEE, U-Boot, TF-A, Linux, Buildroot
  • BL2 (tz.mbn) built by TF-A and signed with sectools (OEM TEST key)
  • All packed into a single UKI EFI binary inside a FAT32 image
  • make all to build everything

Flash

  • make flash-loader — bootloader provisioning only
  • make flash-efi — EFI partition only (fast iteration)
  • Uses QDL / Firehose protocol over USB in EDL mode
  • Blobs: kodiak/input/ → Yocto deploy → make fetch-blobs

Boot

  • PBL → XBL → TF-A (EL3) → OP-TEE (S-EL1) → U-Boot (EL2) → UKI → Linux (EL2)
  • EL2 DTB overlay required for Linux without Gunyah
  • Initramfs is embedded — no block storage mount needed

Key files

  • kodiak.mk — top-level build orchestration
  • kodiak/output/tz.mbn — BL2 signed with sectools (OEM TEST key)
  • kodiak/output/uefi.elf — FIP (OP-TEE + U-Boot)
  • kodiak/output/efi.bin — FAT32 EFI partition image