AidGen SDK Developer Documentation

Introduction

AidGen is an inference framework specifically designed for generative Transformer models, built on top of AidLite. It aims to fully utilize various computing units of hardware (CPU, GPU, NPU) to achieve inference acceleration for large models on edge devices.

AidGen is an SDK-level development kit that provides atomic-level large model inference interfaces, suitable for developers to integrate large model inference into their applications.

AidGen supports multiple types of generative AI models:

Language large models -> AidLLM inference
Multimodal large models -> AidMLM inference

The structure is shown in the diagram below:

💡Note

All large models supported by Model Farm achieve inference acceleration on Qualcomm chip NPUs through AidGen.

Support Status

Model Type Support Status

	AidLLM	AidMLM
Text	✅	／
Image	／	🚧
Audio	／	🚧

✅: Supported 🚧: Planned support

Operating System Support Status

	Linux	Android
C++	✅	／
Python	🚧	／
Java	／	🚧

✅: Supported 🚧: Planned support

Large Language Model AidLLM SDK

Installation

bash

sudo aid-pkg -i aidgen-sdk

Model File Acquisition

Model files and default configuration files can be downloaded directly through the Model Farm Large Model Section

Example

Deploying Qwen2.5-0.5B-Instruct on Qualcomm QCS8550

Step 1: Install AidGen SDK

bash

# Install AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk

# Copy test code
cd /home/aidlux

cp -r /usr/local/share/aidgen/examples/cpp/aidllm .

Step 2: Upload & Unzip Model Resources

Upload the downloaded model resources to the edge device.
Unzip the model resources to the /home/aidlux/aidllm directory:

bash

cd /home/aidlux/aidllm
unzip Qwen2.5-0.5B-Instruct_Qualcomm\ QCS8550_QNN2.29_W4A16.zip -d .

Step 3: Confirm Resource Files

The file distribution is as follows:

bash

/home/aidlux/aidllm
├── CMakeLists.txt
├── test_prompt_abort.cpp
├── test_prompt_serial.cpp
├── aidgen_chat_template.txt
├── chat.txt
├── htp_backend_ext_config.json
├── qwen2.5-0.5b-instruct-htp.json
├── qwen2.5-0.5b-instruct-tokenizer.json
├── qwen2.5-0.5b-instruct_qnn229_qcs8550_4096_1_of_2.serialized.bin
├── qwen2.5-0.5b-instruct_qnn229_qcs8550_4096_2_of_2.serialized.bin

Step 4: Set Dialogue Template

💡Note

For the dialogue template, refer to the aidgen_chat_template.txt file in the model resource package.

Modify the test_prompt_serial.cpp file according to the large model's template:

cpp

    if(prompt_template_type == "qwen2"){
        prompt_template = "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n<|im_start|>user\n{0}<|im_end|>\n<|im_start|>assistant\n";
    }

Step 5: Compile and Run

bash

# install dependency
sudo apt update
sudo apt install libfmt-dev

# compile
mkdir build && cd build
cmake .. && make
# Run test_prompt_serial after successful compilation
./build/test_prompt_serial qwen2.5-0.5b-instruct-htp.json

Enter dialogue content in the terminal.

Model Support Status

Model	Status
Qwen2.5-VL-3B-Instruct	✅
Qwen2.5-VL-7B-Instruct	✅
InternVL3-2B	🚧
Qwen3-VL-4b	🚧
Qwen3-VL-2b	🚧

Installation

bash

sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk

Model File Acquisition

Acquire and download models directly via command line.

bash

sudo aid-pkg -i aidgense
# View supported models
aidllm remote-list api

# Pull a model
aidllm pull api [Url]

# List downloaded models
aidllm list api

# Remove a downloaded model
sudo aidllm rm api [Name]

Example Application

Deploying Qwen2.5-VL-3B-Instruct (392x392) on Qualcomm QCS8550

Install AidGen SDK

bash

# Install AidGen SDK
sudo aid-pkg update
sudo aid-pkg -i aidgen-sdk

# Copy test code
cd /home/aidlux
cp -r /usr/local/share/aidgen/examples/cpp/aidmlm ./

Model Acquisition

bash

# Install aidllm tool
sudo aid-pkg -i aidgense

# Download the model
aidllm pull api aplux/Qwen2.5-VL-3B-392x392-8550

# Move model resources to the directory
mv -r /opt/aidlux/app/aid-openai-api/res/models/Qwen2.5-VL-3B-392x392-8550 /home/aidlux/aidmlm

Create Configuration File

bash

cd /home/aidlux/aidmlm
vim config3b_392.json

Create the following json configuration file:

json

{
    "vision_model_path":"veg.serialized.bin.aidem",
    "pos_embed_cos_path":"position_ids_cos.raw",
    "pos_embed_sin_path":"position_ids_sin.raw",
    "vocab_embed_path":"embedding_weights_151936x2048.raw",
    "window_attention_mask_path":"window_attention_mask.raw",
    "full_attention_mask_path":"full_attention_mask.raw",
    "llm_path_list":[
        "qwen2p5-vl-3b-qnn231-qcs8550-cl2048_1_of_6.serialized.bin.aidem",
        "qwen2p5-vl-3b-qnn231-qcs8550-cl2048_2_of_6.serialized.bin.aidem",
        "qwen2p5-vl-3b-qnn231-qcs8550-cl2048_3_of_6.serialized.bin.aidem",
        "qwen2p5-vl-3b-qnn231-qcs8550-cl2048_4_of_6.serialized.bin.aidem",
        "qwen2p5-vl-3b-qnn231-qcs8550-cl2048_5_of_6.serialized.bin.aidem",
        "qwen2p5-vl-3b-qnn231-qcs8550-cl2048_6_of_6.serialized.bin.aidem"
    ]
}

The file distribution is as follows:

bash

/home/aidlux/aidmlm
├── CMakeLists.txt
├── test_qwen25vl_abort.cpp
├── test_qwen25vl.cpp
├── demo.jpg
├── embedding_weights_151936x2048.raw
├── full_attention_mask.raw
├── position_ids_cos.raw
├── position_ids_sin.raw
├── qwen2p5-vl-3b-qnn231-qcs8550-cl2048_1_of_6.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn231-qcs8550-cl2048_2_of_6.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn231-qcs8550-cl2048_3_of_6.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn231-qcs8550-cl2048_4_of_6.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn231-qcs8550-cl2048_5_of_6.serialized.bin.aidem
├── qwen2p5-vl-3b-qnn231-qcs8550-cl2048_6_of_6.serialized.bin.aidem
├── veg.serialized.bin.aidem
├── window_attention_mask.raw

Compile and Run

bash

sudo apt update
sudo apt-get install libfmt-dev nlohmann-json3-dev

mkdir build && cd build
cmake .. && make

mv test_qwen25vl /home/aidlux/aidmlm/

# Run test_qwen25vl after successful compilation
cd /home/aidlux/aidmlm/
./test_qwen25vl "qwen25vl3b392" "config3b_392.json" "demo.jpg" "Please describe the scene in the picture"

In the test_qwen25vl.cpp test code, model_type is defined for different model types used as the first argument of the executable. Currently supported types are:

Model	Type
Qwen2.5-VL-3B (392X392)	qwen25vl3b392
Qwen2.5-VL-3B (672X672)	qwen25vl3b672
Qwen2.5-VL-7B (392X392)	qwen25vl7b392
Qwen2.5-VL-7B (672X672)	qwen25vl7b672

The running result is shown below:

Quick Start

System Installation

System Usage

40-Pin GPIO

Vision Development

Audio Development

Video Codec Development

Tutorial

Model Farm

GenAI Development

GenAI Inference Toolkit

GenAI HTTP Service

AI Development

API Documentation

Model Conversion

Tutorials

Peripherals

Tool Installation

AidGen SDK Developer Documentation

Introduction

Support Status

Large Language Model AidLLM SDK

Installation

Model File Acquisition

Example

Deploying Qwen2.5-0.5B-Instruct on Qualcomm QCS8550

Installation

Model File Acquisition

Example Application

Deploying Qwen2.5-VL-3B-Instruct (392x392) on Qualcomm QCS8550

GenAI Inference Toolkit

GenAI HTTP Service

API Documentation

AidGen SDK Developer Documentation ​

Introduction ​

Support Status ​

Large Language Model AidLLM SDK ​

Installation ​

Model File Acquisition ​

Example ​

Deploying Qwen2.5-0.5B-Instruct on Qualcomm QCS8550 ​

Multi-modal Vision Model AidMLM SDK ​

Installation ​

Model File Acquisition ​

Example Application ​

Deploying Qwen2.5-VL-3B-Instruct (392x392) on Qualcomm QCS8550 ​

AidGen SDK Developer Documentation

Introduction

Support Status

Large Language Model AidLLM SDK

Installation

Model File Acquisition

Example

Deploying Qwen2.5-0.5B-Instruct on Qualcomm QCS8550

Multi-modal Vision Model AidMLM SDK

Installation

Model File Acquisition

Example Application

Deploying Qwen2.5-VL-3B-Instruct (392x392) on Qualcomm QCS8550