Deploy LLM with AidGen

Introduction

Edge deployment of Large Language Models (LLMs) refers to the process of compressing, quantizing, and deploying models that originally ran in the cloud onto local devices. This enables offline, low-latency natural language understanding and generation. Based on the AidGen inference engine, this chapter demonstrates the deployment, loading, and conversation workflow of LLMs on edge devices.

In this case, the LLM inference runs on the device side. C++ code is used to call relevant interfaces to receive user input and return conversation results in real-time.

• Device: IQ9075
• System: Ubuntu 24.04
• Model: Qwen2.5-0.5B-Instruct

Supported Platforms

Platform	Operation Mode
IQ9075	Ubuntu 24.04

Prerequisites

1. IQ9075 hardware

2. Ubuntu 24.04 system

3. Prepare model files

Visit Model Farm: Qwen2.5-0.5B-Instruct to download the model resource files

💡Note

Select the QCS8550 chip

System Dependency Configuration

Configure the AidLux Package Source

# Download the correct public key
sudo wget -O- https://archive.aidlux.com/ubuntu24/public.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/private-aidlux.gpg > /dev/null

# Edit the source list file
sudo vim /etc/apt/sources.list.d/private-aidlux.list

# Add the repository provided by AidLux to the source file
deb [arch=arm64 signed-by=/etc/apt/trusted.gpg.d/private-aidlux.gpg] https://archive.aidlux.com/ubuntu24 noble main

#  Update the package cache
sudo apt update

After the update is complete, you can use the following command to retrieve the official AidLux SDK dependencies:

sudo apt list | grep aid | grep unknown

# Install software
# Must be installed first (not included with the system)
sudo apt install python3 python3-pip libopencv-dev python3-opencv  net-tools
# Must be installed before aidlite
sudo apt install aidlux-aistack-base aidrtcm

# Install aidlite and dependencies
sudo apt install aid-lms aidlms-sdk aidlite-sdk cmake
sudo apt-get install libfmt-dev nlohmann-json3-dev
sudo apt install aidlite-*

# DSP support
sudo apt-get install qcom-fastrpc1
sudo apt-get install qcom-fastrpc-dev

# Install aidgen-sdk
sudo apt install aidgen-sdk
sudo apt install aidgen-qnn236
sudo apt install aidgen-qnn240

# Install mms service
sudo apt install aid-mms

# GPU support
sudo apt-add-repository -s ppa:ubuntu-qcom-iot/qcom-ppa
sudo apt install qcom-adreno-cl1
sudo ln -s /usr/lib/aarch64-linux-gnu/libOpenCL.so.1 /usr/lib/aarch64-linux-gnu/libOpenCL.so

After installation, check that the aidlite and aidgen directories have been added under /usr/local/share:

Device Authorization

Obtain the Device Serial Number

cat  /sys/devices/soc0/serial_number

Obtain the License File

Provide the serial number to APLUX technical staff to generate a device-specific License file, then place it in the /etc/opt/aidlux/license/AidLuxLics directory.

Activate the License

sudo /opt/aidlux/cpf/aid-lms/manager.sh restart

Case Deployment

Step 1: Copy the AidGen SDK Code Example

# Copy the test code
cd /home/ubuntu/aidllm

cp -r /usr/local/share/aidgen/examples/ ./

Step 2: Upload and Extract Model Resources

• Upload the downloaded model resources to the edge device.

• Extract the model resources to the /home/ubuntu/aidllm directory:

cd /home/ubuntu/aidllm
unzip qnn229_qcs8550_cl4096.zip -d /home/ubuntu/aidllm/
mv qnn229_qcs8550_cl4096/* ./

Step 3: Verify Resource Files

The file layout is as follows:

/home/ubuntu/aidllm
├── aidgen_chat_template.txt
├── chat.txt
├── htp_backend_ext_config.json
├── qwen2.5-0.5b-instruct-htp.json
├── qwen2.5-0.5b-instruct-tokenizer.json
├── qwen2.5-0.5b-instruct_qnn229_qcs8550_4096_1_of_2.serialized.bin
├── qwen2.5-0.5b-instruct_qnn229_qcs8550_4096_2_of_2.serialized.bin
├── examples

Step 4: Set the Conversation Template

💡Note

Refer to the aidgen_chat_template.txt file in the model resource package for the conversation template.

Modify the test_aidgen_text.cpp file according to the model's template:

    // ========================================================================
    // 5. Build the prompt template (Qwen2 format)
    // ========================================================================
    std::string system_prompt =
        "<|im_start|>system\n"
        "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n";

    auto make_user_turn = [](const std::string& text) -> std::string {
        return "<|im_start|>user\n" + text + "<|im_end|>\n<|im_start|>assistant\n";
    };

Step 5: Build and Run

cd /home/ubuntu/aidllm/examples

# Build
mkdir build && cd build
cmake .. && make

mv test_text_only /home/ubuntu/aidllm/

cd /home/ubuntu/aidllm/
./test_text_only qwen2.5-0.5b-instruct-htp.json "hi"