AidGen C++ API

AidGen C++ API Documentation

AidGen 2.x C++ API Documentation

💡Note

Before developing with AidGen-SDK 2.x C++, please be aware of the following basics:

During compilation, include the header file located at /usr/local/include/aidlux/aidgen/aidgen.hpp
During linking, specify the library file located at /usr/local/lib/libaidgen.so
All interfaces are under the aplux::aidgen namespace
AidGen 2.x unifies LLM text generation, multimodal inference, and embedding vectorization capabilities under a consistent API style and programming model
Compared to version 1.x, 2.x integrates aidllm and aidmlm functionality into the unified aidgen namespace, providing a Context → Tokenizer / EmbeddingExtractor / Generator layered architecture

Log Level.enum LogLevel

AidGen-SDK provides an API for setting the log level. You need to specify which log level to use with this log-level enum.

Member Name	Type	Value	Description
INFO	uint8_t	0	Informational message
WARNING	uint8_t	1	Warning
ERROR	uint8_t	2	Error
FATAL	uint8_t	3	Fatal error

Error Code.enum ErrorCode

AidGen-SDK 2.x interface methods uniformly return ErrorCode enum values. Developers can determine whether operations succeeded and locate issues based on the returned error codes.

Member Name	Type	Value	Description
SUCCESS	int32_t	0	Operation succeeded
INVALID_PARAM	int32_t	-1	Invalid parameter
NOT_INITIALIZED	int32_t	-2	Not yet initialized
ALREADY_INITIALIZED	int32_t	-3	Already initialized
OUT_OF_MEMORY	int32_t	-4	Out of memory
UNSUPPORTED	int32_t	-5	Unsupported operation
ABORTED	int32_t	-6	Operation aborted
FILE_NOT_FOUND	int32_t	-10	File does not exist or invalid path
FILE_OPEN_FAILED	int32_t	-11	File cannot be opened (permission denied or locked)
FILE_EMPTY	int32_t	-12	File exists but has no content
FILE_READ_ERROR	int32_t	-13	I/O truncation or system-level error during file read
FILE_PARSE_ERROR	int32_t	-14	File content JSON parsing failed
BACKEND_ERROR	int32_t	-20	Generic runtime error thrown by the backend
BACKEND_NOT_FOUND	int32_t	-21	Backend dynamic library not found
INTERNAL_ERROR	int32_t	-99	Unknown exception or system-level crash fallback

Backend Type.enum BackendType

For AidGen-SDK 2.x, different inference backend frameworks are supported to implement LLM inference tasks. The available inference backends are listed below.

Member Name	Type	Value	Description
DEFAULT	uint8_t	0	Auto-select (priority: QNN248 > QNN240 > QNN236)
QNN236	uint8_t	1	Qualcomm QNN SDK 2.36 inference backend
QNN240	uint8_t	2	Qualcomm QNN SDK 2.40 inference backend
QNN248	uint8_t	3	Qualcomm QNN SDK 2.48 inference backend

Generation State.enum GenState

During inference, the Generator continuously returns generation results to the developer via callback functions. The GenState enum identifies the stage of each callback event, allowing developers to determine generation progress.

Member Name	Type	Value	Description
IDLE	uint8_t	0	Idle, waiting for a new generation round
BEGIN	uint8_t	1	First token fragment of a sentence
CONTINUE	uint8_t	2	Middle continuation fragment of a sentence
END	uint8_t	3	End of sentence
ABORT	uint8_t	4	User interruption or task voluntarily abandoned
ERROR	uint8_t	5	System or hardware-level error

Global Functions

Get Library Version.get_library_version()

Gets the version string of the current AidGen library.

API	get_library_version
Description	Gets the version information of the AidGen library
Parameters	void
Return Value	A string containing the library version, e.g. "2.3.0"

cpp

std::string version = aplux::aidgen::get_library_version();
printf("Current aidgen library version: %s\n", version.c_str());

Get ABI Version.get_abi_version()

Gets the ABI version number for binary compatibility detection.

API	get_abi_version
Description	Gets the ABI version number of the AidGen library
Parameters	void
Return Value	ABI version number (unsigned 64-bit integer)

cpp

uint64_t abi_ver = aplux::aidgen::get_abi_version();
printf("ABI version: %lu\n", abi_ver);

Set Log Level.set_log_level()

Sets the minimum log output level for AidGen. Logs below this level will not be output.

API	set_log_level
Description	Sets the minimum log output level
Parameters	log_level: LogLevel enum value specifying the minimum log level
Return Value	void

cpp

aplux::aidgen::set_log_level(aplux::aidgen::LogLevel::ERROR);

Set Log File Prefix.set_log_file_prefix()

Sets the log file path prefix. The library automatically appends suffixes to generate the actual log file and returns the final full path.

API	set_log_file_prefix
Description	Sets the log file path prefix
Parameters	log_file_prefix: Log file prefix string (must not be empty)
Return Value	The final full log file path generated

cpp

std::string log_path = aplux::aidgen::set_log_file_prefix("./aidgen_log_");
printf("Log file path: %s\n", log_path.c_str());

Context Properties.struct ContextProperties

The ContextProperties struct is used to configure the behavior attributes of the Context, passed when creating a Context instance.

Member List

Member Name	Type	Default Value	Description
stream	bool	true	Whether to enable streaming output
enable_profiler	bool	false	Whether to enable the profiler
enable_prompt_cache	bool	true	Whether to enable Prompt Cache acceleration

Tensor.struct Tensor

The Tensor struct holds a data buffer and its size. It serves as the output container for EmbeddingExtractor and the input container for Generator's multimodal mode. By default, the deleter is nullptr, meaning the unique_ptr automatically calls delete[] to free memory upon destruction. Users can also provide a custom deleter (e.g., std::free) to manage externally allocated memory.

Member List

Member Name	Type	Default Value	Description
data	std::unique_ptr<uint8_t[], TensorDeleter>		Data buffer pointer with support for custom deallocation functions
size	size_t	0	Data size in bytes

Generation Event.struct GenEvent

The data structure returned by the Generator through callback functions during inference, containing the current generation state and text content.

Member List

Member Name	Type	Default Value	Description
state	GenState		Current generation state code
text	std::string		Text fragment from this callback

Meaning of text under each state:

state	Meaning	text Content
IDLE	Not yet started	Empty
BEGIN	First token	First generated text
CONTINUE	Intermediate continuation	Continued generated text
END	Generation complete	Final text
ABORT	Interrupted	Partially generated text before interruption
ERROR	Error occurred	May be empty

Generation Callback Type.GenCallback

Callback function type definition used during Generator inference. Developers need to implement this callback function type to process inference results.

cpp

using GenCallback = std::function<void(const GenEvent& event, void* user_data)>;

⚠️Warning

It is strictly prohibited to call Generator::abort() inside the callback function, as this may cause deadlocks or undefined behavior.

Extractor Profile Data.struct ExtractorProfileData

Performance metrics collected by EmbeddingExtractor during inference.

Member List

Member Name	Type	Default Value	Description
init_time_us	uint64_t	0	Initialization time (microseconds)
prompt_token_count	uint64_t	0	Prompt token count
execute_time_us	uint64_t	0	Inference execution time (microseconds)
prompt_processing_tps	float	0.f	Prompt processing throughput (tokens/s)

Generator Profile Data.struct GeneratorProfileData

Performance metrics collected by Generator during inference.

Member List

Member Name	Type	Default Value	Description
init_time_us	uint64_t	0	Initialization time (microseconds)
prompt_token_count	uint64_t	0	Prompt token count
time_to_first_token_time_us	uint64_t	0	Time to first token latency (microseconds)
prompt_processing_tps	float	0.f	Prompt processing throughput (tokens/s)
generated_token_count	uint64_t	0	Generated token count
generate_time_us	uint64_t	0	Generation phase duration (microseconds)
generate_tps	float	0.f	Generation throughput (tokens/s)

Profile Data.struct ProfileData

Aggregates performance data from both the EmbeddingExtractor and Generator stages.

Member List

Member Name	Type	Default Value	Description
extractor	ExtractorProfileData		Embedding extractor performance data
generator	GeneratorProfileData		LLM generator performance data

Runtime Context Class.class Context

Context is the inference context class that holds model configuration and backend settings, shared by Tokenizer, EmbeddingExtractor, and Generator. Context uses a global singleton pattern — only one Context instance is permitted at a time. Its lifecycle is: create_instance() → initialize() → use → automatic cleanup on destruction. Context is non-copyable and non-movable.

Create Instance Object.create_instance()

Creates the singleton Context instance. If an instance already exists, returns nullptr.

API	create_instance
Description	Creates a Context singleton instance; only one is allowed at a time
Parameters	config: Model configuration file path (JSON), or the JSON content string itself (auto-detected by first non-whitespace character: `{` or `[` means JSON content, otherwise treated as a file path)
	properties: ContextProperties struct reference, configuring streaming output, profiling, Prompt Cache, etc.
	backend_type: BackendType enum value specifying the inference backend, default BackendType::DEFAULT (auto-selects the best available backend)
Return Value	If nullptr, an instance already exists or construction failed; otherwise, a shared_ptr to the Context object

cpp

// Create context properties
aplux::aidgen::ContextProperties props;
props.stream = false;
props.enable_profiler = true;
props.enable_prompt_cache = true;

// Create a Context instance
std::shared_ptr<aplux::aidgen::Context> ctx_ptr = aplux::aidgen::Context::create_instance(
    "qwen2-7b/qwen2-7b.json", props, aplux::aidgen::BackendType::DEFAULT);
if(ctx_ptr == nullptr){
    printf("Context create_instance failed.\n");
    return EXIT_FAILURE;
}

Initialize Operation.initialize()

Performs the complete initialization flow: license validation → search and load backend plugin → parse model configuration JSON → verify backend version compatibility (major version + ABI + minimum minor/patch) → initialize backend handle.

API	initialize
Description	Performs complete initialization including license validation, backend loading, and config parsing
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success. May return BACKEND_NOT_FOUND, FILE_OPEN_FAILED, FILE_EMPTY, FILE_READ_ERROR, FILE_PARSE_ERROR, NOT_INITIALIZED, etc.

💡Tip

If initialization reports Required QNN backend not found (error code BACKEND_NOT_FOUND), the corresponding QNN inference backend SDK is missing from the system. Install the appropriate backend package based on your model and target platform, for example:

aidgen-qnn236 — for Qualcomm QNN SDK 2.36
aidgen-qnn240 — for Qualcomm QNN SDK 2.40
aidgen-qnn248 — for Qualcomm QNN SDK 2.48

After installation, ensure the backend dynamic library (e.g., libaidgen_qnn240.so) is located under /usr/local/lib.

cpp

// Initialize Context; non-SUCCESS return indicates an error
if(ctx_ptr->initialize() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Context initialize failed.\n");
    return EXIT_FAILURE;
}

Get Config.get_config()

Queries all configurable parameters under the backend's "context" scope. Must be called after initialize().

API	get_config
Description	Queries all context-level configurable parameters from the backend
Parameters	void
Return Value	A string containing all context configuration parameters. Returns an empty string if not initialized

cpp

std::string config = ctx_ptr->get_config();
printf("Context config: %s\n", config.c_str());

Set Config.set_config()

Sets the value of a specified configuration item via the backend's "context" scope. Must be called after initialize(), and only effective before Tokenizer, Generator, or EmbeddingExtractor instances are created.

API	set_config
Description	Sets a specified context-level configuration item value
Parameters	key: Configuration item key name
Parameters	value: Configuration value
Return Value	ErrorCode::SUCCESS indicates success. NOT_INITIALIZED if not initialized

cpp

if(ctx_ptr->set_config("key", "value") != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Context set_config failed.\n");
}

Tokenizer Class.class Tokenizer

Tokenizer is a text tokenizer responsible for bidirectional conversion between text and token IDs. It uses a global singleton pattern — only one instance is allowed at a time. Its lifecycle is: create_instance() → initialize() → encode() / decode() → finalize() → destruction. Tokenizer is non-copyable and non-movable.

Create Instance Object.create_instance()

API	create_instance
Description	Creates a Tokenizer singleton instance; only one is allowed at a time
Parameters	context: std::shared_ptr<Context>, a shared initialized Context object (must not be nullptr)
Parameters	reserve: Reserved field, default value is nullptr
Return Value	If nullptr, the context is null, an instance already exists, or construction failed; otherwise, a unique_ptr to the Tokenizer object

cpp

std::unique_ptr<aplux::aidgen::Tokenizer> tokenizer_ptr = aplux::aidgen::Tokenizer::create_instance(ctx_ptr);
if(tokenizer_ptr == nullptr){
    printf("Tokenizer create_instance failed.\n");
    return EXIT_FAILURE;
}

Initialize Operation.initialize()

Initializes the tokenizer backend, loading the tokenizer model with the "NoEmbedding" target (vocabulary only, without embedding weights).

API	initialize
Description	Initializes the tokenizer backend and loads the vocabulary model
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

cpp

if(tokenizer_ptr->initialize() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Tokenizer initialize failed.\n");
    return EXIT_FAILURE;
}

Encode Text to Token.encode()

Encodes a plain text string into a token ID sequence. Must call initialize() first.

API	encode
Description	Encodes a text string into a token ID sequence
Parameters	text: Input text string to be encoded
Parameters	token_ids: std::vector<int32_t>&, output token ID sequence
Return Value	ErrorCode::SUCCESS indicates success. NOT_INITIALIZED if not initialized

cpp

std::vector<int32_t> token_ids;
if(tokenizer_ptr->encode("Hello world!", token_ids) != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Tokenizer encode failed.\n");
    return EXIT_FAILURE;
}
printf("Token count: %zu\n", token_ids.size());

Decode Token to Text.decode()

Decodes a token ID sequence back into human-readable text. Must call initialize() first.

API	decode
Description	Decodes a token ID sequence into a text string
Parameters	token_ids: Input token ID sequence
Parameters	text: std::string&, output decoded text
Return Value	ErrorCode::SUCCESS indicates success. NOT_INITIALIZED if not initialized

cpp

std::string decoded_text;
if(tokenizer_ptr->decode(token_ids, decoded_text) != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Tokenizer decode failed.\n");
    return EXIT_FAILURE;
}
printf("Decoded: %s\n", decoded_text.c_str());

Finalize Release Operation.finalize()

Releases tokenizer backend resources. Must call initialize() first.

API	finalize
Description	Releases tokenizer backend resources
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success. NOT_INITIALIZED if not initialized

cpp

if(tokenizer_ptr->finalize() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Tokenizer finalize failed.\n");
    return EXIT_FAILURE;
}

Embedding Extractor Class.class EmbeddingExtractor

EmbeddingExtractor is used to encode text into a single embedding tensor, suitable for Retrieval-Augmented Generation (RAG), text semantic similarity computation, feature extraction, and other scenarios. It uses a global singleton pattern — only one instance is allowed at a time. Its lifecycle is: create_instance() → initialize() → encode() / get_profiler() → finalize() → destruction. EmbeddingExtractor is non-copyable and non-movable.

Create Instance Object.create_instance()

API	create_instance
Description	Creates an EmbeddingExtractor singleton instance; only one is allowed at a time
Parameters	context: std::shared_ptr<Context>, a shared initialized Context object (must not be nullptr)
Parameters	reserve: Reserved field, default value is nullptr
Return Value	If nullptr, the context is null, an instance already exists, or construction failed; otherwise, a unique_ptr to the EmbeddingExtractor object

cpp

std::unique_ptr<aplux::aidgen::EmbeddingExtractor> extractor_ptr = 
    aplux::aidgen::EmbeddingExtractor::create_instance(ctx_ptr);
if(extractor_ptr == nullptr){
    printf("EmbeddingExtractor create_instance failed.\n");
    return EXIT_FAILURE;
}

Initialize Operation.initialize()

Initializes the embedding extractor backend and loads the embedding model. Must be called after Context::initialize().

API	initialize
Description	Initializes the extractor backend and loads the embedding model
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

cpp

if(extractor_ptr->initialize() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("EmbeddingExtractor initialize failed.\n");
    return EXIT_FAILURE;
}

Finalize Release Operation.finalize()

Releases embedding extractor backend resources. Must call initialize() first.

API	finalize
Description	Releases embedding extractor backend resources
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success. NOT_INITIALIZED if not initialized

Get Property.get_property()

Queries all configurable parameters under the backend's "extractor" scope. Must call initialize() first.

API	get_property
Description	Queries all configurable parameters under the extractor scope
Parameters	void
Return Value	Property value string. Returns an empty string if not initialized

Set Property.set_property()

Sets a specified property value via the backend's "extractor" scope. Must call initialize() first.

API	set_property
Description	Sets a specified property value under the extractor scope
Parameters	key: Property key name
Parameters	value: Property value
Return Value	ErrorCode::SUCCESS indicates success

Encode to Embedding.encode()

Encodes a text string into an embedding tensor. Must call initialize() first.

API	encode
Description	Encodes text into an embedding tensor
Parameters	prompt: Input text string to be encoded
Parameters	embedding: Tensor&, output embedding tensor (data and size are populated by the backend)
Return Value	ErrorCode::SUCCESS indicates success

cpp

aplux::aidgen::Tensor embedding;
if(extractor_ptr->encode("What is the most popular cookie in the world?", embedding) != 
    aplux::aidgen::ErrorCode::SUCCESS){
    printf("EmbeddingExtractor encode failed.\n");
    return EXIT_FAILURE;
}
printf("Embedding size: %zu bytes\n", embedding.size);

Get Profiler.get_profiler()

Retrieves performance profiling data from the last embedding extraction run, only filling the ProfileData::extractor field. Must call initialize() first.

API	get_profiler
Description	Retrieves performance data from the last embedding extraction
Parameters	profile_data: ProfileData&, output performance data (only the extractor field is filled)
Return Value	ErrorCode::SUCCESS indicates success

cpp

aplux::aidgen::ProfileData perf;
if(extractor_ptr->get_profiler(perf) == aplux::aidgen::ErrorCode::SUCCESS){
    printf("Extractor TPS: %.2f tok/s\n", perf.extractor.prompt_processing_tps);
}

Generator Class.class Generator

Generator is an LLM token generator that supports both text-only and multimodal (embedding input) inference modes. It uses a global singleton pattern — only one instance is allowed at a time. Consecutive calls to run() are treated as multi-turn conversations, with internal automatic KV Cache reuse for acceleration; call reset() to clear historical cache and start a fresh conversation. Its lifecycle is: create_instance() → initialize() → run() / reset() / abort() / get_profiler() / get_state() → finalize() → destruction. Generator is non-copyable and non-movable.

Create Instance Object.create_instance()

API	create_instance
Description	Creates a Generator singleton instance; only one is allowed at a time
Parameters	context: std::shared_ptr<Context>, a shared initialized Context object (must not be nullptr)
Parameters	reserve: Reserved field, default value is nullptr
Return Value	If nullptr, the context is null, an instance already exists, or construction failed; otherwise, a unique_ptr to the Generator object

cpp

std::unique_ptr<aplux::aidgen::Generator> generator_ptr = aplux::aidgen::Generator::create_instance(ctx_ptr);
if(generator_ptr == nullptr){
    printf("Generator create_instance failed.\n");
    return EXIT_FAILURE;
}

Initialize Operation.initialize()

Initializes the generator backend and loads the language model. Must be called after Context::initialize().

API	initialize
Description	Initializes the generator backend and loads the language model
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

cpp

if(generator_ptr->initialize() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Generator initialize failed.\n");
    return EXIT_FAILURE;
}

Finalize Release Operation.finalize()

Releases generator backend resources. Must call initialize() first.

API	finalize
Description	Releases generator backend resources
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success. NOT_INITIALIZED if not initialized

Get Property.get_property()

Queries all configurable parameters under the backend's "generator" scope. Must call initialize() first.

API	get_property
Description	Queries all configurable parameters under the generator scope
Parameters	void
Return Value	Property value string. Returns an empty string if not initialized

Set Property.set_property()

Sets generation parameters via the backend's "generator" scope. Must call initialize() first. Common properties include: "stream" (value "0" or "1", controls streaming output), and sampling parameters "temp", "top-k", "top-p" (all values in string format).

API	set_property
Description	Sets a specified property value under the generator scope
Parameters	key: Property key name (e.g. "stream", "temp", "top-k", "top-p")
Parameters	value: Property value (string format)
Return Value	ErrorCode::SUCCESS indicates success

cpp

// Set sampling parameters
generator_ptr->set_property("stream", "1");
generator_ptr->set_property("temp", "0.8");
generator_ptr->set_property("top-k", "20");
generator_ptr->set_property("top-p", "0.9");

Get Embedding Buffer.get_embedding_buff()

Retrieves the raw pointer and size of the model's token embedding weight lookup table. Must call initialize() first.

The returned buffer stores all token embedding vectors contiguously in row-major order. Each token's embedding occupies embedding_size * sizeof(float) bytes. The embedding for the i-th token is located at offset i * embedding_size * sizeof(float). The pointer remains valid for the lifetime of the Generator instance — callers must NOT free this buffer.

Typical use case: manually constructing combined text + vision embedding tensors for multimodal inference.

API	get_embedding_buff
Description	Retrieves the pointer and size of the model's token embedding weight buffer
Parameters	out_ptr: const char*&, output parameter receiving the pointer to the embedding weight buffer; set to nullptr on failure
Parameters	out_size: size_t&, output parameter receiving the total size (in bytes) of the weight buffer
Return Value	ErrorCode::SUCCESS indicates success

cpp

const char* embedding_buf = nullptr;
size_t embedding_buf_size = 0;
if(generator_ptr->get_embedding_buff(embedding_buf, embedding_buf_size) != 
    aplux::aidgen::ErrorCode::SUCCESS){
    printf("get_embedding_buff failed.\n");
}

Text-Only Inference.run()

Starts text-only inference based on a full prompt string. Consecutive calls to run() are treated as multi-turn conversations, with internal automatic KV Cache reuse for acceleration. Call reset() to start a fresh conversation.

⚠️Warning

It is strictly prohibited to call abort() inside the callback function, as this may cause deadlocks or undefined behavior.

API	run
Description	Starts text-only inference based on a full prompt string (caller is responsible for chat template assembly)
Parameters	prompt: Complete prompt string; the developer is responsible for assembling system/user/assistant role markers
	cb: GenCallback type callback function, triggered token-by-token as fragments
	user_data: Pointer to user-defined data, passed through to the callback function; default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

cpp

// Define the callback function
auto dialog_callback = [](const aplux::aidgen::GenEvent& event, void* user_data){
    switch(event.state){
        case aplux::aidgen::GenState::BEGIN:
            printf("[BEGIN]%s", event.text.c_str());
            break;
        case aplux::aidgen::GenState::CONTINUE:
            printf("%s", event.text.c_str());
            fflush(stdout);
            break;
        case aplux::aidgen::GenState::END:
            printf("%s[END]\n", event.text.c_str());
            break;
        case aplux::aidgen::GenState::ABORT:
            printf("\n[ABORT]%s\n", event.text.c_str());
            break;
        case aplux::aidgen::GenState::ERROR:
            printf("\n[ERROR]%s\n", event.text.c_str());
            break;
        default: break;
    }
};

// Execute text-only inference
std::string prompt = "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
                     "<|im_start|>user\nHello<|im_end|>\n"
                     "<|im_start|>assistant\n";
if(generator_ptr->run(prompt, dialog_callback) != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Generator run failed.\n");
    return EXIT_FAILURE;
}

Multimodal Inference.run()

Starts multimodal inference based on a pre-computed embedding tensor. The developer is responsible for generating and assembling text embeddings and image embeddings. Consecutive calls to run() also reuse KV Cache; call reset() to clear.

⚠️Warning

It is strictly prohibited to call abort() inside the callback function, as this may cause deadlocks or undefined behavior.

API	run
Description	Starts multimodal inference based on a pre-computed embedding tensor
Parameters	embedding: Tensor&, the embedding tensor (caller is responsible for generation and assembly)
	cb: GenCallback type callback function
	user_data: Pointer to user-defined data, passed through to the callback function; default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

Reset Conversation State.reset()

Resets the internal KV Cache and conversation state via the backend's "generator_reset" action. Must be called after initialize(). Call this before starting a new topic in multi-turn conversations; after calling, previous KV Cache acceleration becomes invalid.

API	reset
Description	Resets KV Cache and conversation state, preparing for a new conversation round
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

cpp

// Reset before starting a new topic
if(generator_ptr->reset() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Generator reset failed.\n");
    return EXIT_FAILURE;
}
// A fresh conversation can now begin
generator_ptr->run(new_prompt, dialog_callback);

Abort Inference.abort()

Actively terminates the current inference task via the backend's "generator_abort" action. Must be called after initialize(). Typically called from another thread or triggered via a timeout mechanism.

⚠️Warning

It is strictly prohibited to call abort() inside the GenCallback callback function.

API	abort
Description	Actively terminates the currently running inference task
Parameters	reserve: Reserved field, default value is nullptr
Return Value	ErrorCode::SUCCESS indicates success

cpp

// Terminate inference from another thread
if(generator_ptr->abort() != aplux::aidgen::ErrorCode::SUCCESS){
    printf("Generator abort failed.\n");
}

Get Profiler.get_profiler()

Retrieves performance profiling data from the last inference run, only filling the ProfileData::generator field. Must call initialize() first.

API	get_profiler
Description	Retrieves performance data from the last generation inference
Parameters	profile_data: ProfileData&, output performance data (only the generator field is filled)
Return Value	ErrorCode::SUCCESS indicates success

cpp

aplux::aidgen::ProfileData perf;
if(generator_ptr->get_profiler(perf) == aplux::aidgen::ErrorCode::SUCCESS){
    printf("Time to first token: %lu us\n", perf.generator.time_to_first_token_time_us);
    printf("Generate rate: %.2f tok/s\n", perf.generator.generate_tps);
    printf("Generated token count: %lu\n", perf.generator.generated_token_count);
}

Get Current State.get_state()

Queries the current state of the generator (IDLE / BEGIN / CONTINUE / END / ABORT / ERROR). Must be called after initialize().

API	get_state
Description	Queries the current running state of the generator
Parameters	state: GenState&, output parameter receiving the current state value
Return Value	Always returns ErrorCode::SUCCESS when initialized. NOT_INITIALIZED if not initialized

cpp

aplux::aidgen::GenState current_state;
if(generator_ptr->get_state(current_state) == aplux::aidgen::ErrorCode::SUCCESS){
    printf("Current state: %d\n", static_cast<int>(current_state));
}

Typical Usage Flow

Complete Text-Only Inference Flow

Context::create_instance(config, props, backend)
  → Context::initialize()
  → Generator::create_instance(ctx) → Generator::initialize()
  → Generator::set_property("stream", "1")
  → Generator::run(prompt, callback)
  → Generator::get_profiler(profile_data)
  → [Optional] Generator::reset() → Generator::run(new_prompt, callback)
  → Generator::finalize()

Complete example code:

cpp

#include "aidgen.hpp"
using namespace aplux::aidgen;

int main() {
    // 1. Create Context and initialize
    ContextProperties props;
    props.stream = false;
    props.enable_profiler = true;
    auto ctx = Context::create_instance("/path/to/model.json", props, BackendType::QNN240);
    if (!ctx || ctx->initialize() != ErrorCode::SUCCESS) return -1;

    // 2. Create and initialize Generator
    auto generator = Generator::create_instance(ctx);
    if (!generator || generator->initialize() != ErrorCode::SUCCESS) return -1;
    generator->set_property("stream", "1");

    // 3. Execute inference
    std::string prompt =
        "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
        "<|im_start|>user\nHello, introduce yourself<|im_end|>\n"
        "<|im_start|>assistant\n";
    generator->run(prompt, [](const GenEvent& event, void*) {
        printf("%s", event.text.c_str());
        fflush(stdout);
    });

    // 4. Get performance data and release
    ProfileData perf;
    generator->get_profiler(perf);
    generator->finalize();
    return 0;
}

Tokenizer Usage Flow

Context::create_instance() → Context::initialize()
  → Tokenizer::create_instance(ctx) → Tokenizer::initialize()
  → Tokenizer::encode(text, token_ids)   // Text → Token
  → Tokenizer::decode(token_ids, text)   // Token → Text
  → Tokenizer::finalize()

cpp

auto tokenizer = Tokenizer::create_instance(ctx);
tokenizer->initialize();

// Encoding
std::vector<int32_t> tokens;
tokenizer->encode("Hello world!", tokens);

// Decoding
std::string decoded;
tokenizer->decode(tokens, decoded);

tokenizer->finalize();

Embedding Extraction Flow

Context::create_instance() → Context::initialize()
  → EmbeddingExtractor::create_instance(ctx) → EmbeddingExtractor::initialize()
  → EmbeddingExtractor::encode(prompt, embedding)
  → EmbeddingExtractor::get_profiler(profile_data)
  → EmbeddingExtractor::finalize()

cpp

#include "aidgen.hpp"
using namespace aplux::aidgen;

int main() {
    ContextProperties props;
    props.enable_profiler = true;
    auto ctx = Context::create_instance("/path/to/bge-large-htp.json", props, BackendType::QNN240);
    if (!ctx || ctx->initialize() != ErrorCode::SUCCESS) return -1;

    auto extractor = EmbeddingExtractor::create_instance(ctx);
    if (!extractor || extractor->initialize() != ErrorCode::SUCCESS) return -1;

    Tensor embedding;
    extractor->encode("What is the most popular cookie in the world?", embedding);

    ProfileData perf;
    extractor->get_profiler(perf);
    printf("TPS: %.2f tok/s\n", perf.extractor.prompt_processing_tps);

    // Save embedding to file
    std::ofstream out("output.raw", std::ios::binary);
    out.write(reinterpret_cast<const char*>(embedding.data.get()), embedding.size);

    return 0;
}

Error Handling Pattern

All methods returning ErrorCode should have their return values checked:

cpp

if (ctx->initialize() != ErrorCode::SUCCESS) {
    fprintf(stderr, "Context initialization failed.\n");
    return EXIT_FAILURE;
}

Common initialization failure scenarios:

BACKEND_NOT_FOUND: QNN inference backend not found. Install the corresponding backend SDK (e.g., aidgen-qnn240). See the tip under Context::initialize() above for details.
FILE_OPEN_FAILED: Invalid model config file path or insufficient permissions
FILE_PARSE_ERROR: Model config file has malformed JSON
NOT_INITIALIZED: Version incompatibility or license validation failure

Memory Management Principles

Object	Management	Description
Context	shared_ptr	Shared ownership; should be destroyed after Generator / Tokenizer / EmbeddingExtractor
Generator	unique_ptr	Exclusive ownership; destructor automatically releases the singleton lock
Tokenizer	unique_ptr	Exclusive ownership; destructor automatically releases the singleton lock
EmbeddingExtractor	unique_ptr	Exclusive ownership; destructor automatically releases the singleton lock
Tensor::data	unique_ptr + custom deleter	Default `delete[]`; can pass `std::free` or other custom deleters

AidGen 1.x C++ API Documentation (Deprecated)

⚠️Warning

The following is the AidGen 1.x API documentation, covering the aplux::aidllm and aplux::aidmlm namespaces. This version is no longer maintained. New projects should use the AidGen 2.x API above.

AidLLM C++ API Documentation

💡Note

Before developing with AidGen-SDK C++, please be aware of the following basics:

During compilation, include the header file located at /usr/local/include/aidlux/aidgen/aidllm.hpp
During linking, specify the library file located at /usr/local/lib/libaidgen.so
All interfaces are under the aplux::aidllm namespace

Inference Backend Type.enum LLmBackendType

For AidllmSDK, different inference backend frameworks are supported to implement LLM inference tasks. The available inference backends are listed below.

Member Name	Type	Value	Description
TYPE_DEFAULT	uint8_t	0	Unknown backend type
TYPE_GENIE	uint8_t	1	Genie inference backend

Inference Task State.enum LLMSentenceState

During an inference task, a single session may go through multiple stages. Developers can use these state codes to understand the current runtime status of the inference task.

Member Name	Type	Value	Description
BEGIN	enum class	0	Session start segment
CONTINUE	enum class	1	Intermediate content during ongoing session inference
END	enum class	2	Session ending segment
COMPLETE	enum class	3	Current session completed successfully
ABORT	enum class	4	Current session terminated passively
ERROR	enum class	5	Current session inference error

Interpreter Runtime State.enum LLMState

The overall state of the Aidllm interpreter during runtime. Developers can query this state to understand the interpreter's current working status.

Member Name	Type	Value	Description
STANDIDLE	enum class	0	Idle standby state
BUSYING	enum class	1	Busy processing inference
ABORT	enum class	2	Inference has been terminated
ERROR	enum class	3	Inference encountered an error

Log Level.enum LogLevel

AidllmSDK provides an API for logging (introduced later). You need to specify which log level is currently used, so this log-level enum is required.

Member Name	Type	Value	Description
INFO	uint8_t	0	Message
WARNING	uint8_t	1	Warning
ERROR	uint8_t	2	Error
FATAL	uint8_t	3	Fatal error

Global Functions

Get Library Version.get_library_version()

Gets the version information string of the current Aidllm library.

API	get_library_version
Description	Gets the version information of the Aidllm library
Parameters	void
Return Value	A string containing the library version information

cpp

std::string version = aplux::aidllm::get_library_version();
printf("Current aidllm library version: %s\n", version.c_str());

Set Log Level.set_log_level()

Sets the minimum log output level for Aidllm. Logs below this level will not be output.

API	set_log_level
Description	Sets the minimum log output level
Parameters	log_level: LogLevel enum value specifying the minimum log level
Return Value	void

cpp

aplux::aidllm::set_log_level(aplux::aidllm::LogLevel::ERROR);

Set Log File Prefix.set_log_file_prefix()

Sets the log file name prefix for outputting logs to files with the specified prefix.

API	set_log_file_prefix
Description	Sets the log file name prefix
Parameters	log_file_prefix: Log file name prefix string
Return Value	void

cpp

aplux::aidllm::set_log_file_prefix("aidllm_log_");

Inference Callback Function Type.LLMCallback

The callback function type definition used during Aidllm inference. Developers need to implement a callback function of this type to handle inference results.

cpp

using LLMCallback = std::function<int32_t(LLMCallbackData& cb_data, void* user_data)>;

💡Note

The callback function return type is int32_t. Returning 0 indicates normal continuation of inference; a non-zero value can be used to control the inference flow.

Inference Callback Data Type.struct LLMCallbackData

During inference tasks, Aidllm uses developer-provided callback functions. This data type is the argument passed to that callback function, and developers can use it in custom callbacks to process inference results.

Member List

The LLMCallbackData struct contains the following members:

Member	state
Type	enum LLMSentenceState
Default Value
Description	Status code of the current inference session

Member	text
Type	std::string
Default Value
Description	Result text of the inference task / message corresponding to special status codes

Runtime Context Class.class LLMContext

During Aidllm runtime, some configuration information may need to be set, and runtime-related data also needs to be passed around. Objects of this runtime context type are used to complete data flow.

Create Instance Object.create_instance()

To set runtime context information, you first need a configuration instance object. This function is used to create an instance object of type LLMContext.

API	create_instance
Description	Used to construct an instance object of class LLMContext
Parameters	config_file: Initial configuration file, where key information such as backend type and model file names can be configured
Return Value	If it is nullptr, object construction failed; otherwise, it is a pointer to an LLMContext object

cpp

// Create a configuration instance object; report an error if the return value is null
std::unique_ptr<LLMContext> llm_context_ptr = LLMContext::create_instance("qwen2-7b/qwen2-7b.json");
if(llm_context_ptr == nullptr){
    printf("Test sample: LLMContext create_instance failed.\n");
    return EXIT_FAILURE;
}

Member List

The LLMContext object is used to manage runtime configuration information, including the following parameters:

Member	config_file
Type	std::string
Default Value
Description	Initial configuration file, the config file parameter passed when creating the object

Member	backend_type
Type	LLmBackendType
Default Value	LLmBackendType::TYPE_DEFAULT
Description	The developer is required to specify the inference backend in the config file. After initialization parses the config file, this field will be overwritten to indicate the backend type specified by the developer

Member	model_file_vec
Type	std::vector<std::string>
Default Value
Description	The developer is required to specify model files in the config file. After initialization parses the config file, this field will be overwritten to indicate the model files specified by the developer

Member	config_overwrite_options
Type	std::string
Default Value
Description	By setting this field, you can specify certain key parameters in the inference process, thereby affecting inference speed, inference results, etc.

Member	android_tmp_directory
Type	std::string
Default Value
Description	This field is only valid on the Android platform. By setting this field, you can specify a directory for which the system user has valid permissions, for temporary use by the inference program

Interpreter Class.class LLMInterpreter

An object instance of type LLMInterpreter is the main executor of inference operations and is used to carry out specific inference processes.

Create Instance Object.create_instance()

To perform inference-related operations, an inference interpreter is essential. This function is used to construct an instance object of the inference interpreter.

API	create_instance
Description	Uses various data managed by the LLMContext object to construct an object of type LLMInterpreter
Parameters	llm_context: Reference to the unique_ptr of an LLMContext instance object (std::unique_ptr<LLMContext>&)
Parameters	reserve: Reserved field, default value is nullptr
Return Value	If it is nullptr, object construction failed; otherwise, it is a unique_ptr to an LLMInterpreter object

cpp

 // Use the LLMContext object pointer to create the interpreter object; report an error if the return value is null
std::unique_ptr<LLMInterpreter> llm_interpreter_ptr = LLMInterpreter::create_instance(llm_context_ptr);
if(llm_interpreter_ptr == nullptr){
    printf("Test sample: LLMInterpreter create_instance failed.\n");
    return EXIT_FAILURE;
}

Initialization Operation.initialize()

After the interpreter object is created, some initialization operations are required, such as environment checks and resource construction.

API	initialize
Description	Completes the initialization work required for inference
Parameters	enable_profiler: Whether to enable the profiler, default value is false
Parameters	reserve: Reserved field, default value is nullptr
Return Value	A value of 0 indicates successful initialization; otherwise a non-zero value indicates failure

cpp

// Initialize the interpreter; report an error if the return value is non-zero
int init_result = llm_interpreter_ptr->initialize();
if(init_result != EXIT_SUCCESS){
    printf("Test sample: aidllm initialize failed.\n");
    return EXIT_FAILURE;
}

Sampling Parameter Setup Operation.set_sampler()

After initialization completes successfully, sampling parameters can be set with this function to control the randomness, diversity, and quality of generated content.

API	set_sampler
Description	Sets sampling parameters to control the randomness and diversity of LLM outputs.
Parameters	key: Name of the sampling parameter. Currently supported: `"temp"`: Controls output randomness (Temperature); smaller values are more conservative. `"top-k"`: Limits the sampling range to the top K tokens with the highest probability. `"top-p"`: Nucleus Sampling; limits to the token pool whose cumulative probability reaches P.
Parameters	value: Parameter value represented as a string: For `"temp"`: floating-point numeric string (e.g. `"1.2"`). For `"top-k"`: integer numeric string (e.g. `"20"`). For `"top-p"`: floating-point numeric string (e.g. `"0.6"`).
Return Value	A value of 0 indicates success; a non-zero value indicates failure (e.g. invalid key or unsupported value format).

cpp

// Set sampling parameters
llm_interpreter_ptr->set_sampler("temp", "0.8");
llm_interpreter_ptr->set_sampler("top-k", "20");
llm_interpreter_ptr->set_sampler("top-p", "0.9");

Session Inference Operation.run()

After successful initialization, you can run dialog inference with the LLM. Developers provide a custom callback function to handle continuous inference results during the session.

API	run
Description	Executes one session inference
Parameters	prompt: Prompt string
	cb: Callback function of type LLMCallback for handling continuous inference results during the session
	user_data: Pointer to user data, convenient for using this data in custom callback functions, default value is nullptr
Return Value	A value of 0 indicates the inference executed successfully; otherwise a non-zero value indicates failure

cpp

// Define callback function
LLMCallback dialog_callback = [&](LLMCallbackData& cb_data, void* user_data)->int32_t{
    if(cb_data.state == LLMSentenceState::BEGIN){
        printf("%s", cb_data.text.c_str());
    }else if(cb_data.state == LLMSentenceState::CONTINUE){
        printf("%s", cb_data.text.c_str());
        fflush(stdout);
    }else if(cb_data.state == LLMSentenceState::END){
        printf("%s\n", cb_data.text.c_str());
    }else if(cb_data.state == LLMSentenceState::COMPLETE){
        printf("\n[COMPLETE]%s\n", cb_data.text.c_str());
    }else if(cb_data.state == LLMSentenceState::ABORT){
        printf("\n[ABORT]%s\n", cb_data.text.c_str());
    }else if(cb_data.state == LLMSentenceState::ERROR){
        printf("\n[ERROR]%s\n", cb_data.text.c_str());
    }
    return EXIT_SUCCESS;
};

// Execute inference
std::string prompt = "<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n";
int run_result = llm_interpreter_ptr->run(prompt, dialog_callback);
if(run_result != EXIT_SUCCESS){
    printf("Test sample: aidllm run failed.\n");
    return EXIT_FAILURE;
}

Query Inference State.state()

During inference, developers may need to query the current runtime state of the interpreter, such as determining whether it is idle or actively inferring.

API	state
Description	Gets the current runtime state of the inference task
Parameters	state: Reference to an LLMState variable; the function will overwrite this variable with the current state
Return Value	A value of 0 indicates the query executed successfully; otherwise a non-zero value indicates failure

cpp

LLMState current_state = LLMState::STANDIDLE;
llm_interpreter_ptr->state(current_state);
printf("Current state: %d\n", (int)current_state);

Session Termination Operation.abort()

In some situations, users may want to interrupt the session that is currently running inference. This function is used to terminate inference.

⚠️Warning

It is strictly forbidden to call the abort function inside the callback function (LLMCallback), as this may cause deadlocks or undefined behavior.

API	abort
Description	Terminates the currently running inference session
Parameters	reserve: Reserved field, default value is nullptr
Return Value	A value of 0 indicates successful termination; otherwise a non-zero value indicates failure

cpp

// Terminate inference in another thread
int abort_result = llm_interpreter_ptr->abort();
if(abort_result != EXIT_SUCCESS){
    printf("Test sample: aidllm abort failed.\n");
    return EXIT_FAILURE;
}

Final Release Operation.finalize()

As mentioned above, the interpreter object needs to run initialize() for initialization. Correspondingly, the interpreter also needs to run release operations to destroy previously created resources.

API	finalize
Description	Completes necessary de-initialization and release operations
Parameters	reserve: Reserved field, default value is nullptr
Return Value	A value of 0 indicates the release operation executed successfully; otherwise a non-zero value indicates failure

cpp

// Execute interpreter de-initialization; report an error if the return value is non-zero
int fin_result = llm_interpreter_ptr->finalize();
if(fin_result != EXIT_SUCCESS){
    printf("Test : aidllm finalize failed.\n");
    return EXIT_FAILURE;
}

Get Profiler.get_profiler()

When the profiler is enabled during initialization (enable_profiler = true), this function can be used to obtain the profiler object pointer for performance data collection and analysis. For detailed usage, refer to the "Profiler C++ API Documentation" section below.

API	get_profiler
Description	Gets the pointer to the profiler object
Parameters	void
Return Value	If the profiler is enabled, returns a Profiler object pointer; if not enabled, returns nullptr

cpp

// Enable profiler during initialization
int init_result = llm_interpreter_ptr->initialize(true);

// Get the profiler
aplux::aidgen::Profiler* profiler = llm_interpreter_ptr->get_profiler();

AidMLM C++ API Documentation

💡Note

Before developing with AidMLM-SDK C++, please be aware of the following basics:

During compilation, include the header file located at /usr/local/include/aidlux/aidgen/aidmlm.hpp
During linking, specify the library file located at /usr/local/lib/libaidgen.so
All interfaces are under the aplux::aidmlm namespace
AidMLM is designed for multimodal large model (vision-language model) inference, currently supporting Qwen2-VL and Qwen2.5-VL series models

Inference State.enum AidLLMState

During an AidMLM inference task, a single session may go through various stages. Developers can use these state codes to understand the current runtime status of the inference task.

Member Name	Type	Value	Description
STAND	enum class	0	Not yet working
START	enum class	1	Inference started
BUSYING	enum class	2	Inference in progress
FINISH	enum class	3	Inference finished
COMPLETE	enum class	4	Inference completed fully or truncated
WAITING	enum class	5	Current token decoding failed, waiting for next decode
ABORT	enum class	6	Current inference terminated early by developer
ERROR	enum class	7	Inference failed due to exception

Log Level.enum LogLevel

Member Name	Type	Value	Description
INFO	uint8_t	0	Message
WARNING	uint8_t	1	Warning
ERROR	uint8_t	2	Error
FATAL	uint8_t	3	Fatal error

Model Type.enum ModelType

Specifies the type of multimodal model currently in use.

Member Name	Type	Value	Description
RESERVED	enum class	0	Reserved type
QWEN2VL	enum class	1	Qwen2-VL model
QWEN25VL	enum class	2	Qwen2.5-VL model

Inference Callback Data Type.struct AidLLMCBData

During AidMLM inference tasks, developer-provided callback functions are used. This data type is the argument passed to that callback function.

Member List

Member	state
Type	enum AidLLMState
Default Value
Description	Status code of the current inference session

Member	text
Type	std::string
Default Value
Description	Result text of the inference task / message corresponding to special status codes

Inference Callback Function Type.AidLLMCB

The callback function type definition used during AidMLM inference.

cpp

using AidLLMCB = std::function<void(AidLLMCBData& cb_data, void* user_data)>;

Image Data Type.struct ImageData

A struct for passing image data to the multimodal model.

Member List

Member	img_pos
Type	int
Default Value	-1
Description	Position index of the image in the prompt. If -1, the image is appended at the end of the prompt

Member	img_data
Type	uint8_t*
Default Value	nullptr
Description	Image data pointer, pointing to RGB format image pixel data. Developers need to pre-resize to the model's required width and height

Initialization Parameter Type.struct AidmlmInitParam

Configuration parameters required for AidMLM initialization.

Member List

Member Name	Type	Default Value	Description
vision_model_path	std::string		Vision encoder model file path
pos_emb_cos_path	std::string		Position encoding cosine weight file path
pos_emb_sin_path	std::string		Position encoding sine weight file path
embedding_weights_path	std::string		Word embedding weights file path
window_attention_mask_path	std::string		Window attention mask file path (Qwen2.5-VL only)
full_attention_mask_path	std::string		Full attention mask file path (Qwen2.5-VL only)
llm_model_path_vec	std::vector<std::string>		LLM model file path list
dbg_opt	std::string		Debug options string
type	ModelType	ModelType::RESERVED	Multimodal model type
qwen2vl_cfg	Qwen2VLConfig		Qwen2-VL model configuration
qwen25vl_cfg	Qwen25VLConfig		Qwen2.5-VL model configuration
enable_profiler	bool	false	Whether to enable the profiler
genie_log_level	int	1	Genie backend log level (1=ERROR, 2=WARN, 3=INFO, 4=VERBOSE)
use_shared_buffer	bool	false	Whether to use shared buffer
use_mmap	bool	false	Whether to use memory-mapped model loading
use_genie_load_model_ex	bool	false	Whether to use Genie extended model loading

Model Configuration Structs

AidMLM provides predefined model configuration structs to specify vision model configurations for different resolutions and parameter scales. Developers can choose the corresponding configuration based on the model in use.

Config Struct Name	Model	Image Size	Embedding Dim
Qwen2VLConfig	Qwen2-VL	644×644	1536
Qwen25VLConfig	Qwen2.5-VL 3B	392×392	2048
Qwen25VL3B644Config	Qwen2.5-VL 3B	644×644	2048
Qwen25VL3B672Config	Qwen2.5-VL 3B	672×672	2048
Qwen25VL7B392Config	Qwen2.5-VL 7B	392×392	3584
Qwen25VL7B644Config	Qwen2.5-VL 7B	644×644	3584
Qwen25VL7B672Config	Qwen2.5-VL 7B	672×672	3584

Global Functions

Get Library Version.get_library_version()

API	get_library_version
Description	Gets the version information of the AidMLM library
Parameters	void
Return Value	A string containing the library version information

cpp

std::string version = aplux::aidmlm::get_library_version();
printf("Current aidmlm library version: %s\n", version.c_str());

Multimodal Inference Class.class Aidmlm

An object instance of type Aidmlm is the main executor of multimodal inference operations and is used to carry out vision-language model inference processes.

Construction and Destruction

Aidmlm objects are created via the default constructor.

cpp

aplux::aidmlm::Aidmlm mlm_ctx;

Set Log Level.set_log_level()

A static method that sets the minimum log output level for AidMLM.

API	set_log_level
Description	Sets the minimum log output level (static method)
Parameters	log_level: LogLevel enum value
Return Value	void

cpp

aplux::aidmlm::Aidmlm::set_log_level(aplux::aidmlm::LogLevel::INFO);

Set Log File Prefix.set_log_file_prefix()

A static method that sets the log file name prefix.

API	set_log_file_prefix
Description	Sets the log file name prefix (static method)
Parameters	log_file: Log file name prefix string
Return Value	void

cpp

aplux::aidmlm::Aidmlm::set_log_file_prefix("./test_mlm");

Initialization Operation.initialize()

Loads the multimodal model and initializes the inference environment.

API	initialize
Description	Loads the model and completes the initialization work required for inference
Parameters	param: AidmlmInitParam struct reference containing model paths, configurations, and other initialization parameters
Parameters	enable_profiler: Whether to enable the profiler, default value is false
Return Value	A value of 0 indicates successful initialization; otherwise a non-zero value indicates failure

cpp

aplux::aidmlm::AidmlmInitParam init_param;
init_param.type = aplux::aidmlm::ModelType::QWEN25VL;
init_param.vision_model_path = "/path/to/veg.serialized.bin.aidem";
init_param.pos_emb_cos_path = "/path/to/position_ids_cos.raw";
init_param.pos_emb_sin_path = "/path/to/position_ids_sin.raw";
init_param.embedding_weights_path = "/path/to/embedding_weights.raw";
init_param.window_attention_mask_path = "/path/to/window_attention_mask.raw";
init_param.full_attention_mask_path = "/path/to/full_attention_mask.raw";
init_param.llm_model_path_vec.push_back("/path/to/llm_model.serialized.bin.aidem");
init_param.use_genie_load_model_ex = true;

aplux::aidmlm::Aidmlm mlm_ctx;
if(mlm_ctx.initialize(init_param) < 0){
    printf("AidMLM initialize failed.\n");
    return EXIT_FAILURE;
}

Sampling Parameter Setup Operation.set_sampler()

After initialization completes successfully, sampling parameters can be set with this function to control the randomness, diversity, and quality of generated content.

API	set_sampler
Description	Sets sampling parameters to control the randomness and diversity of LLM outputs.
Parameters	key: Name of the sampling parameter. Currently supported: `"temp"`: Controls output randomness (Temperature); smaller values are more conservative. `"top-k"`: Limits the sampling range to the top K tokens with the highest probability. `"top-p"`: Nucleus Sampling; limits to the token pool whose cumulative probability reaches P.
Parameters	value: Parameter value represented as a string: For `"temp"`: floating-point numeric string (e.g. `"1.2"`). For `"top-k"`: integer numeric string (e.g. `"20"`). For `"top-p"`: floating-point numeric string (e.g. `"0.6"`).
Return Value	A value of 0 indicates success; a non-zero value indicates failure (e.g. invalid key or unsupported value format).

cpp

mlm_ctx.set_sampler("top-k", "20");
mlm_ctx.set_sampler("temp", "0.8");

Session Inference Operation.run()

After successful initialization, you can send image-text combined prompts to the multimodal model for inference.

💡Note

This function is not thread-safe. Only one thread can call the run method at a time.

API	run
Description	Executes one multimodal session inference
Parameters	prompt: User prompt string
	sys_prompt: System prompt string
	img_vec: Reference to an ImageData vector containing the image data to input
	cb: Callback function of type AidLLMCB for handling inference results
	starting_round: Whether this is the start of a new conversation round (true for new conversation start)
	user_data: Pointer to user data, convenient for using this data in custom callback functions, default value is nullptr
Return Value	A value of 0 indicates the inference executed successfully; otherwise a non-zero value indicates failure

cpp

// Define callback function
void my_callback(aplux::aidmlm::AidLLMCBData& cb_data, void* user_data){
    if(cb_data.state == aplux::aidmlm::AidLLMState::START){
        printf("[BOS]%s", cb_data.text.c_str());
    }else if(cb_data.state == aplux::aidmlm::AidLLMState::FINISH){
        printf("[EOS]%s\n", cb_data.text.c_str());
    }else if(cb_data.state == aplux::aidmlm::AidLLMState::ERROR){
        printf("[ERROR]%s\n", cb_data.text.c_str());
    }else{
        printf("%s", cb_data.text.c_str());
    }
}

// Prepare image data (pre-resize to model-required dimensions, RGB format)
cv::Mat img = cv::imread("test.jpg");
cv::Mat img_rgb;
cv::cvtColor(img, img_rgb, cv::COLOR_BGR2RGB);
cv::Mat img_resized;
cv::resize(img_rgb, img_resized, cv::Size(392, 392));

aplux::aidmlm::ImageData img_data = {
    .img_pos = -1,
    .img_data = (uint8_t*)img_resized.data,
};
std::vector<aplux::aidmlm::ImageData> img_vec;
img_vec.push_back(img_data);

// Execute inference
std::string sys_prompt = "You are a helpful assistant.";
std::string user_prompt = "Please describe the scene in this image";
int run_result = mlm_ctx.run(user_prompt, sys_prompt, img_vec, my_callback, true);
if(run_result < 0){
    printf("AidMLM run failed.\n");
    return EXIT_FAILURE;
}

Session Termination Operation.abort()

Used to interrupt the currently running inference session.

API	abort
Description	Terminates the currently running inference session
Parameters	reserve: Reserved field, default value is nullptr
Return Value	A value of 0 indicates successful termination; otherwise a non-zero value indicates failure

Reset Operation.reset()

In multi-round conversation scenarios, when you need to process the next image or restart a conversation, call reset to clear internal state.

API	reset
Description	Resets the internal state of the inference engine to prepare for the next inference
Parameters	void
Return Value	A value of 0 indicates successful reset; otherwise a non-zero value indicates failure

cpp

// Reset after processing one image, prepare for the next
if(mlm_ctx.reset() < 0){
    printf("AidMLM reset failed.\n");
    return EXIT_FAILURE;
}

Final Release Operation.finalize()

Releases model resources and completes de-initialization.

API	finalize
Description	Releases model resources and completes necessary de-initialization operations
Parameters	void
Return Value	A value of 0 indicates successful release; otherwise a non-zero value indicates failure

cpp

if(mlm_ctx.finalize() < 0){
    printf("AidMLM finalize failed.\n");
    return EXIT_FAILURE;
}

Get Profiler.get_profiler()

When the profiler is enabled during initialization, this function can be used to obtain the Profiler object pointer. For detailed usage, refer to the "Profiler C++ API Documentation" section below.

API	get_profiler
Description	Gets the pointer to the profiler object
Parameters	void
Return Value	If the profiler is enabled, returns a Profiler object pointer; if not enabled, returns nullptr

cpp

// Enable profiler
aplux::aidmlm::AidmlmInitParam init_param;
init_param.enable_profiler = true;
// ... other parameter setup ...
mlm_ctx.initialize(init_param, true);

// Get performance data after inference
aplux::aidgen::Profiler* profiler = mlm_ctx.get_profiler();
aplux::aidgen::ProfileData data = profiler->get_data();
printf("Init time: %lu us\n", data.init_time_us);
printf("Time to first token: %lu us\n", data.time_to_first_token_us);
printf("Generate rate: %.2f tok/s\n", data.generate_rate);
printf("ViT execute time: %lu us\n", data.vit_execute_time_us);

Profiler C++ API Documentation

💡Note

Profiler-related interfaces are under the aplux::aidgen namespace (independent of aplux::aidllm and aplux::aidmlm).

Header file path /usr/local/include/aidlux/aidgen/profiler.hpp
Both AidLLM and AidMLM use their respective get_profiler() methods to obtain a Profiler object for performance analysis

Performance Data Type.struct ProfileData

During inference, developers may want to monitor performance metrics at each stage. The ProfileData struct stores performance data collected during the inference process.

Member List

Member Name	Type	Description
init_time_us	uint64_t	Initialization time (microseconds)
prompt_token_num	uint64_t	Number of input prompt tokens
prompt_processing_rate	float	Prompt processing rate (tok/s)
time_to_first_token_us	uint64_t	Time to first token (microseconds)
generated_token_num	uint64_t	Number of generated tokens
generate_rate	float	Token generation rate (tok/s)
generate_time_us	uint64_t	Total generation time (microseconds)
vit_execute_time_us	uint64_t	Vision model execution time (microseconds), AidMLM only
vit_init_time_us	uint64_t	Vision model initialization time (microseconds), AidMLM only
vit_preprocess_time_us	uint64_t	Vision model preprocessing time (microseconds), AidMLM only
vit_postprocess_time_us	uint64_t	Vision model postprocessing time (microseconds), AidMLM only

Profiler Class.class Profiler

The Profiler class manages performance data collection during inference. It must be enabled during initialization (enable_profiler = true) to be used.

Get Performance Data.get_data()

Gets the currently collected performance data.

API	get_data
Description	Gets the performance analysis data collected during inference
Parameters	void
Return Value	ProfileData struct containing performance metrics for each stage

Reset Performance Data.reset()

Resets the collected performance data, typically called before starting a new inference round.

API	reset
Description	Clears collected performance data and restores to initial state
Parameters	void
Return Value	void

cpp

// Enable profiler during initialization
int init_result = llm_interpreter_ptr->initialize(true);

// Get the profiler
aplux::aidgen::Profiler* profiler = llm_interpreter_ptr->get_profiler();

// Execute inference...
llm_interpreter_ptr->run(prompt, dialog_callback);

// Get performance data
aplux::aidgen::ProfileData data = profiler->get_data();
printf("Time to first token: %lu us\n", data.time_to_first_token_us);
printf("Generate rate: %.2f tok/s\n", data.generate_rate);
printf("Generated token count: %lu\n", data.generated_token_num);

// Reset data, prepare for next inference round
profiler->reset();

AidGen C++ API ​

AidGen C++ API Documentation ​

AidGen 2.x C++ API Documentation ​

Log Level.enum LogLevel ​

Error Code.enum ErrorCode ​

Backend Type.enum BackendType ​

Generation State.enum GenState ​

Global Functions ​

Get Library Version.get_library_version() ​

Get ABI Version.get_abi_version() ​

Set Log Level.set_log_level() ​

Set Log File Prefix.set_log_file_prefix() ​

Context Properties.struct ContextProperties ​

Member List ​

Tensor.struct Tensor ​

Member List ​

Generation Event.struct GenEvent ​

Member List ​

Generation Callback Type.GenCallback ​

Extractor Profile Data.struct ExtractorProfileData ​

Member List ​

Generator Profile Data.struct GeneratorProfileData ​

Member List ​

Profile Data.struct ProfileData ​

Member List ​

Runtime Context Class.class Context ​

Create Instance Object.create_instance() ​

Initialize Operation.initialize() ​

Get Config.get_config() ​

Set Config.set_config() ​

Tokenizer Class.class Tokenizer ​

Create Instance Object.create_instance() ​

Initialize Operation.initialize() ​

Encode Text to Token.encode() ​

Decode Token to Text.decode() ​

Finalize Release Operation.finalize() ​

Embedding Extractor Class.class EmbeddingExtractor ​

Create Instance Object.create_instance() ​

Initialize Operation.initialize() ​

Finalize Release Operation.finalize() ​

Get Property.get_property() ​

Set Property.set_property() ​

Encode to Embedding.encode() ​

Get Profiler.get_profiler() ​

Generator Class.class Generator ​

Create Instance Object.create_instance() ​

Initialize Operation.initialize() ​

Finalize Release Operation.finalize() ​

Get Property.get_property() ​

Set Property.set_property() ​

Get Embedding Buffer.get_embedding_buff() ​

Text-Only Inference.run() ​

Multimodal Inference.run() ​

Reset Conversation State.reset() ​

Abort Inference.abort() ​

Get Profiler.get_profiler() ​

Get Current State.get_state() ​

Typical Usage Flow ​

Complete Text-Only Inference Flow ​

Tokenizer Usage Flow ​

Embedding Extraction Flow ​

Error Handling Pattern ​

Memory Management Principles ​

AidGen 1.x C++ API Documentation (Deprecated) ​

AidLLM C++ API Documentation ​

Inference Backend Type.enum LLmBackendType ​

Inference Task State.enum LLMSentenceState ​

Interpreter Runtime State.enum LLMState ​

Log Level.enum LogLevel ​

Global Functions ​

Get Library Version.get_library_version() ​

Set Log Level.set_log_level() ​

Set Log File Prefix.set_log_file_prefix() ​

Inference Callback Function Type.LLMCallback ​

Inference Callback Data Type.struct LLMCallbackData ​

Member List ​

Runtime Context Class.class LLMContext ​

Create Instance Object.create_instance() ​

Member List ​

Interpreter Class.class LLMInterpreter ​

AidGen C++ API

AidGen C++ API Documentation

AidGen 2.x C++ API Documentation

Log Level.enum LogLevel

Error Code.enum ErrorCode

Backend Type.enum BackendType

Generation State.enum GenState

Global Functions

Get Library Version.get_library_version()

Get ABI Version.get_abi_version()

Set Log Level.set_log_level()

Set Log File Prefix.set_log_file_prefix()

Context Properties.struct ContextProperties

Member List

Tensor.struct Tensor

Member List

Generation Event.struct GenEvent

Member List

Generation Callback Type.GenCallback

Extractor Profile Data.struct ExtractorProfileData

Member List

Generator Profile Data.struct GeneratorProfileData

Member List

Profile Data.struct ProfileData

Member List

Runtime Context Class.class Context

Create Instance Object.create_instance()

Initialize Operation.initialize()

Get Config.get_config()

Set Config.set_config()

Tokenizer Class.class Tokenizer

Create Instance Object.create_instance()

Initialize Operation.initialize()

Encode Text to Token.encode()

Decode Token to Text.decode()

Finalize Release Operation.finalize()

Embedding Extractor Class.class EmbeddingExtractor

Create Instance Object.create_instance()

Initialize Operation.initialize()

Finalize Release Operation.finalize()

Get Property.get_property()

Set Property.set_property()

Encode to Embedding.encode()

Get Profiler.get_profiler()

Generator Class.class Generator

Create Instance Object.create_instance()

Initialize Operation.initialize()

Finalize Release Operation.finalize()

Get Property.get_property()

Set Property.set_property()

Get Embedding Buffer.get_embedding_buff()

Text-Only Inference.run()

Multimodal Inference.run()

Reset Conversation State.reset()

Abort Inference.abort()

Get Profiler.get_profiler()

Get Current State.get_state()

Typical Usage Flow

Complete Text-Only Inference Flow

Tokenizer Usage Flow

Embedding Extraction Flow

Error Handling Pattern

Memory Management Principles

AidGen 1.x C++ API Documentation (Deprecated)

AidLLM C++ API Documentation

Inference Backend Type.enum LLmBackendType

Inference Task State.enum LLMSentenceState

Interpreter Runtime State.enum LLMState

Log Level.enum LogLevel

Global Functions

Get Library Version.get_library_version()

Set Log Level.set_log_level()

Set Log File Prefix.set_log_file_prefix()

Inference Callback Function Type.LLMCallback

Inference Callback Data Type.struct LLMCallbackData

Member List

Runtime Context Class.class LLMContext

Create Instance Object.create_instance()

Member List

Interpreter Class.class LLMInterpreter