C API

Make sure the VOLLO_TREES_SDK environment variable is set by sourcing setup.sh from the Vollo Trees SDK.

source <path-to-VOLLO_TREES_SDK>/setup.sh

The Vollo runtime API is a C API with simple types and functions in order to be straight forward to use from any language with a C FFI.

• Header file: $VOLLO_TREES_SDK/include/vollo-rt.h
• Dynamic library: $VOLLO_TREES_SDK/lib/libvollo_rt.so
• Static library: $VOLLO_TREES_SDK/lib/libvollo_rt.a

It was built against GLIBC version 2.17.

To compile against Vollo RT with a standard C compiler, you can use the following flags: -I $VOLLO_TREES_SDK/include -L $VOLLO_TREES_SDK/lib -lvollo_rt

These are the main steps (in order) a program using vollo_rt will follow:

1. Initialise the Vollo runtime using vollo_rt_init
2. Add Vollo accelerators to the runtime using vollo_rt_add_accelerator (Note: the current release only supports one accelerator)
3. Load a Vollo program onto the Vollo accelerators with vollo_rt_load_program
4. Optionally, inspect the metadata about the models in the program using API calls such as vollo_rt_num_models and vollo_rt_model_num_inputs
5. Queue and run inference jobs by first calling vollo_rt_add_job with number_format_fp32 inputs and the model's output format (vollo_rt_model_output_format: number_format_fp64 for models compiled from double-precision leaves, number_format_fp32 otherwise), and then polling in a loop for their completion using vollo_rt_poll. You can queue several jobs before calling vollo_rt_poll or add extra jobs at any point.
6. Finally call vollo_rt_destroy to release resources.

The API is designed to explicitly return errors when it can to let the user handle them as they see fit. The metadata functions will instead error out themselves if any of the documented pre-conditions they rely on aren't met. Any other crash is considered a bug and we would be very grateful if you could tell us about it.

Initialisation

A vollo context is created by calling vollo_rt_init. Add an accelerator by using the vollo_rt_add_accelerator function, or vollo_rt_add_device to name a card by its PCI address (see hosts with more than one accelerator).

/**
 * Initialise the vollo-rt context. This must be called before any other vollo-rt functions.
 *
 * Logging level can be configured by setting the environment variable `VOLLO_RT_LOG` to one of:
 * "error", "warn", "info", "debug", or "trace"
 */
vollo_rt_error_t vollo_rt_init(vollo_rt_context_t* context_ptr);

/**
 * Destroy vollo-rt context, releasing its associated resources.
 */
void vollo_rt_destroy(vollo_rt_context_t vollo);

/**
 * Add an accelerator.
 * The accelerator is specified by its index. The index refers to an accelerator in the sorted list
 * of PCI addresses. This should be called after `vollo_rt_init` but before `vollo_rt_load_program`
 */
vollo_rt_error_t vollo_rt_add_accelerator(vollo_rt_context_t vollo, size_t accelerator_index);

Loading a program

A program is loaded onto the Vollo accelerator using the vollo_rt_load_program function.

/**
 * Load a program onto the Vollo accelerators.
 * This should be called after `vollo_rt_add_accelerator`
 *
 * A Vollo program is generated by the Vollo compiler, it is typically named
 * "<program_name>.vollo".
 * The program is intended for a specific hardware config (number of accelerators,
 * cores and other configuration options), this function will return an
 * error if any accelerator configuration is incompatible with the program.
 * Once loaded, the program provides inference for several models concurrently.
 *
 * Note: This should only be called once per `vollo_rt_context_t`, as such if
 * a program needs to be changed or reset, first `vollo_rt_destroy` the current
 * context, then start a new context with `vollo_rt_init`.
 */
vollo_rt_error_t vollo_rt_load_program(vollo_rt_context_t vollo, const char* program_path);

Model metadata

Once a program is loaded, it provides inference for one or more models. Metadata about a model is obtained with vollo_rt_model_* functions.

Each model can have multiple distinct inputs and outputs. Each input and each output has a multi-dimensional shape associated with it. All of the metadata is defined by the program as supplied by the Vollo compiler. All the shapes are statically defined.

Some models can be compiled as streaming statefully over a dimension, that dimension is then erased from the inference shape but its possition can be recovered in the model metadata.

/**
 * Inspect the number of models in the program loaded onto the vollo.
 *
 * Programs can contain multiple models, a `model_index` is used to select a
 * specific model
 */
size_t vollo_rt_num_models(vollo_rt_context_t vollo);

/**
 * Get the number of inputs of a model
 *
 * Each input has its own distinct shape
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 */
size_t vollo_rt_model_num_inputs(vollo_rt_context_t vollo, size_t model_index);

/**
 * Get the number of outputs of a model
 *
 * Each output has its own distinct shape
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 */
size_t vollo_rt_model_num_outputs(vollo_rt_context_t vollo, size_t model_index);

/**
 * Get the shape for input at a given index
 *
 * The return value is an array of dims containing the input shape
 * Use `vollo_rt_model_input_shape_len` to get the number of axes in the shape.
 *
 * For backwards compatibility the array is also 0-terminated, but that should not be relied upon
 * in order to correctly support shapes containing a 0 dimension
 *
 * The value lives for as long as the model
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `input_index < vollo_rt_model_num_inputs`
 */
const size_t* vollo_rt_model_input_shape(
  vollo_rt_context_t vollo, size_t model_index, size_t input_index);

/**
 * Get the number of axes in the shape for the input at a given index
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `input_index < vollo_rt_model_num_inputs`
 */
size_t vollo_rt_model_input_shape_len(
  vollo_rt_context_t vollo, size_t model_index, size_t input_index);

/**
 * Get the shape for output at a given index
 *
 * The return value is an array of dims containing the output shape
 * Use `vollo_rt_model_output_shape_len` to get the number of axes in the shape.
 *
 * For backwards compatibility the array is also 0-terminated, but that should not be relied upon
 * in order to correctly support shapes containing a 0 dimension
 *
 * The value lives for as long as the model
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `output_index < vollo_rt_model_num_outputs`
 */
const size_t* vollo_rt_model_output_shape(
  vollo_rt_context_t vollo, size_t model_index, size_t output_index);

/**
 * Get the number of axes in the shape for the output at a given index
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `output_index < vollo_rt_model_num_outputs`
 */
size_t vollo_rt_model_output_shape_len(
  vollo_rt_context_t vollo, size_t model_index, size_t output_index);

/**
 * Get the number of elements for input at a given index
 *
 * This is simply the product of the dimensions returned by `vollo_rt_model_input_shape`,
 * it is provided to make it easier to allocate the correct number of elements.
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `input_index < vollo_rt_model_num_inputs`
 */
size_t vollo_rt_model_input_num_elements(
  vollo_rt_context_t vollo, size_t model_index, size_t input_index);

/**
 * Get the number of elements for output at a given index
 *
 * This is simply the product of the dimensions returned by `vollo_rt_model_output_shape`,
 * it is provided to make it easier to allocate the correct number of elements.
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `output_index < vollo_rt_model_num_outputs`
 */
size_t vollo_rt_model_output_num_elements(
  vollo_rt_context_t vollo, size_t model_index, size_t output_index);

/**
 * Get the number type of an input at the given index
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `input_index < vollo_rt_model_num_inputs`
 */
number_format vollo_rt_model_input_format(
  vollo_rt_context_t vollo, size_t model_index, size_t input_index);

/**
 * Get the number type of an output at the given index
 *
 * Requirements (panics otherwise):
 * - a program was loaded with `vollo_rt_load_program`
 * - `model_index < vollo_rt_num_models`
 * - `output_index < vollo_rt_model_num_outputs`
 */
number_format vollo_rt_model_output_format(
  vollo_rt_context_t vollo, size_t model_index, size_t output_index);

Running inference

The interface returns results asynchronously so that inference requests can be made as fast as the system can support, without blocking on output data being returned. This way, it also supports running multiple requests concurrently. Before any compute is started a job with associated input and output buffers needs to be registered with the runtime using vollo_rt_add_job with number_format_fp32 inputs and the model's output format — Vollo Trees takes fp32 inputs and delivers each output as the exact 64-bit fixed-point leaf sum converted once to the model's output format: number_format_fp64 for models whose ONNX states its leaves in double precision (the ai.onnx.ml opset 3 *_as_tensor attributes), number_format_fp32 otherwise. Query it with vollo_rt_model_output_format; outputs requested in another format are converted from the native result on the CPU.

/**
 * Sets up a computation on the vollo accelerator where the inputs and outputs number formats are
 * specified. If the number format differs from the model's native format, conversion will be
 * performed on the CPU.
 *
 * The model's native number format can be queried with `vollo_rt_model_input_format` and
 * `vollo_rt_model_output_format`.
 * Vollo Trees natively takes fp32 inputs and produces outputs in the model's own format: fp64
 * when the ONNX model states its leaves in double precision, fp32 otherwise. bf16 inputs are
 * widened losslessly, fp64 inputs are rejected, and outputs requested in another format are
 * converted from the native result on the CPU.
 *
 * Note:
 * - The computation may be started by this call, or only on the next call to vollo_rt_poll:
 *   either way it is possible to set up several computations that are kicked off at the
 *   same time.
 *
 * - vollo:
 *     the context that the computation should be run on
 * - model_index:
 *     the model to run
 * - user_ctx:
 *     a user context that will be returned on completion. This can be used to disambiguate when
 *     multiple models are running concurrently.
 *     NOTE: the jobs for a single model are guaranteed to come back in order, but the jobs for
 *     different models are not.
 * - input_data:
 *     A pointer to the start of an array with pointers to the start of the data to each input.
 *     The number of inputs is given by `vollo_rt_model_num_inputs`. Each input length is the
 *     product of the shape given by `vollo_rt_model_input_shape` (or more convenient:
 *     `vollo_rt_model_input_num_elements`). The number format of each input is given by
 *     `vollo_rt_model_input_format`.
 *     - lifetime:
 *       - The outer array only needs to live until `vollo_rt_add_job` returns
 *       - The input buffers need to live until `vollo_rt_poll` returns with the completion for
 *         this job
 * - output_data:
 *     A pointer to the start of an array with pointers to the start of the data to each output
 *     buffer. The number of outputs is given by `vollo_rt_model_num_outputs`. Each output length
 *     is the product of the shape given by `vollo_rt_model_output_shape`
 *     (or more convenient: `vollo_rt_model_output_num_elements`). The number format of each
 *     output is given by `vollo_rt_model_output_format`.
 *     lifetime:
 *       - The outer array only needs to live until `vollo_rt_add_job` returns
 *       - The output buffers need to live until `vollo_rt_poll` returns with the completion for
 *         this job
 */
vollo_rt_error_t vollo_rt_add_job(
  vollo_rt_context_t vollo,
  size_t model_index,
  uint64_t user_ctx,
  const number_format* input_number_format,
  const void* const* input_data,
  const number_format* output_number_format,
  void* const* output_data);

To actually start and later complete an inference you must use the vollo_rt_poll function multiple times. It is typically called in a loop (with a timeout) until some or all of the jobs are completed.

/**
 * Poll the vollo accelerator for completion.
 *
 * Note: Polling also initiates transfers for new jobs, so poll must be called
 * before any progress on these new jobs can be made.
 *
 *   num_completed: out: the number of completed user_ctx returned
 *   returned_user_ctx: buffer for the returned user_ctx of completed jobs, this will only be
 *                      valid until the next call to vollo_rt_poll.
 */
vollo_rt_error_t vollo_rt_poll(
  vollo_rt_context_t vollo, size_t* num_completed, const uint64_t** returned_user_ctx);