Host functions for sparse Symbol-keyed map creation and unpacking
Specification
CAP: 0086
Title: Host functions for sparse Symbol-keyed map creation and unpacking
Working Group:
Owner: Dmytro Kozhevin <@dmkozh>
Authors: Dmytro Kozhevin <@dmkozh>, Leigh McCulloch <@leighmcculloch>
Consulted:
Status: Final
Created: 2026-07-08
Discussion: https://github.com/orgs/stellar/discussions/1877
Protocol version: 28
Simple Summary
Introduce new host functions that support data migration for map-based Soroban UDTs.
Working Group
As specified in the Preamble.
Motivation
The de facto standard Soroban UDT contracttype structs implementation is based on a MapObject with Symbol keys, backed by the two host functions map_new_from_linear_memory and map_unpack_to_linear_memory that allow converting UDT structs to and from MapObjects, respectively. However, these host functions do strict validation and cause the contract call to trap if the input/output map doesn't exactly match the expected schema (i.e. if it doesn't have the exact same keys as the struct). This makes it impossible to migrate UDTs to a new schema, because the new schema may have additional keys or may omit some of the old keys.
This is problematic for most UDT-based data migration scenarios:
- A contract may need to add a new field to a UDT struct or remove an old unused field when its implementation is updated. There are known cases where this has rendered contracts unusable after an update.
- A struct used in a function interface may need to evolve, but cannot be updated unless all interface-user contracts are updated atomically, which is not feasible if multiple contract instances share the same interface.
- A contract may want to verify that a map matches an expected schema without manually iterating it, but the existing host functions trap on any error instead of returning gracefully.
All of these issues are technically avoidable by either very careful planning of the UDT schema (with some combination of manual versioning and enums), or by implementing a custom map serialization/deserialization logic in the contract code, which requires developers to bypass the existing standard SDK and host functions. Both options are non-obvious and error-prone, and the latter option is also inefficient and requires more code to be written and maintained.
Thus this CAP proposes to provide a new standard way of manipulating the UDTs efficiently and in a migration-friendly manner.
Goals Alignment
This CAP is aligned with the following Stellar Network Goals:
- The Stellar Network should make it easy for developers of Stellar projects to create highly usable products
Abstract
Two new host functions are introduced: sparse_map_new_from_linear_memory and sparse_map_unpack_to_linear_memory. They are mostly similar to the existing host functions map_new_from_linear_memory and map_unpack_to_linear_memory, but they have more relaxed schema validation rules. Specifically, if the expected key is not present in the map, it will be fetched as Void value (which maps to None in the UDTs), and the Void values are never explicitly stored in the map. sparse_map_unpack_to_linear_memory also allows fetching only a subset of the map keys.
Specification
New host functions
The diff is based on commit f0123830e1ee64c993139d86697454680c059cd0 of rs-soroban-env.
diff --git a/soroban-env-common/env.json b/soroban-env-common/env.json
index fa41a862..8e8c6d14 100644
--- a/soroban-env-common/env.json
+++ b/soroban-env-common/env.json
@@ -1047,7 +1047,7 @@
}
],
"return": "MapObject",
- "docs": "Return a new map initialized from a pair of equal-length arrays, one for keys and one for values, given by a pair of linear-memory addresses and a length in Vals."
+ "docs": "Return a new map initialized from a pair of equal `len` length arrays, one for keys and one for values, specified by linear memory addresses. Key strings are specified as `len` 8 byte slices consisting of the 4 byte pointer and 4 byte length. Actual keys must be byte strings sorted in ascending order and be convertible to `Symbol` type. Values may be arbitrary `Val`s. Panics if any of the invariants above are violated."
},
{
"export": "a",
@@ -1071,7 +1071,53 @@
}
],
"return": "Void",
- "docs": "Copy Vals from `map` to the array `vals_pos`, selecting only the keys identified by the array `keys_pos`. Both arrays have `len` elements and are identified by linear-memory addresses."
+ "docs": "Copy all value `Val`s from `map` to the linear memory array at `vals_pos` address. `len` must match the number of entries in `map`. Map keys must be of `Symbol` type and must match the key byte strings in the linear memory array at `keys_pos` address. Key strings are specified as 8 byte slices consisting of the 4 byte pointer and 4 byte length. Keys must be sorted in ascending order. Panics if any of the invariants above are violated."
+ },
+ {
+ "export": "b",
+ "name": "sparse_map_new_from_linear_memory",
+ "args": [
+ {
+ "name": "keys_pos",
+ "type": "U32Val"
+ },
+ {
+ "name": "vals_pos",
+ "type": "U32Val"
+ },
+ {
+ "name": "len",
+ "type": "U32Val"
+ }
+ ],
+ "return": "MapObject",
+ "docs": "Return a new map initialized from a pair of equal `len` length arrays, one for keys and one for values, specified by linear memory addresses. Key strings are specified as `len` 8 byte slices consisting of the 4 byte pointer and 4 byte length. Actual keys must be byte strings sorted in ascending order and be convertible to `Symbol` type. Values may be arbitrary `Val`s. Key-value pairs where the value is `Void` are not included into the final map. Panics if any of the invariants above are violated.",
+ "min_supported_protocol": 28
+ },
+ {
+ "export": "c",
+ "name": "sparse_map_unpack_to_linear_memory",
+ "args": [
+ {
+ "name": "map",
+ "type": "MapObject"
+ },
+ {
+ "name": "keys_pos",
+ "type": "U32Val"
+ },
+ {
+ "name": "vals_pos",
+ "type": "U32Val"
+ },
+ {
+ "name": "len",
+ "type": "U32Val"
+ }
+ ],
+ "return": "Void",
+ "docs": "Fetch value `Val`s from `map` to the linear memory array at `vals_pos` address according to the key byte strings stored in linear memory at `keys_pos`. Key strings are specified as 8 byte slices consisting of the 4 byte pointer and 4 byte length. Keys must be sorted in ascending order. The map keys are expected to have `Symbol` type and its content bytes are matched to the input keys. If there is no matching map key, the corresponding value is set to `Void`. Panics if any of the invariants above are violated.",
+ "min_supported_protocol": 28
}
]
},
Note that the documentation changes to the existing host functions are purely cosmetic and do not change the semantics of the functions, but instead document their behavior more precisely, so that the difference between the existing and new host functions is more clear.
Semantics
sparse_map_new_from_linear_memory function
sparse_map_new_from_linear_memory host function fetches the len elements from the linear memory starting at keys_pos and vals_pos addresses, zips them into key-value pairs, and creates a new map with the fetched key-value pairs, skipping any pairs where the value is Void. The resulting map is returned.
Keys array is specified by len 8 byte slices consisting of the 4 byte pointer followed by 4 byte lengths (low-endian). The actual keys are byte strings identified by the slices. They must be byte strings sorted in the ascending order and must be convertible to Symbol type.
Values array is specified by len Vals starting at vals_pos address. Values may be arbitrary Vals.
The output map will consist of key-value pairs, where the key is a Symbol converted from the corresponding key byte string, and the value is the corresponding Val. Any key-value pair where the value is Void is skipped and not included in the output map.
The function panics if any of its invariants are violated.
sparse_map_unpack_to_linear_memory function
sparse_map_unpack_to_linear_memory host function fetches the len elements from the linear memory starting at keys_pos address, and for each key, it fetches the corresponding value from the input map. If a key is not present in the map, the corresponding value is set to Void. The fetched values are written to the linear memory starting at vals_pos address.
Keys array is specified by len 8 byte slices consisting of the 4 byte pointer followed by 4 byte lengths (low-endian). The actual keys are byte strings identified by the slices. They must be byte strings sorted in the ascending order. Note, that the function doesn't require that the input keys are valid Symbol values for the sake of optimization, but these may never be found in the map, and thus the corresponding values will always be set to Void.
The function tries to find every key in the provided map. Map keys are expected to have Symbol type, and their content bytes are compared with the input keys. If a key is not present in the map, the corresponding value is set to Void.
The function panics if any of its invariants are violated.
Design Rationale
New host functions vs changing the old host functions
The new host functions have exactly the same interface as the existing ones, and there is a consideration to change the behavior of the existing host functions on the protocol boundary, instead of introducing new host functions.
This approach has a benefit of every existing contract to benefit from the new behavior to some degree (as long as they are using the Option fields or just need the ability to not fetch every key from the map). However, there is a subtle risk of performing this change unconditionally: if a contract does actually rely on the strict schema validation for safety, the protocol upgrade would break the assumptions around that, and the contract may become vulnerable to some attacks. It's quite unlikely that it would happen in practice, but we chose to err on the side of caution here.
SDK may adopt the new host functions as soon as they're available, and then the developers will be able to use the migration-friendly behavior in a more explicit way - even if there are some subtle issues with the new logic, they will be able to test it first before mainnet deployment (unlike in case of an unconditional change of the existing host functions).
Protocol Upgrade Transition
The proposed functionality will be available starting from protocol version TBD.
Backwards Incompatibilities
There are no backwards incompatibilities introduced by this CAP.
Resource Utilization
The new host functions are going to be metered using the existing metering primitives. No other resource utilization changes are expected.
Security Concerns
As mentioned in the Design Rationale section, the new host functions are more permissive than the existing ones. However, given that the change requires developers to rebuild their contracts with the new version of the SDK, the subtle validation behavior expectations may be tested and manual verification may be added if necessary.
Test Cases
TBD
Implementation
TBD
Appendix: UDT migration examples
While UDTs are technically not in the CAP scope, the following examples illustrate how UDTs that use the new host functions can be used to perform data migration.
Adding new storage fields to a UDT struct
Consider a contract that has the following UDT struct for storing the admin information:
#[contracttype]
struct AdminInfo {
admin: Address,
is_frozen: bool,
}
In an updated version the contract wants to support multiple admins. This can be achieved by adding a new field to the struct:
#[contracttype]
struct AdminInfo {
admin: Address, // deprecated
is_frozen: bool,
admins: Option<Vec<Address>>,
}
This struct is compatible with the old struct, and the new field admins can be initialized with the existing admin field value after the update or on first access. In the later update, the admin field can be removed from the struct completely, as long as the admins field has been initialized:
#[contracttype]
struct AdminInfo {
admins: Vec<Address>,
is_frozen: bool,
}
This struct is compatible with the previous struct and is sufficient to keep the contract operational (notice, that Option is not really necessary for admins anymore, although it may be kept as well for the sake of flexibility).
Extending the interface of a contract function
Consider a protocol where multiple contracts consult the same policy contract to perform some checks. The policy contract has the following function for checking the policy:
fn check_policy(state: ProtocolState) -> bool {
// some checks
}
#[contracttype]
struct ProtocolState {
total_debt: i128,
total_collateral: i128,
}
There are several different contracts that call check_policy.
Now let's say the protocol state definition has been expanded to include the total_liquidation field, and the policy contract needs to be updated to support it. There are several ways to go about this, for example, the ProtocolState struct can be updated to include the new field as non-optional:
#[contracttype]
struct ProtocolState {
total_debt: i128,
total_collateral: i128,
total_liquidation: i128,
}
Every caller of check_policy can be updated to provide the new field and still remain compatible with the old version of the policy contract. After every caller has been updated, the policy contract itself can be updated as well.
Alternatively, the new field can be added as an Option type, which allows the policy contract to be updated first, and then the callers can be updated one by one:
#[contracttype]
struct ProtocolState {
total_debt: i128,
total_collateral: i128,
total_liquidation: Option<i128>,
}
After every contract has been updated, the policy contract can be updated again to enforce that the total_liquidation field is always present (by unwrapping the Option, or removing it from the type definition altogether).
Preamble
Discussion
0 linked threads