Proposal: Dynamic Userdata

[cheesycod]

Note: I've already implemented this in a fork and it seems really useful as a feature so I wanted to pitch it to mlua upstream
Note 2: All code snippets below are merely illustrative of one possible way of doing this that is already implemented and working

The Problem

Currently, mlua's support for userdata is mostly generic and compile time oriented. While it's super ergonomic when all userdata is defined at compile time (admittedly, most use cases), its not very helpful if the userdata is defined at runtime. While mlua does have some support for runtime defined userdata (e.g. create_any_userdata), these methods are still semigeneric and require all userdata's of type T to share the same metatable. These limitations caused me problems with one of my hobby projects, gluau, in which the userdata is not known at compile time to Rust.

Proposed Solution

To solve this, I propose the addition of another way of creating userdata in mlua called dynamic userdata. A dynamic userdata is a special type of userdata that is fully defined at runtime through the use of a associated data and a metatable that is used to create/define the dynamic userdata and its properties. For example, using the below function signature:

pub fn create_dynamic_userdata<T>(&self, data: T, metatable: &Table) -> Result<AnyUserData>
    where
        T: Send + Sync + 'static,
    {

In short, a dynamic userdata takes an associated data of type T and a metatable (on non-Luau, mlua can add a constraint/error if this metatable contains __gc). Both of these things can be defined at runtime trivially (for an example, see gluau which already uses this to implement userdata). An example implementation of how mlua could interally make a dynamic userdata with a given metatable could look like:

    #[cfg(feature = "dynamic-userdata")]
    pub(crate) unsafe fn make_dyn_userdata(
        &self,
        mt: &Table,
        data: Box<dyn Any + Send + Sync>,
    ) -> Result<AnyUserData> {
        let state = self.state();
        let _sg = StackGuard::new(state);
        check_stack(state, 3)?;

        let protect = !self.unlikely_memory_error();
        crate::util::push_userdata_dyn(state, data, protect)?;
        let ud_ptr = ffi::lua_topointer(state, -1);
        (*self.extra.get()).dyn_userdata_set.insert(ud_ptr as *mut c_void);

        self.push_ref_at(&mt.0, state);
        ffi::lua_setmetatable(state, -2);

        Ok(AnyUserData(self.pop_ref()))
    }

// Internally uses 3 stack spaces, does not call checkstack.
//
// mt_ptr is a pointer to the metatable for this userdata
// which is needed during the destructor call to clear out
// the associated data of the dynamic userdata.
#[inline]
#[cfg(feature = "dynamic-userdata")]
pub(crate) unsafe fn push_userdata_dyn(
    state: *mut ffi::lua_State,
    data: Box<dyn Any + Send + Sync>,
    protect: bool,
) -> Result<*mut DynamicUserDataPtr> {
    let size = const { mem::size_of::<DynamicUserDataPtr>() };

    #[cfg(feature = "luau")]
    let ud_ptr = if protect {
        protect_lua!(state, 0, 1, |state| {
            ffi::lua_newuserdatadtor(state, size, collect_userdata_dyn)
        })?
    } else {
        ffi::lua_newuserdatadtor(state, size, collect_userdata_dyn)
    } as *mut DynamicUserDataPtr;

    let t = DynamicUserDataPtr { data };
    ptr::write(ud_ptr, t);
    Ok(ud_ptr)
}

// This method is called by Luau GC when it's time to collect the userdata.
#[cfg(feature = "dynamic-userdata")]
pub(crate) unsafe extern "C" fn collect_userdata_dyn(
    state: *mut ffi::lua_State,
    ud: *mut std::os::raw::c_void,
) {
    use crate::userdata::registry::DynamicUserDataPtr;

    // Almost none Lua operations are allowed when destructor is running,
    // so we need to set a flag to prevent calling any Lua functions
    let extra = (*ffi::lua_callbacks(state)).userdata as *mut crate::state::ExtraData;
    (*extra).running_gc = true;
    // Luau does not support _any_ panics in destructors (they are declared as "C", NOT as "C-unwind"),
    // so any panics will trigger `abort()`.
    mlua_debug_assert!(
        (*extra).dyn_userdata_set.remove(&ud),
        "did not remove dyn userdata"
    );
    std::ptr::drop_in_place(ud as *mut DynamicUserDataPtr);
    (*extra).running_gc = false;
}

where DynamicUserDataPtr is:

#[cfg(feature = "dynamic-userdata")]
pub(crate) struct DynamicUserDataPtr {
    pub(crate) data: Box<dyn Any + Send + Sync>,
}

Users can then pass this dynamically created userdata to Lua code etc. To retrieve the stored 'associated data', I propose the addition of a new dynamic_data method to AnyUserData which could look like:

    /// For a dynamic userdata, returns the inner data put into the userdata.
    ///
    /// This will return `UserDataTypeMismatch` if the userdata is not dynamic or
    /// if the dynamic userdata's metatable is not associated with the type `T`.
    #[cfg(feature = "dynamic-userdata")]
    pub fn dynamic_data<T: 'static>(&self) -> Result<&T> {
        let lua = self.0.lua.lock();
        // Get the metatable pointer
        let ud_ptr = unsafe { ffi::lua_topointer(lua.ref_thread(self.0.aux_thread), self.0.index) };

        unsafe {
            if !(&(*lua.extra())).is_userdata_dynamic(ud_ptr as *mut c_void) {
                return Err(Error::UserDataTypeMismatch);
            }

            let ud = get_userdata::<DynamicUserDataPtr>(lua.ref_thread(self.0.aux_thread), self.0.index);

            match (&*ud).data.downcast_ref::<T>() {
                Some(data) => Ok(data),
                None => Err(Error::UserDataTypeMismatch),
            }
        }
    }

Of note here is that a dynamic userdata, unlike normal userdata, cannot be borrowed, taken, compared with is (at least without changes to is) or destroyed (outside of GC by Lua/Luau). The job of doing this would fall on the user.

[khvzak]

Currently, mlua's support for userdata is mostly generic and compile time oriented. While it's super ergonomic when all userdata is defined at compile time (admittedly, most use cases), its not very helpful if the userdata is defined at runtime. While mlua does have some support for runtime defined userdata (e.g. create_any_userdata), these methods are still semigeneric and require all userdata's of type T to share the same metatable. These limitations caused me problems with one of my hobby projects, gluau, in which the userdata is not known at compile time to Rust.

There is a Scope::create_any_userdata helper that is designed to support creating dynamic (non-static) userdata with uniq metatable per instance and runtime-defined methods.
Would it cover your case?

T: Send + Sync + 'static

Could you provide more detail please what is associated data of type T ?

[cheesycod]

Currently, mlua's support for userdata is mostly generic and compile time oriented. While it's super ergonomic when all userdata is defined at compile time (admittedly, most use cases), its not very helpful if the userdata is defined at runtime. While mlua does have some support for runtime defined userdata (e.g. create_any_userdata), these methods are still semigeneric and require all userdata's of type T to share the same metatable. These limitations caused me problems with one of my hobby projects, gluau, in which the userdata is not known at compile time to Rust.

There is a Scope::create_any_userdata helper that is designed to support creating dynamic (non-static) userdata with uniq metatable per instance and runtime-defined methods.

Would it cover your case?

T: Send + Sync + 'static

Could you provide more detail please what is associated data of type T ?

Unfortunately, scoped userdata is destroyed when the scope ends making it not fit for my usecase of having long lived dynamic userdata that can be returned from FFI to Go. It also doesn't allow for selectively sharing metatables which is useful to reduce memory usage. Lastly, it's much harder implementation wise coupled with worse performance for my use case (even if it could be used out of scope by e.g. storing a permanent heap allocated scope) as there's more synchronization and FFI calls+callbacks neeeded between the Rust+Go layer mostly (but not fully) due to the use of UserDataRegistry (function callbacks from Rust to Go are hard to get right)

Basically, I'd like for a way to make a userdata using a normal table to act as the userdatas metatable and get stored data (like a cgo handle) from the userdata

Associated data is effectively the data stored with the dynamic userdata and can be fetched by Rust after creation from the AnyUserData. The associated data is static as it must be able to live for the lifetime of the program and should have a type id

If implemented, this should probably be a Luau specific feature as I don't know how to make it safe outside of Luau

BTW, gluau uses my dynamic userdata proposal right now if it'd help to see an actual use case for this.

[cheesycod]

Reading the example, seems this can be solved by a table (or newproxy) with metatable? Any associated items can be stored in the table.

Not if you want typeof to work/return the type name unfortunately (which is admittedly rare but yeah)

[khvzak]

Not if you want typeof to work/return the type name unfortunately (which is admittedly rare but yeah)

Apparently this is done intentionally by Luau developers to avoid spoofing. See this

[cheesycod]

Not if you want typeof to work/return the type name unfortunately (which is admittedly rare but yeah)

Apparently this is done intentionally by Luau developers to avoid spoofing. See this

Yeah, thats why i made the dynamic userdata thing, to allow easily creating userdata with typename etc set and working and set some data inside the userdata to make it more ergonomic to use (you can share metatables on all your userdata's and have each have their own handles/whatever)

[khvzak]

Yeah, thats why i made the dynamic userdata thing, to allow easily creating userdata with typename etc set and working and set some data inside the userdata to make it more ergonomic to use (you can share metatables on all your userdata's and have each have their own handles/whatever)

I'm still not sure that the extra complexity worth it for such a rare use case.
Furthermore, mlua already provides all building blocks to solve the problem elegantly:

struct DynamicUserData(Box<dyn Any>);
impl LuaUserData for DynamicUserData {}

let newproxy_typed = lua.create_function(|lua, (data, mt): (LuaValue, LuaTable)| unsafe {
    let ud = lua.create_userdata(DynamicUserData(Box::new(data)))?;
    lua.exec_raw::<()>((&ud, &mt), |state| {
        mlua::ffi::lua_setmetatable(state, -2);
    })?;
    Ok(ud)
})?;
lua.globals().set("newproxy_typed", newproxy_typed)?;

lua.load(
    r#"
    local ud = newproxy_typed({"arbitrary", "data"}, {
        __type = "MyCustomType",
    })

    print(typeof(ud))
    "#,
)
.exec()?;

Is there anything I missed in the solution above?

[cheesycod]

Yeah, thats why i made the dynamic userdata thing, to allow easily creating userdata with typename etc set and working and set some data inside the userdata to make it more ergonomic to use (you can share metatables on all your userdata's and have each have their own handles/whatever)

I'm still not sure that the extra complexity worth it for such a rare use case.

Furthermore, mlua already provides all building blocks to solve the problem elegantly:

struct DynamicUserData(Box<dyn Any>);

impl LuaUserData for DynamicUserData {}



let newproxy_typed = lua.create_function(|lua, (data, mt): (LuaValue, LuaTable)| unsafe {

    let ud = lua.create_userdata(DynamicUserData(Box::new(data)))?;

    lua.exec_raw::<()>((&ud, &mt), |state| {

        mlua::ffi::lua_setmetatable(state, -2);

    })?;

    Ok(ud)

})?;

lua.globals().set("newproxy_typed", newproxy_typed)?;



lua.load(

    r#"

    local ud = newproxy_typed({"arbitrary", "data"}, {

        __type = "MyCustomType",

    })



    print(typeof(ud))

    "#,

)

.exec()?;

Is there anything I missed in the solution above?

Not really, actually!

Thanks for the help!