Binding Flecs to Lua with Luajit FFI

I've been waiting on multi-hour FPGA runs a lot lately for my thesis, so I picked up a side project to kill the time. I've been building a renderer/game engine/game in C++ using Vulkan. The engine is called "ren", cause it renders and I couldn't come up with anything else at the time. So far, it can load assets from disk and render them with a typical PBR pipeline. This is what it looks like with a poorly lit Sponza scene:

You'll notice it still looks kinda ugly, but that's cause I still don't have things like shadows or any form of GI (of course). I'll get to all that later, but I've been distracted by other stuff. The engine is honestly a little over-engineered for how early on it is. The whole thing is built around the ECS system, flecs, and most game logic (if you can call it that right now) is through systems that operate on the entities.

At its core, Flecs is just like any other ECS (I've tried entt, but I am a big fan of flecs' model of "everything is an entity"). It offers the ability to create 'entities', which are just integers, and you can attach 'components' which are basically just structs. An entity can have any number of components, and the composition of many components is what enables complexity and flexibility. For example, an entity which can move might have the following position and velocity components.

struct position {
    float x, y, z;
};

struct velocity {
    float x, y, z;
};

Then, on every frame, you can (abstractly) update them with a system by querying for all the entities that have a position and velocity component attached.

auto system = make_system<position, velocity>([](position &p, velocity &v) {
    p.x += v.x;
    p.y += v.y;
    p.z += v.z;
});

The power of this is that if you just describe all your game logic according to this system interface, much of the struggles of typical OO programming goes away. So far, all this stuff is done using C++ code in my engine, and frankly I don't want to be writing game logic in C++.

I don't want to write games in C++

Many game engines can have their game logic programmed in a higher level language. Unity lets you program in C# and Unreal lets you use that horrible blueprint system. For my engine, I'm taking the classic approach, and embedding lua as my scripting language. I'm actually using LuaJIT, as the FFI system it provides is pure magic, as you'll soon see.

If I want to program my games in lua in this engine, I'll need to provide some kind of interface to define components and implement systems that act on them. There seems to have been some previous work to get LuaJIT to talk to flecs, but it looks like the project is mostly abandoned -- and I want to do it myself anyways!

So to start, I've been building out an interface to allow lua to create entities:

local ren = require 'ren'

local e = ren.spawn('my_entity')

This is done using luajit's magical FFI:

local ffi = require 'ffi'

-- Just write some C code...
ffi.cdef [[
    typedef struct ecs_world ecs_world_t;
    typedef uint64_t ecs_entity_t;
    ecs_world_t *__ren_get_world(); // defined by my engine, not flecs.
    
    ecs_entity_t ecs_new(ecs_world_t *);
    ecs_entity_t ecs_set_name(struct ecs_world_t *, ecs_entity_t, const char *);
]]

-- Then call it

function ren.spawn(name)
    -- Ask the engine for the ecs world (the thing that manages entities and components)
    local world = ffi.C.__ren_get_world()
    -- Spawn an entity in that world
    local e = ffi.C.ecs_new(world)
    -- If you asked for a name, set it
    if name ~= nil then
        ffi.C.ecs_set_name(world, e, name)
    end
    
    return e
end

You'll notice I didn't write any lua bindings or anything, I'm just calling functions by their name in C code. This is magical. I originally was playing around with binding s7 scheme as a scripting language, but ran up against annoyances (and performance problems) related to binding to the engine. This is clearly much simpler and obviously faster, as luajit compiles this code down into object code which directly invokes these FFI functions as if it was written in C.

I can also just pass values back and forth between lua and C, which is magic. That ecs_entity_t is a 64 bit integer, and I can just use that as any normal lua value.

So, I can spawn entities. The next step is to somehow define components. This is what I came up with. You define a component just like you would a C structure

local position = ren.struct('position', [[
    float x, y z;
]])

Then you can use it to attach/update/get components from an entity.

-- spawn an entity
local e = ren.spawn()

-- set the position component on e.
position.set(e, { x = 1, y = 2, z = 3})

-- Grab a mutable reference to the position component and update it
local p = position.get_mut(e)
p.x = 3

This kinda seems like magic, and I think it actually is magic, but let's delve into the insides of that struct method. It takes a name and the C fields formatted as a string, and some other stuff we'll ignore for now that lets me have things like operator overloading on vec3 types and whatnot.

function ren.struct(name, fields, user_methods, user_metatype)

The first thing it does is clean up the field string. Mainly, we remove C/C++ style comments.

-- remove c-style comments from fields
    fields = fields:gsub("/%*.-%*/", "")
    -- remove c++-style comments from fields
    fields = fields:gsub("//.-\n", "\n")

Then, we use that cleaned up string to define the FFI type by generating some C code.

local cdef = string.format([[
        typedef struct %s {
            %s
        } %s;
    ]], name, fields, name)
    ffi.cdef(cdef)

From there, we can technically do ffi.new(name, ...) to allocate an instance of that struct, but its not a component yet. We then call a function I've thrown together in my engine called __ren_register_component which takes the name of the component, the size, the alignment, and a formatted string of the fields.

local component_id = ffi.C.__ren_register_component("ren::comp::" .. name,
        ffi.sizeof(name),
        ffi.alignof(name),
        "{ " .. fields .. " }")

Internally, that function calls out to flecs using the meta plugin which parses that field string into type information that flecs can use internally. That function returns the id for that component. That id can be used to add/remove/get the component from a struct.

Then, I setup the rest of the interface that is returned from this struct method

--- ...

    -- We return this structure, which has a bunch of useful information for later.
    local info = {
        -- The name of the struct
        name = name,
        -- The C definition of the struct as the user typed.
        fields = fields,
        -- The ID of the component in the ECS world.
        cid = component_id,
        -- The size of the struct in bytes.
        size = ffi.sizeof(name)
    }

    -- We do some lua magic with metatables so I can call the return value to construct
    local thestruct = setmetatable(info, {
        -- call is the constructor
        __call = function(_, ...)
            return ffi.new(name, ...)
        end,
        __index = {
            -- Set the component's value on an entity, adding it if needed.
            set = function(entity, ...)
                return ffi.C.ecs_set_id(ren.world(), entity, info.cid, info.size, info(...))
            end,

            -- Get a mutable reference to the component on an entity
            get_mut = function(entity)
                local ptr = ffi.C.ecs_get_id(ren.world(), entity, info.cid)
                if ptr == nil then
                    return nil
                end
                return ffi.cast(info.name .. "*", ptr)
            end,

            -- Get a copy of the component's current state on an entity
            get = function(entity)
                local comp = info.get_mut(entity)
                -- return a copy of the component
                if comp == nil then
                    return nil
                end
                -- Make a new instance of the struct
                local copy = ffi.new(name)
                -- Copy the data
                ffi.copy(copy, comp, info.size)
                -- ... and return the copy
                return copy
            end
        }
    })

    -- The names of structs in the FFI are in a global C namespace. As a result,
    -- we will put the struct in a global namespace as well to mirror that (If
    -- something is a vec3, just use vec3()).  This might change later, though.
    _G[name] = thestruct

    return thestruct
end

The full source of this function can be found here. The interesting stuff there is the set and get_mut/get methods in the metatable. It is what enables the magic we saw before to attach the component to an entity.

The nice part is that this is all just through the luajit ffi interface, and I had to write only one binding function in C++ to simplify things (__ren_register_component). The rest is just lua.

So yeah, this is still early on, and I should really focus on my thesis, but the next steps are to get queries and systems working, so I can start building game logic :).

I'll follow up with another post once I figure that out, but in the meantime thanks for reading!