This was a super fun project building this with my friend Isaac!!! While I made the animations and set up the projection mapping, he coded all the gesture control, and in game data manipulation. From a previous project- the DMX controllable mahjong table, I had realized that there was a web API available in the lighting software I had used (ELM), which could make for some really cool lighting ideas.

At the time, we were really hooked by the game Balatro, and needed an excuse to play it more… so we thought it would be amazing to try set up a Balatro game via the projector and play it using just our hands- since the game can be played almost entirely with just moving the mouse and left click.

With a web cam above the table, moving your hand across would move the cursor in game. A pinching motion within that space cause the mouse to click! With that, we were able to make a simple "Click" animation that communicated with the mahjong table and activated a pulse animation when the user clicked.

Next, we really tried to level up the immersion. Balatro has these very satisfying animations when the score of your played hand reaches certain thresholds. However, it wasn't easy to get that data, since it was buried within the game files. However, Isaac found a way to create a mod which is able to read in game events, and have it trigger API calls to the mahjong table- activating cool light effects as the score increases.