Projects/

Mixperiment64

Mixperiment64 was a COVID-19 project I wrote over the course of a few weeks to allow folks to play N64 with each other over Microsoft’s Mixer livestreaming service. By browsing to Mixperiments.com, users are able to join a live N64 game and take control of an N64 controller either using keyboard controls or an attached Gamepad.

Motivation

Mixer’s main differentiator in the livestreaming space is that it can deliver streams with ultra-low latency, often below 1 second. This opens up some unique interaction possibilities since viewers can see the impact of their input very quickly.

I’d been looking at crafting some sort of viewer-controlled live streaming experience for a good while, but given the timing of COVID-19 in early 2020 and the need for virtual activities to enjoy with others, the idea of simply sharing N64 emulator input became attractive.

Building It

The Nvidia Jetson board proved to be a capable livestreaming device based on my experience using it for ChickenCam, so I decided to use it as the platform for this project as well. It’s very similar to a Raspberry Pi in that it’s a compact ARM64 Linux Platform, however this unit has a pretty capable GPU built-in for tasks like AI and computer vision, so it lends itself particularly well for running 3D games and transcoding video.

Programmatic Mixer FTL Streaming

The first goal was to get the game actually streaming to Mixer. I could have used a traditional streaming solution like Open Broadcaster Software, however it has pretty limited ARM64 support, and it’s not really built for this sort of scenario - there’s a lot of overhead for the GUI, Scene composition, streamer controls, etc. It’s meant to be interacted with by a person managing the live stream.

Instead, I found this great talk by Francisco Javier Velazquez-Garcia who had worked to develop a plugin for GStreamer to broadcast to Mixer via the low-latency FTL protocol. I was able to pull his patch and build the library locally, enabling me to easily use the GStreamer API to manage a live video stream to Mixer programmatically.

Low-Latency Input from Browser

From there it was just a question of getting input back from the viewers. Getting game-quality low-latency input over the network from a browser is a challenge myself and many others have encountered before - the APIs exposed to you via Javascript largely rely on TCP for transmitting data, which means additional latency for packet ordering, replaying of lost packets, etc.

Recently, however, WebRTC has become a supported standard for things like peer-to-peer video conferencing via the browser. WebRTC is built with the peer-to-peer audio/video conferencing scenario in mind, but there does exist a way to send raw data over a WebRTC transport - it’s just notoriously difficult. As luck would have it, someone has done the hard work of providing a library to harness this transport to send raw data: WebUDP.

Using WebUDP, I created a little Node web service to manage a queue of players, assign players to contollers, and receive input from players using keyboard bindings or Gamepad API bindings.

Injecting Input to Emulator

To get the player input back into the actual N64 emulation I wrote my own Input Plugin for Mupen64Plus. It does little more than listen on loopback interfaces for incoming controller packets.

Sources