As much as I would love to pick up a couple of development kits for the sake of tinkering, I don’t think Nintendo would allow it, and I don’t have many thousands of dollars resting by the fireplace. As such, I’m going to have to work on a lot of assumptions.
Here’s a proposal I made in a previous blog post; pair the WiiU with it’s now popular handheld, the 3DS, and/or Drop the Game Pad for a Pro Controller in a new SKU. Below is a series of baseless mathematical formulas to argue my proof of feasibility.
Most of what I am about to say also has already been said by myself on Twitter but there’s no harm to documenting my rambles in a place less ephemeral like a blog. As one of my many assumptions involves the 3DS not being able to communicate directly to the WiiU via the same Game Pad protocol, what I am proposing here is simply a streaming design for WiiU and 3DS using a home network. I would like to add however that some recent events give me faith that Nintendo could manage to bypass home network requirements by having the 3DS emulate the Game Pad. My only support in that case would be that since the 3DS would be doing little more than decoding an A/V stream that it would have plenty of processing power to handle the additional task of transcribing the packets. The only issue of course is that decoding some proprietary variant of H.264, or whatever codec WiiU uses, may be too heavy for the 3DS to handle without hardware acceleration.
Another assumption is that, since 3DS videos currently are suspect to be M-JPEG, it would be easy enough to transmit JPEG format video frames. It is not the most efficient format, certainly doesn’t compare to motion sensitive formats but JPEG is fairly common and decoding is less process intensive than .264.
Your typically 802.11g WiFi can offer up to 54 Mbps but the increasingly popular 802.11n can offer up to 300 Mbps theoretical bandwidth. The practical and more real application is rarely more than 50% of this, which places us at a comfortable 21 Mbps for maximum throughput, reserving some for packet overhead and input communication.
Assuming that 60 frames per second is the gravy we are looking for, let’s see where that puts us.
(21 Mbps / 8 bits) / 60 fps = roughly 44 KB per frame
A bit rate of 44 KB per frame doesn’t give us much to work with for real-time encoding and decoding. And the low common denominator of 802.11g is not helping. This math also assumes that LAN traffic is relatively low, and audio is not yet being transmitted; image only.
After some fiddling of various video game screenshots I settled on a JPEG quality of 50 (Q50) according to Photoshop. I assume that this is a linear mapping of jpeglib’s settings of 1-99, but who know what magic lies inside of Adobe code. The important thing was that tended to result in images that were roughly between 32 KB and 48 KB; the variance is related to something I’ll get into later.
The 3DS has a resolution of 400×240 for the top screen. This resolution is doubled when rendering in 3D, giving you one image per eye. Before we start cutting back, let’s go for the gold; 3D video streaming. A nice thing about the 800×240 image is that similar pixel patterns are likely to occur since both images are relatively from the view point. These similarities can help during the quantization process.
800×240 @ Q50 JPEG = ~48kB = 22.5 Mbps !!
Though 22.5 Mbps is over-budget, I’m also using the most naive video stream imaginable. What options do we have?
- Lower the resolution to less than native, and upscale.
- Lower the frame rate to 30 =), instant 50% savings!
- Lower the JPEG quality as needed.
- Stream to the smaller touch screen (2D, 320×240)
- Render higher quality images ?!?
Before I get to option #5, let me just say that rendering to 320×240 can be done for either the bottom or top screen (see option #1). Those few pixels could turn a 22 Mbps stream into a 15-18 Mbps stream, which is under-budget. Adding to that, cutting the frame rate to 30 could put you into single digits. Adding to that, reducing the JPEG quality is a last resort but an effective one. Note that input commands can likely still transmit at 60 fps and be processed on the WiiU at 60 fps, but only encode every other frame.
Okay, option #5.
As mentioned before, certain types of games seemed to perform better than others. I found that games that were generally noisy resulted in not only larger JPEG files but worse quality. Noisy or highly aliased images required that I turn the dial up substantially to maintain image quality, resulting in images that were as large as 64 KB in size (~30 Mbps). If the image was strongly anti-aliased or even slightly blurred, the image quality and compression rates gradually improved. These findings are likely a result of JPEGs cosine transforms which like their data to be generally smooth and consistent.
This bit of information might open the door to opportunities to apply a blur filter onto an image, encode, transmit, decode, un-blur/sharpen, to achieve acceptable quality at a better data rate. Regardless, enabling ant-aliasing is highly suggested, which shouldn’t be an issue for the WiiU at 240p. A simple sharpen filter shader on the 3DS might help to improve the perceived image quality as well.
240p is not HD. This definitely something to remember. Though some screenshots may look acceptable at this resolution, the following image is what your Dark Souls inventory might look like on a 3DS. Keep in mind that no compression tests were done and this image was saved as PNG to conserve as much detail as possible.
As a final thought, I’m kind of hoping that Nintendo doesn’t throw in the towel for the WiiU. What I am proposing is merely an option to lowering the price tag by giving users (who may already own a 3DS) and cheap buy into this home console. It would also be a strong argument for buyers who don’t own either hardware to go out and buy both, since the 3DS already has a strong library to support itself.