GGVJ: An Interactive VJing Experience

•

Playlist curated for the interactive experience

Final Project Showcase @NYU ITP Camp 2023

This project was ideated and created over the course of 4 weeks, and showcased at the NYU ITP Camp of summer 2023 in New York City. This project makes use of all the new technologies I picked up on during my time at camp, including Blender, TouchDesigner and MadMapper. I tied it all together with my love for music and the eagerness to create more interactive experiences.

Inspiration

I was inspired by the undulating motions of the sound wave, and I wanted to find ways to visualize this in 3D space. The first prototype created was similar to a topographical landscape. It had an interesting shape to it and I could visualize a correlation to the sound wave, from the low notes being represented by the valleys of the 3D model, and the high notes, the peaks. I started thinking of different directions I could illustrate this correlation more deeply. I added distinct layers of elevation so that I could manipulate each layer in accordance to the sound wave.

3D Modelling

The 3D model was created using geometry nodes in Blender. I used noise to algorithmically produce the different shapes seen in the following prototypes. This process allowed for a quick and convenient workflow, as vastly different shapes could be created in a matter of minutes by only tweaking a few parameters. As this was my first time using Blender, I was unaware of how heavy the model was until I imported it into TouchDesigner to create a point cloud and I saw that the model was made up of over 70,000 points.

As time was a huge constraint on this project, I knew I had to simplify the shape by a lot. I started off by using a 100x100 grid, so for the second prototype I brought it down to a 50x50 grid. This lowered the point cloud to 30,000 points, and proved to be a lot more performant in TouchDesigner, which would be used to create the textures in the next step. I wanted to make it more interesting by applying a mirror effect on both the x and y axis to create symmetry.

By simplifying the shape, the result was a "pixelated" effect which I ended up really liking, so I decided to go for a more "retro" look and feel. For the final prototype, I stuck with the 50x50 grid, and tweaked the shape so it would have more peaks and valleys for a more compelling appearance.

3D Printing

The next step was to 3D print the prototypes. It was my first time using a 3D printer (Ultimaker Cura S3 and S5) and the process was very enjoyable. The first prototype took 9 hours to print, but only came out to be 4x4" in size. The effect of the undulating curves was beautiful, but it was hard to see the grid effect I was trying to accomplish.

The second prototype was much closer to what I had envisioned, as now I could see where each box of the 50x50 grid sat. However it took a long time to print, a total of 1 day and 10 hours for a 7x7" model. As previously mentioned, I was limited on time to experiment, so I had to make a quick decision on whether I should try for another print.

In the end I wanted to create a more captivating profile for the 3D model and also needed it to be as big as I could get it, because I had decided at this point I would use projection mapping on the model to produce some visual effects. I used the biggest 3D printer we had at the camp shop, and was able to get a print of a 9x9" 3D model. It took over 2 days to print, and used over 100m of material. However, I think the wait was definitely worth it.

3D printed model prototypes

Textures

I had to create 8 different textures for the 8 different layers of elevation. The following animated textures were created using TouchDesigner. I started off with making a texture from a 50x50 grid. Then using the noise texture operator and animating it on the z-axis, I was able to create some animated, pixelated textures quite quickly. I wanted to create a palette of very contrasting colours so when projected next to one and on top of another, it would be easy to see the variance of colours.

Projection Mapping

Using MadMapper, I mapped out each layer on the 3D model and overlayed a different texture for each elevation. To really highlight the grid look, I applied a 50x50 grid texture on top of the projection. This made the layers pop even more and really accentuate the "pixel effect". It gave the illusion that the 3D model was comprised of many cubes, and each cube was being lit up individually.

Final 3D model prototype mapped with 8 different layers of textures (static front view)

Interactivity

The last step was the most vital piece of this project. Originally, my plan was to make the projections audio reactive, analyzing the treble and bass notes of any given song, and triggering the opacity and visibility of each layer accordingly. However I felt that this approach was a bit stale. I wanted to get people excited about what they saw, and I thought the best way to do that was to add interactivity to the project.

In spirit of the "retro" theme, I added a gaming controller as an input to manipulate the visibilities of each layer. I connected the SteelSeries wireless controller to my computer and read the OSC values through TouchDesigner. Using MadMapper, I then mapped each of these OSC values to a button on the controller. This way, each layer's visibility was controlled by a button, and multiple layers could be projected together at the same time.

I made a playlist and started experimenting with how the whole projection looked when I started pressing the buttons and trying to sync each button tap to a beat - and that was how the name "GGVJ" came to be. "GG" is a term in gaming which stands for "good game", and literally means what it says. In a way, this was my attempt at gamifying VJing.