This year at SXSW in Austin, we showcased our AI Photobooth.

Our AI Photobooth transforms people into imaginative styles that dynamically change across the event. It uses a Stable Diffusion backend with LCMs, ControlNets and IPAdapters to capture and transform attendees into exciting multiverses beyond our own.

In this T&DA exploration, we have harnessed the power of Stable Audio AI to turn text prompts into 3 unique music tracks that drive audio-reactive AI visuals.

We isolated stems from Stable Audio using Moises.ai’s Audio Seperation capability, and then leveraged Spotify’s Audio Intelligence Lab MIDI conversion tool to transform these stems into MIDI tracks. This gave us complete creative control to compose three original soundscapes inspired by the ocean, viruses, and neon space worlds.

We designed 3D audio visualisers in our custom Houdini AI pipeline that generates an audio-reactive depth pass and uses AnimDiff and ControlNet for the AI reskinning.

Listen in!

In this LABs experiment we have used the recent Stable Video Diffusion to bring album covers to life. Stable Video Diffusion is a foundational model for genAI videos that can be generated from text prompts or images.

In this LABs experiment we have explored the temporal capabilities of AnimDiff in a real world environment.

We used ControlNet to approximate a depth map of the input footage, which we then used to guide AnimDiff alongside various prompts. In comparison to earlier experiments involving img2img: rather than each frame being generated from scratch, AnimDiff enables each frame here to have a context of its previous frame within 16 buffer chunks.

Here is our latest LABs exploration into generating AI videos with AnimDiff and ControlNet.

AnimDiff has the unique ability to retain temporal cohesion between frames through motion LoRAs. When combined with the ControlNet QRCode Monster model, we are able to direct the optical flow of motion to follow the path of our logo and even make it a seamless loop.

We present our latest experimental installation showcased at Limehouse Creative Circle, a real-time AI powered mirror that turns partygoers into a series of styles like Pixar, Watercolour, Studio Ghibli, Barbie, Tom Cruise and more!

Read more about it on The Stable

Pocket Dimensions is an MR/VR experience where players are able to explore various unique dimensions that can adapt to any space. Pocket Dimensions started out as a way create a caving system, using Procedural Mesh Generation, based on user generated 3 dimension splines.  Players draw rifts through their room. They then open up these rifts and step into a procedurally generated cave system that is guided by their defined pathway. Within this Pocket Dimension, they are able to collect energy by mining crystals.  We intend to add a feature that once the experience is completed, users contribute mining crystals into a communal pot, like crowdsourcing, or hive honey pot, and the community can build towards something together.  Currently, a user can enter dimensions they have explored into a virtual passport system, and based on energy they have collected, use this energy to explore further into the worlds that live within all dark matter.

Experience has been developed to be HMD agnostic. Try the experience now for Meta Quest Pro:

https://drive.google.com/file/d/16XOtGtsDryu5ksNRF0dhMBnoPSZFgj5_/view?usp=share_link

Here is our latest AI exploration with Meta’s Meta Quest Pro where we managed to get GravitySketch running live through Stable Diffusion img2img. As you draw, your sketch will turn into the prompt you specify (with a 3-4 second delay). The new details that Stable Diffusion generates in the output will then shape and inspire the way you continue to develop your creation.

This experience explores a whole new paradigm of human-computer interfacing by reinforcing our symbiotic relationship with neural models. It was made possible with Python and our technical artist’s modded version of InvokeAI, “DiffusionCraft AI” which supports realtime prompt hotswapping.

‍

#ar #wearables #metaquest #oculus #arvr #meta #experience

A combination of Wordle, Hangman and Midjourney, Daydreams is the first daily game to use AI synthesised images. In essence, it’s a web-based game with AI image-based clues.

Every new day presents a new ‘daydream’ to bring into reality. All players solve the exact same daydreams each day, making it the perfect way to start your day, readying you for the day’s watercooler conversation. An addictive daily game, Daydreams is suitable for anyone and everyone to play.

Daydreams leverages Midjourney – a powerful CLIP guided diffusion model and service where people enter prompts into the Midjourney Discord bot and images are generated such as ‘disco astronaut painted by Salvador Dali’ or ‘artistic glass shark sculpture with studio lighting, 3D render’. And it really is only limited by people’s imagination.

Each daydream is a Midjourney prompt that players use to guess from the Midjourney generated images – some days are easy, others harder.

Play it here: https://daydreams.ai/

We Are The War | CLIP-Guided Storytelling & Speech Synthesis

At T&DA we have explored leveraging the latest technologies to create ‘We Are The War’, an animated graphic novella poem, where both the images and narration were generated through visual and audio synthesis.

Created using prompts inspired by children’s book illustrations and the lines of our poem, images were generated using Midjourney. The narrator was generated through speech synthesis software that only needed 15 minute sample of talking. A bespoke filter was designed and used for motion capture and applying facial animation with Ebsynth.

To add parallax and depth to the scenes we used neural Z depth extraction.

Using these tools we tell the story of children’s metaversal shenanigans as realities of recent years bleed into their locked down lives. We can think of this technology almost as if it’s a 10 year old expressing itself, which felt like the perfect means to convey the sentiments of the poem.

#midjourney #clip #ai #descript #synthesis

Introducing a new original idea, a first person game that can be played entirely inside Discord!

Created by T&DA’s Developer Lachlan Christophers, users can explore an underground maze privately, or collaborate together with friends.

Inspired by early text adventure games, Discord Dungeons uses embedded messages in a normal discord chat channel, and implements buttons for navigational controls, but only by completing the maze can users discover what awaits them.

Head over to the T&DA Discord server to try it out for yourself!

*once in the server, type either /maze public OR /maze private, in one of the available chats to start a brand new maze game.

Elephantopia is the first step into the web3 and NFT world for T&DA. We’re making a game world with the idea of eventually having dozens of different game modes and activities all within the same world.

An initial set of Trunket profile image NFT’s help establish the concept, and garner interest. These NFT’s will become the access key to the 3D game world we’re working on, currently we’ve started on a racing segment but over time there will be many different things to try as more gets built…

Soul Trader | Virtual Trading Card AR Filter

For the latest #FacebookHackathon, we have designed a filter that explores a future in which our soul can be captured, shared, and commodified into blockchain trading cards. This is a future where our identity can be exchanged for virtual coins with just the swipe of a finger.

T&DA presents Soul Trader.

Try it on Instagram or Facebook

#arfilters #sparkar #facebook #hackathon

Let’s get this ball rolling!

Rollerputt is a collaborative live gameplay event hosted on the #Facebook platform and designed in #UnrealEngine. This idea has potential for very high engagement through social channels.

Players roll a virtual marble through a hand-crafted obstacle course, collecting coins and knocking down walls to reach the final flag, but here’s the catch – they all control the same ball!

Think text adventures meets marble run, players influence the direction of the ball by submitting comments into the Live Chat (such as ‘charge forward!’ or ‘please mr marble, roll right’).

Every 10-30 seconds the commands are tallied, averaged, and normalised, and the marble is fired in that chosen direction.

By utilising Epic’s Unreal Engine we provided high quality visuals and realistic physics for this live experience.

We hope to deliver more #Unreal live games that place the power in the players. We see potential to further develop ideas around this tech, such as pitting fans of competing teams, or groups with polarising opinions, and have them come together to achieve a common goal, or defeat each other and vanquish the enemy!

#gamedev #multiplayer #rollerputt #experience #realtime

Tetris-U | Full Body Tracking with Machine Learning

By harnessing the power of machine learning to track one’s entire body, T&DA has created an experimental Tetris clone.. but with a twist!

To quote Roger Walters: “all in all you’re just another brick in the wall”.. Introducing Tetris-U! Where the player becomes the Tetris bricks by using a webcam.

The player captures real life poses and converts them into stacking bricks. These poses can include twisting and contorting ones body like a dancer, or as if possessed by unnatural forces, or just creating shapes like a budding mathematician in a desperate search to find the next perfect shape.

You can play this while practicing yoga, dancing or any kind of workout!

Watch the demo video to see some Tetris-U shenanigans.

#machinelearning #tetris #ai #unity3d #mlagents #poseestimation #gameification #gamedev #rnd

EnviroCube | Manipulating Virtual Environments with Physical Objects

‍

You’ve heard of Darkseid’s Motherboxes or Thanos’s Infinity Stones…well T&DA want some power of it’s own so have conjured the EnviroCube!

We have successfully digitised a lolly jar by attaching a #quest controller and recreating it in a virtual environment.

When our virtual camera targets a certain face of the EnviroCube the world transforms into that new environment, be it desert, forest, snowy mountains any many more.

Simply put, each face of the cube can trigger various events to occur.

From art gallery installations to futuristic remotes, to abstract puzzles, this tech has boundless potential waiting to be discovered.

It’s wizardry & these cosmic powers are within your reach by harnessing the power of the EnviroCube!

“The Upside Down” – Immersive AR using LiDAR

Harnessing the real-time capability of Unreal Engine, T&DA has rewritten the rules of reality to manipulate the form, interactivity, and atmosphere of any physical space. We have used the Oculus Quest as a virtual camera in Unreal Engine to explore and interact with a LiDAR scan that is mapped onto the real world.

Understanding the measurements of any physical space has huge potential as it allows us to transform your house or office into dreamscapes, theme parks, space stations, jungles and so much more

Here at T&DA we have been experimenting with our #manus gloves as a focal element for environmental storytelling. In this video we show how this real-time tech can be adapted to an underwater scene where we control a swarm of fish at the tip of our fingers.

Swim fishies swim!

Ever since the words “help me Obi Wan…” were uttered, tech geeks everywhere have wondered how can we leap from our Jetson’s Zoom Calls to communicating via Holograms and 3D real-time representations of ourselves.

In the T&DA offices we have reached closer to the future.
We call this technology Shrink-Ray after our very own Creative Director Raymond. Here you can see that big-Ray is puppeteering a mini-Raymoji ‘Shrink Ray’ in real time and able to communicate as a 3d character via an iPhone camera’s AR.

We have done this by successfully uploading real-time mocap data to the cloud. This means that anyone anywhere around the world can access this data in AR and have a 3D person to interact with live.

Huge potential for performance shows/live events, and the future of communications…