Get in touch:
Live Performance Capture -July Update
What is it?
In early 2020 we developed a Live performance capture solution to enable live facial, body and finger tracking.
We use off-the-shelf components and the Unreal Game Engine to allow us to render 3D characters and environments in real time.
Here's some early tests of our Performance capture system:
Whats it for?
We have begun exploring new and innovative applications for our platform. We are interested in exploring:
Streaming into a VR headset to create immersive and interactive theatrical experiences
Training and Simulation technology, allowing a participant to role play and rehearse training scenarios in VR
Live Performance, using powerful visualisation and particle engines.
Developing VR experiences based around themes of Empathy and interacting with the world through different viewpoints.
Virtual Production and new film making techniques, blending real and virtual worlds together
Virtual Production: Music Video Project
We were approached to develop a solution allowing for a performers movement to be captured and combined with audio visualisation to create a visually stunning music video. We are excited to begin this virtual production in July. Here are some of our initial experiments:
Music by Jamie Pasco
Practical Workshops... Over Zoom
A guide to adapting traditional teaching methodologies to an online classroom
If you’ve arrived at this page, you’ve probably delivered a zoom teaching session yourself, or perhaps are considering making the leap to online teaching. In this blog I'll aim to break down the challenges, advantages and how to best prepare for this brave new world of online teaching.
As a freelance CS tutor i’ve managed to get a lot of experience of using zoom in both 1-2-1 sessions and teaching larger groups of pupils, topics have mostly covered practical video game design, using a number of different engines including scratch, construct 3, unreal engine and hammer editor (CSGO). As you’re all aware it’s best to focus on teaching practical skills during a session and drop in some CS concepts along the way, however these methodologies require a lot of feedback and checking in with pupils to address their level of understanding and problem solving any issues they’re having. It can feel very daunting to deliver a step-by-step tutorial online, and your impulse may be to rely on handouts and presentations however the lasting learning outcomes, levels of engagement and sense of achievement that can be experienced using this method, far outweigh taking the easy option.
Pupil “Its not working”
Teacher “What’s not working?”
Pupil “The colour screen thing”
Teacher “What is the colour screen thing?”
Pupil “Where you do the painting”
Teacher “The paint editor?”
Teacher “What’s now working about it?”
Pupil “I don’t know”
Challenge 1 - Pupils Equipment
The first thing to consider is your pupils may be using a wide range of equipment to take part in the lesson, it might be a PC, laptop, Mac or Tablet. They may only have a single screen and need to minimise the video stream to try the steps themselves or they split their screen and have a small view of the video stream and a small view of whatever software they’re using. They may have slow speed or intermittent internet and your video stream keeps freezing or runs in very low resolution. They may have a particular qwerk of their operating system, firewall, browser and they don’t have the vocabulary to communicate what's going wrong. You may ask them to watch a video or download a small file, but due to their internet it’s going to take 15 mins and they might get left behind.
Challenge 2 - Speaking into the abyss
Pupils join the classroom, mute and close their video and you begin your introduction, you have no idea if pupils are activity listening or have wandered off to get a snack. You explain a concept and ask everyone to post in chat if they understand, are they just saying ‘yes’ so you’ll move on? You ask the class to complete a task: are they struggling? Did they finish 10 minutes ago and are bored? How long do you leave it before you move on? Without the visual and audio feedback of a normal classroom, a lot of the time it can feel like you're just speaking into the abyss. Unfortunately this is just something you’ll have to get used to, the more you experience it the more ‘normal’ it will feel. You can help reduce these feeling by having written copies of the tasks or instructions displayed on screen, and really make sure your instructions are simple and clear, if you can’t explain the task in a single sentence, it’s too complicated (at least for most age groups).
Challenge 3 - Progression
Maybe you’re teaching a class you’ve worked with before and are aware of their abilities and skill set, maybe it’s a brand new group with a mixture of ages, abilities and attention levels. If a pupil gets left behind at a certain step, they may not be able to re-engage for the rest of the class, you may wish to pause the lesson and spend some time troubleshooting what the problem is, as we all know what that experience is like... Without being able to see the pupils screen for yourself, diagnosing and solving the problem can take a very long time, especially as you are waiting for the pupil to type out the answer, which can feel like it takes an eternity, and what are the rest of the class doing at this time? What if another question comes up at the same time?
How to make it work
In advance of starting your lessons, advise pupils on how to best experience the workshop, most won't have access to a second monitor, but they may have a tablet or iphone on which they can watch the stream while working on their main pc. Advise them to work as close to their router as possible, or even use a wired ethernet connection if possible.
We tried two different structures in our online teaching project, Traditional teaching structure and a video guide hybrid structure.
In our first online teaching session, we decided to not change too much from our traditional workshops. We went for the following structure:
Tutorial time (where participants watch the stream)
Try it time (where they try to work on the next step independently)
Catch up time (where you can address any question or issues that have come up)
We were only teaching a small group so felt this method would work fine and we had made enough concessions to overcome the differences in this teaching environment...we were wrong. Our first session was filled with issues, the pupils had never used google hangout or google classroom before so all arrived at different times, many had connection or computer issues, although we got through the lesson, and the majority of the class managed to complete the lesson, it had not been a pleasant experience and we hadn’t felt ‘in control'. We decided to try a different approach in the following sessions
Video guide hybrid structure
For the next session we spent some time to produce a series of video tutorials, this is everything we would normally demonstrate live during the lesson, however with this method pupils could watch the videos at their own pace, re-watch sections they missed and even go back and remind themselves of early steps without holding up the lesson. Pupils arrived in the hangout and after a brief introduction we sent them the link to the video guides and informed them we would remain online to answer any specific questions or cover anything that pupils were interested in around the theme of the lesson. Lots of pupils took advantage of being able to ask us questions about specific steps, or try something ambitious, which would not normally have had the capacity to address if we were teaching a whole group at the same time. We really had the time to support pupils who needed help while the majority of the class were working through the online tutorials. All the pupils managed to complete all the steps by the end of the session, and fed back that they had preferred this method to the one we used yesterday. It does require a lot more prep work in producing a series of video guides, however this can be reused in future workshops
Link to our video guides for construct 3: https://www.limbotech.co.uk/construct
Note on 1-2-1 teaching
When you only have one pupil to worry about, a lot of the challenges above don’t affect the lesson as much. It is really important in this case that you can see the pupils screen and they have a way of seeing your screen. So you can always see how they are progressing and you can ask them to watch your stream if you need to demonstrate a complex step.
Invest in a high quality USB microphone, (£20 - £60) don’t use a headset microphone as the class will have to listen to you breathing for the whole lesson, test your microphone before the lesson, use audacity to record your voice to find the ideal distance to be from it. If you hear unpleasant popping sounds on letters like B, P, D you’re too close to the microphone. While you're teaching, be aware of turning your head away from the microphone while speaking as this makes it very hard to be heard. Ask pupils when the lesson begins if they can all hear you clearly.
Monitors and computers
Ideally you want a second monitor, you will demonstrate and screen share on your primary screen and use your second monitor to keep an eye on the chat window, and to check if anyone disconnects. Move your computer as close to the router as you can and if possible use an Ethernet connection. Intermittent internet may cause your voice to distort, your livestream to freeze, or the resolution will drop too low, meaning pupils cant read text on the screen or follow your mouse movements.
I can recommend OBS https://obsproject.com/ as a great tool for streaming, it’s free, simple and it lets you add a little webcam window into a screen share so pupils can see you and your screen all in one window, this saves having to constantly switch between the two and lets you use facial expressions and arm gestures while teaching. While this setup is great, it does get a bit difficult managing all your monitors, as one screen is dedicated to OBS. at this point it feels like you may even need a third monitor.
For recording tutorial videos for the class to follow at their own pace, I use Free Cam https://www.freescreenrecording.com/ another free piece of software which lets you record screen capture and microphone input at the same time and quickly export it into a video.
Be patient and be empathetic, you may get very frustrated, you may feel the pupil is trying to wind you up by asking simple questions and asking you to repeat a step for the fourth time. But it’s really important to think about their perspective, trying to learn a new skill by watching a low resolution video stream which freezes occasionally. Things that seem really obvious and clear to you, may not always be for them.
Keep progressing the workshop, if someones got really stuck, or their computer has crashed and they've lost all their work, it’s tempting to spend the time making sure they’ve caught up, but you will be neglecting the rest of the class. If you can, provide the pupil with a clear step by step guide or even a working template of the project so they can continue participating. Make it clear that they’ll be a recording of the session released after the lesson, so its not the end of the world.
Make it fun, most of your pupils will have been stuck at home the last few weeks, all their learning is coming from computer screens and they haven’t been social with friends for a long time. Fill your presentation with humour, be energetic and enthusiastic (without being patronizing!) even if things go wrong, keep an upbeat attitude and don’t stress out..
In advance of starting your lessons, advise pupils on how to best experience the workshop, most won't have access to a second monitor, but they may have a tablet or iphone on which they can watch the stream while working on their main pc. Advise them to work as close to their router as possible, or even use a wired ethernet connection if possible
Provide a high quality recording of the instructions after the session, in case someone has disconnected and missed out
April update - Virtual Theatre
Body and Face Tracking
We manage to achieve body and face tracking without any lag. We're using the Vive Mocap Kit by Yuri N Kalinin to capture body movements and map them to our skeleton (based on the UE4 mannequin).
The 3D character is from Reallusion. There is a plugin to transmit facial data from their software to unreal, however there was a slight delay in the data coming through. We decided to adapt the FaceAR Sample project from unreal to map to the Reallusion model. This gave us good results without any lag. In the picture to the right you can see how we remapped the Iphone blend shapes to drive the curves on the Reallusion model.
We still need to do a bit of fine tuning to make the character more expressive.