The tech behind, The Live Coders Conference April 2020
Andy Morrell (LuckyNoS7evin)
Posted on April 20, 2020
What can I say about the first Live Coders Conference? I think I have to mention a little back story as well, how I got involved. The end result was a 17hour live stream on Twitch.tv with over 1 million channel views, a peak of more than 15,000 viewers, an average of over 3,000 viewers over that 17 hours and 3.2 million minutes watched (yes 6 years watch time).
At the start of March 2020 a Live Coder team member by the name of Ted Young a.k.a JitterTed wrote a simple line in the Live Coders Discord server. "Should we do a Live Coders Conference?". Things escalated quickly, within the first week we had a spreadsheet full talks, around 38 from 14 speakers.
I volunteered to take a look at the tech, we wanted to broadcast from a single Twitch channel with no breaks, i.e. not turning the stream off between speakers and treating it as a single program or two depending on the number of speakers we had.
Two weeks on the 16th of March came along. It was time to pick our speakers and organize what we were going to give. At this point, I had somehow become a lead organizer of the conference (luckily not for too long). I have no idea how or why but I rolled with it, as it would be a fantastic experience.
We chose our speakers, decided on a single day conference of 14 speakers, on the 9th April split between 2 individual 7hour streams. So we are sorted, right? No, we had theming, advertising, speakers & MCs to set up the tech with, find a way of producing the event, moderation and much more in not a large amount of time with everyone on the team having day jobs and/or little time to commit to the organization of the event.
The Tech
I'm here today not to talk about the organization and lessons learned from the whole event. I'm here to take you through the full tech setup we used and some of the failings myself and the rest of the team would like to look into for future conferences.
Azure
We used Microsoft Azure to host all external services that the speakers or MCs did not need to control.
RTMP & Broadcast Software
With the aim of having speakers and MCs all routed through to a single Twitch channel without disconnecting between each speaker or presenter, we chose to have an RTMP server act as a middleman between the broadcasters and the production broadcast software.
RTMP? What?
Real-Time Messaging Protocol was originally used by Macromedia for audio streaming, however now this includes a lot more. It is almost "the standard way" of broadcasting video. It is also available in most standard broadcast software like OBS, Wirecast, VMix etc.
How to set up an RTMP server
Create the VM
- Select Ubuntu VM
- Basics: something like as follows, it is up to you if you use private key or password for SSH as long as you're able to SSH into the VM after install
- Disks: No changes needed
- Networking: As this is the first of 2 VMs in this group you are fine with creating a new network, keep a note of the network name so we can use it later when adding the 2nd VM
- Management, Advanced and Tags: No changes needed
- Review and create the VM
Install RTMP
- SSH into your new Ubuntu VM
- Run the following:
sudo apt update
sudo apt upgrade
sudo apt install nginx
sudo apt install libnginx-mod-rtmp
- Yes, two separate install steps, I like to know one is installed before the other.
sudo nano /usr/local/nginx/conf/nginx.conf
- add in the following snippet to the end of the nginx.conf we just opened.
rtmp {
server {
listen 1935;
chunk_size 8192;
application live {
live on;
record off;
}
}
}
sudo systemctl restart nginx
- We are done with this server...
The Streaming VM
Software
- OBS (Open Broadcaster Software) - The streaming/broadcast software used.
- OBS Websockets - a Websocket addon for controlling and listening to OBS remotely
- VLC - This was installed prior to the broadcast but wasn't going to be used, it eventually became invaluable.
Creating the Streaming VM
This is done in the same way as creating the RTMP VM however there are several changes to be made while doing this.
- The VM type - We need to go for Windows this time and in this case, we go for Windows 10
- The VM size - The important factor here is CPU, initially go for something that is 16 Core at least. On the day of your conference bump that up to be on the safe side. I went up to a 32 core system on the day of the conference
- When you get to the network tab make sure you use the same VNet you set up when creating the RTMP VM. Otherwise, data transfer between the two boxes is "gonna cost ya!"
Installing the software
OBS - https://obsproject.com/
Download and install the 64 bit version
VLC - https://www.videolan.org/vlc/download-windows.en-GB.html
Again, download and install the 64 bit version
OBS Websockets - https://github.com/Palakis/obs-websocket/releases
That's it software install complete. On install of VLC, you get an added extra in OBS which is VLC Media Source. This can be used to listen to RTMP streams, I was looking into this however as I said before wasn't going to use it. I eventually did during the 17+ hour stream.
Using the broadcast VM as a producer
The broadcast VM and RTMP server were ready to roll, ready to go. So how did I set up OBS and VLC on the server so we could broadcast to the viewers?
OBS
The following scenes where created:
- Starting
- Ending
- Break
- MC
- LuckyNoS7evin
- A scene for each speaker split into 2 groups, in the order they were speaking, and this referenced the 2 scenes mentioned below.
- 2 scenes which were used as the RTMP stream scenes one for stream key A and one for B
The images here is the scene setup and how the Media Source
component was set up for the RTMP streams.
As you can see we set the input to rtmp://{Internal IP}/live/{stream key}. The important thing to NOTE here is: Use the internal IP address of the rtmp server in your VM. So the traffic between RTMP and Broadcast server is not counted as external data transfer out of Azure
My Job as producer
- Scene switch between MC and Speaker
- Make sure stuff didn't break and if it did try and fix and fix fast
- Monitor audio
- Featured.chat - show questions when speakers were ready
- Get speakers and MCs connected with minimal effort
Speaker & MC setup
As a speaker or MC of The Live Coders Conference, you were given a stream key by myself. This is to uniquely identify you in the RTMP streams I was receiving.
Speakers were split into 2 groups and I used 2 stream keys for all 14 speakers (this was an issue, read Issues section) and with those two groups, speakers and the stream keys were playing leapfrog. e.g. speaker 1 used stream key A, speaker 2 used stream key B and speaker 3 used stream key A again and so on.
MCs were given a single stream key as there was going to be a gap between their sections of the conference.
Each speaker or MC would go into OBS or their broadcast software of choice, select a custom server to stream to and use the Public IP address of our RTMP server. The URL used would have been something like rtmp://{IP}/live. The stream key was used as the stream key.
This example is how OBS is the setup:
When the speaker was about to start their talk they would connect their broadcast to the rtmp stream, which I then picked up.
Each speaker and MC was given a package of theme items to use during the broadcast and set up their own scenes within broadcast software to do this.
Each speaker had their own hardware and software which I was not responsible for, however, I will talk about this in issues.
Issues
So down to the issues I faced:
- RTMP - I should have used individual stream keys per user, rtmp was at times being a pain with connections and re-connections to the rtmp streams
- Audio desync - There were times the audio and video desync'd, we believe this is down to RTMP and buffers. More research is needed on this.
- Audio issues - There were a few moments where audio just "broke" now this I believe was down to the RTMP connects and reconnects I mentioned in point 1 of this list but that is unconfirmed.
- Audio Monitoring was hard
- Too many manual processes
What did I learn?
A 20hour shift is not something I want to try again.
I woke up on the morning of the conference at 6am BST, a full 7 hours before I was running a stream that would last 17+ hours. This was the doing of my 3 and a half-year-old and something that I should have planned for. By the time it got to 24 hours later at 6am BST on the 10th of April I was "DONE".
Have more backup, especially for the producer.
Only one person knowing the tech inside out, knowing all the passwords was not a great idea. I had a backup, it was wasn't a great one and very last minute.
RTMP is a bit flaky at times
I need to look at new tech or make sure our RTMP setup is rock solid. I think it was partially RTMP and partially the way I used it which made some things go a little wrong
The Future
I want to do this again, I want to produce a couple more at least. There are plans and more research to be done. Improve on what we did, the stats speak for themselves, it was an amazing event. However, I feel there were definitely points we could improve on and I hope to correct some of the mistakes I made.
If you have any questions please ask, I would love to expand on every section of this article down the line. Take each part and do a full rundown on why.
Posted on April 20, 2020
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.