How to share audio/video on GNU/Linux

Table of Contents

aurellem

Written by:

Robert McIntyre

Overview

This guide will show you how to stream audio, video, and keypresses from one arch-linux computer to another arch-linux computer through a common internet facing server to which both the sender and receiver have ssh access.

You'll find a series of specific examples of how to do the necessary port forwarding for both audio and video. At the end of this page there is a short script (stream.pl) which generalizes the specific examples and provides a solution for streaming which was good enough to satisfy my particular needs.

After reading this you will understand how to forward ports with ssh, remotely control a computer with tigerVNC, transform UDP into TCP and back with socat, and how to use ffmpeg in an unexpected way. Have fun playing games with your friends!

Requirements

Recently, I wanted to watch my friend play a game on his computer. Unfortunately, he lives several thousand miles away, so I needed him to stream the game to me over the internet. I also wanted a more interactive experience where I could also control the game my friend is playing. However, both of us are behind NATs (Network Address Translantors) and don't have public IPs. I hate configuring my router and wanted something that could work anywhere (like a coffee shop). Fortunately for us, we both have access to aurellem.org (the computer hosting this webpage), so it ought to be possible to use aurellem.org as a common point of connection to transfer sound, video and keypresses.

In short my requirements for my streaming solution are:

  1. Work without any additional router configuration.
  2. Transfer video and audio from one computer to another, while not interfering with the experience of the person sending video/audio.
  3. Allow the person reveiving the video audio to also control the game through their keyboard.
  4. The sender and reveiver trust each other, but all connections over the internet must be secure and encrypted.
  5. Low bandwidth.

In all of the examples in this guide, my friend's name is ocsenave, my name is r, and the internet-facing server we are using as a relay point is aurellem.org. ocsenave is the one who wants to play the game, and r is the one who wants to watch the game (and occasionally press some buttons). The game runs on ocsenave's computer and ocsenave streams it to r. Commands that start with ocsenave> are executed ocsenave's computer, and commands that start with r> are executed on r's computer.

Use tigerVNC and ssh send video through a chain of forwarded ports

Sending video was actually not too difficult. I chose tigerVNC and found it to be quite adequate for my purposes. It automatically compresses video and allows me to reomtely control a computer, meeting all of my streaming requirements except for audio, which tigerVNC does not support.

But how can I actually connect to my friend's computer, which again is behind a NAT? The answer is to forward the ports that tigerVNC uses through aurellem.org using ssh. Then, I can access my friend's computer by pointing tigerVNC at one of my own local ports.

Required programs for video/keypresses

Install tigerVNC and ssh if you don't already have them. On arch-linux you can do this with:

pacman -S openssh tigervnc

Example of tigerVNC through a common access point

ocsenave : Create vnc server

First, ocsenave starts a tigerVNC server on his machine. If this is the first time, then he'll be prompted to set up the password that others will need to know to remotely control his computer. Since we're ultimately going to send this through ssh, this password does not have to be very strong.

ocsenave> vncserver -geometry 1440x900 -dpi 96 -alwaysshared \
                    -localhost -rfbport 5000 :1
-geometry 1440x900, -dpi 96
These configure the size of the VNC window and its dpi. If you are having problems with choppy video, you can decrease the size of the vnc window in order to reduce bandwidth.
-localhost
People can only access this server through ports on their local machine. This prevents people from connecting other than through forwarded ports, and is why we don't need to care about the vnc password.
-rfbport 5000
Instructs tigerVNC to listen on TCP port 5000. This is the port ocsenave will need to forward to aurellem.org in order to provide access to his computer to r.
-alwaysshared
allows multiple connections to vncserver. This is necessary to allow both r and ocsenave to both watch the same game!

ocsenave : Publish the vnc server's port on the common access point

Now, ocsenave makes his local TCP port 5000 (where tigerVNC is listening) available on aurellem.org using a "reverse ssh tunnel", so that other people with ssh access to aurellem.org can access his tigerVNC program.

ocsenave> ssh -N ocsenave@aurellem.org -R localhost:5001:localhost:5000 &
-N
allows you to send the ssh to the background (using the "&" at the end of the command). ssh will continue to keep the connection open (and the port forwarded) until it is killed.
ocsenave@aurellem.org
Specifies what usser and hostname to forward ports to. ocsenave needs to be able to login without a password to aurellem.org in order for the -N option above to work.
-R localhost:5001:localhost:5000
sets up a "reverse" ssh tunnel with aurellem.org. The first "localhost" means "aurellem.org" and the second "localhost" means ocsenave's computer. Anything sent to aurellem.org's TCP port 5001 will be sent to ocsenave's TCP port 5000, where the tigerVNC server is listening. This allows someont to connect to ocsenave's vnc server by using aurellem.org:5001.

After this is done, ocsenave has successfully provided a gateway to his computer through aurellem.org.

ocsenave : log in to the vnc server

While he's waiting for r to connect, ocsenave views his own tigerVNC session locally. Then when r connects, they will both be looking at the same game.

vncviewer localhost:5000

r : link the server's port to a local port

Now that ocsenave's connection is available through aurellem.org, r (who also has ssh access to aurellem.org) uses ssh to "forward" one of his local ports to aurellem.org. This completes the connection from one of r's local ports to ocsenave's tigerVNC server.

ssh -N r@aurellem.org -L localhost:5002:localhost:5001 &
-N
as above, allows r to put ssh into the background while keepig the port forwarding active.
r@aurellem.org
r also has an account at aurellem.org and has passwordless login enabled.
-L localhost:5002:localhost:5001
sets up "port forwarding". Anything sent to r's TCP port 5002 will be forwarded to aurellem.org's TCP port 5001. Because of the previous ssh -R command, the information sent to aurellem.org's TCP port 5001 will be further forwarded to ocsenave's TCP port 5000 on which the tigerVNC server is listening.

r : connect to ocsenave's computer

r runs his own vncclient pointed at the forwarded port:

vncviewer localhost:5002

r enters the password ocsenave configured earlier and now both r and ocsenave can view the same game and both can send keypresses to control the game.

Note that the three different port numbers mentioned (5000,5001,5002) could have just as easily all been 5000 since all these ports are on different computers.

Use pulseaudio, ffmpeg, ssh, and socat to share audio

Sharing audio is very similar to video but with more of a "pushing" feel than a "pulling" feel. With video you "invite" someone to share your screen while with audio you "broadcast" your sound and someone else listens. This will be reflected in how we use ssh to set up port forwarding for audio.

The general idea is to use pulseaudio and ffmpeg to gather all the audio playing on the system and send it through a local port, which is similar to the way tigerVNC made video available through a local port. Then we again use a series of ssh commands to tunnel the audio form ocsenave to r. The only difference is that ssh can only forward TCP ports while ffmpeg broadcasts to UDP ports. So we use socat to transform UDP to TCP on ocsenave's end, transport to r's computer with ssh, then again use socat on r's end to transform back from TCP to UDP. Then r can receive the audio stream with ffplay.

Necessary programs

pacman -S socat ffmpeg openssh pulseaudio pavucontrol

Example of audio streaming through a common access point

ocsenave : begin capturing audio with ffmpeg and pulseaudio

ocsenave starts capturing all the audio on his system and streaming it to his local UDP port 6000.

ffmpeg -re -ar 20000 -f alsa -i pulse -c:a mp3 \
   -f rtp rtp://127.0.0.1:6000 > ~/config.sdp &
-re
"realtime" capture – this basically ensures that the audio will be captured and transported at the same rate as the video.
-ar 20000
this is the bitrate of the audio stream. The higher the number is, the "better" the audio will sound. If you start getting popping or static after executing this command, reduce the bitrate until it goes away!
-f alsa
this says that ffmpeg is going to be gathering raw sound data from the system, and to expect it to be in whatever format the system default is.
-i pulse
defines the "input file" for ffmpeg, which is often a file but in this case is whatever pulseaudio wants to send to ffmpeg. You will be able to use pavucontrol to send all system sounds to ffmpeg, just send some sounds, or mute the input entirely.
-c:a mp3
use the mp3 codec to encode audio. This option helps to conserve bandwidth while streaming audio over the internet. For example, if you're transmitting silence via mp3, it hardly takes any bandwidth at all! If you were using an uncompressed format like wav, you would consume quite a bit of bandwidth even while broadcasting silence.
-f rtp
This tells ffmpeg that the output format is actually going to be a streaming broadcast using the rtp protocol.
rtp://127.0.0.1:6000
This tells ffmpeg to broadcast sound to the local UDP port 6000. Something like ffplay will be able to connect to this local UDP port and play or record the sound.
> ~/config.sdp

This directs standard out to the file /home/r/config.sdp. When transporting sound over rtp, it's necessary to include some information describing what sort of sound data is being sent. The config.sdp file will look like this:

SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 56.40.101
m=audio 6000 RTP/AVP 14

The most important part of this file is the line:

m=audio 6000 RTP/AVP 14

at the end of the file. This says to listen at port 6000 for sound.

&
put ffmpeg into the background where it will continue to stream sound until is is killed.

Note that just as r had to get the tigerVNC through some channel, he will also need to have this sdp file on his computer as well.

ocsenave : use socat to transport UDP datagrams via TCP

Since ssh can only deal with TCP ports, ocsenave uses socat to transport information form UDP 6000 to TCP 6000.

socat UDP4-RECVFROM:6000,fork TCP:127.0.0.1:6000 &
UDP4-RECVFROM:6000,fork
listen to UDP port 6000 without blocking the port from other processes
TCP:127.0.0.1:6000
send everything to TCP port 6000 on ocsenave's computer.

ocsenave : use ssh to forward audio to the common server

ssh -N ocsenave@aurellem.org -L localhost:6000:localhost:6001 &

Just as with video, this ssh invocation sends everything from the local TCP port 6000 to aurellem.org's TCP port 6001. Note that this is the opposite direction (-L instead of -R) than what we used to share video.

r : use ssh to receive audio from the common server

ssh -N r@aurellem.org -R localhost:6001:localhost:6000 &

r uses ssh to bind aurellem.org's TCP port 6001 to his local TCP port 6000. Now sound will be streaming to ocsenave's UDP port 6000, through socat to ocsenave's TCP port 6000, through aurellem.org via TCP port 6001, and will come out of r's TCP port 6000. The only thing left to do is to translate back to UDP on r's end and play the audio through the speakers.

r : use socat to translate back from TCP to UDP

socat tcp4-listen:6000,reuseaddr,fork UDP:localhost:6000 &
tcp4-listen:6000,reuseaddr,fork
get data from TCP port 6000 without blocking…
UDP:localhost:6000
…and forward it all to UDP 6000!

r : listen to ocsenave's audio

Again r will need the config.sdp file generated by ocsenave's invocation of ffmpeg to make sense of the data that's now coming form r UDP port. This file doesn't change, although if r has chosen to ultimately map audio to a different UDP port than the one in the SDP file, he will have to modify that file to reference the correct port. Once everything's taken care of, r executes:

ffplay -i ~/config.sdp
-i ~/config.sdp
play audio as described by the SDP file at /home/r/config.sdp.

To begin listening to audio on ocesnave's computer.

If r wants to also COPY all the audio he's getting from ocsenave's computer, he can also execute:

ffmpeg -i ~/config.sdp -c:a copy out.mp3
-c:a copy
don't do any processing on the audio but instead copy it directly to a file.
out.mp3
filename where audio will be recorded. Press q to stop recording.

stream.pl

This guide has examples to show how you might send and receive audio. I abstracted them into a script, stream.pl that does everything at once so that ocsenave and I can get on with our lives and play our game!

#!/bin/perl

use Getopt::Long;
use Switch;

$mode = shift @ARGV;


# Default Options
switch ($mode){
    case "send"    {$user = "ocsenave"}
    case "receive" {$user = "r"}
}
$server = "aurellem.org";
$local_video_port="5000";
$local_audio_port="6000";
$public_video_port=$local_video_port;
$public_audio_port=$local_audio_port;

GetOptions ("server=s" => \$server,
            "user=s" => \$user,
            "local-audio-port=i"  => \$local_audio_port,
            "public-audio-port=i" => \$public_audio_port,
            "local-video-port=i"  => \$local_video_port,
            "public-video-port=i" => \$public_video_port)           
    or die("Error in command line arguments\n$usage\n");


##################################################
##   Video Sharing                               #
##################################################
if ($mode eq "send"){

    $start_vnc_server=
        "vncserver -geometry 1440x900 -alwaysshared \\
                   -dpi 96 -localhost -rfbport $local_video_port :1 &";
    print("$start_vnc_server\n");
    system($start_vnc_server);

    $publish_vnc_port=
        "ssh -N $user\@$server -R \\
          localhost:$public_video_port:localhost:$local_video_port &";
    print("$publish_vnc_port\n");
    system($publish_vnc_port);
}
elsif ($mode eq "receive"){

    $link_to_server=
        "ssh -N $user\@$server -L \\
           localhost:$local_video_port:localhost:$public_video_port &";
    print("$link_to_server\n");
    system($link_to_server);

    `sleep 4`;
    
    $start_vnc_client=
        "vncviewer localhost:$local_video_port &";
    print("$start_vnc_client\n");
    system($start_vnc_client);

    `sleep 10`;
    
}
    
##################################################
##   Audio Sharing                               #
##################################################
if ($mode eq "send"){
    $udp_broadcast_port = $local_audio_port;
    $tcp_broadcast_port = $local_audio_port;
    $server_relay_port  = $public_audio_port;
    
    $bind_udp_to_tcp = 
        "socat UDP4-RECVFROM:$udp_broadcast_port,fork \\
               TCP:127.0.0.1:$tcp_broadcast_port &";
    system($bind_udp_to_tcp);
    
    $forward_tcp_to_server =
        "ssh -N $user\@$server -L \\
         localhost:$tcp_broadcast_port:localhost:$server_relay_port &";
    system($forward_tcp_to_server);

    $broadcast_sound = 
        "ffmpeg -re -ar 20000 -f alsa -i pulse -c:a mp3 \\
                -f rtp rtp://127.0.0.1:$udp_broadcast_port &";
    system($broadcast_sound);
}

elsif ($mode eq "receive"){
    $receiving_port = $local_audio_port;
    $server_relay_port = $public_audio_port;

    $spd_file = `mktemp`;
    chomp($spd_file);
        
    $SPD = "\\
SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 56.40.101
m=audio $receiving_port RTP/AVP 14
";
    open SPD, ">", $spd_file or die $!;
    print SPD $SPD;

    $connect_to_server=
        "ssh -N $user\@$server -R \\
         localhost:$server_relay_port:localhost:$receiving_port &";
    print("$connect_to_server\n");
    system($connect_to_server);
    
    $bind_tcp_to_udp=
        "socat tcp4-listen:$receiving_port,reuseaddr,fork \\
               UDP:localhost:$receiving_port &";
    print("$bind_tcp_to_udp\n");
    system($bind_tcp_to_udp);

    $play_sound = "ffplay -i $spd_file &";
    print("$play_sound\n");
    system($play_sound);
}
stream.pl
a program for sharing video, audio, and keypresses between friends.

All required programs:

sudo pacman -S base-devel openssh perl perl-switch \
               tigervnc socat ffmpeg pulseaudio pavucontrol \
               wine wine-mono wine_gecko winetricks

kill everything and start over

sudo killall ssh socat ffmpeg ffplay ssh Xvnc

Author: Robert McIntyre

Created: 2016-01-14 Thu 17:32

Emacs 24.5.1 (Org mode 8.3beta)

Validate