I've been building a coaching platform where coaches and athletes can interface with each other. Since users need to share exercise videos on the platform, we needed a solution to stream video to the client. The two options I was looking at for this problem were HTTP range requests and HTTP Live Streaming (HLS). Range requests are a good option, but they only serve a single version of the video and can't adapt quality to the user's internet connection. Some of the available options I found for HLS were: Cloudflare Stream, AWS MediaConvert, and Mux. Still, I thought I'd try rolling my own system for this as an exercise.
HLS is an HTTP based adaptive bitrate streaming protocol developed by Apple in 2009. It splits a video into chunks so that users can request only sections of the video that they need, ad hoc. For example, for a 1 minute video split in 6 second chunks, a user watching the video would request each segment when needed (visual is sped up):
When serving video using HLS, you can store these segments in an object store like S3. The folder structure for a single video looks something like the following. You create a master file that tells the player which qualities exist, and you have a directory for each quality, which includes all the segments for that quality in addition to a playlist file that lists the segment files.
master.m3u8 → tells the player which quality levels exist (1080p, 720p, etc).
playlist_X.m3u8 → lists the actual segment files for that resolution.
segment_X.ts → small 2-6s media chunks fetched as you watch.
I decided to set this up myself instead of relying on a third party service. To do this, you need three things:
- Lambda function that transcodes videos to HLS
- Cloudflare R2 bucket to store files
- Cloudflare Worker to authorize user requests while streaming video
For the Lambda function, I created a Lambda Layer with FFmpeg using a static build. Instead of creating your own Lambda Layer, you can use mine. Check the releases tab in my FFmpeg-static-lambda-layer repo here. You can use the ARN provided in the latest release. If you're using AWS SAM, paste it into the template like so:
Resources:
HlsTranscoder:
Type: AWS::Serverless::Function
Properties:
...
Environment:
Variables:
...
Layers:
- arn:aws:lambda:us-east-1:155343614931:layer:FFmpeg-static:1
Events:
...
Policies:
...If you're creating the Lambda function from the AWS console go to Code > Layers > Add a Layer > Specify an ARN and paste in the same ARN.
Lambda function steps:
- Parses the input request and downloads the video from R2
- Extracts video metadata with FFprobe
- Transcodes each quality preset with FFmpeg
- Uploads all HLS segments and playlists
- Generates a master playlist and sends a completion webhook
# transcoder/app.py
import json
import tempfile
import video
import utils.s3 as s3
import json
import os
import shutil
def lambda_handler(event, context):
"""Transcode video to multiple qualities and generate HLS playlist
POST /
{
"video_id": "string",
"key": "string",
"webhook_url": "string"
}
"""
# Don't forget to authorize the request
body = json.loads(event["body"])
temp_dir = tempfile.gettempdir()
video_path = f"{temp_dir}/{body['key']}"
parent_dir = os.path.dirname(video_path)
os.makedirs(parent_dir, exist_ok=True)
s3.client.download_file(os.environ["R2_BUCKET_NAME"], body["key"], video_path)
metadata, qualities = video.metadata.get(video_path)
transcoded_outputs = []
for q in qualities:
out = video.hls.transcode(video_path, q, metadata, temp_dir)
transcoded_outputs.append(out)
_, output_dir, _, _, quality_name = out
video.hls.upload_folder(output_dir, body["video_id"], quality_name)
shutil.rmtree(output_dir)
video.hls.generate_master_playlist(
qualities, transcoded_outputs, video_id=body["video_id"]
)
requests.post(body['webhook_url'], json={"video_id": body['video_id'], "status": "processed"})
return {"statusCode": 204}
This helper extracts metadata from the input video using FFprobe. It determines width, height, duration, and which quality levels can be generated to avoid transcoding to higher resolutions than the source.
# transcoder/video/metadata.py
import subprocess
import json
from typing import Dict
import utils.constants as constants
def get(video_path: str) -> tuple[Dict, list]:
"""
Extract video metadata using ffprobe
"""
cmd = [
constants.FFPROBE_PATH,
"-v",
"quiet",
"-print_format",
"json",
"-show_format",
"-show_streams",
video_path,
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise Exception(f"FFprobe failed: {result.stderr}")
data = json.loads(result.stdout)
video_stream = next(
(s for s in data["streams"] if s["codec_type"] == "video"), None
)
if not video_stream:
raise Exception("No video stream found")
is_portrait = video_stream["width"] < video_stream["height"]
effective_height = video_stream["width"] if is_portrait else video_stream["height"]
available = [
q for q in constants.QUALITY_PRESETS if q["height"] <= effective_height
]
return (
{
"duration": int(float(data["format"].get("duration", 0))),
"width": int(video_stream.get("width", 0)),
"height": int(video_stream.get("height", 0)),
"size": int(data["format"].get("size", 0)),
"aspect_ratio": video_stream.get("width", 1)
/ video_stream.get("height", 1),
},
available if available else [constants.QUALITY_PRESETS[-1]],
)
Constants used across the Lambda, including FFmpeg and FFprobe paths, and a few video quality presets that control bitrate and resolution.
# transcoder/utils/constants.py
FFPROBE_PATH = "/opt/bin/ffprobe"
FFmpeg_PATH = "/opt/bin/FFmpeg"
QUALITY_PRESETS = [
{"name": "1080p", "height": 1080, "bitrate": "5000k", "audio_bitrate": "192k"},
{"name": "720p", "height": 720, "bitrate": "2800k", "audio_bitrate": "128k"},
{"name": "480p", "height": 480, "bitrate": "1400k", "audio_bitrate": "128k"},
]
This handles the actual HLS generation. I use FFmpeg to transcode each quality to segments and playlist, then upload them to R2. Then, create a master.m3u8 file that references all variant playlists.
# transcoder/video/hls.py
from typing import Dict, List, Tuple
from pathlib import Path
import subprocess
import utils.constants as constants
import utils.s3 as s3
import os
def transcode(
video_path: str, quality: Dict, metadata: Dict, tmp_path: str
) -> Tuple[Path, Path, int, int, str]:
"""
Transcode video to HLS for a specific quality.
"""
def _even(n: int) -> int:
return n - (n % 2)
quality_name = quality["name"]
output_dir = Path(tmp_path) / "hls" / quality_name
output_dir.mkdir(parents=True, exist_ok=True)
playlist_path = output_dir / "playlist.m3u8"
segment_pattern = str(output_dir / "segment_%d.ts")
src_w = int(metadata["width"])
src_h = int(metadata["height"])
target_h = min(int(quality["height"]), src_h)
planned_h = _even(target_h)
planned_w = _even(int((src_w * planned_h) / src_h))
vf_chain = f"scale={planned_w}:{planned_h}:force_divisible_by=2,setsar=1"
cmd = [
constants.FFmpeg_PATH,
"-i",
video_path,
"-vf",
vf_chain,
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-preset",
"fast",
"-profile:v",
"main",
"-level",
"4.0",
"-b:v",
quality["bitrate"],
"-maxrate",
quality["bitrate"],
"-bufsize",
f"{int(str(quality['bitrate']).rstrip('kK')) * 2}k",
"-c:a",
"aac",
"-b:a",
quality["audio_bitrate"],
"-ac",
"2",
"-ar",
"48000",
"-hls_time",
"6",
"-hls_playlist_type",
"vod",
"-hls_list_size",
"0",
"-hls_flags",
"independent_segments",
"-force_key_frames",
"expr:gte(t,n_forced*6)",
"-hls_segment_filename",
segment_pattern,
"-f",
"hls",
"-y",
str(playlist_path),
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise Exception(f"FFmpeg transcoding failed: {result.stderr}")
return playlist_path, output_dir, planned_w, planned_h, quality_name
def upload_folder(output_dir: Path, video_id: str, quality_name: str):
"""Upload all files in the HLS output directory to S3"""
for file_path in output_dir.iterdir():
if file_path.is_file():
s3_key = f"videos/{video_id}/hls/{quality_name}/{file_path.name}"
# Determine content type
if file_path.suffix == ".m3u8":
content_type = "application/vnd.apple.mpegurl"
elif file_path.suffix == ".ts":
content_type = "video/mp2t"
else:
content_type = "application/octet-stream"
s3.client.upload_file(
str(file_path),
os.environ["R2_BUCKET_NAME"],
s3_key,
ExtraArgs={"ContentType": content_type},
)
def generate_master_playlist(
qualities: List[Dict],
transcoded_outputs: List[Tuple[Path, Path, int, int, str]],
video_id: str,
) -> str:
"""
Build master playlist referencing all variants.
"""
def _kbps_to_bps(k: str) -> int:
return int(str(k).rstrip("kK")) * 1000
lines = [
"#EXTM3U",
"#EXT-X-VERSION:3",
"",
]
for quality, (_, _, w, h, quality_name) in zip(qualities, transcoded_outputs):
bw_bps = _kbps_to_bps(quality["bitrate"]) + _kbps_to_bps(
quality["audio_bitrate"]
)
avg_bw_bps = max(int(bw_bps * 0.85), 1)
lines.append(
f"#EXT-X-STREAM-INF:BANDWIDTH={bw_bps},AVERAGE-BANDWIDTH={avg_bw_bps},"
f'RESOLUTION={w}x{h},NAME="{quality_name}"'
)
lines.append(f"{quality_name}/playlist.m3u8")
lines.append("")
playlist = "\n".join(lines)
s3.client.put_object(
Bucket=os.environ["R2_BUCKET_NAME"],
Key=f"videos/{video_id}/hls/master.m3u8",
Body=playlist.encode("utf-8"),
ContentType="application/vnd.apple.mpegurl",
)
Since HLS generates multiple segment files, you'd typically need a signed URL for a whole directory for a video. This is not possible with the S3 API. We can work around this using Cloudflare Workers.
- Create a short lived token and use that to authorize requests.
- Verify the request is coming from a trusted source
- Return the segment object
// src/index.ts
type Env = {
R2: R2Bucket; // Bind your R2 bucket in wrangler.toml
};
export default {
async fetch(request, env): Promise<Response> {
const url = new URL(request.url);
// Auth (swap with your own logic)
const token = url.searchParams.get("token");
if (verify(token)) {
return new Response("Unauthorized", { status: 401 });
}
const key = url.pathname.replace(/^/+/, "");
const object = await env.R2.get(key);
if (!object) return new Response("Not found", { status: 404 });
const headers = new Headers();
object.writeHttpMetadata(headers);
if (!headers.has("Content-Type")) headers.set("Content-Type", guessContentType(key));
if (!headers.has("Cache-Control")) headers.set("Cache-Control", "public, max-age=60");
return new Response(object.body, { headers });
},
} satisfies ExportedHandler<Env>;
function guessContentType(key: string): string {
if (key.endsWith(".m3u8")) return "application/vnd.apple.mpegurl";
if (key.endsWith(".ts")) return "video/mp2t";
return "application/octet-stream";
}
Note: This is not a production setup. However, you can extend it to make something more robust.