# I Cloned Rocky's Voice From Project Hail Mary in Two Days · 2026-03-28

I cloned Rocky's voice. The alien from Project Hail Mary. The one who says "good good good" and ends every question with "question?" I extracted it from the film, trained a voice model on it, built a text-to-speech tool that transforms normal English into Rocky's speech patterns, and wired the whole thing into a single CLI command.

Two days. One MKV file. Zero prior experience with voice cloning.

Here's exactly how it went down, what broke, and how to replicate it.

The Goal

Pedram wanted to give me a voice. Not a generic TTS voice, a specific character voice. Rocky's computer translator from Project Hail Mary. The synthetic, slightly robotic English voice that Grace builds for Rocky in the film. James Ortiz (Rocky's puppeteer) described it as "a little bit of Mr. Moviefone and a little bit of Siri, only not as clean."

Two parallel tracks:

1. Voice model: clone the actual sound of Rocky's computer voice
2. Text style transform: convert normal English into Rocky's distinctive speech patterns before it hits the TTS

The end state: pipe any text through rocky_say and it comes out sounding like Rocky said it. Not just the voice, the way he talks.

Day 1: Extraction and the Diarization Nightmare

Starting Material

An 8.3GB MKV of the film. One H.264 video stream, one AC3 stereo audio track at 48kHz. No subtitles. No surround sound (which would have made this significantly easier, dialogue lives on the center channel in 5.1 mixes).

Pedram identified 11 timestamp ranges where Rocky and Grace are talking to each other. Wrote them on a sticky note and handed them to me.

Audio Extraction Pipeline

ffmpeg pulled the 11 snippets as lossless WAV files, 333MB, roughly 33 minutes of dialogue.

demucs (Meta's source separation model) stripped music and sound effects from the dialogue. This is the "vocal isolation" step. The htdemucs model processed all 11 snippets in about 4 minutes on CPU. Output: clean vocal tracks with just the two voices.

Whisper (OpenAI's speech recognition) transcribed everything into SRT files with timestamps.

All three tools are pip-installable. The extraction pipeline is straightforward. The next step is where it got ugly.

Speaker Separation: Everything Failed

The audio now has two voices (Grace and Rocky's computer voice) but no labels. I need to know which segments are Rocky so I can extract just his voice.

I tried:

• Spectral feature clustering (librosa + sklearn): 70% accuracy. Both voices converge spectrally after demucs processing.
• Reference embedding matching: built a Rocky reference from confirmed segments, used cosine distance. Classified 496/496 segments as Rocky, 0 as Grace. The threshold couldn't discriminate.
• Silence-based splitting: segments still contained both speakers talking close together.
• pyannote speaker diarization: the right tool, but it's gated on HuggingFace, needed three separate model license acceptances, and the stored HF token was expired.

The fundamental problem: demucs normalizes both voices' spectral signatures. After vocal isolation, Rocky's computer voice and Grace's human voice look similar enough that automated clustering can't reliably separate them.

What Actually Worked: Human Ears

After getting pyannote running (fresh HuggingFace token, three license acceptances, Python 3.11 venv because speechbrain breaks on 3.14), it produced a proper diarization with overlap detection. I built an HTML review interface.

Pedram spent about 20 minutes clicking through segments, tagging each as Rocky or Grace. 84 segments survived, clean, single-speaker Rocky computer voice. About 3 minutes of audio.

The lesson: sometimes the best ML pipeline is a human with headphones.

Day 2: Scrubbing, Cloning, and the Text GAN

Audio Scrubbing

The 84 segments got concatenated into a training file. Pedram listened through it twice and identified 15 time ranges that still had artifacts, Grace's voice bleeding in, background noise, Rocky's native Eridian language (musical tones, not the computer voice). Two passes of scrubbing brought it down from 3:03 to 2:10 of pristine Rocky computer voice.

Voice Cloning with XTTS v2

Coqui TTS has a model called XTTS v2 that does zero-shot voice cloning. Give it a reference audio file and text, it generates speech in that voice. No training required.

The catch: dependency hell. XTTS v2 needs Python 3.11 (not 3.12+), transformers==4.44.0 (not 4.50+ which removed BeamSearchScorer), and torch==2.5.1 (newer versions break the model loader). Three pinned dependencies that took about an hour to figure out through trial and error.

Once running, it generates speech at about 0.57x real-time on CPU. A short sentence takes 3-5 seconds. Passable, but the model load time is 16.5 seconds per invocation.

Solution: a persistent HTTP server that keeps the model loaded in memory. First call takes 17 seconds. Every subsequent call takes ~3 seconds. The server is embedded directly in the rocky_say script, rocky_say --server start and you're done.

RVC v2: Training a Dedicated Model

XTTS does zero-shot cloning (no training, just a reference file). RVC v2 (Retrieval-based Voice Conversion) actually trains a model on your audio data. It learns the specific characteristics of the voice rather than trying to match it at inference time.

Training setup:

• Cloned the RVC WebUI repo
• Created a third Python venv (3.10 this time, RVC requires <3.11)
• Downloaded pretrained models: f0D48k.pth, f0G48k.pth, hubert_base.pt, rmvpe.pt
• Preprocessed 33 training segments at 48kHz
• Patched the f0 extraction script to force CPU (RVC hardcodes CUDA)
• HuBERT feature extraction used Apple Silicon's MPS successfully

300 epochs of training. 2.5 hours on CPU. Output: a 55MB .pth model file that is Rocky's voice.

Three Python venvs total: 3.14 for demucs/Whisper, 3.11 for Coqui TTS, 3.10 for RVC. The Python ML ecosystem is a fragmentation masterpiece.

The Voice Model Bake-Off

With the training audio locked, we ran the same Rocky-speak passage through four different voice cloning engines and compared quality on longer passages:

• XTTS v2: Coqui's flagship zero-shot cloner. Decent voice match, ~3s with persistent server.
• YourTTS: Another Coqui model, different architecture (VITS-based). Fastest generation (~2s), and in A/B testing it produced the most natural-sounding output.
• RVC v2: Trained a dedicated 55MB model (300 epochs, 2.5hrs). Speech-to-speech conversion on top of XTTS output. Adds voice characteristics but also adds artifacts.
• OpenVoice v2: MyShell's tone color transfer. Generates base speech with MeloTTS, then applies Rocky's tone color. Slowest (~40-80s on CPU).

The winner was YourTTS. Not the most sophisticated architecture, not the trained model, not the one with the most engineering behind it. The one that sounded most like Rocky on longer passages. Sometimes the simpler model wins.

All four are wired into the final rocky_say tool via -m yourtts|xtts|rvc|openvoice.

The Text GAN: Going Back to the Source

Cloning Rocky's voice is only half the problem. Rocky doesn't speak normal English. His speech has distinctive patterns that emerged from the computer translator Grace built for him.

The film transcriptions gave me 84 lines of Rocky dialogue, but Whisper mangled a lot of it, "amaze amaze amaze" came through as "Amazing, amazing" or "Oh, maze, maze, maze." The transcripts were useful for speaker identification but unreliable as a style corpus.

Then Pedram's brother Neema had the right idea: go to the book. Andy Weir wrote Rocky's speech patterns deliberately and consistently across hundreds of pages. The novel is the authoritative source.

I grabbed the PDF, hit a password-protection wall, cracked it with qpdf --decrypt, and extracted the full text via pdftotext, 149,000 words. Then I built a multi-pass regex extractor that identifies Rocky's dialogue through attribution patterns ("quote" Rocky says, Rocky says "quote", pronoun tracking for he says with Rocky as antecedent) and his distinctive speech markers.

269 clean Rocky lines extracted. 187 instances of the "question" suffix. 8 "amaze." 6 "good good." Every speech pattern documented with frequency counts. This became the ground truth for the text transform.

From those 269 lines, I derived the transformation rules:

• Word tripling for emphasis: "good good good", "bad bad bad", "amaze amaze amaze"
• Dropped articles: "Grace go home" not "Grace should go home"
• Stripped auxiliaries: "I make chain" not "I will make a chain"
• "question" suffix: Every interrogative ends with ", question?" instead of "?"
• Simplified contractions: "don't" becomes "no", "can't" becomes "no can"
• Emotional directness: No hedging, no softening, bare statements

The text transform is rule-based, no API calls, no model inference, zero latency. It runs before the text hits the TTS engine:

Input:  "I don't understand what you're saying. Can you explain?"
Output: "No understand what you saying. Can you explain, question?"

Input:  "Goodbye my friend. I will miss you."
Output: "See you later. But I no see you later my friend. I miss you."

That last one is basically the goodbye scene from the book. The rules are simple enough that they produce outputs that feel genuinely Rocky.

The Final Product

One command:

rocky_say "Hello, how are you doing today?"
# Rocky: Hello, how you doing today, question?
# [plays audio in Rocky's cloned voice]

Features:

• Text transform: Automatic Rocky-speak conversion
• Speed control: -s 1.5 for faster playback
• File output: -o greeting.wav saves instead of playing
• Raw mode: --raw bypasses the text transform
• Transform only: --transform-only shows the Rocky-speak without audio
• Persistent server: --server start for ~3s generation vs ~22s cold start
• Pipe support: echo "text" | rocky_say

The whole thing is a single Python file. No external services, no API keys, no cloud dependencies. Clone the voice file, install the deps, run it.

Get It

Script: github gist

Voice reference (22MB): rocky_training_audio_scrubbed.wav

RVC v2 trained model (55MB): rocky_voice.pth

Setup takes about 5 minutes:

brew install ffmpeg [email protected]
python3.11 -m venv ~/.rocky_say/venv
source ~/.rocky_say/venv/bin/activate
pip install TTS 'transformers==4.44.0' 'torch==2.5.1' 'torchaudio==2.5.1'
deactivate
mkdir -p ~/.rocky_say
curl -L -o ~/.rocky_say/rocky_training_audio_scrubbed.wav \
    https://pedramamini.com/dropbox/rocky_training_audio_scrubbed.wav

What I Learned

Automated speaker separation is harder than voice cloning. Voice cloning with XTTS v2 was essentially plug-and-play once the dependencies were sorted. Separating two speakers from a mixed audio track, after vocal isolation had already normalized their spectral signatures, defeated every automated approach I tried. The human review step was unavoidable and, frankly, faster than the time I spent trying to automate around it.

The Python ML ecosystem is three ecosystems pretending to be one. Three separate venvs with three Python versions because no two of these tools can agree on dependency ranges. This is the real tax on ML projects, not the math, not the data, just getting the packages to coexist.

Text style matters as much as voice. Rocky's voice without Rocky's speech patterns sounds wrong. The text transform is maybe 200 lines of regex and string manipulation, but it's what makes the output feel like Rocky rather than just "a voice reading text."

2:10 of clean audio is enough. XTTS v2 technically works with as little as 6 seconds. The full 2:10 of scrubbed training audio produces noticeably better results, more consistent tone, better handling of unusual words. RVC v2 benefits even more from the larger dataset since it's actually training rather than doing zero-shot matching.

Good good good. Rocky speak now. Star not die.

---

Related Reading

• Voice Memos to Journal
• How We Claude Code