18
Public whispers
7
Contributors
2026-07-15 09:44:14
First whispered
Discussion activity
Public whispers on lilianweng.github.io over the last 17 weeks
LessMore
Recent public whispers
Text Highlight2026-07-28 10:19:28
Original Highlight Excerpt
"The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory"
Whisper Note
Yep, my last model OOM'd even after offloading to CPU. Brutal.
Text Highlight2026-07-28 10:10:28
Original Highlight Excerpt
"The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory"
Whisper Note
Couldn't afford those GPUs anyway, so I just rent and pray.
Text Highlight2026-07-28 10:01:28
Original Highlight Excerpt
"The main bottleneck for training very large neural network models is the intense demand for a large amount of GPU memory"
Whisper Note
True, but the real pain is the optimizer states, they quietly eat everything.
Text Highlight2026-07-27 13:22:22
Original Highlight Excerpt
"stochastic gradient Langevin dynamics injects Gaussian noise into the parameter updates to avoid collapses into local minima."
Whisper Note
Huh, but doesn't the noise mess up the final accuracy sometimes?
Text Highlight2026-07-27 13:13:22
Original Highlight Excerpt
"stochastic gradient Langevin dynamics injects Gaussian noise into the parameter updates to avoid collapses into local minima."
Whisper Note
So it's like adding a little randomness to not get stuck, makes sense.
Text Highlight2026-07-27 10:19:22
Original Highlight Excerpt
"Diffusion models are inspired by non-equilibrium thermodynamics."
Whisper Note
But how is that different from just adding noise, though?
Text Highlight2026-07-27 10:10:22
Original Highlight Excerpt
"Diffusion models are inspired by non-equilibrium thermodynamics."
Whisper Note
Makes sense why my physics background finally pays off.
Text Highlight2026-07-27 10:01:22
Original Highlight Excerpt
"Diffusion models are inspired by non-equilibrium thermodynamics."
Whisper Note
So basically they're just fancy thermodynamics, huh.
Text Highlight2026-07-15 13:22:31
Original Highlight Excerpt
"The idea of Variational Autoencoder (Kingma & Welling, 2014), short for VAE, is actually less similar to all the autoencoder models above, but deeply rooted in the methods of variational bayesian and graphical model."
Whisper Note
Wait, so calling it an autoencoder is kind of a misnomer then?
Text Highlight2026-07-15 13:13:31
Original Highlight Excerpt
"The idea of Variational Autoencoder (Kingma & Welling, 2014), short for VAE, is actually less similar to all the autoencoder models above, but deeply rooted in the methods of variational bayesian and graphical model."
Whisper Note
So it's more Bayesian than autoencoder, that explains the math-heavy derivations.
Text Highlight2026-07-15 13:05:14
Original Highlight Excerpt
"The attention mechanism was born to help memorize long source sentences in neural machine translation (NMT)"
Whisper Note
Makes sense—memory was the bottleneck, not the model itself.
Text Highlight2026-07-15 12:56:14
Original Highlight Excerpt
"The attention mechanism was born to help memorize long source sentences in neural machine translation (NMT)"
Whisper Note
So attention is basically a cheat code for long sentences, huh?
Text Highlight2026-07-15 10:19:31
Original Highlight Excerpt
"Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer"
Whisper Note
Haha 'probably not true' — honest but then why even mention it?
Text Highlight2026-07-15 10:10:31
Original Highlight Excerpt
"Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer"
Whisper Note
The bottleneck trick is neat, but I always wonder how much info we lose.
Text Highlight2026-07-15 10:02:14
Original Highlight Excerpt
"attention in deep learning can be broadly interpreted as a vector of importance weights"
Whisper Note
I always thought of it as a spotlight, but weights work too.
Text Highlight2026-07-15 10:01:31
Original Highlight Excerpt
"Autocoder is invented to reconstruct high-dimensional data using a neural network model with a narrow bottleneck layer"
Whisper Note
Wait, is it autocoder or autoencoder? typo? anyway good intro
Text Highlight2026-07-15 09:53:14
Original Highlight Excerpt
"attention in deep learning can be broadly interpreted as a vector of importance weights"
Whisper Note
That's a neat way to frame it—makes the math less scary.
Text Highlight2026-07-15 09:44:14
Original Highlight Excerpt
"attention in deep learning can be broadly interpreted as a vector of importance weights"
Whisper Note
So basically it's just fancy weighted averaging, huh.
Ranked nearby
See what people are saying on lilianweng.github.io
Install DomWhisper to view live whispers as you browse, and join the discussion.
Get the extension