|
/
|
Home
|
|
|
/post
|
Posts
|
|
|
/post/swoosh
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Swoosh: Rethinking Activation Functions</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2023-04-01T00:00:00.000Z">April 1, 2023</time><!--]--></div> <!--[!--><!--]--> <!--[--><div class="text-gray-700"><!----><p>Introducing the new Swoosh activation function. Perfect test set generalization
guaranteed.
</p>
<!----></div><!--]-->
|
|
|
/post/einsum
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Einsum is easy and useful</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2022-11-05T00:00:00.000Z">November 5, 2022</time><!--]--></div> <!--[!--><!--]--> <!--[--><div class="text-gray-700"><!----><p><code>einsum</code> is one of the most useful functions in Numpy/Pytorch/Tensorflow and yet many people don't use it. It seems to have a reputation as being difficult to understand and use, which is completely backwards in my view: the reason <code>einsum</code> is great is precisely because it is <em>easier</em> to use and reason about than the alternatives. So this post tries to set the record straight and show how simple <code>einsum</code> really is.
</p>
<!----></div><!--]-->
|
|
|
/post/distributions-operations
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Distributions Part II: What can we do with distributions?</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2022-03-05T23:00:00.000Z">March 5, 2022</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Math</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> As promised in part I, we can do a lot of the same things with Schwartz
distributions as with classical functions. To see how, we'll cover
derivatives, convolutions, and Fourier transforms of distributions.
</p>
<!----></div><!--]-->
|
|
|
/post/random-vs-grid-search
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Visualizing random vs grid search</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-12-22T00:00:00.000Z">December 22, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Machine learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p>Random search usually works better than grid search
for hyperparameter optimization. This brief post suggests a way
to visualize the reason for this geometrically.
</p>
<!----></div><!--]-->
|
|
|
/post/karger-extensions
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Extensions of Karger's algorithm</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-09-10T09:41:00.000Z">September 10, 2021</time><!--]--></div> <!--[!--><!--]--> <!--[--><div class="text-gray-700"><!----><script>
import Spoiler from '$lib/components/Spoiler.svelte';
import Figure from '$lib/components/Figure.svelte';
</script>
<p><em>If you prefer videos, check out our <a href="https://www.youtube.com/watch?v=1sURbgamyvU">ICCV presentation</a>, which covers similar content as this blog post.
For more details, see <a href="http://arxiv.org/abs/2110.02750">our paper</a>.</em>...</p>
<!----></div><!--]-->
|
|
|
/post/distributions-intro
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Distributions Part I: the Delta distribution</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-07-06T12:15:00.000Z">July 6, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Math</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Did you always want to know kind of object this weird Dirac delta "function"
actually is? Well, it's a Schwartz distribution. If that doesn't help much,
then keep reading.
</p>
<!----></div><!--]-->
|
|
|
/post/automation-productivity
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Scripting for personal productivity</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-04-14T17:42:00.000Z">April 14, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Productivity</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> If you can program, you can use that to support your habits and automate
some routines. This post gives a few examples.
</p>
<!----></div><!--]-->
|
|
|
/post/bias-variance-tradeoff
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Troubles with the Bias-Variance tradeoff</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-04-07T16:19:00.000Z">April 7, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Machine learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> The bias-variance tradeoff is a key idea in machine learning. But I'll
argue that we know surprisingly little about it: when does it hold?
How does it relate to the Double Descent phenomenon? And what do we
even formally mean when we talk about it?
</p>
<!----></div><!--]-->
|
|
|
/post/computer-tips
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Collection of quick computer tips</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-03-31T12:49:00.000Z">March 31, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Productivity</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Many of us spend a lot of time working with our computer, so it's worth
spending some time to make that experience as pleasent and productive
as possible. This is a collection of tips that are relatively quick
to implement and still very valuable in the long run in my opinion.
Mainly geared towards developers and others who work with the shell
a lot.
</p>
<!----></div><!--]-->
|
|
|
/post/state-formally-reason-informally
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">State formally, reason informally</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-03-24T10:00:00.000Z">March 24, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Math</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> There's a style of teaching mathematics that I really like: stating definitions
and theorems as formally as in any textbook, but focusing on informal arguments
for why they should be true.
</p>
<!----></div><!--]-->
|
|
|
/post/latex-emacs
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Emacs as an amazing LaTeX editor</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-03-17T13:14:00.000Z">March 17, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Productivity</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Emacs has some really amazing features for writing LaTeX; this post gives
an overview of some of them, either to convince you to give Emacs a try,
or to make you aware that these features exist if you're already using
Emacs but didn't know about them.
</p>
<!----></div><!--]-->
|
|
|
/post/spherical-harmonics
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Perspectives on spherical harmonics</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-03-10T16:07:00.000Z">March 10, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Math</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Spherical harmonics are ubiquitous in math and physics, in part because
they naturally appear as solutions to several problems; in particular they
are the eigenfunctions of the spherical Laplacian and the irreducible
representations of SO(3). But why should the solutions to these problems
be the same? And why are they called spherical harmonics?
</p>
<!----></div><!--]-->
|
|
|
/post/implicit-layers
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Deep Implicit layers</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-03-03T13:48:00.000Z">March 3, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Deep learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Several new architectures for neural networks, such as Neural ODEs and
deep equlibirum models can be understood as replacing classical layers
that explicitly specify how to compute the output with implicit layers.
These layers describe which conditions the output should specify but
leave the actual computation up to some solver that can be chosen arbitrarily.
This post contains a brief introduction to the main ideas behind implicit layers.
</p>
<!----></div><!--]-->
|
|
|
/post/rl-building-blocks-3
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Building Blocks of RL Part III: Model-based RL</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-02-24T09:41:00.000Z">February 24, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Reinforcement learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Reinforcement Learning consists of a few key building blocks that can be combined to create
many of the well-known algorithms. Framing RL in terms of these building blocks
can give a good overview and better understanding of these algorithms. This is
the conclusion of a series with such an overview, covering model-based RL.
</p>
<!----></div><!--]-->
|
|
|
/post/sparsity-singularities
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">L1 regularization: sparsity through singularities</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-02-17T08:33:00.000Z">February 17, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Machine learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> L1 regularization is famous for leading to sparse optima, in contrast to
L2 regularization. There are several ways of understanding this but I'll
argue that it's really all about one fact: the L1 norm has a singularity
at the origin, while the L2 norm does not. And this is not just true
for L1 and L2 regularization: singularities are always necessary to get sparse weights.
</p>
<!----></div><!--]-->
|
|
|
/post/boring-numbers
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Boring numbers, complexity and Chaitin's incompleteness theorem</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-02-10T15:27:00.000Z">February 10, 2021</time><!--]--></div> <!--[!--><!--]--> <!--[--><div class="text-gray-700"><!----><p> There is a "complexity barrier": a number such that we can't prove
the Kolmogorov complexity of any specific string to be larger than
that. The proof of this astonishing fact is closely related to some
famous paradoxa and we'll use this connection to get a better intuition
for why the complexity barrier exists.
</p>
<!----></div><!--]-->
|
|
|
/post/rl-building-blocks-2
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Building Blocks of RL Part II: Policy Optimization</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-02-03T06:39:00.000Z">February 3, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Reinforcement learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Reinforcement Learning consists of a few key building blocks that can be combined to create
many of the well-known algorithms. Framing RL in terms of these building blocks
can give a good overview and better understanding of these algorithms. This is part 2
of a series with such an overview, covering some policy optimization methods.
</p>
<!----></div><!--]-->
|
|
|
/post/too-much-structure
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Too much structure</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-01-27T07:53:00.000Z">January 27, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Structure</span><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Math</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Proving things for object that have a lot of structure can be harder
than for object with less structure, simply because the tree of possible
proofs is much wider. This is probably why trying to prove a more general
case is sometimes a helpful strategy.
</p>
<!----></div><!--]-->
|
|
|
/post/position-momentum-asymmetry
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Asymmetry between position and momentum in physics</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-01-19T09:52:00.000Z">January 19, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Physics</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> In both classical mechanics and QM, there are transformations between position-based
and momentum-based representations that preserve the dynamical laws. So from
a mathematical perspective, position and momentum seem to play equivalent roles
in physics. But they don't play equivalent roles in our cognition, which is part of
the physical universe -- seemingly a paradox.
</p>
<!----></div><!--]-->
|
|
|
/post/rl-building-blocks-1
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Building Blocks of RL Part I: Value-based methods</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-01-13T15:58:00.000Z">January 13, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Reinforcement learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Reinforcement Learning consists of a few key building blocks that can be combined to create
many of the well-known algorithms. Framing RL in terms of these building blocks
can give a good overview and better understanding of these algorithms. This is part 1
of a series with such an overview, covering value-based methods (mainly in a tabular
setting).
</p>
<!----></div><!--]-->
|
|
|
/post/vae-generative
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">VAEs from a generative perspective</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2021-01-06T13:45:00.000Z">January 6, 2021</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Deep learning</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> Variational autoencoders are usually introduced as a probabilistic extension of autoencoders
with regularization. An alternative view is that the encoder arises naturally as a tool
for efficiently training the decoder. This is the perspective I take in this post, deriving
VAEs without assuming an autoencoder architecture a priori.
</p>
<!----></div><!--]-->
|
|
|
/post/perspectives-on-structure
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Ways to think about structure in mathematics</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2020-12-29T13:03:00.000Z">December 29, 2020</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Structure</span><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Math</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> "Structure" is a concept that keeps popping up when thinking about mathematics
but it's hard to pin down what it is exactly. I discuss several different perspectives
for thinking about it.
</p>
<!----></div><!--]-->
|
|
|
/post/karger-stein
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Trading off speed against the probability of success in the Karger-Stein Algorithm</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2020-12-06T17:00:00.000Z">December 6, 2020</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Graphs</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> The Karger-Stein algorithm is an improvement over Karger's beautiful contraction
algorithm for minimum graph cuts. In this post, I show how it finds the perfect
tradeoff between finding a mincut with high probability and finding it quickly.
In the course of doing so, we will also understand where the somewhat opaque
factor of sqrt(2) comes from.
</p>
<!----></div><!--]-->
|
|
|
/post/discounting-relativistic-universe
|
<h2 class="mb-2 text-2xl font-bold tracking-tight text-gray-900">Discounting in a relativistic universe</h2> <div class="mb-3 flex items-center gap-4 text-sm text-gray-600"><!--[--><time datetime="2020-06-20T10:25:00.000Z">June 20, 2020</time><!--]--></div> <!--[--><div class="mb-3 flex flex-wrap gap-2"><!--[--><span class="rounded-full bg-blue-100 px-3 py-1 text-xs font-medium text-blue-800">Physics</span><!--]--></div><!--]--> <!--[--><div class="text-gray-700"><!----><p> For people who want to discount the future, special relativity creates
some challenges. There are different ways to handle those but none
seem completely satisfactory which may be yet another argument against
discounting pure utilities.
</p>
<!----></div><!--]-->
|
|
