Neural Networks, Manifolds, and Topology

Recently, there’s been a great deal of excitement and interest in deep neural networks because they’ve achieved breakthrough results in areas such as computer vision.

However, there remain a number of concerns about them. One is that it can be quite challenging to understand what a neural network is really doing. If one trains it well, it achieves high quality results, but it is challenging to understand how it is doing so. If the network fails, it is hard to understand what went wrong.

While it is challenging to understand the behavior of deep neural networks in general, it turns out to be much easier to explore low-dimensional deep neural networks – networks that only have a few neurons in each layer. In fact, we can create visualizations to completely understand the behavior and training of such networks. This perspective will allow us to gain deeper intuition about the behavior of neural networks and observe a connection linking neural networks to an area of mathematics called topology.

A number of interesting things follow from this, including fundamental lower-bounds on the complexity of a neural network capable of classifying certain datasets.

Read more on my new blog!

Advertisement

Topology Notes

I’ve been talking about writing a topology textbook introductory notes on topology for years. Basically since I wrote my Rethinking Topology (or a Personal Topologodicy) post 2 years ago — it’s hard to believe it’s been that long!

In any case, I finally started writing it. I’ve done a mild review of existing introductions to general topology (ie. I skimmed through the first few chapters of a dozen topology textbooks), so I feel somewhat comfortable contrasting my work to existing literature. It’s quite a different approach.

Topological Anatomy: Closure, Interior, and Boundary

I initially develop topology based on closures and adherant points. Kuratowski’s closure axioms are then built up with natural explanations. Emphasis is given to the variety of possible definitions (along the lines of Lakatos et al’s Proofs and Refutations) and exercises encourage the reader to explore the variety of possible definitions. I attempt to justify the axiomatic approach in a manner similar to Pinter’s wonderful A Book of Abstract Algebra, though I may fall very short. From here, we build intuition for closure, boundary, and interior with some diagrams and proofs of identities. Finally, we wrap up the first chapter with a visual interpretation of the closure axioms.

The indiscrete closure operator on {1,2}

(You can find the most recent version of the book on github.)

Rethinking Topology (or a Personal Topologodicy)

(This document was typeset in unicode. This may cause problems for some people. A PDF is available as an alternative for them.)

When I was originally introduced to topology, I simply accepted most of its properties as generalizations of ℝⁿ. I didn’t give it any serious thought until about a month ago when I read an excellent thread on math overflow about it. Since then, its been one of the things I often find myself thinking about when I’m trying to fall asleep. Given the amount of thought I’ve put into it, and the fact that I feel I should be answer questions like this about topology, given that it’s one of the areas of math I spend a lot of time on, I thought I’d write up my thoughts. They lent themselves well to being written in the form of an introduction to topology, so that’s what I did.

(After finishing this essay I decided to reread the MO thread. The first comment — not answer, a comment — mentions the Kuratowski closure axioms and closure axioms sounded like one might call what I came up with. Sure enough, they’re the exact same, down to the ordering. Are all attempts to make mathematical contribution’s this frustrating? I’m posting this because of the amount of work I put in, but there’s nothing new here.)

Consider 1 with respect to [0,1). It isn’t part of the set, but in a sort of intuitive sense it almost is. And knowing which points are `almost in’ a set gives us lots of information, for example notions of boundaries and connectedness.  Topology is based on us formalizing this notion of `almost in’ and once we formalize it, we can consider non-standard notions of being `almost in’ or apply these ideas to spaces that we don’t typically associate them with.

Separation Axiom Visualisations

A couple days a go, I saw some nice visualisations of separation axioms on Wikipedia. Unfortunately, it wasn’t a full set. Well, here is a full set (well, T0, T1, T2, T2 1/2, T3, T4, T5):

The Mandelbrot Set: Compact?

Several weeks ago, I read something on Wikipedia that shocked me: “The Mandelbrot set is a compact set.”

At first I didn’t believe it. How could the Mandelbrot set, in its infinite complexity, be compact?

Compactness Graph

Here’s the first revision of a graph of the implications of topological properties that I made:

(Click on it to see a better version!)

It’s mostly based off the stuff in Counterexamples In Topology (great book, BTW) but I did add some stuff (like Baire!) and merged/reorganised it. Diagram was made by Graphviz.

Most of the implications are trivial, but there are a few I haven’t prooved yet (most of the ones involving seperation axioms).

Limits and the Infinitesimal Number

I’ve been thinking about the infinitesimal number, , recently. In particular, that one could use it to evaluate limits.

What is a limit, really? I’ve been reading some topology recently and I think that it really is a function that returns an accumulation point (hint: these are alternatively known as limit points). More specifically, I believe that is an attempt to find a value such that is a limit point of the graph of the the function .

But I’ve digressed since the simpler, “It’s the value as we approach the point” is perhaps more useful to us…

Consider . How is this defferent from ? The difference is that we’re looking for the hypothetical value that the function is becoming (also the value of the point which any open set containing it intersected with the graph is not null), not its value when it is infinitly close. Consider : the difference is versus . So, we need to get rid of the infintesimal difference. Let represent the rounding of anumber to the nearest real number. Then,

Does this have any applications? I beleive it may provide a more elegant way to present Calculus.

Why can’t I take the first principles defenition of a derivative and:

? Because the defenition of a limit would give us zero, that being the closest real number. But instead, we could just say that a value is close to delta and use that. Not useing because I see no reason to use a random symbol when there is a logical one.

And we have a differential form, let’s add vectors (for more dimensions):

and for yet more clarity:

which is far less cumborsome than

And has almost identical properties (I suppose that since it is it will be forward facing, eg, instead of undefined).

Just some random thoughts.