Fanfiction, Graphs, and PageRank

On a website called fanfiction.net, users write millions of stories about their favorite stories. They have diverse opinions about them. They love some stories, and hate others. The opinions are noisy, and it’s hard to see the big picture.

With tools from mathematics and some helpful software, however, we can visualize the underlying structure.

Graph of Harry Potter Fanfiction, colored by ship

 

In this post, we will visualize the Harry Potter, Naruto and Twilight fandoms on fanfiction.net. We will also use Google’s PageRank algorithm to rank stories, and perform collaborative filtering to make story recommendations to top fanfiction.net users.

 

Read the post on my new blog!

Advertisement

Visualizing Functions On Groups

Functions of the form or , where is a group, arise in lots of contexts.

One very natural way this can happen is to have a probability distribution on a group, . The probability density of group elements is a function .

Another way this can happen is if you have some function and has a natural action on ‘s domain – if you care about the values takes at a particular point , you are led to consider functions of the form . For a specific example, the intensity of a particular pixel, , in a square gray-scale image, , subject to flips and rotations, can be considered as a function .

Basic Visualization

Recall that we can visualize finitely generated groups by drawing Cayley Diagrams. (There’s a nice book, Visual Group Theory by Nathan Carter, that teaches a lot of basic group theory from the perspective of Cayley Diagrams.)

The natural way to visualize functions on groups is to picture them as taking values on the nodes of the Cayley Diagrams. One way to do this is by coloring the nodes. In the following visualization of a real-valued function on , dark colors represent a value being close to zero and light colors close to one.

Order Statistics

What is the distribution of the maximum of random variables? What started out a utilitarian question in my exploration of some generalized versions of the secretary problem turns out to be quite a deep topic.

(Note that I have little background in probability and statistics. Please forgive (and inform me of, so I can fix!) errors of notation or language. Or general confusion on my part.)

Topology Notes

I’ve been talking about writing a topology textbook introductory notes on topology for years. Basically since I wrote my Rethinking Topology (or a Personal Topologodicy) post 2 years ago — it’s hard to believe it’s been that long!

In any case, I finally started writing it. I’ve done a mild review of existing introductions to general topology (ie. I skimmed through the first few chapters of a dozen topology textbooks), so I feel somewhat comfortable contrasting my work to existing literature. It’s quite a different approach.

Topological Anatomy: Closure, Interior, and Boundary

I initially develop topology based on closures and adherant points. Kuratowski’s closure axioms are then built up with natural explanations. Emphasis is given to the variety of possible definitions (along the lines of Lakatos et al’s Proofs and Refutations) and exercises encourage the reader to explore the variety of possible definitions. I attempt to justify the axiomatic approach in a manner similar to Pinter’s wonderful A Book of Abstract Algebra, though I may fall very short. From here, we build intuition for closure, boundary, and interior with some diagrams and proofs of identities. Finally, we wrap up the first chapter with a visual interpretation of the closure axioms.

The indiscrete closure operator on {1,2}

(You can find the most recent version of the book on github.)

Singularity Summit Talk

As readers of this blog are probably aware, I’m a rather big fan of multiplicative calculus, an obscure mathematical tool that is very useful in reasoning about quasi-exponential trends. And I’m rather sad that I get few occasions to apply it. So, it should come as no surprise that when I was given an opportunity to speak at Singularity Summit last fall, a conference largely concerned with exponential trends in technology, I decided to try to persuade people of the utility of multiplicative calculus and the value of mathematical abstractions.

Now, there was a bit of confusion because a lot of people thought that I was going to be speaking about my work — that’s what all the other Thiel Fellows did — but I think this was much more valuable. The things I do are generally of very domain specific interest and don’t have super deep implications world-changing for the future of technology, just some minor, incremental improvements. In any case, you can watch me and the other fellows talk.

My public speaking skills aren’t the greatest, and it wasn’t the best talk I’ve given… but it was pretty cool when Vernor Vinge came up to me and told me that he liked my talk! (And I think Ray Kurzweil may have said so as well, but I wasn’t sure if it was him.) And then I and the other fellows had dinner with Eliezer Yudkowsky (and lots of other awesome people, but he sat beside us). So it was a crazy awesome experience in general!

Monads For The Terrified

It seems like everyone writes a monad tutorial in the Haskell community… Well, here’s mine.

People learning Haskell generally seem to get tripped up about monads. There are good reasons for this.

1. Monads are, really, needed to write a serious Haskell program. As a consequence, people generally try to learn them fairly shortly after beginning to use Haskell.
2. Monads are defined in Haskell and have a very short definition. As such, it is very tempting to not just teach someone how to use monads, but to also try to teach them the underlying definition. The definition involves a bunch of rather advanced Haskell. In combination with (1), this is a big problem.
3. Monads are a very abstract idea. I’m not really sure that the motivation can be properly grasped by anything other than using them.
4. Monads are generally interweaved with the idea of an IO object.

This introduction attempts to avoid some of these problems. In particular, it is only going to teach you how to use monads and not what they are. This avoids (2) and, to a lesser extent, (3). I’ve tested this approach on a number of people in the last few months, and it seems to be quite effective. That said, I understand that some people may prefer an introduction focusing on the what and why, in which case you may wish to look Chris Smith’s Why Do Monad’s Matter?.

Some day, you will need to learn more. But, if you find you understand the contents of this tutorial, I’d encourage you to wait and play around with monads for a while. Then, when the time comes to learn the in full, they’ll be natural and obvious ideas. We can talk more about that later. For now: onwards!

(Read more on github)

Math Notation Experiment “Paper”

A while back, I wrote a post on some unusual math notation I was playing with. I actually took it much further than that and drafted up a paper full of different ideas over the following months. At the time, I was hoping it might be publishable but then got discouraged and never did anything with it… Except show it to people in person and promise I’d put it online soon. In any event, it’s one of those things I’ve been meaning to put up for forever.

So here it is. I’m particularly proud of the quantifier stuff and the numerals.

When one considers how complicated the ideas mathematical notation must represent are, it clearly does quite a good job. Despite this, the author believes that any claim that modern mathematical  notation is the best should be met with extreme scepticism.

Mathematical notation is a natural language: no one sat down and constructed it, but rather it formed gradually by people making changes that are adopted. Most changes are not adopted; whether they are depends on a variety of factors including: mathematical utility, ease of adoption (the average individual doesn’t want to spend hours learning), dissemination, and shear dumb luck. The first two of these qualities are associated to real properties in notation, forming the necessary selective pressure for evolution to occur.

Evolution is a blind watchmaker: the world around us is filled with examples of the stupidity of biological evolution (the classic example being halibut). Similarly, evolution is also a blind language and notation designer. In particular, it is held back by a strong selective force against change, since people would need to adopt it, and so evolution doesn’t effectively explore the full notation-space. This staticism means that even the most outrageous notations remain unchallenged by virtue of age.

Read More…

Please keep in mind that I wrote this (except for the redaction of some sillier sections and some minor improvements) three years ago, when I was quite a bit younger, and it isn’t representative of my present abilities. That said, I do still think that the fundamental idea, that notation is important and should be explored, is very true.

(I’ve also put the paper on github, for those who are interested.)

HaskSymb: An Experiment in Haskell Symbolic Algebra

One interesting detour I took this last month was writing a toy Haskell Symbolic Algebra library, HaskSymb. It takes advantage of Quasi-Quoters and View Patterns to do awesome math pattern matching and enable one to write beautiful code.

An example of its use:

> -- Let's make some variables!
> let (a,b)  = (V "a", V "b")
> -- Basic Expression manipulation
> (a+1)^3
(a + 1)³
> expand $ (a+1)^3
a³ + a² + a² + a² + a + a + a + 1
> collectTerms $ expand $ (a+1)^3
a³ + 3a² + 3a + 1

The interesting part, however, is the code for expand, collectTerms and friends. It’s extremely pretty. For example:

expand [m| a+b |] = expand a + expand b
expand [m|a*(b+c)|] = expand (a*b) + expand (a*c)
expand a = a

Read more on github. You can also find some discussion on the Haskell reddit.

Fractal Holiday/Solstice/Christmas Cards!

It’s that time of the year again: I’m making my fractal Holiday cards! I slacked last year and only made a few, but I’m back at it.

Previous years, I’d made up fractals for my cards. This year I just explored the Mandelbrot set and found cool looking regions. Since I obsessed over my choice of pictures, it took me a while to make them  — I’ve made about twenty so far, and still need to make more!

Manipulation of Implicit Functions (With an Eye on CAD)

Most people are familiar with the equation for a circle, :

I prefer to think of it as the curve where is zero — the locations where the Euclidean distance from is one.

Understanding Pascal’s Triangle

How many ways are there to arrange “ABC“? ABC, ACB, BAC, BCA, CBA, CAB. So 6 different ways. But could we have determined it in a way that didn’t involve us listing all the cases? The answer is yes, and welcome to the wonderful world of combinatorics.

YAKC: Differential (One) Forms

In my previous post in this series, I introduced the idea of a derivative, and we realized a number of rules regarding them. In this post, we’re going to give some thought as to what derivatives are, look at them from from a rather different perspective, and realize several more rules regarding them.

You Already Know Calculus: Derivatives

Calculus is made to be a whole big hoopla in high school and first year university. It’s supposedly the hardest math class in high school, notoriously complicated and unintuitive.

I blame this on bad education, not just because I’ve observed so much bad math education at the high school level, but because I can’t see any other way anyone could conclude that calculus is difficult. Because I believe that everyone already knows calculus. They just never connect what they already know to the symbols they’re manipulating in math class.

(I’m writing this series of posts — yes, this is only the first of a number of posts! — in a didactic form, but I think they may be of interest to mature mathematicians. While they are (will be?)  informal and unrigorous, they provide intuitive reasons for why everything in basic calculus is true. The results in it are certainly valuable to me: they’re the result of me spending some time trying to answer ‘why’ everything is true at a compelling level. If nothing else, they may be useful in teaching calculus.)

So what is calculus? It’s the mathematical study of rates of change, nothing more and nothing less. We call the rate of change of a function its derivative.

Surface-Oriented CAD, Math, & Telescopes

surfcad is a python proof-of-concept CAD library I’ve been writing for what I call “surface-oriented CAD”.

The idea came to me when I realized that some of things I’d been trying to make in openscad, a CSG based CAD program, would have been much easier to do in the software I’d written for making models of mathematical objects (eg. this model of a solution the diffusion equation). In this software, surface were constructed based on mathematical functions. One had to think to make sure that their object was closed, but it was possible to describe things that would have been prohibitively difficult in a CAD program like openscad.

Rethinking Topology (or a Personal Topologodicy)

(This document was typeset in unicode. This may cause problems for some people. A PDF is available as an alternative for them.)

When I was originally introduced to topology, I simply accepted most of its properties as generalizations of ℝⁿ. I didn’t give it any serious thought until about a month ago when I read an excellent thread on math overflow about it. Since then, its been one of the things I often find myself thinking about when I’m trying to fall asleep. Given the amount of thought I’ve put into it, and the fact that I feel I should be answer questions like this about topology, given that it’s one of the areas of math I spend a lot of time on, I thought I’d write up my thoughts. They lent themselves well to being written in the form of an introduction to topology, so that’s what I did.

(After finishing this essay I decided to reread the MO thread. The first comment — not answer, a comment — mentions the Kuratowski closure axioms and closure axioms sounded like one might call what I came up with. Sure enough, they’re the exact same, down to the ordering. Are all attempts to make mathematical contribution’s this frustrating? I’m posting this because of the amount of work I put in, but there’s nothing new here.)

Consider 1 with respect to [0,1). It isn’t part of the set, but in a sort of intuitive sense it almost is. And knowing which points are `almost in’ a set gives us lots of information, for example notions of boundaries and connectedness.  Topology is based on us formalizing this notion of `almost in’ and once we formalize it, we can consider non-standard notions of being `almost in’ or apply these ideas to spaces that we don’t typically associate them with.