Previously, I discussed training a neural net to clean up images. I’m pleased to say that, using more sophisticated techniques, I’ve since achieved much better results. My latest approach is a four layer convolutional network. Sadly, the convolution throws away the sides of the images, so we get a black margin. In any case, compare the results:
Original Image
Attempt 1: 3 fully connected layers
Attempt 2: 4 convolutional layers.
Much better!
Now, all these results are based on images from the same source, Dror Bar-Natan’s Pensieve. Thus, even though I’ve used separate training and test images, they aren’t really independent. It would be much more interesting to look at how well it does when seeing a very different image… The following is Richard Feynman’s blackboard when he died.
Original image (Feynman’s blackboard)
Cleaned version of Feynman’s blackboard
(I find Caltech’s copyrighting an image of dead persons’s blackboard and coating most of the images with “Caltech Archive”, to be in really poor taste and rude.)
The only preprocessing was scaling the image by a factor of two so that things were of similar scale.
I’m trying to write a super easy to use neural net library. We’ll see where this goes.
Tags: blackboards, neural nets
Christopher Olah's Blog 
June 9, 2013 at 17:53 |
that’s quite interesting. I’m curious about the reverse: analyzing silent films, processing to find the grain and streaks, then applying them to modern digial video. Sometimes contemporary digital images/video are too “stiff”, too sharp and clear; wheras film-based media can be interesting and engaging.
June 9, 2013 at 20:46 |
Thanks!
> then applying them to modern digial video.
You shouldn’t need neural nets to acheive this sort of thing. It should be pretty easy to figure out an algorithm to create appropriate noise. 🙂
June 9, 2013 at 19:17 |
I know next to nothing about neural networks, but I always wanted to learn about them, so I have a couple of questions.
First, could you recommend any decent books about neural networks for someone with 0 knowledge about the subject and very mediocre programming skill? (I only know c++ and Pascal, and not especially well either.)
Second, about cleaning up images. This may be a stupid and potentially a little insulting question, but what’s so impressive about that? It seems like it would be trivial to record a Photoshop action that does practically the same thing. (Maybe not, I didn’t actually try…) I fail to see what makes this example of usage so cool, but that’s almost surely because I am missing something important, so don’t take this question the wrong way.
-George G
June 9, 2013 at 20:54 |
Hey George! Good to see you again!
> I know next to nothing about neural networks, but I always wanted to learn about them…
You should! They’re super cool!
> First, could you recommend any decent books…
I’ve had the tremendous advantage of Michael Nielsen’s guidance in learning this, through a reading group he’s running.
He’s presently writing a book, and I’m very confident that it will be awesome when it comes out! In the mean time, I’m told the standard books on the subject are by Haykin and by Bishop.
> Second, about cleaning up images. This may be a stupid and potentially a little insulting question, but what’s so impressive about that? It seems like it would be trivial to record a Photoshop action that does practically the same thing. (Maybe not, I didn’t actually try…) I fail to see what makes this example of usage so cool, but that’s almost surely because I am missing something important, so don’t take this question the wrong way.
Firstly, my primary motivation is that I want a problem to play around with neural nets with.
But I think it’s a lot less trivial than you might think! It took me quite a bit of time to do the training examples by hand in GIMP, and I tried applying some standard computer vision techniques, to very limited success.
I’d be very curious what you come up with by trying to clean an image in photosphop.
Chris
June 9, 2013 at 22:13 |
After 5 minutes in PS I produced this:
http://imgur.com/VtA0HB7
It’s roughly as good as the output of your network. I think the network (especially the second one) is slightly better than me, because the pictures come out neater and cleaner than mine. I know that if I put a little more effort into it, I could match the quality of the network’s output, though.
BUT! I shamelessly cheated. I used a cleaning procedure that is easy for a human, but would be non-trivial to script: I cut the image into 2-3 slices that had roughly the same level of darkness, and then thresholded each of the pieces individually. Cutting into even greater number of pieces would give an even nicer result, but I was lazy.
I suppose it might be possible to program that action. You could make the program blur the image considerably, and then cut it across the level lines of luminosity value. But it would not be completely trivial.
So, I can say that this exercise has given me a greater appreciation of the usefulness of your network. Plus, I realize that you were just messing around and wasn’t trying to code anything groundbreaking, anyway.
I actually have to clean images like that fairly often. It occurred to me that if you turned your code into a tiny, easy to use program with even a rudimentary GUI and made it available for download it might actually make my life easier.
-GG
June 10, 2013 at 00:44 |
Neat!
I’ll probably make this avialable, or something that willl make making things like this easy, or both, once I have things working a bit better.
July 22, 2014 at 22:56 |
Nice post. Do you provide source code for the 4 layer convolutional NN someplace (messy code is also fine)? Thanks