ImageMonkey - Blog

ImageMonkey reaches the 100k milestone

Sun, 09 Feb 2020 13:34:25 +0100

ImageMonkey is a public open source image dataset with powerful APIs and a tight integration of existing machine learning frameworks.

As of last week it seems that we’ve finally cracked the 100k milestone - the ImageMonkey dataset now contains over 100k CC0 licensed images, ~140k labeled objects, ~25k validations and over 100k annotated objects.

It’s now almost three years ago since I started working on ImageMonkey - at that time mainly to scratch my own itch. Back then, I was working on another project, which I thought had the potential to blow up (actually it flopped totally, but that’s a different story) and for that project I was looking for labeled gym equipment datasets to train a neural net on it. As it turned out annotated images are really hard to find - and it gets even harder if you are looking for a public domain dataset.

That’s when I thought to myself: Wouldn’t it be great if there exists such a public dataset? A dataset that’s not owned by a big company where you have to fear that at some point they will shut down the service and everything is lost (yeah, looking at you Yahoo). A dataset created by people for people. A dataset where eyerbody can contribute to. Something that’s easy to use.

There are millions of people in the world running around with their smartphones, snapping pictures of their dogs, cats and what they have for lunch. Wouldn’t that be great image material for a public domain dataset?

When creating ImageMonkey one of the primary goals was always simplicity and ease of use. ImageMonkey wasn’t primarily designed to be used by the “hardcore data labeler” (that’s not to say that that we do not provide tools for those as well), it was rather designed to be used the ordinary man and and woman. In order to accomplish that, the whole process of collecting data was split up in different tasks (“task based approach”).

The traditional worflow of annotating an object usually looks like this:

collect an image
draw a bounding box around the object of interest
label it appropriately.

With ImageMonkey it’s different. Every of those tasks can be done individually. You love taking pictures, but don’t want to tag them with labels? No problem, just upload the pictures without labels - that’s perfectly fine. You enjoy labeling images (I’ve heard that some people find that quite relaxing) but don’t want to draw bounding boxes around the objects? That’s fine, just do the labeling. As you can see, the complex task of annotating objects is broken down in much simpler tasks. The cool thing about that is, that it’s now possible to build APIs around those functionalities and use those APIs to create other applications.

In the Last Lecture (if you haven’t seen it, I highly encourage you to do so), Randy Pausch talks about head fake - a situation in which someone believes they are learning one thing, but are really learning something different. I am a huge fan of this concept and personally I think it not only works for learning something, but also for doing something.

While working on ImageMonkey I tried to make use of head fakes and incorporate them into the software I was building. Then, let’s be honest here: Collecting data for a dataset can be tedious and boring - so why not make it more fun?

With the ImageMonkey Browser Extension and ImageMonkey - The Game I tried exactly that.

The ImageMonkey Browser Extension shows a random validation each time you open a new browser tab.

ImageMonkey - The Game uses gamification concepts to make image collecting more fun.

Powerful API and tight integrations of existing ML frameworks

ImageMonkey’s task based approach made it possible to build a REST API around all its functionality, which not only allows developers to easily export data, but also gives them the possibility to feed back data their own.

Due to the tight integration of existing machine learning frameworks like Tensorflow and Mask RCNN it’s possible to train a neural net for image classification, object detection and object segmentation with just a handful of commands.

e.g If you want to train a cat/dog image classifier via transfer learning on a pre-trained inception-v3 model, all you need to do is:


      docker pull bbernhard/imagemonkey-train:latest-gpu
      docker run --runtime=nvidia -it bbernhard/imagemonkey-train:latest-gpu


      monkey train --labels="cat|dog" --type="image-classification"

The monkey script automatically downloads the necessary data from ImageMonkey, configures the tensorflow pipeline and then starts tensorflow for training. When the training is done, the script spits out a ready-to-go graph.pb file.

What exactly is a public open source image dataset?

Every line of code that was ever written for the ImageMonkey project is available on Github - i.e if you want to spin up your own (private) ImageMonkey instance for collecting data you can easily do that. But it’s not only that. Every item in the ImageMonkey dataset is licensed under the CC0 license. Which basically means you can do the f*ck you want with it - no strings attached. Furthermore, the whole dataset (i.e the images + an obfuscated database dump) gets uploaded to the internet archive on a regular basis (see here for details).

What’s next?

Hosting ImageMonkey (and all it’s services) costs money. At the moment everything’s running on two big Hetzner bare metal servers (one of them being a GPU instance). And while the server costs are still affordable for me, I would really like to see ImageMonkey getting self sustainable.

Apart from that, I’ll be working on:

a better annotation editor
improve the ML pipeline to provide models for download on a regular basis
some general improvements

Thanks

Last but not least I want to thank @dobkeratops - ImageMonkey wouldn’t be where it is right now without your help. Thanks a lot for all your contributions!

Want to contribute?

Any help is really appreciated - whether it is help with the codebase, contributions to the dataset or financial support. Let’s create our public image dataset together!

Want to read more about ImageMonkey?

Subscribe now!

We made some progress

Sat, 28 Oct 2017 14:34:25 +0200

It’s now almost 30 days since the last blog post and we made some really good progress. If you haven’t kept up with us, then here’s a quick summary:

Content Moderation

One of the core principles of ImageMonkey is: Keep it open and accessible for everybody - without the need to sign up. While this is great from the user experience point of view, it brings some serious problems with it:

How can we make sure the uploaded content doesn’t contain any sensitive material?

Manual content moderation is possible, but doesn’t scale. That’s why we are investing some time and energy into mechanisms which (hopefully) can assist us with content moderation. One of those mechanisms is a NSFW detection engine, which pre-classifies each uploaded image with a NSFW score. This for sure can’t replace manual content moderation entirely, but definitely could help us to identify problematic images early in the process.

We started our experiment with a handful more or less randomly selected P**rnhub and Youtube videos, which were split up in frames and fed to a neural net. With now 10k frames in each category (NSFW/SFW) we already get some pretty decent results. Of course it is still far away from being perfect and produces some false positives, but as an initial attempt I am pretty satisfied with the result. The whole thing is still in an alpha phase and was only trained on nudity, but if you want to give it a try: It’s available here.

Dataset Statistics

The dataset statistics tab was completely reworked and now provides more detailed statistical insight into the dataset.

Important Info At ImageMonkey we are taking your privacy seriously, that means we do not store any personal information (like your IP address) in our database. Neither do we send your IP address to some external service to determine your geolocation. Instead, ImageMonkey uses a local geolocation database for the geolocation lookup.

Multi Validation Mode

Another feature that’s still in beta mode, but already available for try out, is the Multivalidation Mode. The Multivalidation Mode is inspired by that reddit comment and allows users to validate multiple images in one go. You can enable it in the validation tab by clicking ‘m’ on your keyboard.

Detecting malicious behavior

While a completely open and publicly modifiable dataset has it’s benefits, it also gives users the possibility to destroy the dataset’s quality by maliciously voting for the wrong label on purpose.We for sure can’t prevent that, but we can try to make it harder. For that reason a history table was introduced, which now keeps track of every single action that’s performed on the dataset and in which way it affected the dataset.

Together with a browser fingerprint we get some pretty good information about the image’s state and how it changes over time (e.q if an image transitions from being classified as an ‘apple’ to being ‘not an apple’ in a very short time, it could be an malicious attempt).

Important info We do not store any personal information (like your IP address or your browser version) of you. The only thing we are storing is an anonymized hash.

Possibility to choose a label on upload

Up to now it wasn’t possible to select a label other than the one that was randomly chosen by the system. As this was quite inflexible and resulted in users pressing the refresh button of their browsers until the desired label would eventually show up, we decided to change that behavior.

Experiments

There are also some ongoing experiments and proof of concepts, which haven’t made it into an ImageMonkey release yet.

Pre-annotation of video frames (based on background subtraction):

Using the grabcut algorithm to select objects

Gamification

Validate an image each time you open a new browser tab

Want to contribute?

Once again, I would like to thank @dobkeratops for all the awesome ideas, the feedback and technical advice. Many thanks for your contribution!

As ImageMonkey and all it’s parts are completely OpenSource, every contribution is highly appreciated. In case you want to check it out, here are some links:

https://github.com/bbernhard/imagemonkey-core

https://github.com/bbernhard/imagemonkey-client

https://github.com/bbernhard/imagemonkey-playground

https://github.com/bbernhard/imagemonkey-chrome-extension

https://github.com/bbernhard/imagemonkey-admin

Want to read more about ImageMonkey?

Subscribe now!

Let's create our own image dataset

Fri, 06 Oct 2017 20:34:25 +0200

It’s now a week since I posted about ImageMonkey on reddit. While the initial rush on ImageMonkey has slowed down and the load on the server stabilized again, it’s time to reflect and plan the next development steps.

Thanks reddit

First of all, I would like to thank the reddit community for all the great feedback and suggestions. It was my very first post on reddit and I wasn’t expecting much, so I was really blown away to see that much load on the server. Hell, you guys even managed to take the server down temporarily!

As it turned out there was a bug in the database connection pool handling which caused the number of open connections to grow rapidly until the point there were no connections left anymore. I am really glad that the restart-the-application-every-10-minutes-cronjob-hack “worked”, otherwise I would have bitten myself in the ass for not doing any load testing beforehand.

To give you some figures: In the first two days after the reddit post we got an incredible amount of 3002 image validations, 696 annotations and 20 donations.

Thanks, reddit!

Goals

How do you want to convince people to label images?

I think that’s the question it all comes down to in the end. I am pretty sure there aren’t many people out there that would use the words ‘labeling images’ and ‘having fun’ in one sentence. So in order to make the whole project work, we need to make the process of labeling images a painless and smooth as possible.

The following is a list of things that I would like to accomplish with ImageMonkey:

KISS As I would assume there are not many people out there that actually have fun in labeling images, I think it’s very important to make the process of labeling images as painless and fun as possible. If we make the labeling process too complicated people probably won’t use it.

It would for sure be awesome to have a really fine granual labeled dataset, but in my opinion it would already be a huge accomplishment to identify some simple things (apple, banana, smartphone) reliable.
Contributer friendly Another point that’s really important for me is the possibility for users to contribute. As most of the backend stuff is written in Golang and not everybody knows Golang we should try to move as much “logic” a possible to config files. Scenes, Objects and Labels should be defined in (JSON like?) config files, so that users can easily create a pull request if they want to add something.
Tight feedback loop I think it could be beneficial to have some sort of online playground where users can upload pictures and the system tries to classify them. Over time we should see that the model gets better and better in classifiying images. Such a feedback loop could motivate users to actually invest time in improving the dataset’s quality, as they see a direct impact.
Open for contribution but closed for malicious modification Everybody should be able to contribute (preferable without registration). Malicious attempts should be (automatically) detected and (if possible) prevented. If we can’t detect it, we should at least be able to revert the malicious changes.
Easy wins/fail fast As this is just a hobby project, we should focus on the “easy wins” first and try to rule out ideas, that need huge implementation effort but won’t get us the desired results. I am usually not a fan of mockups, but I think here it could really save us some time in evaluating ideas.
“Do one thing philosophy” I am a big fan of Unix’ “Do one thing and do it well philosophy”. I think with such a complex project it’s pretty easy to “overdo things” in order to cover every possible use case. So it might be a good idea to figure out what we want in the end and break down the functionality in tiny apps/extensions with each of them fulfilling one specific purpose. I am afraid that if we put all the functionality in one application, that users will feel overwhelmed and don’t know what they need to do. What I want to avoid is, that we make the base application so complicated and bloated that only “hardcore users” want to use it.
Aim for fast results If we want to get some traction and recoginition we probably also need to attract users that are not technicians. So it could be beneficial to concentrate on some really “basic” image recognition first (with keeping the big picture in mind). For a normal user it’s probably already pretty impresive if they can upload a picture of a dog and the dog gets marked. They most probably don’t care if it’s a labrador or a shepherd.

What’s next?

There are a lot of things going on in our issues tracker. The following list should give you an overview about the most important actions we want to tackle.

Online Playground As already mentioned, I think this one could add some playful character to the site and motivate people to participate. I recently added a proof of concept (you can try it out here). The model uses inception-v3 as base and was re-trained with the dog, cat and apple pictures in our dataset. The idea is to daily (at least as long as our dataset is so small) re-train the model to always serve the latest build. I haven’t yet tried to performance test the implementation, so at that point it’s really more a proof-of-concept ;)
Report Button Just to be on the save side (with the law) we should add a button where users can report inappropriate content.
(Mockup-based) Evaluation @dobkeratops added some really nice ideas that we should consider implementing. But as most of them probably require some fundamental changes in the UI, the database layout and our config file structure, I think it could make sense to evaluate those ideas by either by implementing a proof-of-concept or some (UI) Mockups.
Statistics Rich statistics could help guide people to do labelling that increases the breadth of the database.
Make project more contributer friendly Currently the README is pretty detailed, which could scare off contributers. I think it could make sense to invest some time in making the “onboarding” more smoothly (e.q automate the manual steps; create a Dockerfile…)
Remove manual unlocking Currently every uploaded donation gets unlocked manually in order to make sure that the uploaded content doesn’t contain any sensitive material. While this is a good idea in ensuring that we don’t accidentally become a hoster of sensitive material (nudity, racist content..) it simply doesn’t scale. In order to keep ImageMonkey open and accessible for everybody we should start investigating early in methods to prevent malicious attempts (i.e nudity detection, blocking users with malicious intentions..)
Logo Every project needs a logo, right?

Want to contribute?

First of all, I would like to thank @dobkeratops for all the awesome ideas and suggestions - very much appreciated! It’s always great to see such passion and contribution in Open Source projects - that’s what Open Source is all about.

As ImageMonkey and all it’s parts are completely OpenSource, every contribution is highly appreciated. In case you want to check it out, here are some links:

https://github.com/bbernhard/imagemonkey-core

https://github.com/bbernhard/imagemonkey-client

https://github.com/bbernhard/imagemonkey-playground

https://github.com/bbernhard/imagemonkey-chrome-extension

https://github.com/bbernhard/imagemonkey-admin