Droplets and deep learning
Artificial intelligence and machine learning are becoming a part of our everyday life in the form of self-driving cars or smart (digital) assistants. Here, we used artificial intelligence for detecting droplets, recorded with high-speed imaging, to speed up the entire analysis process.
After all, getting a computer to recognize a circle is a decades old task, which many people have worked on. However, when there are a lot of circles and they are all slightly different than each other, with very various backgrounds, the task for the computer and the AI becomes quite tricky.
We think that the use of artificial intelligence in the study of mini-droplets can quickly provide valuable knowledge in various fields of research.
So small you can barely see it
Everything is uglier up close. Except microfluidics! This scientific and technological discipline utilizes its small dimensions in a variety of applications, ranging from cell studies in biology to investigating chemical reactions in flow. By small, we mean less than 1 mm! Have a look at your fingreprints for a comparison, many microfluidic channels are not larger than the ridges on your fingers. One of the subdisciplines of microfluidics is called droplet microfluidics, which essentially is based on generating discrete droplets of one fluid in another.
The study of mixtures and droplets
At the Ugelstad Laboratory, Department of Chemical Engineering, we work a lot with dispersions (one phase mixed in another, for example milk, blood or paint). We study them in different conditions to see how they behave, if they are stable over time or rather quickly separate into individual phases, like salad dressing left alone in a bottle for a long period. Recently, we started to use droplet microfluidics as an alternative to more traditional methods. It saves us a lot of time and reduces the amount of sometimes dangerous chemicals, as well as allows to precisely control the conditions of the experiment. What exactly are we studying? Droplets! We want to know everything about them: Where are they in the channel? How big they are? Are they all of the same size? Are they moving? If so, how fast? This information enables us to see, for example, 1 if the droplets are stable in different conditions, which we can later relate to a certain process or product.
Fast (and furious) droplets
There are few things you should know about microfluidics. First of all, everything is happening in microscale, so we need microscopes to visualize our droplets. Secondly, things often occur quite fast in the channels and for that reason we use high-speed cameras. How fast are they? Modern mobile phones can record videos up to 100 frames per second. Multiply this by a number between 10 and 100, and you will get the average recording speed for droplet microfluidics. This is also connected to the last point: the generated amount of images to process. Thousands of images needed to be analyzed. Sounds tough? Yes, it actually is.
Tedious work with thousands of frames
Thankfully, many image analysis techniques these days support batch analysis, meaning you can load all of your images, process and analyze them as one set. Some tools are simpler, for example ImageJ, where many things have to be done manually. Some software, for instance Matlab, is more advanced and allows for a higher degree of automation. Regardless of which tool you use, the flow is often similar: load images; pre-process them; convert to a simpler format, for instance binary images; and finally detect some features. If it sounds tedious and lengthy, you have the right hunch. On standard lab computers, image analysis can take as long as the experiment. Which is WAY too long.
Artificial intelligence to the rescue!
So what can we do about it? Or rather, what WE have done about it. At Ugelstad Lab, we have started to implement a certain branch of artificial intelligence to remove the human factor from this time-consuming analysis. To be more precise, we are using several types of (deep) neural networks (DNNs) to speed things up. Essentially, it is a machine learning approach to analyze data.
Main thing you need to know is that DNNs work somewhat like a human brain – they very quickly process huge amount of information that is presented to them, for example in the form of images. You also ”teach” DNNs what should be recognized by ”showing” images of the objects. In this way you create a database, so called weights, that are later used to detect specific features.
Is there a banana in the picture?
Let’s illustrate this with an example. Here are a few pictures of the authors of this post, holding everyday objects.
As you can see, even though the DNN does reasonably well with many of the objects in the images, such as people, computer equipment, cups or bottles, it still fails to detect some objects (e.g. a banana in two pictures on the left) or it misdetects some objects (”remote” and ”chair” in the right image). In other words, if the training of the networks is insufficient, the results will also be far from the truth. In practical application, one of the most advanced uses of DNNs is for self-driving cars. There, their ability to distinguish between a stationary tree and a moving person is crucial, from the safety point of view.
Teaching AI to detect droplets
Ok, but what does this have to do with our microscopic droplets? Well, it turns out you can also train CNNs for detecting and tracking droplets! We have prepared a bunch of ”training” images, which were examples of the kinds of objects we wanted the neural network to identify. In this way, the DNN learned how to distinguish between certain droplets or specs of dust that might have gotten into the system. In the end, our neural network can give you an image with all of the droplets perfectly identified with their unique sizes and positions along with the neural network’s confidence in its prediction. We take this information and process it further.
100 images per second
And one more thing – all of this is done SUPER fast. Analysis of one image takes around 10 ms. We can therefore process approx. 100 images per second, which is a fantastic result for people having thousands of frames to analyze per one recording.
”Every step you take, I’ll be watching you…”
Our next goal is to add tracking to this toolbox. We want the algorithm to assign each individual droplet a label and ”follow” it. In that way we can easily count all the droplets and learn about their behaviour (which path did it take, how fast did it go, etc.). While it is not so important for straight channels, this information can have a large impact when more complex geometries are used, for example a combined network of channels, imitating porous medium. With this, we can identify the preferential pathways in such structures.
A sea of possibilities
Overall, we are quite satisfied with the results obtained so far. We have also started to reach out to other research groups that work in biology and physics for example, that deal with dispersed systems, as the method is much faster, often more precise and requires less user involvement. We hope that the neural network we trained can eventually be used in a very wide variety of applications, such as monitoring exactly how much oil is left in a microscopic pore or keeping track of various biological markers in a single cell.
This blog post was written by
