It was not good enough to simply turn on and off our coffee pot. We wanted to indicate the status of the pot. Questions like “is it empty?” “is it brewing"?” “should I wait for a new pot?” all had to be answered.
To answer these questions we needed to be able to detect the water and coffee levels of our coffee pot. We discussed using water level detectors of all sorts and linking them to the Arduino. Since I’m seeking my masters degree in image processing and Silverlight 4 enables the use of a webcam, I decided to try and build a passive detection system using a webcam.
Silverlight 4 and Webcams
Connecting to a webcam is quite easy in Silverlight 4. This can be accomplished inside or outside the browser. Most examples on the web use a VideoBrush to set the background of any object in the system to be a live webcam feed. We needed to actually process the image so we used the AsyncCaptureImage method which returns a WriteableBitmap which we can then use to process our image. At the end of processing the image AsyncCaptureImage is simply called again. Surprisingly with all of our processing we were still able to achieve frame rates of 30 frames per second.
private void connect_Click(object sender, RoutedEventArgs e)
{
webcam = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
if (CaptureDeviceConfiguration.RequestDeviceAccess())
{
captureSource = new CaptureSource();
captureSource.VideoCaptureDevice = webcam;
captureSource.Start();
captureSource.AsyncCaptureImage(snap);
}
}
public void snap(WriteableBitmap b)
{
//processing code here
this.img.Source = b;
captureSource.AsyncCaptureImage(snap);
}
Coffee level sensing
To sense the level of the coffee we needed first, to decide where the camera should look for information. We did this by constructing a bounding box. In our application the box could be easily configured so if the pot had been moved, we could simply realign the camera or configure the software.
Upon sampling an image, a simple for-loop runs over the bounds set by the bounding box. To track our coffee level we had several options, but since we are only calculating coffee we chose the simplest and less accurate method. Our method simply counts the pixels that are black and divides that by the total amount of pixels to give us a percentage of coffee. To decide whether something is “black” we have a configurable threshold.
List<int> values = new List<int>();
int coffee = 0;
for (int x = (int)cx1.Value; x < cx2.Value; x++)
{
for (int y = (int)cy1.Value; y < cy2.Value; y++)
{
Color c = b.GetPixel(x, y);
values.Add(val);
if (c.B < this.cthreshhold.Value && c.R < cthreshhold.Value && c.G < cthreshhold.Value)
{
c.B = c.R = c.G = 255;
coffee++;
}
else { c.B = c.R = c.G = 0; }
b.SetPixel(x, y, c);
}
}
We ended up using a special carafe (pot) to make this system work. Most have a steel reflective band around the bottom. We lucked out and bought a universal carafe that did not have the steel band around it. If you are not able to do this you could simply construct 2 bounding boxes and add some logic that if the top box has coffee in it add x amount to the results.
Presence detection
A problem we ran into is that when someone pours some coffee our system registers an empty pot which in fact is wrong. So we needed to detect the presence of the pot. We simply adapted some code from the coffee level sensing to determine if the pot was present. If we saw more black then white pixels (the top of the pot is black) then the pot is present and we can continue calculating the amount of coffee that is in our pot.
List<int> pvalues = new List<int>();
int presense = 0;
for (int x = (int)px1.Value; x < px2.Value; x++)
{
for (int y = (int)py1.Value; y < py2.Value; y++)
{
Color c = b.GetPixel(x, y);
if (c.B < pthreshhold.Value && c.R < pthreshhold.Value && c.G < pthreshhold.Value)
{
c.B = c.R = c.G = 255; presense++;
}
else
{
c.B = c.R = c.G = 0;
}
b.SetPixel(x, y, c);
pvalues.Add(c.B);
}
}
Water level detecting
Detecting the water level turned out to be quite a challenge. We chose a coffee maker that had a red ball indicating the water level. One of the most common and frustrating things in image processing is to assume since your eye can see it, so can the computer. Our final solution was solved in a multistep process. The first step was determining if a pixel was red. This became a problem because we simply could not look at the red channel as white registers on the red channel as well. Instead we decided that if a pixel’s red value was greater than the threshold and the pixels blue and green values were below the threshold then the pixel was predominantly red. We actually need to know where the ball is rather than the count of red pixels. To do this we recorded the height (y axis) of each pixel. We then took the median of those values to determine where the most red pixels were located.
List<int> red = new List<int>();
for (int x = (int)wx1.Value; x < wx2.Value; x++)
{
for (int y = (int)wy1.Value; y < wy2.Value; y++)
{
Color c = b.GetPixel(x, y);
if (wthreshhold.Value > c.B && wthreshhold.Value > c.G && c.R > wthreshhold.Value)
{
c.R = 255;
c.B = 0;
c.G = 0;
red.Add(y);
}
else
{
c.B = 255;
c.G = 255;
c.R = 255;
}
b.SetPixel(x, y, c);
}
}
There are several improvements to this that we did not get time to implement. Mainly, averaging the bounding box for multiple frames so that static red dots would be minimized. We could have also chosen the Y value that is most frequent to calculate the water level. In general this was the most fussy part of our project, but it did work.
Integrating into RiteTrack 4
To get this working in RiteTrack we simply wrote code in the “User” class in the code behind. We processed the image at near 30 frames per second but we only updated the database with the current status of the pot every 30 seconds. We constructed two tables to track the coffee status. The first was the current status for storing the image and the current status. The other was a history table. Using some clever scripting our application only needed to write to the current status table, the archiving method was handled at the server level using scripting.
public class dtCoffeeStatusCurrent : dtCoffeeStatusCurrentBase
{
public override void Update()
{
base.Update();
dtCoffeeStatusHx nhx = new dtCoffeeStatusHx() { CoffeeLevel = this.CoffeeLevel, IsBrewing = this.isBrewing, IsOn = this.isOn, StatusDate = DateTime.Now, WaterLevel = this.WaterLevel };
nhx.Update();
}
}
Conclusion
CoffeeTrack running on RiteTrack 4 provided a very convenient and easy platform to build our coffee sensing system on top of. Looks like I get to keep my image processing license and we get to reap the rewards of being able to interface RiteTrack with webcams and more thanks to Silverlight 4.