Think your FitBit does a good job counting calories when you’re working up a sweat at the gym? Better think again, because it doesn’t. Neither do six other popular fitness devices, according to a just published study out of Stanford University School of Medicine.
The study did contain a bit of good news. The devices do a decent job keeping track of your heart rate. In fact, that data is so good it would be useful to a cardiologist looking for heart-rate irregularities, known as arrhythmia, in a patient, says Anna Shcherbina, a co-author of the study.
The team tested seven wearable devices – the Apple Watch, Basis Peak, Fitbit Surge, Microsoft Band, Mio Alpha 2, PulseOn, and Samsung Gear S2 – and conducted tests with the help of 60 volunteers who walked, ran, and sat while wearing them.
At the same time, the volunteers were fitted with medical devices known to be extremely accurate. The data collected by the commercial fitness bands was then compared to that of the medical devices. When it came to the heart rate test, the commercial devices had a median error rate of about 5 percent, which is quite good.
But the error rate when measuring calories ranged from 27 percent for the Fibit to 93 percent for the PulseOn. “That’s not very useful,” Shcherbina told me. “People are using these devices to make important decisions about diet and exercise. They should have good information about how well they work.” The researchers said an error rate of under 10 percent would make the devices a better buy for consumers.
The wearable devices have all been heavily promoted and are often used to plan diet and exercise regimens. Although the heart rate information is important, many people who use the fitness bands are very concerned, if not obsessed, about how many calories they’re burning.
To measure the accuracy of the wearables, the researchers had the volunteers don a mask that collects and measures the carbon dioxide they are exhaling, a measurement of calories burned that constitutes “the gold standard” for lab work, Shcherbina says.
To estimate the number of calories burned, the devices use a combination of accelerometer data which measures motion, and data inputted by the users, including age, weight, gender, and other factors. Using a proprietary algorithm, the devices then combine those factors and make an estimate of how many calories were burned in a given period. Measuring heart rate is less complex, so it’s not surprising that the wearables did a decent job tracking it.
The researchers conducted the tests independently, with no input from the device makers, Shcherbina says. A follow up study will evaluate newer versions of the devices and conduct the tests outside of the lab in a real-world environment.
The study was published in the Journal of Personalized Medicine. The lead researcher was Euan Ashley, a professor of medicine at Stanford. Co-authors included Heidi Salisbury, Jeffrey Christle, Trevor Hastie, Matthew Wheeler, and Shcherbina, a graduate student in in biomedical informatics.