University of Utah
School of Computing


Measuring Fingertip Force by Imaging the Fingernail

The goal of this project is to measure fingertip forces by using a multi-camera system to image coloration changes in the fingernail and surrounding skin. Key advantages of this transduction method are that objects do not have to be instrumented with force sensors, and relatively unconstrained grasps can be measured.

Grasping studies to date have required the design of special test objects using key-like touch pads instrumented with force sensors, but the new technique now allows the characterization of grasp forces for ordinary objects of arbitrary shape. It also allows the measurement of fingertip forces when regrasping and manipulating objects, where the contact points change in an unpredetermined manner.

Key steps are the following.

2D to 3D Image Registration [Haptics06] [CVPR07][TBME08]

Fingernail locations will vary depending on the grasp and on the relative locations of the cameras. As a particular fingernail is imaged, it will be necessary to correspond points in the image to a reference image so that calibration results can be applied.

Fingernails are curved, so the analysis of the coloration changes has to be carried out in 3D. Currently there are many registration methods relying on naturally present features, such as edges, corners or texture. However, the fingernail is relative featureless and some features, related to color, change with force. Pairing of those features with neighborhood correlation gives a great number of outliers (wrong pairs). We currently use RANSAC to find the inlier pairs and use those pairs to calculate the tranform matrix. The 3D model of the fingernail is obtained with a Bumblebee Stereo camera from Point Grey Research.

Bunblebee (Point Grey) Flea (Point Grey) Lens (Tomron)

Elastic Warping [TRO09][WHC07][CVPR07]

To compare fingernails from different subjects, finger images are registered to a finger atlas with simplified reference shapes. The fingernail is modeled as a disk with 70 pixel radius. The surrounding skin region is composed of a ring and an isosceles trapezoid. Ideally, the fingernails are mapped to the fingernail atlas, and the surrounding skin is mapped to the surrounding skin atlas. To do so, the fingernails in the reference images need to be segmented from the surrounding skin.

We use a Canny edge filter to automatically detect the boundary of the fingernail. However, because of the broken skin around the fingernail, the automatically detected boundary is noisy and can rarely form a smooth curve. We use cubic B-splines to fit the edges and achieve a close-loop contour. The region inside of the contour is the segmented nail.

A finger Canny edge Bspline Boundary Segmented nail

The nail and the surrounding skin are transformed to the atlas image with boundary-based elastic deformation transformation. We model both the fingernail and surrounding skin regions as elastic sheets that are warped by an external force field applied to the boundaries. Since elastic warping tends to preserve color pattern shapes and the relative position of the patterns, it suits color pattern comparison across subjects.

Elastic nail warping Elastic skin warping Combination

Weighted Least Squares Model [Haptics06][TBME08]

Our research has identified that certain areas of the fingernail show a strong linear response of coloration to fingertip force, others do not. Not just the fingernail areas show this effect, certain areas of the surrounding skin particularly at the base of the nail show a strong linear response as well. The location of the good areas depends on the contact conditions.

Generally speaking, regions within the fingernail saturate at lower force levels than regions in the surrounding skin. The front of the fingernail saturates at higher force levels than the middle of the fingernail. The skin at the sides of the fingernail saturates at the highest force levels. The relatively low saturation level of the middle of the fingernail emphasizes the need to image the whole fingernail and surrounding skin.

Using the good mesh elements, a generalized least squares estimator is applied to predict fingertip force.

Automated Calibration [WHC09][Haptics10]

To improve our calibration procedure, we are developing a robot that will apply forces in three directions independently. The z-direction force is supplied by a single motor with a linear stage, on which a platform is mounted. A two-DOF pantograph is attached to the platform, on which is mounted a six-axis force sensor. The fingertip rests against the force sensor, while the first and second links of the finger are held in place by a small finger brace. A series of force controllers determine the input signals to each of the three motors. With this design, the test subject can hold the finger still while the machine provides the desired force, allowing for more methodical collection of calibration data.


Faculty Graduate Students
John M. Hollerbach, PI Thomas Grieve
Stephen A. Mascaro


Yu Sun, University of South Florida
Martha Flanders, University of Minnesota
John F. Soechting, University of Minnesota

This project was supported by NIH Grant 1R21EB004600-01A2.


Haptics10 Grieve, T., Lincoln, L., Sun, Y., Hollerbach, J.M., and Mascaro, S.A., ``3D force prediction using fingernail imaging with automated calibration,'' Haptics Symposium, March 25-26, 2010, Waltham, MA, pp. 113-120.
WHC09 Grieve, T., Sun, Y., Hollerbach, J.M., and Mascaro, S.A., ``3-D force control on the human fingerpad using a magnetic levitation device for fingernail image calibration,'' World Haptics Conference, March 18-20, 2009, pp. 411-416.
TRO09 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., ``Estimation of finger force direction with computer vision,'' IEEE Trans. Robotics, 25, 2009, pp. 1356-1369.
TBME08 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., " Predicting fingertip forces by imaging coloration changes in the fingernail and surrounding skin," IEEE Trans. Biomedical Engineering, 55, 2008, pp. 2363-2371.
CVPR07 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., "Imaging the finger force direction," IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 18-23, 2007, Minneapolis, MN.
ICRA07 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., "EigenNail for finger force direction recognition," Proc. IEEE Intl. Conf. Robotics and Automation, April 10-14, 2007, Rome, Italy, pp. 497-502.
WHC07 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., "Finger force direction recognition by principal component analysis of fingernail coloration pattern," Proc. World Haptics Conference, March 22-24, 2007, Tsukuba, Japan, pp. 243-248.
ICRA06 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., "Features and prediction model for imaging the fingernail to measure fingertip forces," Proc. IEEE Intl. Conf. Robotics and Automation, May 15-19, 2006, Orlando, FL, pp. 2813-2818.
Haptics06 Sun, Y., Hollerbach, J.M., and Mascaro, S.A., "Measuring fingertip forces by imaging the fingernail," Proc. 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 757-762, 2006.


  • Manual control box of a PUMA robot simulated by fingernail imaging [wmv]. This video shows finger tracking by a camera, and detection of "intent to click" by color changes in the fingernail.
  • 2D control of a mouse in a simulated object capture game [wmv]. This video demonstrates the ability to detect fingertip force direction.