Transparency
Note 
Azure Cognitive Services: Face API 
Last Updated 3/29/19

Contents 
About this Transparency Note .................................................................................................................................................. 2 
The basics of Face API ................................................................................................................................................................. 2 
Key facial recognition terms ...................................................................................................................................................................... 2 
Face API functions ......................................................................................................................................................................................... 3 
Understanding accuracy and errors ........................................................................................................................................ 3 
How accurate is Face API? .......................................................................................................................................................................... 3 
The language of accuracy........................................................................................................................................................................... 4 
Match scores, match thresholds, and candidate lists ......................................................................................................... 5 
Best practices for improving accuracy .................................................................................................................................... 6 
Plan for an evaluation phase ..................................................................................................................................................................... 6 
Meet image quality specifications .......................................................................................................................................................... 6 
Control image capture environment...................................................................................................................................................... 7 
Plan for variations in subject appearance and behavior ................................................................................................................ 7 
Design the system to support human judgment .............................................................................................................................. 8 
Use multiple factors for authentication ................................................................................................................................................ 8 
Deploying responsible facial recognition systems .............................................................................................................. 9 
Evaluate stakeholder concerns and design the experience to address them........................................................................ 9 
Develop transparent communication and escalation processes for stakeholder concerns............................................. 9 
Provide training and evaluate the effectiveness of people who make final judgments based on facial
recognition ....................................................................................................................................................................................................... 9 
Update privacy policies and implement necessary changes ........................................................................................................ 9 
Learn more about Face API ....................................................................................................................................................... 10 
Contact us ..................................................................................................................................................................................... 10 
About this document ................................................................................................................................................................. 10 






1



About this Transparency Note 
Accessible through Azure Cloud Services, Azure Face API ("Face API") detects, recognizes, and analyzes human
faces in images using pre-trained machine learning models that have been developed by Microsoft. Developers
can integrate Face API functions into their systems without creating their own models. 
Facial recognition1 is an important and useful technology that can improve efficiency, security,
and customer experiences. 
Face API is a building block for creating a facial recognition system. A facial recognition system includes the
technology as well as the people who will use it, the people who will be subject to it, and the environment in
which it is deployed. Creating a system that is fit for purpose requires an understanding of how the technology
works, its capabilities and limitations, and how to achieve the most accurate results. This Transparency Note is
intended to help you understand how Face API works, the choices you can make as a system owner that
influence accuracy, and the importance of thinking about the whole system, including the technology, the
people, and the environment. It is part of a broader effort at Microsoft to implement our Facial Recognition
Principles, which set out how we approach the development and deployment of facial recognition technology. 
We encourage you to use the principles to guide your deployment efforts too. 

The basics of Face API 
Key facial recognition terms 
Template        Images of people are converted to templates, which are then used for facial recognition.
Machine-interpretable features are extracted from one or more images of an individual to
create that individual's template. The images themselves  whether enrollment or probe 
images (see below)  are not stored by Face API and the original images cannot be
reconstructed based on a template. Template quality is a key determinant of how accurate
your results will be. 
Enrollment      Enrollment is the process of enrolling images of individuals for template creation so they
can be recognized. When a person is enrolled to a verification system used for
authentication, their template is also associated with a primary identifier2 that will be used
to determine which template to compare with the probe template (see below). High-quality
images and images representing natural variations in how a person looks (for instance
wearing glasses, not wearing glasses) yield high-quality enrollment templates. 


1 This Transparency Note addresses Face API's recognition functions. Other functions, such as predicting attributes, including 
gender and age, and the limitations of these predictions are not addressed here. 
2 Face API does not store primary identifiers, such as customer IDs, alongside facial templates. Instead, Microsoft associates
stored facial templates with random GUIDs or globally unique identifiers. System developers can associate the GUID
generated by Microsoft with an individual's primary identifier to support verification of that individual. 
2


Probe image     A probe image is an image submitted to a facial recognition system to be compared to
enrolled individuals. Probe images are also converted to probe templates. As with
enrollment templates, high-quality images result in high-quality templates. 

Face API functions 
Face API Detection ("Detection") answers the question, 
Face API can answer the
"Are there one or more human faces in this image?" Detection finds human
questions: 
faces in an image and returns bounding boxes indicating their locations. All
other functions are dependent on Detection: before Face API can identify or       1.  Are there one or more
human faces in this image? 
verify a person (see below), it must know the locations of the faces to be
2.  Are these two images the
recognized. 
same person? 
The Detection function can also be used to predict attributes about each face,     3.  Can this unknown person 
be matched to an enrolled
including age and gender. These attribute prediction functions are completely
template? 
separate from the verification and identification functions of Face API. Face API
does not predict an individual's age or gender as a precursor to verifying or
identifying them. 
Face API Verification ("Verification") builds on Detection and addresses the question, "Are these two images
the same person?". In security or access scenarios, Verification relies on the existence of a primary identifier (such
as a customer ID) and facial recognition is used as a second factor to verify the person's identity. Verification is
also called "one-to-one" matching because the probe template (one person) is only compared to the template
stored for the (one) person associated with the identification presented. 
Face API Identification ("Identification) also starts with Detection and
Face API documentation 
answers the question, "Can this unknown person be matched to an enrolled 
template? Identification compares a probe template to all enrollment templates   For more information on all of
stored in your private repository, so it is also called "one-to-many" matching.      the functions of Face API, see
the Face API documentation 
Candidate matches are returned based on how closely the probe template 
matches each of the enrolled templates. 
Understanding accuracy and errors 
How accurate is Face API? 
Because Face API is a building block for creating a facial recognition system to which other building blocks must
be added, it is not possible to provide a universally applicable estimate of accuracy for the actual system you are
planning to deploy. Companies may share accuracy as measured by public benchmark competitions, but these
accuracies depend on details of the benchmark and therefore won't be the same as the accuracy of a deployed
system. Ultimately, system accuracy depends on a number of factors, including the technology and how it is
configured, environmental conditions, the use case for the system, how people to be recognized interact with
the camera, and how people interpret the system's output. The following section is intended to help you
understand key concepts that describe accuracy in the context of a facial recognition system. With that
understanding, we then describe system design choices and how they influence accuracy. 

3

The language of accuracy 
The accuracy of a facial recognition system is based on a combination of two things: how often the system
correctly identifies a person who is enrolled in the system and how often the system correctly finds no match for
a person who is not enrolled. These two conditions, which are referred to as the "true" conditions, combine with
two "false" conditions to describe all possible outcomes of a facial recognition system: 
True positive or true accept      The person in the probe image is enrolled and they are correctly matched. 

True negative or true reject      The person in the probe image is not enrolled and they are not matched. 

False positive or false accept     Either the person in the probe image is not enrolled but is matched to an
enrolled person OR the person in the probe image is enrolled but is
matched with the wrong person. 

False negative or false reject     The person in the probe image is enrolled, but they are not matched. 

The consequences of a false positive or a false negative vary depending on the purpose of the facial recognition
system. The examples below illustrate this variation and how choices you make in designing the system affect
the experience of those people who are subject to it. 
Logging into a banking app                    Organizing photographs 
Facial recognition can provide an added layer of         Many photo organizing apps help you find
security in addition to a PIN or other primary             pictures of a specific person across your photo
identification. A false positive for this application         collection using facial recognition. In this instance,
reduces customer security because it results in an        the customer is using the app to choose photos
incorrect match, while a false negative could             for a retirement party. Because the customer will
prevent the customer from accessing their              be reviewing the photos and choosing photos
account. Because the purpose of the system is           they wish to use, false positives may not be
security, false positives must be minimized and as       particularly important: facial recognition is
a result, most errors will be false negatives                making the search task easier for the customer,
(account access fails). To address this limitation,          and if they review a few more photos than
system owners can provide a fallback mechanism,       necessary, they can still easily complete their task.
like pushing a notification to the customer's              On the other hand, if they have scanned old
phone with an access code. The customer's             family photographs that are somewhat degraded,
experience may be less efficient in this case, but          the app may not be able to find relatives in these
account access is not blocked, and security is            photographs (false negatives) and the customer
prioritized.                                                   may be frustrated with the app. 

4

These two examples illustrate that the impact of false positives and false negatives vary depending on the
purpose of the system, and that the design of the whole system, including fallback mechanisms, determine the
consequences for people when errors occur. 
Match scores, match thresholds, and candidate lists 
The purpose of this section is to help you understand how system configuration influences system accuracy and 
the trade-off between false positives and false negatives. 
Match score      A match score describes the similarity between a probe template and an enrolled template.
Match scores range from 0 to 1. High match scores indicate that it is more likely that the
two images are of the same person. 
Match         A match threshold is a configurable value between 0 and 1 that determines the match score
threshold        required to be considered a positive match. If the match threshold is set to 0, then any
probe template will match any enrollment template. Face API has a default match threshold
that you can change to suit your application. 
When using the Verification function for authentication, if the match score between the probe template and the
enrollment template associated with the primary identifier is at least as high as the match threshold, Face API
will indicate that the probe image represents the person presenting identification. 
When using the Identification function, it can be useful for a person to review a list of candidates ranked by
match scores to determine the final match. Face API customers can choose how many candidate templates that
reach the match threshold will be returned in ranked order of similarity to the probe template. These matches
are referred to as a "candidate list". Face API will only return candidates with match scores at least as high as
the match threshold. When no templates have match scores that reach the match threshold, no matches are
returned. 
Why choose a match threshold less than one? 
Setting a match threshold allows for balancing the errors between false positives and false negatives to best
address your specific scenario. The overall accuracy of the system is unlikely to be 100% and when the match 
threshold is set to 1, the strictest value, virtually all errors that occur will be false negatives: the system will return
"no match" because the submitted probe template will not perfectly match any enrolled templates. Because
match scores are affected by the quality of the probe and the enrollment images, a lower match score can
indicate poor quality images, rather than less similarity between people in the images. When using Identification,
if the match threshold is set too high, the system may not return enough candidates to find the true match. On 
the other hand, a low match threshold may return low quality matches and can reduce the efficiency and
accuracy of the humans reviewing the matches. 
How should a match threshold be selected? 
The best match threshold for your system is based on: 
The system purpose 
The impact of false positives and false negatives on the people who will be subject to facial recognition 
Whether final judgments are made by a human 
How the whole system, including the experience design, supports resolution of errors. 

5

Before selecting a match threshold, Microsoft recommends that you, as a facial recognition system owner,
collect ground truth evaluation data on site to determine how the match threshold affects the achievement of
your goals and affects people subject to and interpreting the output of the system. 
Ground truth evaluation data is data that is collected to evaluate a system. Critically, the true identity for each
person represented in a probe photo is known and can be correctly matched to their enrollment template.
Ground truth labels can be compared to the output of the system to establish the overall accuracy and error
rates, and the distribution of errors between false positives and false negatives. Ground truth evaluation data
should include adequate sampling of diverse people who will be subject to recognition so that performance
differences can be understood, and corrective action taken. 
Based on the results of this evaluation, you can iteratively adjust the match threshold until the trade-off between
false positives and false negatives meets your objectives. 
Best practices for improving accuracy 
Facial recognition technology is improving and many systems, including Microsoft's Face API, can perform well
even when conditions are not ideal. However, these are specific actions that you can take to ensure best-quality
results from your facial recognition system. 
Plan for an evaluation phase 
Before a large-scale deployment or rollout of any facial recognition system, Microsoft strongly recommends that
system owners conduct an evaluation phase in the context where the system will be used and with the people
who will interact with the system. 
You should work with your analytics and research teams to collect ground truth evaluation data to: 
Establish baseline accuracy, false positive and false negative rates. 
Choose an appropriate match threshold to meet your objectives. 
Determine whether the error distribution is skewed towards specific groups of people. 
This is likely to be an iterative process with adjustments to sensor position, lighting, and other factors that
influence accuracy, as discussed in this section. This evaluation should reflect your deployment environment and
any variations in that environment, such as lighting or sensor placement, as well as ground truth evaluation data
that represents the diversity of people who will interact with your system. 
In addition to telemetry data, you may also want to analyze feedback from the people making judgments based
on the system output, satisfaction data from the people who are subject to recognition, and feedback from
existing customer voice channels to help tune the system and ensure successful engagement. 
Meet image quality specifications 
Image quality is critical to quality facial recognition so you should ensure that both the images used to enroll
people and the probe images meet the following specifications: 
Full-frontal head and shoulder view without obstruction. 
Face size is at least 200x200 pixels with at least 100 pixels between eyes. Faces are detectable when their
size is as small as 36x36 pixels, but for best performance Microsoft recommends a minimum size of
200x200 pixels when using Face API. 

6

Enroll multiple images of each person. Include images that represent typical variations in how the person's
face appears to the camera, for instance, with and without glasses, from varied angles. 
Control image capture environment 
Lighting and camera calibration 
Pay attention to how well the detail of people's faces can be seen in images taken with the camera you are
intending to use in the locations where you will use it. Face API uses RGB images. 
Capture images in appropriate lighting conditions. Is the lighting too bright, too dark? Are faces backlit? Is
there too much light from one side and not enough from the other? When possible, place sensors away
from areas with extreme lighting. 
Is the lighting adequate to accurately capture the details of people's faces with different 
skin tones? 
Backgrounds 
Strive for neutral, non-reflective backgrounds. Avoid backgrounds containing faces, for instance where
there are pictures of people displayed, or where people other than the person to be recognized are 
prominent in the photo. 
Sensor placement and maintenance 
Position sensors at face-level to best capture images that meet the quality specifications. 
Ensure sensors are regularly checked for dust, smudges, and other obstructions. 
Plan for variations in subject appearance and behavior 
Facial occlusions 
Facial recognition works best when the person's entire face is visible. Faces may be partially or entirely occluded
for a variety of reasons, including: 
Religion: Headwear that covers or partially obscures faces. 
Weather: Garments like scarves wrapped across the face. 
Injury: Eye patches or large bandages. 
Vision Disability: Very opaque glasses and pinhole glasses (other glasses and lenses should be fine). 
Personal style: Bangs over eyebrows, baseball caps, large facial tattoos, etc. 
It is not always possible to avoid occlusions: removing glasses may be unsafe and requiring removal of religious
headwear may be impermissible. In addition to sensor placement, the following actions can help to address
occlusion challenges: 
A fallback method, such as a non-biometric alternative, is critical. For some people, the fallback may be the
option they consistently use. 
Pay attention to challenges that people face during evaluation and deployment to identify remediations
that work best for your environment. 
Abrupt changes in appearance 
Dramatic changes in appearance, like the removal of a full beard or many years passing between enrollment and
probe images (for adults), or even short periods of time between photos of children, can result in errors. 
7

In addition to supporting a fallback method, designing the user experience to support immediate reenrollment
following a recognition failure can improve user satisfaction. 
Facial recognition systems are generally less accurate for children. Microsoft recommends using Face API
for recognition of people over 18. Facial recognition can be especially challenging with people 13 and
younger. 
Subject behavior 
Image quality may be low when subjects are not facing the camera, occluding their face with their hands (such
as brushing hair out of their eyes), moving too fast for the sensor to capture their image), or when their
expression is extreme (like yawning widely with their eyes closed). To address these challenges: 
Design the user experience so people understand how to provide high-quality images. 
Create an environment where people naturally face the camera and slow down. 
Provide clear instructions for how people should behave during recognition (eyes open, mouth closed,
stand still, etc.). 
Biometric Twins 
Twins, family members, and other people who look very similar to each other will be difficult for facial
recognition systems to distinguish from one another. This is another reason to support a fallback method. 
Design the system to support human judgment 
In most cases, Microsoft recommends using Face API's facial recognition capabilities to support people making
more accurate and efficient judgements rather than fully automating a process. Meaningful human review is
important to: 
Detect and resolve cases of misidentification or other failures. 
Provide support to people who believe their results were incorrect. 
Identify and resolve changes in accuracy due to changing conditions (like lighting or sensor cleanliness). 
For example, when using Face API for building security, a trained security officer can help when the facial
recognition fails to match someone who believes they are enrolled by deciding whether a person should be
admitted to the building. In this case, Face API helps the security officer work more efficiently, requiring a
judgment to admit someone only when the person is not recognized. 
The user experience that you create to support the people who will use the system output should be designed
and evaluated with those people to understand how well they can interpret the output, what additional
information they might need, how they can get answers to their questions, and ultimately, how well the system
supports their abilities to make more accurate judgments. 
Face API supports facial recognition with still images: there are no anti-spoofing countermeasures built into Face
API, such as depth or motion detection. In cases where facial recognition is supporting human judgment and
improving efficiency, this is generally not a key limitation: humans can easily detect when a person is holding up
a picture to a camera. 
Use multiple factors for authentication 
Use Face API along with one or more other factors when creating authentication systems, such as confirming
passengers who are about to board a plane or confirming a banking transaction. As discussed above,
Verification makes use of facial recognition as a second factor for identifying someone rather than a single or
8

primary factor. Identification does not require another factor; however, Identification is a more technically
difficult problem because the probe template is compared to ALL enrolled templates instead of just the template
for the primary identifier associated with the probe template. It is often still possible to use other signals to
support authentication when using Identification, such as narrowing the set of enrolled templates to compare by
limiting the search to people who have a ticket for a specific flight. While it is not possible to choose Verification
for all scenarios, Microsoft recommends Verification for uses including secure access to buildings and for other
key business and security functions. 
Deploying responsible facial recognition systems 
In addition to addressing accuracy, here are some additional considerations for successful deployment. 
Evaluate stakeholder concerns and design the experience to 
address them 
Understand both the perceived value of the facial recognition system and the concerns that people may have 
about it. Engage your research team to help understand how your customers, employees, and other
stakeholders can help you deploy a system that supports your critical needs and those of the people who will be
involved. 
Develop transparent communication and escalation processes for 
stakeholder concerns 
People may still have questions and concerns. Part of any release plan should include both proactive and
reactive communication, a documented escalation process, and clear explanations for how feedback will be
addressed. 
Provide training and evaluate the effectiveness of people who make final
judgments based on facial recognition 
Microsoft strongly recommends that customers develop training for people who will use the output of systems
or who will decide whether the system output is correct. Customers should also evaluate whether these
employees can make correct judgments based on the output of the system and determine whether any unfair
biases are introduced. 
Update privacy policies and implement necessary changes 
Microsoft strongly recommends that private sector customers provide conspicuous notice to and secure consent
from individuals before capturing their images for use with facial recognition technology. System owners should
also esta blish  responsible data handling practices (including limits on retention and reuse of images) and ensure
that those practices are communicated clearly to individuals subject to the system. Remember to include
considerations for children who may be subject to recognition. In some jurisdictions, there may be additional
legal requirements, and customers are responsible for compliance with all applicable laws. 



9







Learn more about Face API 
Azure Cognitive Services Compliance and Privacy 
Face API Technical Documentation 
Learn more about how others are using Face API 
Review Microsoft's Facial Recognition Principles 
Contact us 
Give us feedback on this document 
Find out about support options 
About this document 
(c)2019 Microsoft Corporation. All rights reserved. This document is provided "as-is." and for informational
purposes only. Information and views expressed in this document, including URL and other Internet Web site
references, may change without notice. You bear the risk of using it. Some examples are for illustration only and
are fictitious. No real association is intended or inferred. 
Last updated: 3/29/2019 










10