FUJIFILM Software Co., Ltd.

Since FUJIFILM Software debuted IMAGE WORKS in 2006, the image file management and sharing service quickly became a global standard for enterprises sharing large volumes of content. In 2016, the Nippon Professional Baseball Organization (NPB) adopted the solution to centrally manage game photos. At the time, the organization was faced with a new challenge of efficiently tagging photos. To provide game photos to corporate users, the players in each image needed to be tagged, which took considerable effort. FUJIFILM Software developed a player name auto-tagging function that employs an original image classification model using Microsoft Azure Cognitive Services and Microsoft Cognitive Toolkit. The company succeeded in drastically reducing the tagging workload. In addition, Azure Durable Functions was used to shorten processing times.

Tagging 300 photos from 3,000 creates enormous workloadThe NPB oversees the Central and Pacific Japanese professional baseball leagues and

contributes to the development of baseball culture in a variety of roles. In 2016, it began

operating the NPB Contents and Images Center (NPB CIC) to streamline the work of lending

out photos of professional baseball teams. The service centrally manages the photo assets of

each team, lends out team photos to users, and manages the associated invoices. NPB IMAGE

WORKS from FUJIFILM Imaging Systems provides the platform for the service.

IMAGE WORKS is a cloud service for sharing and managing large volumes of content, including

images and video. The solution enables safe and efficient management throughout the content

lifecycle, from sharing data during planning and creation to centralized management of

deliverables, data distribution, and deletion from the archives. Since its debut in 2006, IMAGE

WORKS has been highly regarded for its numerous impressive results, including adoption in

2016 for the G7 Hiroshima Foreign Ministers’ Meeting and the G7 Ise-Shima Summit.

NPB was also impressed with these results and adopted the solution for the launch of NPB CIC.

In the past, the photo management performed by each team had been implemented on a

unified platform.

CustomerFUJIFILM Software Co., Ltd.

Products and Services・ Azure・ Azure Cognitive Services・ Azure Functions・ Microsoft Cognitive Toolkit

IndustryDiscrete Manufacturing

SizeLarge (over 10,000 employees)

CountryJapan

Published December 2018

FUJIFILM boosts NPB player recognition, reduces photo tagging costs with Microsoft Azure

FUJIFILM Software Co., Ltd.

“Even after NPB adopted IMAGE WORKS, we would conduct

interviews to discover the company’s user needs,” says Riki Satou,

Manager of the Image Works Team at FUJIFILM Software.

According to Satou, an enormous amount of work time was spent

on tagging photos. “In order to lend a photo to a corporate user, the

players shown in each photo need to be tagged so that the photo of

the player can be found quickly. However, there can be as many as

3,000 photos per game and around 300 of those need to be selected

and determined as suitable for commercial use. NPB required an

automated and efficient system which automatically determined the

names of the players in the photos. To make the work more efficient,

we developed a player name auto-tagging function using AI.”

In December 2016, FUJIFILM Software adopted Microsoft Azure

Cognitive Services and other Microsoft AI services.

Riki SatouTeam ManagerImage Works TeamAdvanced Solution Development GroupServices DivisionFUJIFILM Software Co., Ltd.

Daichi HayataMCSE Cloud Platform and InfrastructureImage Works TeamAdvanced Solution Development GroupServices DivisionFUJIFILM Software Co., Ltd.

<IMAGE WORKS Screen>

Samples where faces cannot be identified

Samples where positions can be inverted

Samples where numbers cannot be identified

Samples where handedness can be determined

※ This pictures is for sample image. Actual product may vary.

FUJIFILM Software Co., Ltd.

Eventually, the company matched the players shown in photos by

comparing them against NPB BIP information, the official professional

baseball records provided by NPB, to reduce the number of possible

selections.

NPB BIP offered robust information to narrow down player names,

including the results of pitching and batting during a game, positions,

and handedness. By using AI, the company developed several models

to classify the players in the photos. These models extrapolated the

names of the players by categorizing them as pitchers or batters, or

whether they were right- or left-handed, then matching them to the

time that the photo was taken and the NPB BIP data, which narrowed

the candidates to no more than five players. To create these multiple

image classification models, the company used Microsoft Cognitive

Toolkit.

“We ultimately decided to create our own model with Cognitive

Toolkit; we were able to develop the model efficiently as it was

possible to conduct transfer learning based on Microsoft Research

ResNet (Deep Residual Net) model. Although it was the first time that

I had dealt with deep learning, I was able to use it proficiently with just

some basic knowledge, such as the kind of training data that should

be prepared and the amount needed. By this point, Microsoft had

held numerous hackathons for us and we had also received advice

from Microsoft partners. This support helped us to understand that

combining multiple models with limited functions would be more

accurate than trying to do everything with a single model. This

approach increased the probability that the correct selection would

be included in the final list of player names to more than 90 percent,” explains Hayata.

Creating multiple models for high-accuracy tag determinationsThere were two reasons why FUJIFILM Software adopted Azure AI

services for the player name auto-tagging function. First, IMAGE

WORKS has run on Azure since 2016, when Azure was initially adopted

as the platform. In 2017, SCIM (System for Cross-domain Identity

Management) had been deployed to enable system-linked user ID

management, and Azure Active Directory was included in this process.

Since the company was already using several Azure platform as a

service (PaaS) functions, using Azure AI services was a natural next

step.

Second, Azure AI services include a wide array of services, notably

Azure Cognitive Services, that can be used by those who are not AI

experts. Also, according to Satou, the company was impressed with

the support Microsoft provided for modernizing IMAGE WORKS and

using SCIM.

However, using AI to automatically extrapolate the names of players

in photos was no easy task. Daichi Hayata, from the MCSE Cloud

Platform and Infrastructure Image Works Team, and the developer

lead on the player name auto-tagging function, explains the

challenges.

“Many game photos are taken at an angle or from the side. Face API [a

function of Azure Cognitive Services] is extremely accurate if a photo

of a face is taken from the front, but it is rather limited if the photo is

from an angle. In fact, when we used Face API to determine the faces

in photos from games, the recognition rate was only about 20 percent.

We also considered a method in which determinations were made

using the names and numbers on uniforms,” says Hayata. “But there

were few photos that clearly showed the name or number together

with the player’s face, so it did not provide any major improvements in

accuracy. We worried about this dilemma for several months.”

“ We ultimately decided to create our own model with Cognitive Toolkit; we were able to develop the model efficiently as it was possible to conduct transfer learning based on Microsoft Research ResNet (Deep Residual Net) model. Although it was the first time that I had dealt with deep learning, I was able to use it proficiently with just some basic knowledge.”

Daichi Hayata MCSE Cloud Platform and Infrastructure Image Works Team Advanced Solution Development Group Services Division

　FUJIFILM Software Co., Ltd.

“ In order to lend a photo to a corporate user, the players shown in each image need to be tagged…. To make the work more efficient, we developed a player name auto-tagging function using AI.”

Riki Satou Team Manager Image Works Team Advanced Solution Development Group Services Division

　FUJIFILM Software Co., Ltd.

FUJIFILM Software Co., Ltd.

Azure Durable Functions reduces processing time, enables same-day taggingIn January 2018, FUJIFILM Software decided to combine multiple

determination models. From there, model development progressed

rapidly and was completed in June 2018, barely six months later.

However, just before completion, the company faced another

challenge. Running multiple classification models increased the

processing load. It took 40 minutes to determine 100 photos, which

the NPB and the teams using the service felt was slow.

“We initially thought that we could perform batch processing

overnight and have someone make the final determination the

next day, but we found out that the photographers of some teams

were tagging the photos themselves immediately after uploading.

Therefore, we needed to improve real-time performance,” explains

Satou.

The solution was Azure Durable Functions, which is provided by

a serverless compute service, Azure Functions. The service is an

extension of Azure Functions and Azure

WebJobs, and enables stateful functions to

be described in serverless environments.

Using the service for asynchronous

distributed processing of multiple functions

can shorten processing times dramatically.

Using this service with the player name auto-

tagging function reduced the processing

time to one-twentieth of the original time

and enabled same-day tagging.

“The cloud services of the different providers

generally offer the same functions these

days, but only Azure has Durable Functions,

which made us very happy to be using Azure.

There are about 3,000 photos per game on

average, and processing is completed within

about five minutes even if we register all

3,000 in NPB CIC,” says Hayata.

The following figure illustrates processing flow of the player name

auto-tagging function. First, photos are imported into IMAGE

WORKS, and Durable Functions is run on each image, performing

preprocessing tasks such as image resizing and rotation. Extrapolation

is then performed by combining Face API processing with classification

through the multiple models built using Cognitive Toolkit, and the

results are saved in a database.

NPB began trial use of the tagging function in June 2018, and currently

five teams are testing it. Users select the correct player name from

the list of candidates; tagging work that once took up to 3 to 4 hours

can now be completed in 30 minutes. In 2019, all teams that have

introduced NPB CIC are scheduled to make full use of the function.

“Although we are using AI in a form specifically for professional

baseball in this instance, a similar mechanism can be applied to other

sports,” says Satou. “We are already working on initiatives to apply

it to other sports, as well as considering expanding the function to

enterprises engaged in advertising and PR, and even adding video

analysis.”

The processing flow for the player name auto-tagging function provided to NPB. Multiple classification models created with Cognitive Toolkit are combined with Face API processing to extrapolate player names. In addition, running Durable Functions on the number of images has greatly reduced processing time.

①PhotoImport

⑤ClassificationModel Analysis

⑥ExtrapolationProcessing

Client

②Search ③

④Face API

BIP DataImport and formatting of

official professional baseball records

Durable Functions (Image Units)

AnalysisResult DB

Acquisition of images to

be analyzed

Durable Functions is run on the number

of images

・Exif Data Acquisition・Resizing・Rotation

Storage Face List Player Data

Preprocessing of Images

Implementation InquiryThis case study can also be accessed via the internet.https://customers.microsoft.com/Please note that the information contained in this case study was current as of the time of writing (December 2018) but may no longer be up to date.This case study is provided for information purposes only. Microsoft makes no warranties whatsoever, express or implied, in relation to the content of this document.

For product inquiries, please use the following contacts:■Website: http://www.microsoft.com/ja-jp/* All other company names, product names, and logos are registered trademarks or trademarks of their respective companies.* Please note that product features and specifications are subject to change without notice.

Microsoft Corporation

SE