POST
SHARE
Sign Language Learning Game SignTown launched by The Nippon Foundation and The Chinese University of Hong Kong Google-developed AI-based recognition technology offers enjoyable “gateway to deaf culture”
A beta version of the SignTown game for teaching sign language, jointly developed by The Nippon Foundation and The Chinese University of Hong Kong in collaboration with Google and Kwansei Gakuin University, was released today. Improvements will be made based on users’ feedback over the next few months, with the target launch date for the full version set for September 23, the International Day of Sign Languages.
Most sign language teaching materials use visual images to show signs to the learner, who memorizes the signs and gestures. SignTown, however, operates as a game in which the learner makes the signs and gestures while facing a camera connected to a personal computer, and the program uses artificial intelligence (AI) to determine whether the signs and gestures were made correctly. The game also uses avatars in a fictional town where everyone communicates using sign language as their native language, providing insights into deaf culture.
About SignTown
SignTown is an online game set in a fictional town where sign language is the official language, and is used to gather items. The menus are available in English, Japanese, and traditional Chinese, and the game can be used to learn Japanese or Hong Kong Sign Language.
The learner faces the camera and uses sign language to indicate items used to perform tasks like packing bags for a trip, finding a hotel, and ordering food in a café.
Project roles
| The Nippon Foundation | Overall project management, expertise regarding sign language and deaf persons, development funding |
|---|---|
| The Chinese University of Hong Kong | Joint overall project management, academic oversight of sign language linguistics, collection of training data for Hong Kong Sign Language, expertise regarding deaf persons |
| Project concept proposal, research and development of sign language recognition technologies using artificial intelligence | |
| Kwansei Gakuin University | Collection of training data for Japanese Sign Language, expertise regarding deaf persons |
The source code for the sign language recognition technology on which SignTown is built is being made publicly available free of charge as open-source code. This will allow researchers and developers around the world to develop recognition technologies for sign languages used in other countries and regions.
- * Source code: The plain text written in a programming language that constitutes the underlying structure of software and computer programs. The source code for SignTown’s sign language recognition technology is available (in English only) at: https://github.com/google/shuwa (external site)
Overview of sign language recognition technology
In addition to hand gestures, various body movements like leaning forward and facial expressions play an important grammatical role in sign language. In addition, conventional cameras are only able to recognize flat images in two dimensions, meaning that sign language recognition has required special equipment including specialized cameras and gloves to recognize three-dimensional movements and positions, which has held back its widespread use. This project, however, has developed a recognition model that is able to recognize these body movements and facial expressions using a conventional camera. (Please refer to the attachment for details.)
About SignTown’s Sign Language Recognition Technology（PDF/164KB）Automated translation model
This project aims to develop an automated translation system that is able to use an everyday webcam or smartphone camera to recognize sign language as natural speech and convert it into an audio format. SignTown represents an intermediate success toward this goal, with the aim of recognizing the special features of sign language grammar.
Social background
Sign language is a language for mutual understanding among deaf persons, and an environment in which sign language can be used is essential to the promotion of deaf persons’ participation in society. The Convention on the Rights of Persons with Disabilities, adopted by the United Nations in 2006, clearly states that “‘Language’ includes spoken and signed languages,” and in Japan, the Basic Act for Persons with Disabilities as amended in 2011 recognizes sign language as a language. Nevertheless, understanding of sign language and deaf persons is still limited and there are barriers to the use of sign language to participate in various aspects of society. At the same time, the global coronavirus pandemic has increased interest in sign language with the inclusion of sign language interpretation at televised press conferences held by public officials, creating an ideal opportunity to promote sign language. The widespread use of information and communications technology has made it possible to develop a beta version of SignTown, the world’s first game to use AI for teaching sign language, with the aim of using the latest technologies to promote an understanding of sign language and deaf culture.
About The Chinese University of Hong Kong
Founded in 1963, CUHK is a leading comprehensive research university with a global reputation. Located in the heart of Asia, CUHK has a vision and a mission to combine tradition with modernity, and to bring together China and the West. Under the University’s unique collegiate system, the programs and activities offered by its nine colleges complement the formal curricula by delivering whole-person education and pastoral care. The University has eight faculties: Arts, Business Administration, Education, Engineering, Law, Medicine, Science, and Social Science. Together with the Graduate School, the University offers over 300 undergraduate and postgraduate programs. All faculties are actively engaged in research in a wide range of disciplines, with an array of research institutes and research centers specializing in interdisciplinary research of the highest quality.
CUHK is recognized as the most innovative university in Hong Kong for the fourth consecutive year and is ranked 26th in the Asia-Pacific region by Reuters’ latest “Asia Pacific Region’s Most Innovative Universities” listing. The University currently has more than 1,000 granted patents in different jurisdictions worldwide. Some of these patents have been licensed to relevant industries that help bring these innovations to the market to benefit society. In academic year 2019-20, CUHK has received 257 granted patents and filed 386 patent applications for inventions developed in the areas of medical technology, biotechnology, information technology, telecommunications, and materials science.
The Chinese University of Hong Kong websiteAbout Kwansei Gakuin University
Kwansei Gakuin University, founded in 1889, is a comprehensive university that provides education based on the principles of Christianity. The Sign Language Research Center (SLRC) was established in April 2015 as the first institute researching sign language at a Japanese university, and has been active since 2016 with support from The Nippon Foundation. The SLRC has been contributing to this project by selecting Japanese Sign Language (JSL) words and collecting JSL data for use in machine learning.
Contact
Public Relations Team
The Nippon Foundation
- Email: cc@ps.nippon-foundation.or.jp
