SignTown Officially Released on Sept. 22AI recognition sign language learning game offers gateway to Deaf culture

A media event was held at The Nippon Foundation’s head office in Tokyo on September 22 to mark the official launch of the SignTown game for teaching sign language, jointly developed by The Nippon Foundation and The Chinese University of Hong Kong in collaboration with Google and Kwansei Gakuin University. Since the launch of the beta version in May, roughly 8,500 people have used the site, and various improvements have been made based on their feedback. The launch was timed to coincide with the International Day of Sign Languages on September 23, which was a public holiday in Japan.

The event began with opening remarks by The Nippon Foundation Chairman Yohei Sasakawa, who noted that although the United Nations has recognized sign language as a language, Japan has yet to ratify a Sign Language Act and greater efforts are needed to promote the use of sign language in daily life. He also pointed out that having more of the general public conversant to some degree in sign language would lower barriers to employment for persons who are deaf and hard of hearing, and make it easier for these persons, who also represent a large potential market for goods and services, to participate more fully in society.

Mr. Sasakawa’s remarks were followed by a presentation by The Nippon Foundation’s Ikumi Kawamata. Ms. Kawamata, who is Deaf, used sign language with spoken interpretation to provide background on sign language itself and an overview of the SignTown project. She explained that 5% of the world’s population has some degree of hearing loss, and that 70 million people – close to 1% of the world’s population – use sign language. Ms. Kawamata explained that like a spoken language, sign language enables people to convey and understand emotions and nuances that are lost when communicating by written text alone. For these people, sign language is more than just a means of communication, it includes culture and manners and is an integral part of their identity.

SignTown was developed as an easy and enjoyable way to experience sign language for the first time. From this starting point, people may be encouraged to improve their skills further and use it in their daily lives. In addition to being able to act as interpreters in general social and work situations, this could lead to people like doctors, teachers, and store employees being able to communicate with patients, students, and customers who are deaf using sign language.

SignTown marks a breakthrough in sign language recognition technology, with the ability to recognize signs using a conventional two-dimensional camera attached to a personal computer. Improvements made during the beta phase based on user feedback include a tutorial demonstrating how to set the camera, slow-motion and enlarged size replay functions for the instructor’s signs, and side-by side replay showing the correct sign together with the user’s sign. Going forward, plans are to increase the number of words in the SignTown database, and make the game compatible with smartphones and tablet devices. SignTown is part of a larger project that seeks to use artificial intelligence sign language recognition to create sign language dictionaries, and eventually to develop automated interpretation between sign language and speech.

Following short video messages from project partners The Chinese University of Hong Kong, Google, and Kwansei Gakuin University, Ms. Kawamata was joined on the podium by popular 19-year-old YouTuber Chloe of Yurima Girl. Ms. Chloe, who is Deaf and fluent in Japanese Sign Language, demonstrated SignTown. After setting the camera to synchronize the artificial intelligence recognition, she set the module for Hong Kong Sign Language and learned signs that would be useful for ordering in a café and checking into a hotel. Clicking on an icon also displayed cultural insights, showing that Deaf people prefer bright settings to dark ones, and sitting facing each other rather than side-by-side when dining together.

Ms. Chloe commented that the site is enjoyable to use and the side-by-side replay function is particularly helpful, and that she looks forward to using it as a game she can play with her young sister. Also, while she has seen speech-to-sign language interpretation and speech-to-text conversion, this is the first time she has seen sign language recognition and was impressed by the technology. She also said that she is interested in traveling to other countries, and hopes that SignTown will be expanded going forward to include more countries’ sign languages.