Are you interested in social robots? Do you have two Android devices laying around? Do love cute little robot faces?
Then this is the software for you!
Robot face with simple webcam face tracking enabled.
The Phone-Face-Tablet-Controller for Social Robotics is an open, expandable software framework compatible with a wide variety of social therapy robots for Windows and Android. Built in Unity 2017.1.5 and a variety of commercial-off-the-shelf parts and plugins, this allows you to connect an Android tablet or phone displaying an animated robot's face paired with a Windows or Android laptop, tablet, or smart phone as the controller. The controller app, controlled by a human operator, sends commands to the face app that control Speech-To-Text, expression, face color, and more via either a Bluetooth or LAN WiFi connection. If connected to an Arduino USB interface, we can also control 2-wheeled base motion for moving forwards, backwards, and turning left and right. The ultimate use case of this software is to teach and inspire children with autism and other special needs.
Tablet Controller button board view. See list of Palettes on the left, button grid in middle, and individual Button Speech parameters on right.
- Emotion Display ( Concerned | Surprise | Happy | Sad )
- Face tracking with eyes looking at user.
- Portrait or Landscape mode
- Face Color Customization
- Speech to Text Voice
- Lip Sync Vocals
- File Save, Load and Selection with speech phrase list as comma-delimited CSV
- Speech to text entry, adding and deleting with quick phrase selection
- Bluetooth discovery and connection of Client Face
- Expression control / Face color control
- Arduino robot hardware control
One of the coolest features of this social robot tablet controller phone face framework is that it allows the user to connect an Arduino-based board via USB directly to the phone face Android device (given that the device has support for USB OTG devices.) It is possible to send Arduino control signals from the tablet controller to the phone face either through LAN WiFi or Bluetooth, where the phone face is directly connected to the Arduino via USB OTG.
A full example setup with tablet controller and phone face connected via serial USB to an Arduino-based tank platform.
Spinning the USB Arduino tank with Bluetooth Control from the tablet controller app!
Download the Arduino Code for Robot Tank Base here:
https://github.com/johnchoi313/Phone-Face-Tablet-Controller/tree/master/ArduinoBluetoothTankControl
We've been working with the awesome team at Misty Robotics to integrate the same tablet face controller app with their awesome line of social robots, in particular, the Misty I and Misty II.
The main HTTP REST API wrapper code we created for connecting Unity to the Misty I Robot (which can be re-used in other Unity C# projects) can be found here:
https://github.com/johnchoi313/Phone-Face-Tablet-Controller/tree/master/ArduinoBluetoothTankControl
Phone Face Compiled APK
https://github.com/johnchoi313/Phone-Face-Tablet-Controller/blob/master/APKs/FAM%20Tank%20Face.apk
Tablet Controller Compiled APK
https://github.com/johnchoi313/Phone-Face-Tablet-Controller/blob/master/APKs/FAM%20Tank%20Controller.apk
Misty I Controller Compiled APK
https://github.com/johnchoi313/Phone-Face-Tablet-Controller/blob/master/APKs/FAM%20Misty%20I%20Controller.apk
Misty II Controller Compiled APK
https://github.com/johnchoi313/Phone-Face-Tablet-Controller/blob/master/APKs/FAM%20Misty%20II%20Controller.apk
There are quite a few third party plugins that were used to create this project, some free, and some paid. These plugins significantly expand the functionality of Unity (in particular, Unity for Android), that make this project possible. In order to fully compile this project correctly, you will need to download and load each of the following plugins in Unity:
WIZAPPLY CO., LTD. Serial Port Utility Pro ($78)
https://assetstore.unity.com/packages/tools/utilities/serial-port-utility-pro-125863
CROSSTALES LLC RT-Voice PRO ($65)
https://assetstore.unity.com/packages/tools/audio/rt-voice-pro-41068
UTMail - Email Composition and Sending Plugin ($25)
https://assetstore.unity.com/packages/tools/integration/utmail-email-composition-and-sending-plugin-90545
TECH TWEAKING Android & Microcontrollers / Bluetooth ($20)
https://assetstore.unity.com/packages/tools/input-management/android-microcontrollers-bluetooth-16467
SHATALMIC, LLC Bluetooth LE for iOS, tvOS and Android ($20)
https://assetstore.unity.com/packages/tools/network/bluetooth-le-for-ios-tvos-and-android-26661
PIOTR ZMUDZINSKI Mobile Speech Recognizer ($15)
https://assetstore.unity.com/packages/tools/audio/mobile-speech-recognizer-73036
SÜLEYMAN YASIR KULA Runtime File Browser (FREE)
https://assetstore.unity.com/packages/tools/gui/runtime-file-browser-113006
SÜLEYMAN YASIR KULA Simple Input System (FREE)
https://assetstore.unity.com/packages/tools/input-management/simple-input-system-113033
PAPER PLANE TOOLS OpenCV plus Unity (FREE)
https://assetstore.unity.com/packages/tools/integration/opencv-plus-unity-85928
CLAYTON INDUSTRIES Http Client (FREE)
https://assetstore.unity.com/packages/tools/network/http-client-79343
azixMcAze Unity-UIGradient (FREE)
https://github.com/azixMcAze/Unity-UIGradient
YOON CHANGSIK CSV2Table (FREE)
https://assetstore.unity.com/packages/tools/utilities/csv2table-36443
IMPHENZIA Gradient Sky (FREE)
https://assetstore.unity.com/packages/2d/textures-materials/sky/gradient-sky-109899
(Note: We do not maintain or offer support for any of these third party plugins or libraries. The usage of each third party plugin and library is governed with respect to their own individual terms and licenses.)
Special thanks to Fine Arts Miracles and their wonderful work in discovering methodologies to implement social robotics for therapy and early childhood education for making this project possible! Additional special thanks to the incredible previous work of Origami Robotics and their open source Romibo robots in creating a sounding board for the application of social robots everywhere!
The original Romibo robot from Origami Robotics.